Traffic flow optimization method and device based on multiple agents

By building a traffic network map and using multi-agent methods to dynamically adjust traffic light control, the problem of insufficient real-time and systematicity of traditional traffic management methods is solved, and intelligent optimization of traffic flow and efficient utilization of resources are achieved.

CN118918714APending Publication Date: 2024-11-08北京大学武汉人工智能研究院
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410746659.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional traffic management methods lack real-time and systematicity, and cannot effectively deal with fluctuations in traffic flow and optimize the entire transportation network, resulting in congestion at the intersection and waste of resources.

Method used

Based on multi-agent traffic flow optimization method, by building a traffic network map, using geographical information and traffic flow information, using random walk algorithm and Attention network for information fusion, designing an intelligent model to dynamically adjust traffic light control and optimize traffic lights phase selection and switching.

Benefits of technology

Dynamic and intelligent traffic light control is realized, real-time response capabilities of traffic flow are improved, resource utilization of the entire transportation network is optimized, and congestion and waiting time are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918714A_ABST
    Figure CN118918714A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-agent-based traffic flow optimization method and device, and relates to the field of road traffic optimization, and the device comprises the steps: based on a traffic network diagram corresponding to a target path diagram, selecting a plurality of adjacent nodes with the minimum distance between the embedding vectors of each node, and obtaining each node and the embedding vectors of the corresponding adjacent nodes; splicing the state vector and the embedding vector of each node, and carrying out information fusion through an Attention network to obtain a vector of each node after information fusion; performing state design, action design and reward design based on the information fusion vector and the state vector of the node to obtain a corresponding agent model; and traffic signal lamp control is carried out based on the designed intelligent agent model. Modeling is carried out based on the road geographic information and the traffic flow information, model construction is carried out on the obtained traffic network diagram, and dynamic and intelligent traffic signal lamp control is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of road traffic optimization, and in particular to a traffic flow optimization method and device based on multi-agents. Background Art

[0002] With the acceleration of urbanization, urban traffic congestion is becoming increasingly serious. Traditional traffic management methods mainly rely on manual control and fixed-time traffic lights. These methods may be effective when the traffic volume is small and the traffic conditions are relatively simple, but with the rapid development of urbanization and the sharp increase in traffic demand, the limitations of these methods are gradually becoming apparent.

[0003] First, the traditional method lacks real-time performance. The fixed-time traffic lights cannot be adjusted according to the real-time traffic conditions, resulting in unreasonable traffic light timing during peak traffic hours, which cannot effectively cope with traffic flow fluctuations and easily leads to intersection congestion.

[0004] Secondly, traditional methods lack systematicity. The urban traffic system is a complex network, with various road sections and intersections affecting each other. Traditional methods often only focus on a single intersection or road section, lacking global optimization of the entire traffic network, and failing to make full use of road resources, resulting in congestion on some sections while others are relatively idle.

[0005] In order to solve the problem that traditional traffic management methods lack real-time and systematicity, a traffic flow optimization technology based on multi-agent is provided. Summary of the invention

[0006] In view of the defects existing in the prior art, the purpose of the present invention is to provide a traffic flow optimization method and device based on multi-agent, modeling based on road geographic information and traffic flow information, modeling the obtained traffic network diagram, and realizing dynamic and intelligent traffic light control.

[0007] In order to achieve the above purpose, the technical solution adopted by the present invention is: In a first aspect, the present application provides a multi-agent based traffic flow optimization method, the method comprising the following steps: Based on the traffic network graph corresponding to the target path graph, multiple adjacent nodes with the smallest distance between the embedding vectors of each node are selected, and then the embedding vectors of each node and the corresponding adjacent nodes are obtained; The state vectors and embedding vectors of each node are concatenated, and information is fused through the Attention network to obtain the information fused vectors of each node; Based on the information fusion vector and state vector of the node, perform state design, action design and reward design to obtain a corresponding intelligent agent model; Traffic lights are controlled based on the designed intelligent agent model; The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

[0008] On the basis of the above technical solution, based on the information fusion vector and state vector of the node, state design, action design and reward design are performed to obtain the corresponding agent model, including the following steps: Based on the information fusion vector and state vector of the node, perform state design on each of the nodes; For a node in the traffic network diagram that is a traffic light, designing a working action according to a phase, wherein the working action includes phase selection or phase switching; Select all roads connected by nodes with traffic lights in the traffic network diagram, and design the total waiting time of all vehicles on the corresponding road as a negative reward.

[0009] On the basis of the above technical solution, the method further includes the following steps before selecting a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network graph corresponding to the target path graph, and then obtaining the embedding vectors of each node and the corresponding adjacent nodes: Constructing a traffic network diagram based on the intersection location information and traffic flow information corresponding to the target path diagram; For each node in the traffic network graph, the window size, the wandering length, the return parameter and the entry and exit parameters are set.

[0010] On the basis of the above technical solution, the method further includes an embedding vector acquisition process, and the embedding vector acquisition process includes the following steps: Using a random walk algorithm, a node sequence corresponding to each node in the traffic network diagram is calculated; Based on the traffic network graph and the node sequence of each node, an embedding vector of each node is obtained; wherein, The node sequence includes nodes and walker movement probabilities of adjacent nodes arranged according to numerical values.

[0011] On the basis of the above technical solution, the method is further configured with a movement probability calculation formula for the traffic network diagram, and the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

[0012] On the basis of the above technical solution, the method is further configured with a calculation formula for the traffic flow intensity at the intersection of the traffic network diagram, and the calculation formula for the traffic flow intensity at the intersection is: ;in, is the traffic flow intensity at intersection i, is the corresponding time window, To pass the intersection within the time window The total number of vehicles.

[0013] On the basis of the above technical solution, the method further includes a return parameter calculation process, and the return parameter calculation process includes the following steps: Based on the intersection traffic flow intensity, the minimum intersection traffic flow intensity and the maximum intersection traffic flow intensity of each intersection, a normalized intersection traffic flow intensity is obtained; Based on the normalized intersection traffic flow intensity of each intersection, the return parameters corresponding to each intersection are obtained.

[0014] On the basis of the above technical solution, the method is further configured with a calculation formula for the traffic flow intensity between intersections of the traffic network diagram, and the calculation formula for the traffic flow intensity between intersections is: ;in, For intersection The traffic flow intensity between intersections j and intersection j, is the corresponding time window, To pass the intersection within the time window The total number of vehicles arriving at intersection j.

[0015] On the basis of the above technical solution, the method further includes an in-out parameter calculation process, and the in-out parameter calculation process includes the following steps: Based on the inter-intersection traffic flow intensity between the intersections, the minimum value of the inter-intersection traffic flow intensity and the maximum value of the inter-intersection traffic flow intensity, obtaining the normalized inter-intersection traffic flow intensity between the intersections; Based on the normalized inter-intersection traffic flow intensity between the intersections, the corresponding entry and exit parameters between the intersections are obtained.

[0016] In a second aspect, the present application provides a multi-agent-based traffic flow optimization device, the device comprising: The first processing module is used to select a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network diagram corresponding to the target path diagram, and then obtain the embedding vectors of each node and the corresponding adjacent nodes; An information fusion module is used to concatenate the state vectors and embedding vectors of each node and perform information fusion through an Attention network to obtain the information fusion vectors of each node; A model design module, which is used to perform state design, action design and reward design based on the information fusion vector and state vector of the node to obtain a corresponding intelligent agent model; A traffic flow prediction module is used to control traffic lights based on the designed agent model; wherein, The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

[0017] On the basis of the above technical solution, the model design module is further used to perform state design on each of the nodes based on the information fusion vector and state vector of the node; The model design module is also used to design working actions according to phases for nodes in the traffic network diagram that are traffic lights, wherein the working actions include phase selection or phase switching; The model design module is also used to select roads that are commonly connected to all nodes of traffic lights in the traffic network diagram, and design the overall waiting time of all vehicles on the corresponding roads as a negative reward.

[0018] On the basis of the above technical solution, the device further comprises a traffic network diagram construction module, which is used to construct a traffic network diagram based on the intersection location information and traffic flow information corresponding to the target path diagram; The traffic network diagram construction module is also used to set the window size, wandering length, return parameter and entry and exit parameters for each node in the traffic network diagram.

[0019] On the basis of the above technical solution, the device further includes an embedding vector acquisition module, which is used to calculate and obtain the node sequence corresponding to each node in the traffic network diagram using a random walk algorithm; The embedding vector acquisition module is also used to obtain the embedding vector of each node based on the traffic network diagram and the node sequence of each node; wherein, The node sequence includes nodes and walker movement probabilities of adjacent nodes arranged according to numerical values.

[0020] On the basis of the above technical solution, the device is also configured with a movement probability calculation formula for the traffic network diagram, and the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

[0021] On the basis of the above technical solution, the device is also configured with a calculation formula for the traffic flow intensity at the intersection of the traffic network diagram, and the calculation formula for the traffic flow intensity at the intersection is: ;in, is the traffic flow intensity at intersection i, is the corresponding time window, To pass the intersection within the time window The total number of vehicles.

[0022] On the basis of the above technical solution, the device further includes a return parameter calculation module, which is used to obtain the normalized intersection traffic flow intensity of each intersection based on the intersection traffic flow intensity, the minimum intersection traffic flow intensity and the maximum intersection traffic flow intensity of each intersection; The return parameter calculation module is also used to obtain the return parameter corresponding to each intersection based on the normalized intersection traffic flow intensity of each intersection.

[0023] On the basis of the above technical solution, the device is also equipped with a calculation formula for the traffic flow intensity between intersections of the traffic network diagram, and the calculation formula for the traffic flow intensity between intersections is: ;in, For intersection The traffic flow intensity between intersections j and intersection j, is the corresponding time window, To pass the intersection within the time window The total number of vehicles arriving at intersection j.

[0024] On the basis of the above technical solution, the device further includes an entry and exit parameter calculation module, which is used to obtain the normalized traffic flow intensity between each intersection based on the traffic flow intensity between each intersection, the minimum traffic flow intensity between each intersection, and the maximum traffic flow intensity between each intersection; The entry and exit parameter calculation module is also used to obtain the corresponding entry and exit parameters between the intersections based on the normalized inter-intersection traffic flow intensity between the intersections.

[0025] Compared with the prior art, the advantages of the present invention are: This application is modeled based on road geographic information and traffic flow information, and a model is constructed on the obtained traffic network diagram to achieve dynamic and intelligent traffic light control. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a flowchart of the steps of the multi-agent-based traffic flow optimization method in an embodiment of the present application; Figure 2 A schematic diagram of the movement probability calculation principle of a traffic network diagram in a multi-agent-based traffic flow optimization method in an embodiment of the present application; Figure 3 Schematic diagram of the AAC algorithm principle of the multi-agent-based traffic flow optimization method in the embodiment of the present application; Figure 4 A schematic diagram of the overall architecture design of the multi-agent-based traffic flow optimization method in an embodiment of the present application; Figure 5 It is a structural block diagram of a multi-agent-based traffic flow optimization device in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0029] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0030] The embodiment of the present application provides a multi-agent-based traffic flow optimization method and device, which performs modeling based on road geographic information and traffic flow information, constructs a model of the obtained traffic network diagram, and realizes dynamic and intelligent traffic light control.

[0031] In order to achieve the above technical effects, the overall idea of ​​this application is as follows: A traffic flow optimization method based on multi-agent, the method comprising the following steps: S1. Based on the traffic network graph corresponding to the target path graph, multiple adjacent nodes with the smallest distance between the embedding vectors of each node are selected to obtain the embedding vectors of each node and the corresponding adjacent nodes; S2, concatenate the state vector and embedding vector of each node, and perform information fusion through the Attention network to obtain the information fusion vector of each node; S3, based on the information fusion vector and state vector of the node, perform state design, action design and reward design to obtain the corresponding intelligent agent model; S4. Control traffic lights based on the designed intelligent agent model; wherein, The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

[0032] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0033] First, see Figures 1 to 4 As shown, the embodiment of the present application provides a traffic flow optimization method based on multi-agent, and the method includes the following steps: S1. Based on the traffic network graph corresponding to the target path graph, multiple adjacent nodes with the smallest distance between the embedding vectors of each node are selected to obtain the embedding vectors of each node and the corresponding adjacent nodes; S2, concatenate the state vector and embedding vector of each node, and perform information fusion through the Attention network to obtain the information fusion vector of each node; S3, based on the information fusion vector and state vector of the node, perform state design, action design and reward design to obtain the corresponding intelligent agent model; S4. Control traffic lights based on the designed intelligent agent model; wherein, The traffic network graph includes multiple nodes and paths between nodes. One node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

[0034] In the embodiment of the present application, modeling is performed based on road geographic information and traffic flow information, and a model is constructed on the obtained traffic network diagram to achieve dynamic and intelligent traffic light control.

[0035] Further, based on the information fusion vector and the state vector of the node, state design, action design and reward design are performed to obtain the corresponding agent model, including the following steps: Based on the information fusion vector and state vector of the node, perform state design on each of the nodes; For a node in the traffic network diagram that is a traffic light, designing a working action according to a phase, wherein the working action includes phase selection or phase switching; Select all roads connected by nodes with traffic lights in the traffic network diagram, and design the total waiting time of all vehicles on the corresponding road as a negative reward.

[0036] Furthermore, before selecting a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network graph corresponding to the target path graph, and then obtaining the embedding vectors of each node and the corresponding adjacent nodes, the method further includes the following steps: Constructing a traffic network diagram based on the intersection location information and traffic flow information corresponding to the target path diagram; For each node in the traffic network graph, the window size, the wandering length, the return parameter and the entry and exit parameters are set.

[0037] Furthermore, the method also includes an embedding vector acquisition process, and the embedding vector acquisition process includes the following steps: Using a random walk algorithm, a node sequence corresponding to each node in the traffic network diagram is calculated; Based on the traffic network graph and the node sequence of each node, an embedding vector of each node is obtained; wherein, The node sequence includes nodes and walker movement probabilities of adjacent nodes arranged according to numerical values.

[0038] Furthermore, the method is also configured with a movement probability calculation formula for the traffic network diagram, and the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

[0039] Furthermore, the method is also configured with a calculation formula for the traffic flow intensity at the intersection of the traffic network diagram, and the calculation formula for the traffic flow intensity at the intersection is: ;in, is the traffic flow intensity at intersection i, is the corresponding time window, To pass the intersection within the time window The total number of vehicles.

[0040] Furthermore, the method is also configured with a normalized intersection traffic flow intensity calculation formula for the traffic network diagram, and the normalized intersection traffic flow intensity calculation formula is: ;in, is the normalized intersection traffic flow intensity at intersection i, is the traffic flow intensity at intersection i, is the minimum traffic flow intensity at the intersection in the traffic network diagram, It is the maximum value of the traffic flow intensity at the intersection in the traffic network diagram.

[0041] Furthermore, the method also includes a return parameter calculation process, and the return parameter calculation process includes the following steps: Based on the intersection traffic flow intensity, the minimum intersection traffic flow intensity and the maximum intersection traffic flow intensity of each intersection, a normalized intersection traffic flow intensity is obtained; Based on the normalized intersection traffic flow intensity of each intersection, the return parameters corresponding to each intersection are obtained.

[0042] Furthermore, the method is also configured with a return parameter calculation formula for the traffic network diagram, and the return parameter calculation formula is: ;in, is the return parameter of intersection i, is the normalized intersection traffic flow intensity at intersection i.

[0043] Furthermore, the method is also configured with a calculation formula for the traffic flow intensity between intersections of the traffic network diagram, and the calculation formula for the traffic flow intensity between intersections is: ;in, For intersection The traffic flow intensity between intersections j and intersection j, is the corresponding time window, To pass the intersection within the time window The total number of vehicles arriving at intersection j.

[0044] If the traffic flow intensity of an intersection node is high, it means that the intersection may be a congestion point or an important traffic hub. In this case, the designer hopes that the random walk can return to this node more frequently to better capture its characteristics. Therefore, the value of p can be set lower.

[0045] In order to ensure the stability and effectiveness of the algorithm, the traffic flow intensity needs to be normalized.

[0046] For intersections The traffic flow intensity of all intersections can be calculated by and maximum value , then the intersection The traffic flow intensity is normalized:

[0047] For the traffic flow intensity of edge (i, j), the minimum value of the traffic flow intensity of all edges is also calculated and maximum value , and then normalize: .

[0048] If the traffic flow intensity at a node (intersection) is high, it means that the intersection may be a congestion point or an important traffic hub. In this case, designers hope that the random walk can return to this node more frequently to better capture its characteristics. Therefore, designers can set p to be lower.

[0049] intersection The return parameter p can be calculated as: .

[0050] If the slave node To Node The traffic flow intensity of is high, which means that the connection between these two nodes is important. In this case, the designer wants the random walk to be more inclined to explore such a connection. Therefore, q can be set lower so that the random walk is more likely to move along the edges with high traffic flow intensity.

[0051] In the Node2Vec method, the node parameter q is generally set to a fixed value, and different values ​​are set according to the normalized traffic flow intensity between adjacent nodes, and the sum of the transition probabilities of all adjacent nodes is guaranteed to be 1.

[0052] Adjacent intersection nodes The in-out parameter q can be calculated as: ; .

[0053] Furthermore, the method is also configured with a normalized intersection traffic flow intensity calculation formula for the traffic network diagram, and the normalized intersection traffic flow intensity calculation formula is: ; in, For intersection The normalized inter-intersection traffic flow intensity between intersection j and intersection j, For intersection The traffic flow intensity between intersections j and intersection j, is the minimum traffic flow intensity between intersections in the traffic network diagram, It is the maximum value of traffic flow intensity between intersections in the traffic network diagram.

[0054] Furthermore, the method also includes an in-out parameter calculation process, and the in-out parameter calculation process includes the following steps: Based on the inter-intersection traffic flow intensity between the intersections, the minimum value of the inter-intersection traffic flow intensity and the maximum value of the inter-intersection traffic flow intensity, obtaining the normalized inter-intersection traffic flow intensity between the intersections; Based on the normalized inter-intersection traffic flow intensity between the intersections, the corresponding entry and exit parameters between the intersections are obtained.

[0055] Furthermore, the method is also configured with an entry and exit parameter calculation formula for the traffic network diagram, and the entry and exit parameter calculation formula is: ; ;in, For intersection The entry and exit parameters between intersection j, For intersection The normalized inter-intersection traffic flow intensity between intersection j, n means there are n intersection nodes adjacent to i, k means it is the kth one, and the sum is taken from the 0th to the kth one.

[0056] Based on the technical solution of the embodiment of the present application, in actual implementation, the specific situation is as follows: Node2Vec is a graph-based representation learning algorithm that can be used to learn low-dimensional representations of nodes in a graph. When intersection location information and traffic flow information are collected, the Node2Vec method can be used for modeling to better understand and analyze the traffic network.

[0057] The following are the steps to model intersection location information and traffic flow information based on the Node2Vec method: The first step is to build a graph: the collected intersection location information and traffic flow information need to be converted into a graph. Each intersection can be represented as a node, and the road can be represented as an edge between nodes.

[0058] Step 2, parameter setting: Before using Node2Vec, you need to set some parameters, such as window size (using the default parameter 5, which means that each node is affected by the node with the farthest distance of 5), walk length (using the default parameter 80, which means that each random walk takes 80 steps), return parameter p (this parameter is calculated based on the traffic flow intensity of the current intersection node, and the calculation method is given below) and entry and exit parameter q (this parameter is calculated based on the traffic flow intensity of the current intersection node, and the calculation method is given below). These parameters will affect the behavior of random walks and thus affect the learning of node embedding.

[0059] Among them, the window size and walk length are given values ​​and will be used for each random walk.

[0060] The return parameter p and the entry and exit parameter q are both calculated based on the traffic flow intensity of the intersection and the node, and are the attributes of the node. The return parameter p determines the tendency of the walker to return to the previous node when it is at the current intersection node, while the entry and exit parameter q determines the tendency of the walker to explore a new node away from the previous node when it is at the current intersection node.

[0061] Specifically, as shown in the accompanying drawings of the specification Figure 2 As shown, for the movement probability calculation formula of the traffic network diagram, the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

[0062] In specific operations, these two parameters are dynamically adjusted according to the traffic flow intensity to better capture the characteristics of the traffic network.

[0063] Assume there is an intersection and intersection , and in the time window Through the intersection The total number of vehicles is , from the intersection To the intersection The number of vehicles is .

[0064] intersection The traffic flow intensity can be defined as: .

[0065] The traffic flow intensity from intersection i to intersection j can be defined as: .

[0066] The third step is random walk: use the random walk algorithm to perform multiple walks on the graph. Specifically, it starts from any point in the graph and transfers between nodes probabilistically. Repeat the above steps a finite number of times, and finally collect a series of node sequences.

[0067] It should be noted that a node sequence, such as a number of node sequences such as abcbd, starts from any point in the graph. If it starts from a, the first node is a. The probability of going to each node is different, so the node sequence is different, and there may be repeated node sequences.

[0068] The fourth step is to train the model: use the SkipGram model to learn the embedding vector of the node, similar to the way of processing text, input the node sequence collected by random walk into the SkipGram model, and finally obtain the embedding vector. This embedding vector can capture the local and global characteristics of the intersection, such as its neighborhood structure and its position in the entire traffic network.

[0069] Actor-Attention-Critic (AAC) is a multi-agent reinforcement learning algorithm that combines the Actor-Critic framework and the attention mechanism to effectively handle decision-making problems in multi-agent environments. In a multi-agent environment, each agent needs to consider the behavior of other agents to make the best decision, which increases the complexity of learning.

[0070] The Actor-Critic framework is a reinforcement learning method in which "Actor" represents a policy function, which is responsible for selecting actions based on the current state; "Critic" represents a value function, which is responsible for evaluating the quality of the current strategy and providing feedback to the Actor to improve the strategy. In a multi-agent environment, each agent has its own Actor and Critic.

[0071] Attention Mechanism is a technique that selectively pays attention to information from sequence data, which allows the model to focus on the most relevant information for the current decision. In multi-agent reinforcement learning, the attention mechanism can help agents more effectively process the states and behaviors of other agents, thereby improving the overall collaborative performance.

[0072] Key features of the AAC algorithm Figure 3 As shown, through the centralized attention mechanism, information from other agents is effectively processed and focused on the factors that are most important to the current decision maker.

[0073] The AAC algorithm converts a single agent into (Observation Information) and (Action information) is encoded into Vector representation, information fusion with other intelligent agent vectors through Attention network, combined with the contribution of other intelligent agents through weighted summation and then used In this way, the state-action-value function of the agent is It takes into account both its own observation and action information as well as the contributions of other intelligent agents.

[0074] The embedding vectors used in the above article to model intersection location information and traffic flow information based on the Node2Vec method can also be fused with the embedding vectors of other agents. In this way, the Attention network considers the agent's geographical location and traffic flow information when fusing information, which helps the traffic light agent understand the status and actions of the surrounding intersections and better coordinate the switching of traffic lights at each intersection.

[0075] Based on the technical solution of the embodiment of the present application, the overall architecture design is as shown in the accompanying drawings of the specification. Figure 4 The specific situation is as follows: Step 1: Select an agent, that is, select a node: select k nodes with the smallest distance between node embedding vectors, as shown in the figure of the specification. Figure 4 As shown in the figure, k (k=4) nodes with the smallest vector distance to Node_0 are selected. At this time, the embedding vector v_0 of Node_0 and the embedding vectors of its surrounding nodes are obtained. This step can be completed offline.

[0076] Step 2, information fusion: concatenate the state vector s observed by all agents with the embedding vector v, and perform information fusion through the Attention network. The Attention network calculates the weight of each input data, realizes information fusion, and thus improves the model's ability to understand and process data.

[0077] Step 3: Actor-Critic reinforcement model training: (1) State design: The vector fused by the Attention network is concatenated with the agent state vector s. The concatenated vectors are passed to the Actor network to predict the agent’s next action and to the Critic network to judge the state of the agent.

[0078] (2) Action design: Each traffic light agent has at most n phases, and the actions are the selection and switching between these n phases.

[0079] (3) Reward design: Select the roads that are connected by all traffic light agents in step 1, and use the total waiting time of all vehicles on the road as the negative reward.

[0080] Among them, the intelligent agent mentioned above can be understood as a selected node for predicting its traffic conditions.

[0081] It should be noted that the execution steps of the agent model are similar to the model training steps. In the end, there is no need to pass it to the Critic network to judge the state of the agent. Instead, it can be passed to the Actor network to predict the next action of the agent.

[0082] In summary, the technical solution of the embodiment of the present application regards traffic lights within a certain range as multiple intelligent agents, and realizes real-time dynamic adjustment and optimization of traffic flow by learning the collaboration and information exchange between intelligent agents in a simulated environment.

[0083] In addition, the technical solution of the embodiment of the present application also has other situations: Methods based on graph neural networks: Graph Neural Networks (GNNs) can be learned directly on traffic network graphs without first using Node2Vec to extract features. GNNs can capture complex relationships between nodes and can be combined with reinforcement learning to control traffic lights, but learning the relationship between nodes and optimizing traffic light control at the same time may make model training extremely difficult.

[0084] Multi-agent reinforcement learning (without attention mechanism): Other multi-agent reinforcement learning frameworks such as Independent Q-Learning (IQL) or Multi-Agent Deep Deterministic Policy Gradient (MADDPG) can be used for traffic light control. These frameworks do not rely on the attention mechanism and directly use the embedding vector obtained by the Node2Vec method as the feature input, which may be inefficient when coordinating multiple agents.

[0085] It should be noted that the key points of the technical solution of the embodiment of the present application are: Construction and feature extraction of traffic network graphs: How to convert the actual traffic network into a graph and use the Node2Vec algorithm to effectively extract the feature representations of intersection nodes. These feature representations need to be able to reflect the relationship between intersections and traffic flow intensity information.

[0086] Integration of attention mechanism: How to integrate traffic network graph features into the AAC framework so that each agent can dynamically adjust its behavior according to the surrounding environment and the state of other agents, and how this mechanism can improve the collaborative efficiency of the entire system.

[0087] Second, see Figure 5 As shown, an embodiment of the present application provides a multi-agent-based traffic flow optimization device, the device comprising: The first processing module is used to select a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network diagram corresponding to the target path diagram, and then obtain the embedding vectors of each node and the corresponding adjacent nodes; An information fusion module is used to concatenate the state vectors and embedding vectors of each node and perform information fusion through an Attention network to obtain the information fusion vectors of each node; A model design module, which is used to perform state design, action design and reward design based on the information fusion vector and state vector of the node to obtain a corresponding intelligent agent model; A traffic flow prediction module is used to control traffic lights based on the designed agent model; wherein, The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

[0088] In the embodiment of the present application, modeling is performed based on road geographic information and traffic flow information, and a model is constructed on the obtained traffic network diagram to achieve dynamic and intelligent traffic light control.

[0089] Furthermore, the model design module is also used to perform state design on each of the nodes based on the information fusion vector and state vector of the node; The model design module is also used to design working actions according to phases for nodes in the traffic network diagram that are traffic lights, wherein the working actions include phase selection or phase switching; The model design module is also used to select roads that are commonly connected to all nodes of traffic lights in the traffic network diagram, and design the overall waiting time of all vehicles on the corresponding roads as a negative reward.

[0090] Furthermore, the device also includes a traffic network diagram construction module, which is used to construct a traffic network diagram based on the intersection location information and traffic flow information corresponding to the target path diagram; The traffic network diagram construction module is also used to set the window size, wandering length, return parameter and entry and exit parameters for each node in the traffic network diagram.

[0091] Furthermore, the device also includes an embedding vector acquisition module, which is used to calculate and obtain a node sequence corresponding to each node in the traffic network diagram using a random walk algorithm; The embedding vector acquisition module is also used to obtain the embedding vector of each node based on the traffic network diagram and the node sequence of each node; wherein, The node sequence includes nodes and walker movement probabilities of adjacent nodes arranged according to numerical values.

[0092] Furthermore, the device is also configured with a movement probability calculation formula for the traffic network diagram, and the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

[0093] Furthermore, the device is also configured with a calculation formula for the traffic flow intensity at the intersection of the traffic network diagram, and the calculation formula for the traffic flow intensity at the intersection is: ;in, is the traffic flow intensity at intersection i, is the corresponding time window, To pass the intersection within the time window The total number of vehicles.

[0094] Furthermore, the device is also configured with a normalized intersection traffic flow intensity calculation formula for the traffic network diagram, and the normalized intersection traffic flow intensity calculation formula is: ;in, is the normalized intersection traffic flow intensity at intersection i, is the traffic flow intensity at intersection i, is the minimum traffic flow intensity at the intersection in the traffic network diagram, It is the maximum value of the traffic flow intensity at the intersection in the traffic network diagram.

[0095] Furthermore, the device also includes a return parameter calculation module, which is used to obtain the normalized intersection traffic flow intensity of each intersection based on the intersection traffic flow intensity, the minimum intersection traffic flow intensity and the maximum intersection traffic flow intensity of each intersection; The return parameter calculation module is also used to obtain the return parameter corresponding to each intersection based on the normalized intersection traffic flow intensity of each intersection.

[0096] Furthermore, the device is also configured with a return parameter calculation formula for the traffic network diagram, and the return parameter calculation formula is: ;in, is the return parameter of intersection i, is the normalized intersection traffic flow intensity at intersection i.

[0097] Furthermore, the device is also configured with a calculation formula for the traffic flow intensity between intersections of the traffic network diagram, and the calculation formula for the traffic flow intensity between intersections is: ;in, For intersection The traffic flow intensity between intersections j and intersection j, is the corresponding time window, To pass the intersection within the time window The total number of vehicles arriving at intersection j.

[0098] Furthermore, the device is also configured with a calculation formula for the normalized traffic flow intensity between intersections for the traffic network diagram, and the calculation formula for the normalized traffic flow intensity between intersections is: ; in, For intersection The normalized inter-intersection traffic flow intensity between intersection j and intersection j, For intersection The traffic flow intensity between intersections j and intersection j, is the minimum traffic flow intensity between intersections in the traffic network diagram, It is the maximum value of traffic flow intensity between intersections in the traffic network diagram.

[0099] Furthermore, the device also includes an entry and exit parameter calculation module, which is used to obtain the normalized traffic flow intensity between each intersection based on the traffic flow intensity between each intersection, the minimum traffic flow intensity between each intersection, and the maximum traffic flow intensity between each intersection; The entry and exit parameter calculation module is also used to obtain the corresponding entry and exit parameters between the intersections based on the normalized inter-intersection traffic flow intensity between the intersections.

[0100] Furthermore, the device is also configured with an entry and exit parameter calculation formula for the traffic network diagram, and the entry and exit parameter calculation formula is: ; ;in, For intersection The entry and exit parameters between intersection j, For intersection The normalized inter-intersection traffic flow intensity between intersection j and intersection j.

[0101] It should be noted that the multi-agent-based traffic flow optimization device provided in the embodiment of the present application, and its corresponding technical problems, technical means and technical effects are similar to the principles of the multi-agent-based traffic flow optimization method in terms of principle.

[0102] In the description of the present application, it should be noted that the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0103] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0104] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A traffic flow optimization method based on multi-agent, characterized in that: The method comprises the following steps: Based on the traffic network graph corresponding to the target path graph, multiple adjacent nodes with the smallest distance between the embedding vectors of each node are selected, and then the embedding vectors of each node and the corresponding adjacent nodes are obtained; The state vectors and embedding vectors of each node are concatenated, and information is fused through the Attention network to obtain the information fused vectors of each node; Based on the information fusion vector and state vector of the node, perform state design, action design and reward design to obtain a corresponding intelligent agent model; Traffic lights are controlled based on the designed intelligent agent model; The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.

2. The multi-agent based traffic flow optimization method according to claim 1, characterized in that: Based on the information fusion vector and the state vector of the node, state design, action design and reward design are performed to obtain the corresponding agent model, including the following steps: Based on the information fusion vector and state vector of the node, perform state design on each of the nodes; For a node in the traffic network diagram that is a traffic light, designing a working action according to a phase, wherein the working action includes phase selection or phase switching; Select all roads connected by nodes with traffic lights in the traffic network diagram, and design the total waiting time of all vehicles on the corresponding road as a negative reward.

3. The multi-agent based traffic flow optimization method according to claim 1, characterized in that: Before selecting a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network graph corresponding to the target path graph, and then obtaining the embedding vectors of each node and the corresponding adjacent nodes, the method further includes the following steps: Constructing a traffic network diagram based on the intersection location information and traffic flow information corresponding to the target path diagram; For each node in the traffic network graph, the window size, the wandering length, the return parameter and the entry and exit parameters are set.

4. The multi-agent based traffic flow optimization method according to claim 1, characterized in that: The method further includes an embedding vector acquisition process, and the embedding vector acquisition process includes the following steps: Using a random walk algorithm, a node sequence corresponding to each node in the traffic network diagram is calculated; Based on the traffic network graph and the node sequence of each node, an embedding vector of each node is obtained; wherein, The node sequence includes nodes and walker movement probabilities of adjacent nodes arranged according to numerical values.

5. The multi-agent based traffic flow optimization method according to claim 1, characterized in that: The method is also configured with a movement probability calculation formula for the traffic network diagram, and the movement probability calculation formula is: ;in, The current node is denoted as v, the previous node is denoted as t, the next node is denoted as x, and the distance between the previous node and the next node is , p is the return parameter, which is used to determine the tendency to return to the previous node, and q is the entry and exit parameter, which is used to determine the tendency to explore new nodes far away from the previous node.

6. The multi-agent based traffic flow optimization method according to claim 3, characterized in that: The method is also configured with a calculation formula for the traffic flow intensity at the intersection of the traffic network diagram, and the calculation formula for the traffic flow intensity at the intersection is: ;in, is the traffic flow intensity at intersection i, is the corresponding time window, To pass the intersection within the time window The total number of vehicles.

7. The multi-agent based traffic flow optimization method according to claim 6, characterized in that: The method further includes a return parameter calculation process, wherein the return parameter calculation process includes the following steps: Based on the intersection traffic flow intensity, the minimum intersection traffic flow intensity and the maximum intersection traffic flow intensity of each intersection, a normalized intersection traffic flow intensity is obtained; Based on the normalized intersection traffic flow intensity of each intersection, the return parameters corresponding to each intersection are obtained.

8. The multi-agent based traffic flow optimization method according to claim 3, characterized in that: The method is also configured with a calculation formula for the traffic flow intensity between intersections of the traffic network diagram, and the calculation formula for the traffic flow intensity between intersections is: ;in, For intersection The traffic flow intensity between intersections j and intersection j, is the corresponding time window, To pass the intersection within the time window The total number of vehicles arriving at intersection j.

9. The multi-agent based traffic flow optimization method according to claim 8, characterized in that: The method further includes an in-out parameter calculation process, wherein the in-out parameter calculation process includes the following steps: Based on the inter-intersection traffic flow intensity between the intersections, the minimum value of the inter-intersection traffic flow intensity and the maximum value of the inter-intersection traffic flow intensity, obtaining the normalized inter-intersection traffic flow intensity between the intersections; Based on the normalized inter-intersection traffic flow intensity between the intersections, the corresponding entry and exit parameters between the intersections are obtained.

10. A traffic flow optimization device based on multi-agent, characterized in that: The device comprises: The first processing module is used to select a plurality of adjacent nodes with the smallest distance between the embedding vectors of each node based on the traffic network diagram corresponding to the target path diagram, and then obtain the embedding vectors of each node and the corresponding adjacent nodes; An information fusion module is used to concatenate the state vectors and embedding vectors of each node and perform information fusion through an Attention network to obtain the information fusion vectors of each node; A model design module, which is used to perform state design, action design and reward design based on the information fusion vector and state vector of the node to obtain a corresponding intelligent agent model; A traffic flow prediction module is used to control traffic lights based on the designed agent model; wherein, The traffic network graph includes multiple nodes and paths between nodes, one node corresponds to an intersection in the target path graph, and one path between nodes corresponds to a road between two intersections in the target path graph.