Distributed multi-agent regional traffic control method and system and electronic equipment

Abstract

The invention provides a distributed multi-agent regional traffic control method and system and electronic equipment, and the method comprises the steps that a sub-region agent determines target intersection agents according to current traffic information, and transmits a regional cooperative control instruction to each target intersection agent, the target intersection intelligent body receives association strategy information of the association intersection intelligent body, determines target strategy information according to the current traffic information, the regional cooperative control instruction, the preference information and the association strategy information, and sends a target control instruction to the lane intelligent body and / or the flow direction intelligent body according to the target strategy information, and the lane intelligent agent controls the controlled equipment according to the target control instruction and / or the flow direction intelligent agent controls the controlled equipment according to the target control instruction. According to the invention, the transmission cooperation capability between the intelligent agents is improved, and adaptive adjustment is realized, so that global optimization can be carried out, and the adaptability and the intelligent level of the whole traffic system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of traffic control, and in particular to a distributed multi-agent regional traffic control method, system and electronic equipment. Background Art

[0002] In modern transportation systems, urban traffic control has become increasingly complex. How to improve traffic management efficiency and the operational efficiency of urban traffic has become an urgent problem to be solved.

[0003] In the prior art, a solution for traffic coordination control using multi-agents is proposed. Specifically, multi-agents are used to simulate information interaction between traffic participants, and each traffic participant is controlled based on a preset control strategy.

[0004] However, the existing technology of traffic control through multi-agents cannot achieve adaptive strategy adjustment and cannot adapt to complex traffic environments. In addition, the coordination and communication between various agents are restricted by rules or models, which limits the overall performance of the control system. Summary of the invention

[0005] The purpose of this application is to provide a distributed multi-agent regional traffic control method, system and electronic equipment to address the deficiencies in the above-mentioned prior art, so as to solve the problem that adaptive traffic control cannot be achieved in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a distributed multi-agent regional traffic control method, the method is applied to a regional traffic coordination control system, the regional traffic coordination control system includes: a sub-area agent, a plurality of intersection agent sets and controlled devices corresponding to each of the intersection agent sets, each of the intersection agent sets includes: an intersection agent, at least one lane agent and at least one flow direction agent, the method includes:

[0008] The sub-area agent selects at least one intersection agent as a target intersection agent according to the acquired current traffic information of the target area through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each of the target intersection agents, wherein the current traffic information includes the congestion status of each lane;

[0009] The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent;

[0010] The target intersection agent determines target strategy information according to the current traffic information, the regional collaborative control instruction, the preference information and the associated strategy information through the intersection decision model pre-deployed in the target intersection agent, wherein the target strategy information indicates the control strategy of the target intersection agent;

[0011] The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information;

[0012] The lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through a flow direction decision model pre-deployed in the flow direction agent.

[0013] Optionally, the training process of the sub-area decision model, the intersection decision model, the lane decision model and the flow direction decision model includes:

[0014] Using progressive transfer learning, the pre-acquired agent parameter library is transplanted into an initial sub-area decision model, an initial intersection decision model, and an initial lane flow direction decision model, wherein the initial sub-area decision model, the initial intersection decision model, and the initial lane flow direction decision model are all deployed on a cloud server;

[0015] Based on the distributed algorithm, the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction agent decision model are iteratively fused and trained according to the preset action set, the preset feedback set and the preset association relationship between the decision models to obtain the sub-area intermediate model, the intersection intermediate model and the lane flow direction decision model;

[0016] Deploy the intersection intermediate model to each intersection agent, deploy the lane intermediate model to each lane agent, and deploy the flow intermediate model to each flow agent;

[0017] Iteratively adjusting the sub-area intermediate model to obtain the sub-area decision model;

[0018] The intersection intermediate model, lane intermediate model and flow direction intermediate model are iteratively adjusted respectively to obtain intersection decision models corresponding to each intersection intermediate model, lane decision models corresponding to each lane intermediate model and flow direction decision models corresponding to each flow direction intermediate model.

[0019] Optionally, the iteratively adjusting the sub-area intermediate model to obtain the sub-area decision model includes:

[0020] Determine the weights corresponding to multiple intersection agents according to the preset learning rate of the intersection, the reward function parameters corresponding to each intersection agent, and the minimum batch processing quantity;

[0021] Determine the value of each intersection control strategy in each intersection control strategy set according to the weight corresponding to each intersection intelligent agent, the number of intelligent agents at each intersection, the current traffic information of each intersection, the current traffic information of the target area, the preset regional control strategy set, the regional control strategy set corresponding to each intersection control strategy set, and the evaluation coefficient corresponding to each intersection control strategy set;

[0022] The sub-area intermediate model is iteratively modified based on the value of each intersection control strategy in each intersection control strategy set to obtain a sub-area decision model.

[0023] Optionally, iteratively adjusting the intersection intermediate models to obtain intersection decision models corresponding to the intersection intermediate models includes:

[0024] Determine a predicted traffic situation based on current traffic information, and determine a current area control state based on the predicted traffic situation and historical traffic information;

[0025] Determining whether the current regional control state is included in the policy knowledge base, wherein the policy knowledge base includes a set of regional control states and policies corresponding to each of the regional control states;

[0026] If so, determining the current regional control strategy that matches the current regional control state in the strategy knowledge base, and iteratively adjusting the intersection intermediate model based on the current regional control state and the current regional control strategy to obtain the intersection decision model corresponding to each of the intersection intermediate models;

[0027] If not, the intersection intermediate model is iteratively adjusted according to the current area control state to obtain the intersection decision model corresponding to each intersection intermediate model.

[0028] Optionally, the iterative adjustment of the intersection intermediate models according to the current area control state to obtain intersection decision models corresponding to each of the intersection intermediate models includes:

[0029] Based on the multi-agent reinforcement learning strategy, an objective function is generated according to the current regional control state, the preset regional control strategy set, the preset value function and the model parameter set corresponding to the control strategy set;

[0030] The intersection intermediate model is iteratively modified according to the objective function to generate an intersection decision model.

[0031] Optionally, the flow direction intermediate model is iteratively adjusted to obtain a flow direction decision model corresponding to the flow direction intermediate model, including:

[0032] Based on the Markov decision strategy, the cumulative reward of each flow agent is determined according to the current traffic information, the flow action set, the flow control strategy set, the green light extension time corresponding to the flow control strategy set, the activation time of each flow agent, the occupancy rate of each flow agent, the flow rate of each flow agent, the weight of each flow agent under the current traffic information and the association relationship of each flow agent;

[0033] The flow direction intermediate model is iteratively modified according to the accumulated reward to obtain a flow direction decision model.

[0034] Optionally, the lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, including:

[0035] The lane agent controls the controlled device through the lane decision model according to the target control instruction and the current traffic information.

[0036] In a second aspect, the present application provides a regional traffic coordination control system, the regional traffic coordination control system comprising: a sub-area agent, a plurality of intersection agent sets and controlled devices corresponding to each of the intersection agent sets, each of the intersection agent sets respectively comprising: an intersection agent, at least one lane agent and at least one flow direction agent, wherein:

[0037] The sub-area agent selects at least one intersection agent as a target intersection agent according to the acquired current traffic information of the target area through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each of the target intersection agents, wherein the current traffic information includes the congestion status of each lane;

[0038] The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent;

[0039] The target intersection agent determines target strategy information according to the current traffic information, the regional collaborative control instruction, the preference information and the associated strategy information through the intersection decision model pre-deployed in the target intersection agent, wherein the target strategy information indicates the control strategy of the target intersection agent;

[0040] The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information;

[0041] The lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through a flow direction decision model pre-deployed in the flow direction agent.

[0042] In a third aspect, the present application provides an electronic device comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the distributed multi-agent regional traffic control method as described above.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distributed multi-agent regional traffic control method as described above are executed.

[0044] The beneficial effect of the present application is that the present application is applied to a regional traffic coordination control system, the system includes a sub-area intelligent agent, a plurality of intersection intelligent agent sets and controlled devices corresponding to each intersection intelligent agent set, each of the intersection intelligent agent sets includes an intersection intelligent agent, at least one lane intelligent agent and at least one flow direction intelligent agent. This multi-level control system can enable the intelligent agent at the high level to effectively coordinate the overall strategy, and the intelligent agent at the low level to perform specific assigned tasks, ensure information sharing and consistency strategies between intelligent agents, and be able to achieve rapid response in a highly dynamic traffic environment, and reduce the impact of information delay and inconsistency on the task execution effect. The specific method includes: the sub-area intelligent agent selects at least one intersection intelligent agent as the target intersection intelligent agent according to the current traffic information of the target area, and sends a regional collaborative control instruction to the target intersection intelligent agent. After the target intersection intelligent agent receives the associated strategy information sent by the associated intersection intelligent agent, the target strategy information is determined according to the current traffic information, the regional collaborative control instruction and the associated strategy information, so that each intelligent agent can dynamically adjust the decision strategy according to the traffic situation, improve the intelligent decision-making level of the system, better cope with the complex and changeable traffic environment, and improve the efficiency of task execution. After determining the target strategy information, the target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow agent according to the target strategy information, so that the lane agent and the flow agent control the controlled device according to the target control instruction. In the process of issuing multi-level regional coordination instructions and formulating target strategy information, this method fully considers the association relationship of the global agent, realizes the formulation of the global optimal strategy, and is not limited to local optimization. It can effectively avoid the global suboptimal problem caused by local optimization and realize the global optimal control of the entire target area. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a schematic diagram of the architecture of a regional traffic coordination control system provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of control areas of each intelligent agent provided in an embodiment of the present application;

[0048] Figure 3 It is a flow chart of a distributed multi-agent regional traffic control method provided in an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of the training process of a sub-area decision model, an intersection decision model, a lane decision model and a flow direction decision model provided in an embodiment of the present application;

[0050] Figure 5 It is a flowchart of a method for determining a sub-area decision model provided in an embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of a flow chart of determining an intersection decision model provided by an embodiment of the present application;

[0052] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0054] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0055] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0056] In order to improve the efficiency of traffic management and the operational efficiency of urban traffic, the existing technology usually uses multi-agent traffic coordination control solutions. Specifically, multi-agents are used to simulate the information interaction between traffic participants, and each traffic participant is controlled based on a preset control strategy. Among them, traffic participants can be vehicles and traffic lights. Although this method can achieve coordinated control of traffic, the coordination and cooperation between agents depend on pre-set rules or models, and it is impossible to fully utilize the collaborative potential between agents, unable to achieve adaptive strategy adjustment, and unable to adapt to complex traffic environments. The coordination and communication between agents are restricted by rules or models, which limits the overall performance of the control system.

[0057] In the existing technology, deep reinforcement learning technology is also used for traffic control to optimize traffic light control and vehicle route planning. However, deep reinforcement learning technology requires all data in the controlled area to be used as training samples for model training. This may encounter computing resource limitations and data transmission delays when dealing with large-scale and dynamically changing traffic networks, making it difficult to update and optimize control strategies.

[0058] Based on this, the present application proposes a distributed multi-agent regional traffic control method, which is applied to a regional traffic coordination control system. The system includes a sub-area agent, a plurality of intersection agent sets, and controlled devices corresponding to each intersection agent set. Each of the intersection agent sets includes: an intersection agent, at least one lane agent, and at least one flow agent. This multi-level control system can enable the agent at the high level to effectively coordinate the overall strategy, and the agent at the low level to perform specific assigned tasks, ensuring information sharing and consistency strategies between agents, and can achieve rapid response in a highly dynamic traffic environment, reducing the impact of information delay and inconsistency on task execution. The specific method includes: the sub-area agent selects at least one intersection agent as the target intersection agent according to the current traffic information of the target area, and sends a regional collaborative control instruction to the target intersection agent. After the target intersection agent receives the associated strategy information sent by the associated intersection agent, it determines the target strategy information according to the current traffic information, regional collaborative control instructions, preference information and associated strategy information, so that each agent can dynamically adjust the decision strategy according to the traffic situation, and improve the intelligent decision-making level of the system. After determining the target strategy information, the target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow agent according to the target strategy information, so that the lane agent and the flow agent control the controlled device according to the target control instruction. In the process of issuing regional coordination instructions and formulating target strategy information, the application fully considers the association relationship of global agents to achieve global optimal strategy formulation, which is not limited to local optimization, and can effectively avoid the global suboptimal problem caused by local optimization, and achieve global optimal control of the entire target area.

[0059] Before introducing the distributed multi-agent regional traffic control method, the architecture of the regional traffic coordination control system is introduced first. Figure 1 Schematic diagram of the architecture of a regional traffic coordination control system provided by an embodiment of the present application, such as Figure 1 As shown, the regional traffic coordination control system includes: a sub-area intelligent agent, a plurality of intersection intelligent agent sets and controlled devices corresponding to each intersection intelligent agent set, each intersection intelligent agent set includes an intersection intelligent agent, at least one lane intelligent agent and at least one flow direction intelligent agent, each intelligent agent in each intersection intelligent agent set and the controlled devices corresponding to the intersection intelligent agent set are deployed in the intersection area corresponding to the intersection intelligent agent set.

[0060] Among them, sub-area agents, intersection agents, lane agents and flow agents are agents that can perceive the environment and take actions to achieve specific goals, and have autonomy, adaptability and interaction capabilities. Agents can be software, hardware or systems. If the agent is software, it can be deployed on cloud servers and edge devices. If the agent is hardware, the intersection agent, lane agent and flow agent are deployed in the corresponding intersection area. It is worth mentioning that the intersection agent set can include not only intersection agents, lane agents and flow agents, but also other agents that control traffic, such as signal agents.

[0061] For example, Figure 2 Schematic diagram of each intelligent agent control area provided in an embodiment of the present application. Figure 2 As shown, the sub-area agent can send a regional cooperative control instruction according to the current traffic information of the target area, the target area includes at least one intersection area, and the target intersection agent sends a target control instruction to the lane agent and the flow direction agent, so that the lane agent can control the control of the variable lane in the intersection area, and the flow direction agent can control the control of the lane flow direction in the intersection area. The variable lane and the flow direction are respectively part of the intersection area.

[0062] Next, refer to Figure 3 The distributed multi-agent regional traffic control method is introduced.

[0063] S301. The sub-area agent selects at least one intersection agent as a target intersection agent according to the current traffic information of the target area obtained through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each target intersection agent.

[0064] Optionally, the current traffic information includes the congestion status of each lane at each intersection. Exemplarily, the congestion status can be expressed in levels, and the current traffic information can include that the congestion level of lane a at intersection A is level one, and the congestion level of lane b at intersection A is level two.

[0065] Optionally, the sub-area decision model can implement an adaptive allocation mechanism for the contribution weights of the regional collaborative control effects of intelligent agents, strengthen the contribution weights of strategies that have a greater impact on the global goal, and realize adaptive distributed cooperative decision-making.

[0066] As an optional implementation, the sub-region decision model can be an actor-critic model. This model can accelerate the learning process of the agent through the critic's effective evaluation and feedback of the strategy, make the algorithm converge quickly, and reduce the consumption of computing resources in deployment and application.

[0067] Specifically, the current traffic information of the target area is input into the sub-area decision model, and the sub-area decision model outputs the identification of the target intersection agent and the regional collaborative control instructions for each target intersection agent according to the current traffic information. The regional collaborative control instructions received by each target intersection agent are instructions for the current traffic conditions of each target intersection agent.

[0068] It is worth mentioning that the regional collaborative control instructions usually include the task type or task goal for each target intersection agent, focusing on the control mode and relationship of each target intersection agent. For example, the regional collaborative control instructions can allow the target intersection agent to complete edge real-time sensing, collaborative green wave control between intersections, etc.

[0069] Among them, the intersection intelligent agents that are not the target intersection intelligent agents will not receive the regional collaborative control instructions, and will perform control work according to the regional collaborative control instructions of the previous stage or the pre-set control instructions.

[0070] S302: The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent.

[0071] The associated intersection agent may be an agent that has a preset association relationship with the current target intersection agent. For example, intersection agents in a set of adjacent intersection agents have a preset association relationship.

[0072] Optionally, the control strategy of the associated intersection agent is determined by the associated intersection agent according to current traffic information, regional collaborative control instructions and associated strategy information. It is worth noting that the associated strategy information here is the control strategy, current task and preference information corresponding to the intersection agent that has an associated relationship with the associated intersection agent.

[0073] The current task is the action currently performed by the associated intersection agent, that is, the action indicated by the control strategy performed by the associated intersection agent in the previous stage.

[0074] The preference information is the preset preference of the associated intersection agent. For example, the preference information of the associated intersection agent is that if the regional coordination instruction indicates to perform coordinated green wave control, the direction of the variable lane is adjusted first.

[0075] S303, the target intersection intelligent agent determines the target strategy information according to the intersection decision model pre-deployed in the target intersection intelligent agent, the current traffic information, the regional collaborative control instructions, the preference information and the associated strategy information, and the target strategy information indicates the control strategy of the target intersection intelligent agent.

[0076] Optionally, the intersection decision model can be a model that applies a deep reinforcement learning algorithm.

[0077] Optionally, the traffic information, regional collaborative control instructions, preference information and associated strategy information of the intersection where the target intersection agent is located in the current traffic information are input into the target intersection agent, and the target strategy information is output. The target strategy information can be used as the control strategy in the associated strategy information of the intersection agent that has an associated relationship with the current target intersection agent.

[0078] S304: The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information.

[0079] Specifically, the target control instructions received by each lane agent and the flow agent may be different. For example, the target control instruction may indicate that the left turn release time is extended, or the variable lane is switched between a straight lane and a left turn lane.

[0080] Optionally, some lane agents and some flow direction agents may not receive the target control instruction, or the received target control instruction instructs them to execute the action of the previous stage.

[0081] S305. The lane agent controls the controlled device according to the target control instruction through the lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through the flow direction decision model pre-deployed in the flow direction agent.

[0082] Optionally, the lane agent and the flow agent respectively determine the optimal lane control action and the optimal flow control action according to the received target control instruction, and send the above control actions to the controlled device so that the controlled device performs traffic control based on the received control actions.

[0083] Optionally, the controlled device may be, for example, a signal light, a turn signal light, and a variable lane indicator light.

[0084] The method in this embodiment is applied to a regional traffic coordination control system, which includes a sub-area intelligent agent, a plurality of intersection intelligent agent sets, and controlled devices corresponding to each intersection intelligent agent set, wherein each intersection intelligent agent set includes an intersection intelligent agent, at least one lane intelligent agent, and at least one flow direction intelligent agent. This multi-level control system can enable the intelligent agent at the high level to effectively coordinate the overall strategy, and the intelligent agent at the low level to perform specific assigned tasks, thereby ensuring information sharing and consistency strategies between intelligent agents, and can achieve rapid response in a highly dynamic traffic environment, and reduce the impact of information delay and inconsistency on the task execution effect. The specific method includes: the sub-area intelligent agent selects at least one intersection intelligent agent as the target intersection intelligent agent according to the current traffic information of the target area, and sends a regional coordinated control instruction to the target intersection intelligent agent. After the target intersection intelligent agent receives the associated strategy information sent by the associated intersection intelligent agent, the target strategy information is determined according to the current traffic information, the regional coordinated control instruction, and the associated strategy information, so that each intelligent agent can dynamically adjust the decision strategy according to the traffic situation, improve the intelligent decision-making level of the system, better cope with the complex and changeable traffic environment, and improve the task execution efficiency. After determining the target strategy information, the target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow agent according to the target strategy information, so that the lane agent and the flow agent control the controlled device according to the target control instruction. In the process of issuing multi-level regional coordination instructions and formulating target strategy information, this method fully considers the association relationship of the global agent, realizes the formulation of the global optimal strategy, and is not limited to local optimization. It can effectively avoid the global suboptimal problem caused by local optimization and realize the global optimal control of the entire target area.

[0085] Next, refer to Figure 4 The training process of the sub-area decision model, intersection decision model, lane decision model and flow direction decision model in the above steps is introduced. Figure 4 It is a training process diagram of a sub-area decision model, an intersection decision model, a lane decision model and a flow direction decision model provided in an embodiment of the present application.

[0086] S401. Using progressive transfer learning, the pre-acquired agent parameter library is transplanted into the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction decision model. The initial sub-area decision model, the initial intersection decision model and the initial lane flow direction decision model are all deployed on a cloud server.

[0087] Among them, progressive transfer learning is a machine learning method that can gradually transfer the learned knowledge to the target domain after learning in one or more source domains, and continuously optimize and adjust in the process to improve the performance of the target domain. This method is particularly suitable when the amount of data in the target domain is small. It can be learned in the data-rich source domain first, and then the model can be gradually adjusted to adapt to the target domain.

[0088] Specifically, the pre-acquired agent parameter library can be the model parameters of a trained model with similar functions to the initial sub-area decision model, the initial intersection decision model and the initial lane flow decision model. Transplanting the parameters in the agent parameter library to the initial sub-area decision model, the initial intersection decision model and the initial lane flow decision model can avoid the problem of inaccurate model output results due to the small number of model training samples. It is worth noting that when transplanting the parameters in the agent parameter library to the initial sub-area decision model, the initial intersection decision model and the initial lane flow decision model, only the model parameters corresponding to the homogeneous agents are transplanted.

[0089] Before starting training, the initial sub-area decision model, the initial intersection decision model, and the initial lane flow decision model are all deployed on the cloud server. The cloud server deploys parallel training tasks with a large computing load. After the model is trained on the cloud server, it can be sent to each edge device.

[0090] S402. Based on a distributed algorithm, the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction agent decision model are iteratively fused and trained according to a preset action set, a preset feedback set and the preset association relationship between each decision model to obtain a sub-area intermediate model, an intersection intermediate model and a lane flow direction decision model.

[0091] As an optional implementation, a distributed Max-plus algorithm can be used to accelerate the convergence of each model, thereby obtaining the optimal joint action of the intelligent agent in each model. Specifically, the optimal action of the node is transferred between the nodes of the undirected graph through the maximum a posteriori probability of the undirected graph model to solve the problem.

[0092] Among them, all intersections are regarded as a collaboration graph, in which each intersection is a vertex of the collaboration graph, and the roads between the intersections are the edges of the collaboration graph.

[0093] Each agent i selects an action a from the preset action set i , all agents can form a joint action a=(a 1 ,a 2 ,…,a n). Each agent can be a sub-area agent, an intersection agent, a lane agent, and a flow agent. In the iterative process, the optimal action a is determined among all joint actions by combining the preset feedback set. * , the optimal action is one of the multiple joint actions. When the Max-plus algorithm iteration is terminated, each agent i can obtain the optimal action according to the following formula (1):

[0094]

[0095] Among them, μ ij (a i ) represents the action sent by agent j to agent i, Γ(i) represents all neighboring agents of agent j, and the neighboring agents are determined according to the preset association relationship between the decision models. The termination condition of the iteration is ij Converge to a fixed value. Determine the sub-area intermediate model, intersection intermediate model and lane flow decision model based on the optimal actions corresponding to each agent.

[0096] The Q function of each intersection defined in the Max-plus algorithm can be expressed by the following formula (2):

[0097] f ij (a i ,a j )=Q i (s i ,a i |θ i )+Q j (s j ,a j |θ j ) (2)

[0098] Among them, f ij (a I ,a j ) is the joint function of agent i and agent j, that is, the action sent by agent i to agent j includes the maximum value of the joint function of agent i and agent j and the optimal action received by agent i from other nodes except agent j. According to the preset association relationship between the decision models, each agent only receives action information from agents with preset association relationship. i (s I ,a i |θ i ) and Q j (s j ,a j |θ j ) are the Q functions of agent i and agent j respectively.

[0099] In the process of iterative fusion training of the initial sub-area decision model, the initial intersection decision model and the initial lane flow agent decision model, each time an agent receives action information from an adjacent agent, the agent immediately updates its own optimal action information and transmits the updated optimal action information. The algorithm ends when the agent receives the end flag, and the current action of each agent is the optimal joint action.

[0100] In the above process, the distributed method is used to transmit messages, which can effectively speed up the convergence of the algorithm.

[0101] S403, deploying the intersection intermediate model to each intersection agent, deploying the lane intermediate model to each lane agent, and deploying the flow intermediate model to each flow agent.

[0102] As an optional implementation, the intersection intermediate model, lane intermediate model and flow intermediate model can be deployed on the edge device.

[0103] S404: Iteratively adjust the sub-area intermediate model to obtain a sub-area decision model.

[0104] Optionally, after the intersection intermediate model, lane intermediate model and flow direction intermediate model are deployed to the intelligent agent, the sub-area intermediate model is fine-tuned according to actual traffic information to obtain a sub-area decision model.

[0105] S405, iteratively adjust the intersection intermediate model, lane intermediate model and flow direction intermediate model respectively to obtain the intersection decision model corresponding to each intersection intermediate model, the lane decision model corresponding to each lane intermediate model and the flow direction decision model corresponding to each flow direction intermediate model.

[0106] When iteratively adjusting the sub-area intermediate model, the intersection intermediate model, lane intermediate model and flow direction intermediate model are also iteratively adjusted respectively, so as to realize the strategy of centralized training and distributed execution, and achieve mutual coordination between intersections within the sub-area and between sub-areas.

[0107] In this embodiment, the agent parameter library is transplanted to the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction decision model by using progressive transfer learning, thereby shortening the training process and avoiding the phenomenon that insufficient road network data leads to unsatisfactory model training results. Then, based on the distributed algorithm, the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction agent decision model are iteratively fused and trained according to the preset action set, the preset feedback set and the preset association relationship between the decision models to obtain the sub-area intermediate model, the intersection intermediate model and the lane flow direction decision model, thereby accelerating the convergence of the algorithm and obtaining the optimal joint action of multiple agents. After the intersection intermediate model, the lane intermediate model and the flow direction intermediate model are deployed, the sub-area intermediate model is iteratively adjusted to obtain the sub-area decision model, and the intersection intermediate model, the lane intermediate model and the flow direction intermediate model are iteratively adjusted respectively to obtain the intersection decision model corresponding to each intersection intermediate model, the lane decision model corresponding to each lane intermediate model and the flow direction decision model corresponding to each flow direction intermediate model, so that the model can make more flexible and accurate decisions in a complex and changeable traffic environment, and improve the adaptability and intelligence level of the overall traffic system.

[0108] Further, Figure 5 1 is a flow chart of a method for determining a sub-area decision model provided in an embodiment of the present application. Figure 5 As shown, next, the sub-area intermediate model in the above step S404 is iteratively adjusted to obtain a sub-area decision model for introduction.

[0109] S501, determining the weights corresponding to multiple intersection agents according to the preset learning rate of the intersection, the reward function parameters corresponding to each intersection agent, and the minimum batch processing quantity.

[0110] Optionally, the influence weight of the intersection agent on the global road network can be expressed by the following formula (3):

[0111]

[0112] Among them, λ j,i (t) is the current weight of the ith agent at the jth intersection, λ j,i (t-1) is the weight of the i-th agent at the j-th intersection in the previous stage, η is the preset learning rate of the intersection; θ Q is the reward function parameter corresponding to each intelligent agent at each intersection; B represents the minimum batch size, L b is the loss function under the current weights and model parameters. As an optional implementation, the loss function under each weight and model parameter can be calculated first, and then the influence weight of the global road network can be calculated and obtained.

[0113] S502. Determine the value of each intersection control strategy in each intersection control strategy set based on the weight corresponding to each intelligent agent at each intersection, the number of intelligent agents at each intersection, the current traffic information at each intersection, the current traffic information of the target area, the preset regional control strategy set, the regional control strategy set corresponding to each intersection control strategy set, and the evaluation coefficient corresponding to each intersection control strategy set.

[0114] Specifically, the value function considering the contribution allocation mechanism based on the symmetric key block encryption algorithm architecture can be the following formula (4):

[0115]

[0116]

[0117] Among them, s j,i represents the current traffic information of the ith agent at the jth intersection, s represents the overall traffic status in the sub-area, a represents the set of regional control strategies for the sub-area, and n j represents the number of agents in the jth intersection, which can be the number of all agents in the intersection agent set; μ j,i and The regional control strategy set of the ith agent in the jth intersection corresponds to the control strategy set of each intersection and the preset parameters of each strategy, Q j,i and is the evaluation coefficient corresponding to each intersection control strategy set and the related parameters of each evaluation coefficient, λ j,i Represents the weight corresponding to the i-th agent in the j-th intersection.

[0118] As an optional implementation, the regional control strategy set may be a strategy set established in advance using the optimal traffic flow allocation strategy, based on historical traffic conditions and prediction results generated based on historical traffic conditions. The regional control strategy set includes control strategy sets for each intersection. The prediction result may be a traffic situation after a preset time period. As another implementation, a strategy knowledge base may be established based on the optimal strategy during operation, and the strategy knowledge base may be used as part of the regional control strategy set.

[0119] S503 , iteratively modifying the sub-area intermediate model based on the value of each intersection control strategy in each intersection control strategy set to obtain a sub-area decision model.

[0120] Optionally, the parameters corresponding to the strategy with the highest value among the intersection control strategies in the intersection control strategy set may be used as the parameters of the sub-area decision model.

[0121] In this embodiment, the weights corresponding to multiple intersection agents are determined according to the preset learning rate of the intersection, the reward function parameters corresponding to the intelligent agents at each intersection, and the minimum batch processing quantity. Then, the value of each intersection control strategy in each intersection control strategy set is determined according to the weights corresponding to the intelligent agents at each intersection, the number of intelligent agents at each intersection, the current traffic information of each intersection, the current traffic information of the target area, the preset regional control strategy set, the regional control strategy set corresponding to each intersection control strategy set, and the evaluation coefficient corresponding to each intersection control strategy set. Based on the value of each intersection control strategy in each intersection control strategy set, the sub-area intermediate model is iteratively corrected to obtain the target sub-area decision model, so that the sub-area decision model adapts to the current traffic status of the intersection, and enables the intelligent agent to output the optimal strategy to achieve efficient traffic control.

[0122] Next, refer to Figure 6 How to iteratively adjust the intermediate models of intersections in the above step S404 to obtain the intersection decision models corresponding to the intermediate models of each intersection is introduced. Figure 6 It is a flow chart of determining an intersection decision model provided in an embodiment of the present application.

[0123] S601. Determine a predicted traffic situation based on current traffic information, and determine a current area control state based on the predicted traffic situation and historical traffic information.

[0124] Optionally, the predicted traffic situation may be the congestion situation of each lane after a preset time period, and the historical traffic information may be the congestion situation of each lane before a preset time period.

[0125] Optionally, the current regional control status may be the form of the current regional coordination work, completion time, preset requirements, etc.

[0126] S602: Determine whether the policy knowledge base contains the current regional control state, where the policy knowledge base includes a set of regional control states and policies corresponding to each regional control state.

[0127] Among them, each time each intelligent agent executes a strategy, the regional control state is obtained, and the regional control state and the strategy corresponding to the regional control state are counted to generate and update the strategy knowledge base.

[0128] Specifically, each regional control state in the regional control state set in the traversal strategy knowledge base is traversed, and for the traversed regional control state, it is determined whether the traversed regional control state is the same as the current regional control state.

[0129] S603: If yes, determine the current regional control strategy that matches the current regional control state in the strategy knowledge base, and iteratively adjust the intersection intermediate model based on the current regional control state and the current regional control strategy to obtain the intersection decision model corresponding to each intersection intermediate model.

[0130] If so, the current regional control strategy is determined. Based on the current regional control strategy and the current regional control state, the optimal action plan is determined from the action set of the intelligent agents at each intersection, and then the intermediate model of the intersection is iteratively adjusted based on the optimal action plan to obtain the decision model of each intersection.

[0131] S604: If not, iteratively adjust the intersection intermediate model according to the current area control state to obtain the intersection decision model corresponding to each intersection intermediate model.

[0132] If not, an objective function is generated according to the current regional control state, and then the intersection intermediate model is iteratively corrected according to the objective function to generate an intersection decision model.

[0133] It is worth mentioning that step S603 converges faster than step S604 and has a higher model training efficiency. Therefore, pre-establishing and updating the strategy knowledge base in real time can accelerate the training process of the intersection decision model.

[0134] In this embodiment, the predicted traffic situation is determined based on the current traffic information, and the current regional control state is determined based on the predicted traffic situation and historical traffic information, and it is determined whether the current regional control state is included in the strategy knowledge base. The intersection intermediate model is iteratively adjusted according to the judgment result to obtain the intersection decision model corresponding to each intersection intermediate model. This embodiment accelerates the training process of the intersection decision model through the strategy knowledge base, and improves the algorithm coordination success rate and application efficiency.

[0135] Furthermore, in the above step S604, the specific steps of iteratively adjusting the intersection intermediate model according to the current regional control state to obtain the intersection decision model corresponding to each intersection intermediate model are as follows:

[0136] Optionally, based on a multi-agent reinforcement learning strategy, an objective function is generated according to the current regional control state, a preset regional control strategy set, a preset value function, and a model parameter set corresponding to the control strategy set. The intersection intermediate model is iteratively corrected according to the objective function to generate an intersection decision model.

[0137] Specifically, for the i-th agent, set the estimation function of its Q function to Q θ , based on deep reinforcement learning, we find a set of parameter sets θ = (θ V ,θ A ), so that the estimated function Q of the Q function θSatisfies the Nash-Bellman equation. Taking the actor-critic model as an example, the actor is the control strategy, the critic is the value function, and the parameter set θ is the parameter set of the value function and the control strategy, where θ V Represents the model parameters of the value function; θ A The model parameters representing the control strategies of each intelligent agent. The algorithm of the objective function is as shown in the following formula (5). The parameters corresponding to the minimum objective function are used as the parameters of the intersection decision model to generate the intersection decision model.

[0138]

[0139] Among them, A θ is the preset advantage function, which is used to improve the algorithm training efficiency. θ is the value function for evaluating the reward; x m is the current regional control state of each agent; u m is the set of regional control strategies of agent m. The agents in this embodiment include intersection agents, lane agents and flow agents.

[0140] In this embodiment, a multi-agent reinforcement learning strategy is used to generate an objective function based on the current area control status, and the intermediate model of the intersection is iteratively corrected according to the objective function to generate an intersection decision model, thereby realizing the function of automatically following changes in traffic conditions and dynamically switching corresponding coordination goals and coordination strategies based on a deep reinforcement learning algorithm.

[0141] The iterative optimization process of the flow direction intermediate model is similar to the iterative optimization process of the lane intermediate model. Therefore, taking the iterative optimization process of the flow direction intermediate model as an example, the process of determining the flow direction decision model in the above step S404 is introduced below:

[0142] Optionally, based on the Markov decision strategy, the cumulative reward of each flow direction agent is determined according to the current traffic information, the flow action set, the flow control strategy set, the green light extension time corresponding to the flow control strategy set, the activation time of each flow direction agent, the occupancy rate of each flow direction agent, the flow rate of each flow direction agent and the weight of each flow direction agent under the current traffic information. The flow intermediate model is iteratively corrected according to the cumulative reward to obtain the flow decision model.

[0143] The Markov decision strategy is used to optimize the flow intermediate model. The state variables in the flow intermediate model are: in The green light time after deducting the minimum green light time of the flow direction from the waiting time of the flow direction agent i, that is, the activation time of the flow direction agent, o iis the ratio of the occupancy rate of the flow to agent i to the average occupancy rate of the flow to agents with associated relationships with agent i, q i is the ratio of the flow to agent i to the average flow to agents with associated relationships to agent i, w i is the weight of the flow direction corresponding to the flow direction agent i under the current traffic information.

[0144] Set the control variable to in Indicates the green light extension time. Set the initial state action pair of each flow agent i to (x i,0 ,u i,0 ), x i is the set of flow actions flowing to the agent, in the set of flow control strategies π i = {u i,1 ,u i,2 ,…}, the cumulative reward flowing to agent i The calculation formula is as follows (6):

[0145]

[0146] Among them, R i,t is the current traffic information flowing to agent i at t, which comes from the optimization target of the intersection; x i is the set of flow actions flowing to agent i; p i (x i,t+1 |x i,0:t ,u i,0:t ) is the flow to agent i in (x i,0:t ,u i,0:t ) is transformed into state x i,t+1 γ is the discount factor, which ranges from 0 to 1 and reflects the importance of future rewards; σ i,t is a binary variable used to indicate whether the current flow agent i is activated.

[0147] The optimal duration and the optimal action can be determined according to the following formula (7):

[0148]

[0149] Where n is the number of flow agents in the intersection agent set, u i,t is the set of flow actions performed by flow agent i, To solve the optimal action for agent i.

[0150] Finally, the parameters for determining the optimal duration and the optimal action are used as parameters in the flow direction decision model, and finally the flow direction decision model is obtained.

[0151] In this embodiment, based on the Markov decision strategy, the cumulative reward of each flow agent is determined according to the current traffic information, etc., and the flow intermediate model is iteratively corrected according to the cumulative reward to obtain the flow decision model, so as to realize specific flow control in combination with the agents with associated relationships.

[0152] Next, the process in which the lane agent controls the controlled device according to the target control instruction in the above step S305 through the lane decision model pre-deployed in the lane agent is introduced.

[0153] Optionally, the lane agent controls the controlled device according to the target control instructions and current traffic information through the lane decision model.

[0154] Optionally, the target control instruction includes a lane control instruction and a flow direction control instruction.

[0155] As an optional implementation, the lane agent determines the optimal lane control action through the lane decision model according to the lane control instructions and the current traffic information, and sends the optimal lane control action to the controlled device, thereby controlling the controlled device to perform traffic control.

[0156] As another optional implementation, the lane agent and the flow agent have an interactive relationship. After the lane agent determines the optimal lane control action, it sends the optimal lane control action to the flow agent. The flow agent determines the optimal flow control action through the flow decision model according to the flow control instruction, the optimal lane control action and the current traffic information, and sends the optimal flow control action to the controlled device, thereby controlling the controlled device to perform traffic control.

[0157] In this embodiment, the controlled equipment is controlled by the lane agent and the flow direction agent to achieve the specific implementation of traffic control.

[0158] The present application provides a regional traffic coordination control system, which includes: a sub-area intelligent agent, a plurality of intersection intelligent agent sets, and controlled devices corresponding to each intersection intelligent agent set, wherein each intersection intelligent agent set includes: an intersection intelligent agent, at least one lane intelligent agent, and at least one flow direction intelligent agent, wherein:

[0159] The sub-area agent selects at least one intersection agent as the target intersection agent according to the current traffic information of the target area acquired through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each target intersection agent, and the current traffic information includes the congestion status of each lane;

[0160] The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent;

[0161] The target intersection agent determines the target strategy information according to the current traffic information, the regional cooperative control instruction, the preference information and the associated strategy information through the intersection decision model pre-deployed in the target intersection agent, and the target strategy information indicates the control strategy of the target intersection agent;

[0162] The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information;

[0163] The lane agent controls the controlled device according to the target control instruction through the lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through the flow direction decision model pre-deployed in the flow direction agent.

[0164] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0165] The present application also provides an electronic device, such as Figure 7 As shown, it is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, including: a processor 701, a memory 702 and a bus. The memory 702 stores machine-readable instructions executable by the processor 701. When the computer device is running, the processor 701 communicates with the memory 702 through the bus. When the machine-readable instructions are executed by the processor 701, the processing of the above-mentioned distributed multi-agent regional traffic control method is performed.

[0166] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned distributed multi-agent regional traffic control method are executed.

[0167] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0168] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0169] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A distributed multi-agent regional traffic control method, characterized in that: The method is applied to a regional traffic coordination control system, which includes: a sub-area intelligent agent, a plurality of intersection intelligent agent sets, and controlled devices corresponding to each of the intersection intelligent agent sets, each of the intersection intelligent agent sets includes: an intersection intelligent agent, at least one lane intelligent agent, and at least one flow direction intelligent agent, and the method includes: The sub-area agent selects at least one intersection agent as a target intersection agent according to the acquired current traffic information of the target area through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each of the target intersection agents, wherein the current traffic information includes the congestion status of each lane; The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent; The target intersection agent determines target strategy information according to the current traffic information, the regional collaborative control instruction, the preference information and the associated strategy information through the intersection decision model pre-deployed in the target intersection agent, wherein the target strategy information indicates the control strategy of the target intersection agent; The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information; The lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through a flow direction decision model pre-deployed in the flow direction agent.

2. The distributed multi-agent regional traffic control method according to claim 1, characterized in that: The training process of the sub-area decision model, the intersection decision model, the lane decision model and the flow direction decision model includes: Using progressive transfer learning, the pre-acquired agent parameter library is transplanted into an initial sub-area decision model, an initial intersection decision model, and an initial lane flow direction decision model, wherein the initial sub-area decision model, the initial intersection decision model, and the initial lane flow direction decision model are all deployed on a cloud server; Based on the distributed algorithm, the initial sub-area decision model, the initial intersection decision model and the initial lane flow direction agent decision model are iteratively fused and trained according to the preset action set, the preset feedback set and the preset association relationship between the decision models to obtain the sub-area intermediate model, the intersection intermediate model and the lane flow direction decision model; Deploy the intersection intermediate model to each intersection agent, deploy the lane intermediate model to each lane agent, and deploy the flow intermediate model to each flow agent; Iteratively adjusting the sub-area intermediate model to obtain the sub-area decision model; The intersection intermediate model, lane intermediate model and flow direction intermediate model are iteratively adjusted respectively to obtain intersection decision models corresponding to each intersection intermediate model, lane decision models corresponding to each lane intermediate model and flow direction decision models corresponding to each flow direction intermediate model.

3. The distributed multi-agent regional traffic control method according to claim 2, characterized in that: The iterative adjustment of the sub-area intermediate model to obtain the sub-area decision model includes: Determine the weights corresponding to multiple intersection agents according to the preset learning rate of the intersection, the reward function parameters corresponding to each intersection agent, and the minimum batch processing quantity; Determine the value of each intersection control strategy in each intersection control strategy set according to the weight corresponding to each intersection intelligent agent, the number of intelligent agents at each intersection, the current traffic information of each intersection, the current traffic information of the target area, the preset regional control strategy set, the regional control strategy set corresponding to each intersection control strategy set, and the evaluation coefficient corresponding to each intersection control strategy set; The sub-area intermediate model is iteratively modified based on the value of each intersection control strategy in each intersection control strategy set to obtain a sub-area decision model.

4. The distributed multi-agent regional traffic control method according to claim 2, characterized in that: Iteratively adjusting the intersection intermediate models to obtain intersection decision models corresponding to the intersection intermediate models, including: Determine a predicted traffic situation based on current traffic information, and determine a current area control state based on the predicted traffic situation and historical traffic information; Determining whether the current regional control state is included in the policy knowledge base, wherein the policy knowledge base includes a set of regional control states and policies corresponding to each of the regional control states; If so, determining the current regional control strategy that matches the current regional control state in the strategy knowledge base, and iteratively adjusting the intersection intermediate model based on the current regional control state and the current regional control strategy to obtain the intersection decision model corresponding to each of the intersection intermediate models; If not, the intersection intermediate model is iteratively adjusted according to the current area control state to obtain the intersection decision model corresponding to each intersection intermediate model.

5. The distributed multi-agent regional traffic control method according to claim 4 is characterized in that: The iterative adjustment of the intersection intermediate model according to the current area control state to obtain the intersection decision model corresponding to each intersection intermediate model includes: Based on the multi-agent reinforcement learning strategy, an objective function is generated according to the current regional control state, the preset regional control strategy set, the preset value function and the model parameter set corresponding to the control strategy set; The intersection intermediate model is iteratively modified according to the objective function to generate an intersection decision model.

6. The distributed multi-agent regional traffic control method according to claim 2, characterized in that: The flow direction intermediate model is iteratively adjusted to obtain the flow direction decision model corresponding to the flow direction intermediate model, including: Based on the Markov decision strategy, the cumulative reward of each flow agent is determined according to the current traffic information, the flow action set, the flow control strategy set, the green light extension time corresponding to the flow control strategy set, the activation time of each flow agent, the occupancy rate of each flow agent, the flow rate of each flow agent, the weight of each flow agent under the current traffic information and the association relationship of each flow agent; The flow direction intermediate model is iteratively modified according to the accumulated reward to obtain a flow direction decision model.

7. The distributed multi-agent regional traffic control method according to claim 1, characterized in that: The lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, including: The lane agent controls the controlled device through the lane decision model according to the target control instruction and the current traffic information.

8. A regional traffic coordination control system, characterized in that: The regional traffic coordination control system includes: a sub-area agent, a plurality of intersection agent sets and controlled devices corresponding to each of the intersection agent sets, each of the intersection agent sets includes: an intersection agent, at least one lane agent and at least one flow direction agent, wherein: The sub-area agent selects at least one intersection agent as a target intersection agent according to the acquired current traffic information of the target area through the sub-area decision model pre-deployed in the sub-area agent, and sends a regional collaborative control instruction to each of the target intersection agents, wherein the current traffic information includes the congestion status of each lane; The target intersection agent receives the associated strategy information sent by the associated intersection agent, where the associated strategy information is used to indicate the control strategy, current task and preference information of the associated intersection agent; The target intersection agent determines target strategy information according to the current traffic information, the regional collaborative control instruction, the preference information and the associated strategy information through the intersection decision model pre-deployed in the target intersection agent, wherein the target strategy information indicates the control strategy of the target intersection agent; The target intersection agent sends a target control instruction to at least one lane agent and / or at least one flow direction agent according to the target strategy information; The lane agent controls the controlled device according to the target control instruction through a lane decision model pre-deployed in the lane agent, and / or the flow direction agent controls the controlled device according to the target control instruction through a flow direction decision model pre-deployed in the flow direction agent.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the distributed multi-agent regional traffic control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the distributed multi-agent area traffic control method as described in any one of claims 1 to 7.

Images