Multi-agent based multi-objective traffic path guidance method and system

Abstract

The application belongs to the technical field of traffic management, in particular to a multi-target traffic path induction method and system based on multiple agents, a simulation module for calibrating simulated traffic demand, an optimization control module for maximizing the cumulative reward of the entire system over time, an iterative optimization module for realizing policy iteration optimization in information interaction, a judgment module for judging whether the system benefit tends to be stable through the information of the simulation module, and an output module for outputting the control scheme. The application provides differentiated path induction information, avoids traffic congestion caused by homogeneous navigation information, realizes dynamic balance of traffic flow, significantly improves the overall efficiency of the traffic network and reduces vehicle emissions, and provides strong support for the development of green traffic.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of traffic management, in particular to a multi-agent based multi-objective traffic path induction method and system. BACKGROUND

[0002] With the popularity of mobile communication technology, path induction, which provides real-time traffic information and optimal traffic routes for drivers through information systems (such as smartphone applications, variable message signs, and vehicle applications), plays an increasingly important role in urban traffic. Studies have shown that path induction can significantly affect the efficiency and greenhouse gas emissions of the traffic system. In the case of dense urban road traffic networks, improving the efficiency of the traffic system and reducing traffic carbon emissions through path induction methods has become a feasible approach.

[0003] However, traditional path induction methods often focus on a single objective (such as minimizing travel time), ignoring the balance between traffic efficiency and greenhouse gas emissions, as well as the diverse needs of urban management or the preferences of travelers. Therefore, how to establish an effective multi-agent reinforcement learning method to provide differentiated traffic path induction for travelers and flexibly switch between different induction targets to better meet the traffic induction needs in different situations and grasp and enhance the regulatory capacity of multi-objective traffic induction on the entire urban traffic system is a key problem that needs to be solved. SUMMARY

[0004] The present application aims to solve the problem of single induction target in existing navigation systems, leading to unbalanced utilization of traffic resources, and proposes a multi-agent based multi-objective traffic path induction method and system. The present application considers multiple objectives such as travel time and emissions to provide differentiated induction paths for travelers with different origin-destination pairs (OD pairs) and different travelers with the same OD pair, improving the efficiency of the traffic system while reducing system emissions and improving the overall performance of the traffic system. At the same time, the present application can also flexibly adjust the induction strategy according to the needs of urban management, thereby improving the efficiency and response capability of urban traffic management. The specific technical solutions are as follows:

[0005] In one aspect of the present application, a multi-agent based multi-objective traffic path induction system is provided, which comprises:

[0006] A simulation module for calibrating simulated traffic demand;

[0007] An optimization control module for maximizing the cumulative reward of the entire system over time;

[0008] An iterative optimization module for implementing policy iteration optimization in information interaction;

[0009] A judgment module is configured to determine whether the system benefit tends to be stable by using the simulation module information.

[0010] An output module is configured to output the management and control scheme.

[0011] Specifically, in the simulation module, SUMO is used as a micro-simulation platform, traffic demand is calibrated, and 3-4 candidate paths are pre-screened for each OD pair according to feasibility.

[0012] Specifically, in the optimization control module, a multi-agent Markov decision process model is included, and the optimization goal is to maximize the cumulative reward of the entire system over time by improving the strategy of the agent.

[0013] Specifically, the decision variables of the agent include state, action and reward.

[0014] The state is a measure of the traffic condition, represented by a vector S t =(b t,i ,v t,i ,l t,i ), where b t,i represents the number of vehicles on each edge at a given time step, v t,i represents the average vehicle speed, and l t,i represents the average travel distance. The action of each agent j is a split rate, which is proportionally allocated among the pre-determined candidate paths.

[0015] The action vector is represented as where m(j) represents the number of candidate paths. For example, for a specific OD pair d, the action represents the proportion of vehicles allocated to candidate paths 1, 2 and 3.

[0016] The reward is defined as a comprehensive indicator reflecting traffic efficiency (TT) and environmental impact (CE), represented as:

[0017] r t =-αTT t -(1-α)*γ*CE t

[0018] where TT t =∑TT t,e is the travel time of vehicle e under the current traffic condition, and CE t =∑CE t,i is the carbon dioxide emission of road segment i.

[0019] The parameter gamma is used to unify the order of magnitude of the two values, and the specific value is estimated based on the total travel time and total emissions in the simulation scenario, and the total travel time and total emissions of a single target optimization;

[0020] The optimization target of the model is adjusted by the weight alpha: when alpha = 0, the model target is to reduce emissions; when alpha = 1, the model prioritizes travel time and efficiency;

[0021] When alpha = 0.5, the model will reduce emissions and improve travel efficiency as a common goal, aiming to find an induced optimization solution that balances the two goals.

[0022] Specifically, in the optimization control module, an initialization reinforcement learning algorithm is also used to optimize the traffic efficiency and emissions of the entire network;

[0023] The initialization reinforcement learning algorithm is a VDA2C algorithm that uses transitions obtained from a multi-agent Markov decision process as input to learn the state-action mapping of the agent, and returns the reward action according to the updated policy.

[0024] Specifically, in the optimization control module, a traveler preference attribute is also set.

[0025] Specifically, the implementation of policy iteration optimization in information interaction includes: selecting OD pairs and determining corresponding candidate paths according to traffic demand analysis and simulation scenarios;

[0026] An information interaction mechanism is established between the simulation module and the optimization control module, and reward information obtained by the agent taking various actions in different states is collected through the simulation process;

[0027] A dynamic adjustment mechanism in reinforcement learning is introduced, and a policy gradient method is used to optimize the policy.

[0028] Specifically, the system revenue stability criterion is determined by the simulation module information: when the reward value output by the simulation module tends to be stable, or the algorithm reaches the maximum number of iterations, the optimization process is considered to be completed.

[0029] Specifically, the output control scheme includes verifying the effectiveness of the optimization policy by comparing the efficiency and emissions of the traffic system before and after optimization after the policy optimization is completed.

[0030] Another aspect of the present application provides a multi-agent based multi-objective traffic path induction method, comprising the following steps:

[0031] S1: Initialize the simulation module to calibrate the simulated traffic demand;

[0032] S2: Initialize the optimization control module to maximize the cumulative reward of the entire system over time;

[0033] S4: implementing policy iteration optimization in information interaction by using an iteration optimization module;

[0034] S5: realizing whether the system benefit tends to be stable by using a simulation module information judging module;

[0035] S6: outputting a control scheme by an output module.

[0036] Beneficial effects:

[0037] 1. The application provides differentiated path guidance information, avoids traffic congestion caused by homogeneous navigation information, realizes dynamic balance of traffic flow, significantly improves the overall efficiency of the traffic network and reduces vehicle emissions, and provides strong support for the development of green traffic. Based on the traffic data of Liushi Expressway in Hangzhou and the surrounding area, the improvement effect of the guidance strategy optimization on the system performance is proved under multiple targets. Under the ecological target, the overall efficiency of the road network is improved by 13.90%, and the carbon dioxide emissions are reduced by 16.09%, which achieves a significant win-win effect in realizing the two targets of improving efficiency and reducing emissions.

[0038] 2. The application provides a multi-agent Markov decision framework suitable for real-time traffic guidance, which can flexibly adapt to different guidance targets and respond to diversified urban traffic management needs. The framework improves the accuracy and adaptability of traffic management, and provides a solid theoretical and practical foundation for efficient guidance in complex traffic scenarios.

[0039] 3. The application fully considers the individual preferences of travelers, proposes differentiated and customized path guidance strategies, can deeply explore the potential laws of traveler demand and behavior patterns, and significantly improves the satisfaction and compliance of travelers in path selection. Under the optimized guidance strategy, the paths allocated to travelers with different preference attributes show significant inclination, opening up a new direction for personalized traffic management. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a structural schematic diagram of the guidance optimization method of the application;

[0041] Figure 2 is a schematic diagram of the execution process of the simulation module of the application;

[0042] Figure 3 is a schematic diagram of the execution process of the optimization control module of the application;

[0043] Figure 4 is a schematic diagram of the multi-agent Markov decision process framework of the application;

[0044] Figure 5is a schematic diagram of the calculation process of the algorithm. DETAILED DESCRIPTION

[0045] The technical solutions of the present application will be described in detail below with reference to the drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application. In the case of no conflict, the technical features in the embodiments and the embodiments can be combined with each other.

[0046] The term "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents that the front and rear associated objects are in an "or" relationship.

[0047] Embodiment 1, a multi-agent-based multi-objective traffic path induction system, the system comprising: a simulation module for calibrating simulated traffic demand;

[0048] An optimization control module for maximizing the cumulative reward of the entire system over time;

[0049] An iterative optimization module for implementing policy iteration optimization in information interaction;

[0050] A judgment module for judging whether the system benefit tends to be stable through the simulation module information;

[0051] An output module for outputting the control scheme.

[0052] Specifically, in actual application, first, the simulation module is initialized for calibrating the simulated traffic demand, that is, selecting SUMO (a open source microscopic traffic simulation tool) as a microscopic simulation platform, calibrating the traffic demand, that is, calibrating the simulated traffic demand according to the road flow, speed and other data of the actual traffic environment, in order to improve the calculation efficiency and ensure the acceptance of the driver to the induction, according to the feasibility (i.e. the acceptable detour degree) for each OD, 3-4 preset alternative paths are encoded into the routing file. Write a Python script for communication with the TraCI interface of the platform, use commands such as getLastStepMeanSpeed to record specific indicators (such as road travel time, carbon emissions, etc.); according to the induction strategy, that is, the vehicle shunting rate between a plurality of alternative paths obtained through the control module, the setRouteID command is used to adjust the vehicle path.

[0053] Then, the optimization control module is initialized, which contains a multi-agent Markov decision process model. The optimization objective is to maximize the cumulative reward of the whole system over time by improving the strategy of the agent. The decision variables of the agent include the current state and the selected action, and the strategy is the mapping rule from state to action. Considering the complexity of the traffic problem, each OD pair is responsible for decision-making and optimization by an agent. According to the induction target (such as the shortest travel time or the least emission) and the corresponding alternative path, the state, action and reward of each agent in the Markov state transition equation are defined. Compared with the single-agent model, the multi-agent design can significantly reduce the action dimension of each agent (the split rate of all alternative paths), thereby improving the learning efficiency of the model.

[0054] More specifically, in the present application, the state, action and reward are defined as follows:

[0055] The state is a measure of the traffic condition, represented by a vector S t = (b t,i , v t,i , l t,i ), where b t,i represents the number of vehicles on each edge at a given time step, v t,i represents the average vehicle speed, and l t,i represents the average travel distance. The action of each agent j is an induction split rate, which is a proportional allocation among the pre-determined alternative paths. The action vector is represented as where m(j) represents the number of alternative paths. For example, for a specific OD pair d, the action represents the proportion of vehicles allocated to alternative paths 1, 2 and 3.

[0056] The reward is defined as a comprehensive indicator reflecting the travel time (TT) of traffic efficiency and the carbon dioxide emission (CE) reflecting environmental impact, represented as:

[0057] r t = -αTT t -(1-α)*γ*CE t

[0058] where TT t =∑TT t,e is the travel time of vehicle e to complete the whole journey under the current traffic condition, and CE t =∑CE t,iis the CO2 emission of link i. Parameter γ is used to unify the order of magnitude of the two values, and the specific value is estimated based on the total travel time and total emission in the simulation scenario, as well as the total travel time and total emission of the single target optimization. The optimization objective of the model is adjusted by weight α: when α = 0, the model objective is to reduce emissions; when α = 1, the model prioritizes travel time and efficiency; when α = 0.5, the model will reduce emissions and improve travel efficiency as a common goal, aiming to find an induced optimization solution that can achieve a balance between the two goals.

[0059] where initializing the reinforcement learning algorithm includes employing the VDA2C algorithm to enhance the cooperation between multiple OD pairs to optimize the traffic efficiency and emissions of the entire network. The state-action mapping (i.e., policy) of the agent is learned using the transitions (i.e., state, action, and reward) obtained from the multi-agent Markov decision process as input, and the reward action is returned according to the updated policy. The value function V is employed to evaluate the expected cumulative reward of the policy π after the agent takes an action in a given state, and the value function V is calculated by the formula where represents the local value function of the i-th agent in the next state s', r i (s) is the immediate reward of agent i in state s, φ i is the parameter of agent i. The global value function is then calculated by mixing the local value functions of multiple agents: where V tot is the global value function, representing the comprehensive evaluation of the joint action of all agents, g ψ is the mixing network used to combine local values, ψ is the parameter of the mixing network, and u is the joint action of all agents. VDA2C updates the value function through the following steps by continuously interacting with the environment and iterating within a given time range:

[0060] ① Sampling phase: the agent interacts with the environment and collects data (i.e., state, action, reward, etc.), which is called the sampling phase.

[0061] ② Feedback phase: the agent calculates the local value function based on its own experience and shares information. The local value functions of all agents are combined by the mixing network to generate the global value function.

[0062] ③ Update phase: according to the global value function and the immediate reward, the advantage function A t is calculated, and the policy is updated. The advantage function is calculated by the following formula:

[0063] A(s') = r(s') + λV tot (s') - V tot (s)

[0064] where r(s') is the immediate reward of the next state s', V tot (s') and V tot (s) is the value of the global value function at the next state s' and the current state s, and λ is the discount factor that balances the immediate reward and future rewards.

[0065] Based on the advantage function A(s'), the policy gradient is calculated by the policy gradient method and the policy parameters θ are updated. The specific calculation method is as follows:

[0066]

[0067] where σ is the learning rate, which controls the step size of parameter update.

[0068] Through the above steps, the agent gradually improves the decision-making ability, thereby improving the traffic efficiency and emission reduction effect of the entire network.

[0069] When setting the traveler preference attributes, when the traveler preferences are different, each agent corresponds to a group of travelers with a preference characteristic. Different preference attributes are set for different agents, such as time-sensitive and environmentally friendly. According to the preference attributes, the weights in the reward function are adjusted to reflect the needs of different travelers. For example, time-sensitive travelers value the reduction of travel time more, while environmentally friendly travelers pay more attention to the reduction of carbon dioxide emissions.

[0070] In the process of information interaction, policy iteration optimization is a crucial step in the process of SUMO traffic simulation environment optimization. In order to ensure the effectiveness and adaptability of the policy, a series of information interaction steps are taken.

[0071] According to the traffic demand analysis and simulation scenario, the OD pair is selected and the corresponding alternative paths are determined. In the process of path selection, the current traffic conditions and road characteristics need to be considered comprehensively to ensure the rationality and effectiveness of the selected path in practical application. An information interaction mechanism is established between the simulation module and the optimization control module, and the reward information obtained by the agent taking various actions in different states is collected through the simulation process. These reward information provides key data support for subsequent policy optimization. The agent updates the policy using the collected reward information, thereby improving the decision-making quality in specific situations. Through continuous policy optimization, the agent gradually adapts to complex traffic environments and makes more effective decisions. To further improve the learning ability of the agent, a dynamic adjustment mechanism in reinforcement learning is introduced, and a policy gradient method is used to optimize the policy. The optimized policy is transmitted back to the simulation module through the information interaction mechanism to generate new states and reward information, thereby further updating the policy.

[0072] Wherein, when the reward value output by the simulation module tends to be stable, or the algorithm reaches the maximum iteration number, it is considered that the optimization process is completed.

[0073] When the output control scheme, that is, after the strategy optimization is completed, the effectiveness of the optimization strategy is verified by comparing the efficiency and emission of the traffic system before and after optimization. The influence of different strategies on different paths and different OD vehicles is analyzed to ensure that the proposed path induction strategy can improve the overall traffic management effect in practical application.

[0074] Embodiment 2, a multi-agent based multi-objective traffic path induction method, comprising the following steps:

[0075] S1: initializing the simulation module to calibrate the simulation traffic demand;

[0076] S2: initializing the optimization control module to maximize the cumulative reward of the entire system over time;

[0077] S4: using the iterative optimization module to realize strategy iteration optimization in information interaction;

[0078] S5: using the simulation module information to judge whether the system benefit tends to be stable through the judgment module;

[0079] S6: outputting the control scheme through the output module.

[0080] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-agent based multi-objective traffic path induction system, characterized in that: The system comprises: a simulation module for calibrating simulated traffic demand; an optimal control module for maximizing cumulative rewards of the entire system over time; an iterative optimization module for implementing policy iteration optimization in information interaction; a judgment module for judging whether the system revenue tends to be stable through simulation module information; an output module for outputting a control scheme; In the simulation module, SUMO is used as a micro-simulation platform, traffic demand is calibrated, and 3-4 candidate paths are pre-screened for each OD pair according to feasibility; In the optimal control module, a multi-agent Markov decision process model is included, and the optimization goal is to maximize the cumulative rewards of the entire system over time by improving the strategy of the agent; The decision variables of the agent include state, action and reward; where the state is a measure of the traffic conditions, represented by a vector where, denotes the number of vehicles on each edge at a given time step, denotes the average speed, denotes the average travel distance, and the action of each agent is a diversion rate, which proportionally allocates among pre-determined alternative routes; The action vector is represented as wherein represents the number of alternative paths; The reward is defined as travel time reflecting traffic efficiency CO2 emissions influenced by the reaction environment The integrated index is represented as: wherein, is a vehicle a travel time to complete the journey under current traffic conditions, is a road segment a carbon dioxide emission; Parameters The order of magnitude for unifying two values is estimated based on the total travel time and total emissions in the simulation scenario, and the total travel time and total emissions of individual target optimization; By weight Adjust the optimization goal of the model: when emissions; when travel time and efficiency; When The model will reduce emissions and improve traffic efficiency as a common goal, aiming to find induced optimization solution that can achieve a balance of the two goals; Implementing policy iteration optimization in information interaction includes: selecting OD pairs and determining corresponding candidate paths according to traffic demand analysis and simulation scenarios; An information interaction mechanism is established between the simulation module and the optimal control module, and reward information obtained by the agent in different states by taking various actions is collected through the simulation process; A dynamic adjustment mechanism in reinforcement learning is introduced, and a policy gradient method is used to optimize the strategy; The standard for judging whether the system revenue tends to be stable through simulation module information is: when the reward value output by the simulation module tends to be stable, or the algorithm reaches the maximum number of iterations, the optimization process is considered to be completed; Outputting a control scheme includes verifying the effectiveness of the optimization strategy by comparing the efficiency and emissions of the traffic system before and after optimization after the completion of the policy optimization.

2. The multi-agent based multi-objective traffic path induction system of claim 1, wherein: In the optimal control module, an initialization reinforcement learning algorithm is also used to optimize the traffic efficiency and emissions of the entire network; The initialization reinforcement learning algorithm is VDA2C algorithm, which uses the transition obtained from the multi-agent Markov decision process as input to learn the state-action mapping of the agent, and returns the reward action according to the updated strategy.

3. The multi-agent based multi-objective traffic path induction system of claim 1, wherein: In the optimal control module, a traveler preference attribute is also set.

4. A multi-agent based multi-objective traffic path guidance method using the multi-agent based multi-objective traffic path guidance system according to claim 1, characterized by: The steps include: S1: initialize the simulation module to calibrate the simulated traffic demand; S2: initialize the optimal control module to maximize the cumulative rewards of the entire system over time; S4: use the iterative optimization module to implement policy iteration optimization in information interaction; S5: use the judgment module to judge whether the system revenue tends to be stable through simulation module information; S6: output the control scheme through the output module.

Images