Multi-mode traffic simulation method based on multi-agent sequential decision

Abstract

The invention discloses a multi-mode traffic simulation method based on multi-agent sequential decision, and the method comprises the steps: building a traffic simulation model comprising a road network and a plurality of vehicles, each vehicle being an agent, and building an individual objective function and a global objective function of each vehicle; a sequential interaction model is established, all vehicles execute actions according to the strategy in each time step, then the road traffic state and the vehicle state are updated, Q values of all vehicles are calculated based on the sequential interaction model, the individual target function and the global target function, and the strategy is updated according to the Q values; and after simulation calculation meets a termination condition, stopping calculation and outputting a result. According to the method, all vehicles make sequential decisions and execute actions, the obtained traffic phenomenon is more authentic and reasonable, the method has two termination conditions of a congestion judgment mechanism and a convergence judgment mechanism, the model has more comprehensive functions, and the temporary state in the model can more accurately simulate the influence of emergency situations on the traffic flow.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent traffic simulation technology, specifically relating to a multi-modal traffic simulation method based on multi-agent sequential decision-making. Background Technology

[0002] With the continuous growth of motor vehicle ownership, urban traffic congestion has become increasingly serious. To find ways to alleviate traffic congestion, it is necessary to first study the phenomenon of traffic congestion. By modeling, reproducing, and analyzing complex traffic phenomena, researchers can better understand the formation mechanism of traffic congestion. Simulating urban traffic patterns is a commonly used method for studying traffic congestion. With the popularization and development of reinforcement learning algorithms, simulation methods based on intelligent agents have become a research hotspot. For example, Chinese patent titled "A Large-Scale Intelligent Management and Control Method for Urban Road Networks Based on Mobile Phone Signaling Data" (application number: 202511015135.X) proposes a road network management and control method based on an Actor-Critic architecture. This method treats the entire road network as an intelligent agent and simulates the overall traffic conditions of the road network based on the dynamic changes in the states of vehicles and intersection traffic lights, thereby determining traffic flow allocation strategies. Another example is Chinese patent titled "A Public Transportation Modal Prediction Method Based on Multi-Agent Simulation" (application number: 201910666673.3), which proposes a simulation method that treats both public transportation vehicles and urban residents as intelligent agents and simulates population flow through the interactive actions of residents and public transportation vehicles.

[0003] Both of the above methods improve simulation accuracy through agent-based policy optimization mechanisms, but they also have certain limitations. Treating the entire road network as a single agent results in a simplistic representation of vehicle types and behavioral patterns, deviating significantly from reality. Schemes that treat both public transportation vehicles and urban residents as agents focus more on simulating population flow, excluding vehicles other than buses and failing to reflect the overall traffic situation. Furthermore, current agent-based simulation models often suffer from a common flaw: the use of a synchronous update mechanism. In each simulation time step, all agents simultaneously make decisions and update their states based on a global snapshot at the same moment. While this synchronous update mechanism reduces the complexity of the simulation computation, it ignores the fundamental fact that vehicles in traffic flow are sequentially influenced by traffic conditions, simultaneously affecting the traffic flow itself. This leads to simulation results that fail to accurately capture the dynamic changes in traffic flow and are ill-suited to reflecting the formation process of traffic congestion.

[0004] Therefore, it is necessary to propose a simulation method for traffic flow dynamics that can set up multiple types of vehicles such as private cars and buses, simulate the sequential actions of all vehicles, and consider the mutual influence between different vehicles, in order to simulate multi-modal traffic flow, that is, traffic flow that includes multiple types of vehicles. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-modal traffic simulation method based on multi-agent sequential decision-making. This method treats each vehicle in the traffic flow as an agent, and all vehicles make sequential decisions based on their current driving state and related vehicle actions, and then execute actions to achieve dynamic simulation of multi-modal traffic flow changes.

[0006] The technical solution adopted in this invention is as follows: A multi-agent sequential decision-making-based multi-modal traffic simulation method includes the following steps: S1. A traffic simulation model is established based on the urban road network structure, road characteristics, and vehicle features. The traffic simulation model includes a road network and various vehicles. The road network includes multiple roads, each with a unique serial number. Each vehicle in the model is an intelligent agent, and all vehicles travel along the roads and each vehicle has a unique serial number.

[0007] S2. Set the initial driving state and action space for each vehicle, wherein the action space is a set of actions that a vehicle can choose; establish an agent state vector to describe the vehicle driving state and a road state vector to describe the road traffic state.

[0008] S3. Establish an individual objective function for each vehicle, which is used to calculate the individual reward value of the vehicle; at the same time, establish a global objective function to calculate the overall reward value of the road network.

[0009] S4. Set the time step of the traffic simulation model and establish a sequential interaction model; within each time step, all vehicles execute actions in sequence based on the current strategy, and each vehicle updates the road state vector after executing an action. In addition, within each time step, all vehicles update the agent state vector, and then, based on the sequential interaction model, the strategy of all vehicles is updated by combining the updated road state vector and the updated agent state vector.

[0010] S5. Set the termination condition of the traffic simulation model, simulate traffic phenomena based on the traffic simulation model, and determine whether the obtained simulation result meets the termination condition at each time step. If it meets the condition, stop the calculation and output the simulation result; otherwise, proceed to the next time step.

[0011] Further optimization, the road network is distributed on a plane. Step S2 specifically includes the following steps: S2.1. Establish a Cartesian coordinate system for measuring the entire road network; if a certain position is the end of multiple roads, the position is taken as a node; if a certain position is only the end of one road, the position is taken as an endpoint; all roads are connected to other roads only through nodes; the road network also includes multiple fixed points, each fixed point is located at a non-end position on a certain road, and any two fixed points are located at different positions.

[0012] S2.2. The endpoints and fixed points are collectively referred to as route base points. The process of determining the route includes multiple stages, one of which is as follows: take a route base point without a start point mark as the start point and another route base point as the end point. Then determine one or more optional routes from the start point to the end point. Each optional route includes one road or multiple connected roads. After determining all optional routes, mark the route base point that is the end point as the end point. Take a route base point without an end point mark as the end point and continue to determine optional routes until all route base points except the start point have end point marks. Then mark the route base point that is the start point in the current stage as the start point and remove all end point marks. Then proceed to the next stage and repeat the above process until all route base points have start point marks. Each optional route has a unique number.

[0013] S2.3. Determine the action type corresponding to each vehicle type. The action is represented as a vector containing multiple elements, and each action corresponds to a unique number for the same vehicle. The traffic simulation model includes two vehicle types: private cars and buses. The actions of private cars are... a i,b =[p i,j , v i, adj ], i For vehicle serial number, b Number the action. a i,b For vehicles i Selectable actions b p ij For vehicles i Selected route feature values, j v is the optional route number. i, adj For vehicles i The target speed of the vehicle; the bus's movement is a i,b = [a i, stop , a i, speed , p i,j ], a i, stop For docking items, a i, stop ∈{ c 1, c 2}, c 1. c Both are constants andc 1≠ c 2, a i, speed For speed adjustment, -V d, max ≤ a i, speed ≤ V a, max V d, max For the maximum deceleration rate, V a, max Maximum acceleration rate; vehicle i All selectable actions are grouped into a set, which is the action space A. i .

[0014] S2.4. The positions of all vehicles are described by point coordinates, and an agent state vector is established. s i,t The state vector of a private car is represented as s i,t = [ x i,t , y i,t , v speed i,t , q i,t , s k,t The state vector of the bus is represented as: s i,t = [ x i,t , y i,t , v speed i,t , q i,t , s k,t , cap rem i,t ], t For a moment, x i,t , y i,t vehicles i exist t Location at any given moment x axis, y Axis coordinates v speed i,t For vehicles i exist t The speed at any given moment q i,t For vehicles i exist t The remaining journey at that time, sk,t for t The road where the moment is k The characteristic parameters, whose values ​​are determined by the road state vector. S road k,t Decide, k For vehicles i exist t The road number at which the time is located. cap rem i,t For buses i exist t The remaining passenger capacity at any given time. cap rem i,t = cap i - pass i,t , cap i For buses i Rated passenger capacity, pass i,t for t Bus schedule i The actual passenger capacity.

[0015] S2.5. Set the basic parameters for each road. The basic parameters for a given road include... or k With cap k , or k For roads k The congestion coefficient, cap k For roads k Rated traffic flow; establish road state vector S road k,t , S road k,t = [ v k,t , t k,t ,cap k - v k,t ], v k,t for t Road at all times k Traffic flow on the road t k,t for t Vehicles pass through the road at any time k The estimated time required.

[0016] In reality, when driving a private car, people usually only consider two factors: which route to take and how fast to drive. However, buses have restrictions on stopping at stations and maintaining a relatively stable speed. Their routes are usually fixed, but considering that buses may need to change routes in case of emergencies, the actions of a bus involve three elements: stopping, etc. stop The two optional parameters represent stopping at the next station and not stopping at the next station, respectively. (Speed ​​adjustment item a) speed This is used to control bus travel time within a certain range. Buses usually travel along fixed routes, but may temporarily change routes in case of congestion or other emergencies. Therefore, the action includes p. ij Since the weight and driving requirements of a bus change with the number of passengers, the state vector of a bus includes the remaining passenger capacity. Except when performing actions, the speed of all vehicles remains constant.

[0017] Further optimization, step S3 specifically includes the following steps: S3.1. Establish the individual objective function for all vehicles, the specific process is as follows: S3.1.1. Establish the individual objective function for private cars. The individual objective function for private cars is expressed as follows: (1) in, f i,t (·) represents the individual objective function. t i,t for t Time vehicle i Time already traveled within the road network, d i,t for t Moment Private Car i The difference in distance between the current route and the shortest route. t i,t , d i,t All are with s i,t Relevant parameters, α , β These are the first type of weighting coefficients. α , β >0 and α + β = 1.

[0018] S3.1.2. Establish the individual objective function for the bus. The individual objective function for the bus is expressed by the following formula: (2) in, t i,planFor buses i The planned travel time, o i,t for t Bus schedule i The level of crowding inside the car , c , d , e These are the second type of weighting coefficients. c , d , e All are greater than 0 and c + d + e = 1.

[0019] S3.2. Establish the global objective function, which is expressed as follows: (3) in, S t It is a set of road state vectors. S t = { S road 1,t , S road 2,t , …, S road M,t}, F (·) represents the global objective function. N t for t The total number of vehicles in the road network at any given time. M The total number of roads, g The congestion penalty coefficient, g >0.

[0020] The individual objective function is used to measure the difference between the actual driving state of a vehicle and the ideal driving state, where the ideal driving state corresponds to the expectation of driving state by private car owners or bus drivers in reality. The global objective function is used to measure the traffic flow carried by the entire road network.

[0021] Further optimization involves updating the agent's state vector and the road state vector in step S4, specifically including the following steps: S4.1. Determine the initial strategy for all vehicles and set the number of time steps for the simulation calculation; within each time step, the vehicles... i Before executing the action, a random number is generated within the range [0, 1] and compared with the critical value. i t In comparison, if the selected random number is less than i t Then the vehiclei Execute the action according to the current strategy, if the selected random number is not less than i t Then the vehicle i Randomly select one action from all currently available actions to execute. , T total This represents the number of time steps in the simulation calculation. Within the initial time step, the current policy is the initial policy.

[0022] S4.2. Each time step is divided into... N t Each step involves one vehicle performing an action; within any given step, all road traffic flow is updated using the following formula: (4) in, t 1 + ( t 0) and t 1 + ( t 0 - 1 represent the current step and the previous step, respectively. For the current steps within the road k Traffic flow changes l Entering or leaving the road within the current step k Vehicle serial number, For the set, all paths entering or leaving the current step. k All vehicle serial numbers are included in this set, Δ v i For vehicles i Road occupancy, for private cars |Δ v i | = C , C The bus Δ is a constant. v i The calculation formula is , cap private The standard passenger capacity for private cars is for vehicles entering the road. k vehicles i Δ v i >0, for leaving the road k vehicles i Δ v i <0.

[0023] The estimated time required for vehicles to travel on any road is updated using the following formula: (5) t When 0 = 0, the traffic flow of any road is the initial traffic flow of that road at the current time step. The estimated time for vehicles to travel on any road is the initial estimated time for that road at the current time step. .

[0024] S4.3 Road State Vector Update at Each Time Step N t After that, based on the current time step vehicle i of s i,t With the action performed a i,b and vehicles i Current location k Initial traffic flow in the next time step v k,t+1 The initial estimated time required for vehicles to pass through this road. t k,t+1 The vehicle was obtained i Agent state vector at the next time step s i,t+1 .

[0025] When the vehicle executes actions, it uses a greedy algorithm. The vehicle may execute actions based on the current policy or randomly, avoiding policy bias towards a particular action during long computation periods. This helps in arriving at the optimal policy. Furthermore, considering that the policy becomes increasingly adapted to the agent's operating environment as computation time increases, therefore... i t The value increases with time, causing the agent to rely more on current decisions to execute actions in later stages. Each time step is divided into... N t The simulation is divided into two steps: first, all vehicles need to perform actions sequentially in the method of this invention; second, since vehicles are constantly moving and the overall state of traffic flow is constantly changing, setting up steps allows for a more accurate description of the changing distribution of vehicles in the road network. Because vehicles are constantly moving, while the size of the road network is fixed, and vehicles are considered to have left the network once they reach their destination, in order to maintain the continuity of traffic flow, vehicles originating from the starting point must continuously exist throughout the simulation process. Therefore, the total number of vehicles in the road network... N t It is a time variable.

[0026] Further optimization involves the sequential interaction model in step S4, which is based on the Q-learning algorithm. The process of constructing the sequential interaction model specifically includes the following steps: S4.4. After a vehicle performs an action, its mixed reward is calculated using the following formula: (6) in, , Vehicles at a certain time step i Mixed rewards and individual rewards, for t Global reward at any moment l As a reward weighting coefficient, l ∈[0, 1], F ( S t The objective function value is calculated based on the road traffic flow and estimated travel time within the target step, where the target step refers to the number of vehicles within that time step. i The steps of performing the action.

[0027] S4.5. Define the neighborhood of all vehicles. Within a certain time step, the neighborhood of a vehicle includes the road where the vehicle was located before performing the action, as well as the upstream and downstream roads of the road where the vehicle is located according to the current route. When no vehicles in the road network have performed any actions, all vehicles in the neighborhood are considered related vehicles of that vehicle. Calculate the average action of the vehicle using the following formula: (7) in, a ' i,t for t Time vehicle i Average movement, A i,t For a set, the set includes t Time vehicle i The actions performed by all relevant vehicles respectively n For the relevant vehicle serial number, for t Time vehicle i The total number of related vehicles.

[0028] S4.6. Establish a Q-function for calculating the policy Q-value, and update the policy Q-value for all vehicles using the following formula: (8) in, Q i,t and Q i,t+1 They are respectively t Time and t + 1 moment vehicle i The Q-value of the strategy, α 0 represents the learning rate. c Let be the discount factor, and max be a function. Q (·) is the Q function. Q ( s i,t+1 , ai,b , a ' i,t )express t Time vehicle i In state s i,t+1 With average movement a ' i,t Next action a i,b Q value at that time a i,b ∈A i , Q i, 0 = 0.

[0029] S4.7. According to Q i,t+1 For vehicles i Update the current strategy, and use the updated strategy as the vehicle's strategy. i The current strategy for the next time step.

[0030] Traditional traffic flow simulation models rely solely on vehicle states to determine actions. However, in reality, drivers' decision-making processes are often influenced by the actions of surrounding vehicles. For instance, if a large number of vehicles at the front of the traffic flow turn onto a particular road, drivers behind may perceive that road as too congested and choose alternative routes. This invention's method considers not only vehicle states but also the actions of nearby vehicles when formulating strategies, resulting in simulated traffic conditions that more closely resemble reality.

[0031] Further optimization is achieved by setting the traffic simulation model termination conditions in step S5 as follows: S5.1. Establish a congestion determination mechanism, which includes road network saturation conditions, congested road ratio conditions, and total delay convergence conditions, as detailed below: S5.1.1. Set the sample stage, wherein the sample stage includes the current time step. W For each consecutive time step, the average road saturation at that time step is calculated based on the simulation results obtained at a certain time step in the sample phase, and an overall saturation threshold is set. e ρ The average road saturation is calculated using the following formula: (9) in, r t for t The average road saturation at each time step; the average road saturation at each time step within the sample phase needs to be calculated using formula (9), and then the standard deviation of the average road saturation at all time steps is calculated. If the obtained standard deviation is less than e ρIf the simulation results meet the road network saturation condition, then the simulation results are considered to meet the condition; otherwise, they are not.

[0032] S5.1.2. Calculate the saturation of each road within a certain time step based on the simulation results obtained in the sample phase, and set an individual saturation threshold. e c and saturation ratio threshold e ratio The formula for calculating the saturation of a certain road is: (10) in, r k,t for t The Road of Time k The saturation level; the saturation level of all roads at a given time step is compared with... e c Compare and select those with saturation greater than 100%. e c The roads that are considered congested are determined after all roads have been compared, resulting in the number of congested roads within that time step. M The ratio of these ratios is called the saturation ratio. The saturation ratio needs to be calculated for each time step within the sample phase, and then the standard deviation of the saturation ratios for all time steps in the sample phase is calculated. If the obtained standard deviation is less than... e ratio If the simulation results meet the congested road ratio condition, then the simulation results are considered to meet the condition; otherwise, they are not.

[0033] S5.1.3. Calculate the total delay of the road network within a certain time step based on the simulation results obtained in the sample phase, and set a total delay threshold. e D The formula for calculating the total delay of the road network is: (11) in, D total,t for t Total network delay at each time step; the total network delay for each time step within the sample phase must be calculated using formula (11), and then the standard deviation of the total network delay for all time steps is calculated. If the obtained standard deviation is less than e D If the simulation results meet the total delay convergence condition, then the simulation results are considered to have met the condition; otherwise, they are not.

[0034] S5.1.4. If the network saturation condition, the congested road ratio condition, and the total delay convergence condition are all satisfied at a certain time step, then congestion is considered to have occurred in the simulation results obtained at that time step; otherwise, congestion is considered not to have occurred. After a certain number of time steps in the simulation calculation, the simulation results are judged by the aforementioned congestion determination mechanism at each time step. If congestion has occurred at a certain time step, or the number of time steps has reached a certain threshold, the congestion determination mechanism is used to determine the congestion determination. T total If the simulation calculation meets the termination condition, it is considered to have met the termination condition; otherwise, it is considered not to have met the termination condition.

[0035] Further optimization is achieved by setting the traffic simulation model termination conditions in step S5 as follows: S5.2. Establish a convergence determination mechanism, which includes policy convergence conditions, traffic distribution convergence conditions, and global convergence conditions, as detailed below: S5.2.1. Calculate the global policy change degree based on the simulation results of the sample stage, and set the global policy threshold. e π The formula for calculating the global policy variability is as follows: (12) in, Q average,t For the degree of change in global strategy, t Let be the ordinal number of the time step in the sample phase. t For the current moment; if Q average,t < e π If the simulation results meet the convergence conditions of the strategy, then the simulation results are considered to meet the convergence conditions of the strategy; otherwise, they are not.

[0036] S5.2.2. Based on the simulation results of the sample stage, establish the road network traffic vector for each time step. The road network traffic vector at a certain time step is represented as V. t = [ v 1,t , v 1,t ,…, v M,t It calculates the global traffic variability based on all road network traffic vectors and sets a global traffic threshold. e v The global traffic variability is calculated using the following formula: (13) in, V average,t The degree of global traffic change; if V average,t < e vIf the simulation results meet the convergence condition for the flow distribution, then the simulation results are considered to meet the convergence condition; otherwise, they are not.

[0037] S5.2.3. Calculate the global target change during the sample phase and set the global target change threshold. e F The formula for calculating the global target change is as follows: (14) in, F average,t This represents the change in the global objective; if F average,t < e F If the simulation results meet the global convergence condition, then the simulation results are considered to meet the global convergence condition; otherwise, they are not.

[0038] S5.2.4. If all three convergence conditions—strategy convergence condition, flow distribution convergence condition, and global convergence condition—are met at a certain time step, the traffic flow in the simulation results obtained at that time step is considered stable; otherwise, it is considered not yet stable. After a certain number of time steps in the simulation calculation, the simulation results are judged through the aforementioned convergence determination mechanism at each time step. If the traffic flow is stable at a certain time step, or the number of time steps has reached a certain threshold, the simulation results are considered stable. T total If the simulation calculation meets the termination condition, it is considered that the simulation calculation has met the termination condition; otherwise, it is considered that the termination condition has not yet been met.

[0039] Considering that the traffic simulation model is used to study traffic congestion, it needs to have two functions: one is to simulate the formation of traffic congestion, and the other is to determine whether changing certain factors can alleviate or eliminate congestion given that it is known to occur. Therefore, the method of this invention includes two termination conditions: a congestion determination mechanism and a convergence determination mechanism. The congestion determination mechanism is used to confirm the formation of traffic congestion in the simulation results, while the convergence determination mechanism is used to confirm that a stable traffic flow has been formed in the simulation results.

[0040] Further optimization involves a traffic simulation model that includes normal and temporary states. In the temporary state, the road network contains one or more special roads, each with a temporary rated traffic flow cap. new k With temporary congestion coefficient or new k Under normal conditions, the formula for calculating the Q function is described as follows: The traffic simulation model under temporary conditions includes a temporary reward mechanism, in which cap new k , or new kReplace cap respectively k , or k Substituting into steps S2.5-S4.2, the temporary vehicle mixed reward calculated using formula (6) is denoted as... The formula for calculating the temporary state Q function is: (15) in, oh For incremental learning rate, oh ∈(0, 0.5).

[0041] In reality, traffic conditions are frequently affected by various factors such as sudden congestion, traffic accidents, road construction, and adjustments to traffic policies. These factors often cause congestion, and when these factors occur, the road conditions change accordingly. Drivers passing through such roads will make decisions based on their past knowledge of the road and the specific on-site conditions. Therefore, this invention sets up normal and temporary states. In the normal state, all roads in the road network are in a normal state. In the temporary state, some roads in the road network experience state changes due to the aforementioned factors. The impact of these changes on the agent's decision-making is reflected in the temporary vehicle hybrid reward.

[0042] Further optimization involves determining the optional route in step S2.2 based on the shortest path algorithm, specifically including the following steps: S2.2.1. Establish the road weight calculation formula, which is expressed in the following form: (16) in, C k For roads k The weight value, t k To pass through the road k Estimated time required l k For roads k Length, oh 1. oh 2. oh 3 are all road weight coefficients, and oh 1+ oh 2+ oh 3 = 1.

[0043] S2.2.2. Based on the road network topology, determine all routes connecting the starting point and the ending point, denoted as initial routes. Calculate the weight value of each initial route. The weight value of a given initial route is the sum of the weight values ​​of all roads along that route. Then, find the initial route with the smallest weight value and denote it as the baseline route. Denote the number of nodes in the baseline route as K. In order from the starting point to the ending point, all nodes in the baseline route are sequentially designated as node 1, node 2, ..., node K. ψ The portion between nodes is denoted as L( ψ ), ψ ∈[1, K] and ψ The integer is used to determine all connections based on the road network topology. ψ The route between the node and the destination is denoted as ψ any type of route ψ Class of routes and L( ψ All routes formed by connecting these points are categorized into set A. ψ ),like ψ If the number of routes of type 1 is 1, then A( ψ -1), A( ψ -2), ..., A(1) are all optional routes. If the number of routes of type 1 is 1, then all initial routes are optional routes. In any of the above routes, the same road can be passed at most once.

[0044] Before setting off on a drive, people often choose a route. If there are no congestion or other factors that might interfere with their decision-making, they will continue along that route. If they encounter interference, they will choose another route based on the situation. The above method of determining the available routes helps to better simulate the process of changing routes during real-world driving.

[0045] The beneficial effects of the method of the present invention are as follows: 1. In the traffic simulation model established by the method of the present invention, all vehicles make sequential decisions and execute actions, and the decision-making process considers not only the vehicle's own state, but also the influence of the actions of nearby vehicles on the decision, making the traffic phenomena simulated by the method of the present invention more realistic and reasonable. 2. The method of the present invention has two termination conditions: a congestion determination mechanism and a convergence determination mechanism, which enables the traffic simulation model to simulate both stable traffic conditions and congested traffic conditions, thus giving the traffic simulation model more comprehensive functions. 3. Traffic simulation models include normal and temporary states. Temporary states allow for a more accurate simulation of the impact of emergencies on traffic flow. Attached Figure Description

[0046] Figure 1 A schematic diagram of the overall process of the traffic simulation method of this invention.

[0047] Figure 2 A schematic diagram of the agent policy update process. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below through specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example 1: A multi-agent sequential decision-making method for multi-modal traffic simulation. The overall process of this method is as follows: Figure 1 As shown, the specific steps include: S1. A traffic simulation model is established based on the real road structure and distribution and vehicle characteristics of a certain area of ​​the city. The traffic simulation model includes a road network and various vehicles. The road network includes multiple roads, which are distributed on a plane. Each road corresponds to a unique number. Each vehicle in the model is an intelligent agent. All vehicles travel along the roads and each vehicle has a unique number.

[0050] S2. Set the initial driving state and action space for each vehicle. The initial driving state is determined based on the actual driving state of the vehicle when it enters the area. The action space is a set of actions that a vehicle can choose. Establish an agent state vector to describe the vehicle's driving state and a road state vector to describe the road traffic state. This includes the following steps: S2.1. Establish a Cartesian coordinate system for measuring the entire road network; if a certain position is the end of multiple roads, the position is taken as a node; if a certain position is only the end of one road, the position is taken as an endpoint; all roads are connected to other roads only through nodes; the road network also includes multiple fixed points, each fixed point is located at a non-end position on a certain road, and any two fixed points are located at different positions.

[0051] S2.2. The endpoints and fixed points are collectively referred to as route base points. The process of determining the route includes multiple stages, one of which is as follows: take a route base point without a start point mark as the start point and another route base point as the end point. Then determine one or more optional routes from the start point to the end point. Each optional route includes one road or multiple connected roads. After determining all optional routes, mark the route base point that is the end point as the end point. Take a route base point without an end point mark as the end point and continue to determine optional routes until all route base points except the start point have end point marks. Then mark the route base point that is the start point in the current stage as the start point and remove all end point marks. Then proceed to the next stage and repeat the above process until all route base points have start point marks. Each optional route has a unique number.

[0052] The optional routes are determined based on the shortest path algorithm, and the specific process includes the following steps: S2.2.1. Establish the road weight calculation formula, as shown in formula (16).

[0053] S2.2.2. Based on the road network topology, determine all routes connecting the starting point and the ending point, denoted as initial routes. Calculate the weight value of each initial route. The weight value of a given initial route is the sum of the weight values ​​of all roads along that route. Then, find the initial route with the smallest weight value and denote it as the baseline route. Denote the number of nodes in the baseline route as K. In order from the starting point to the ending point, all nodes in the baseline route are sequentially designated as node 1, node 2, ..., node K. ψ The portion between nodes is denoted as L( ψ ), ψ ∈[1, K] and ψ The integer is used to determine all connections based on the road network topology. ψ The route between the node and the destination is denoted as ψ any type of route ψ Class of routes and L( ψ All routes formed by connecting these points are categorized into set A. ψ ),like ψ If the number of routes of type 1 is 1, then A( ψ -1), A( ψ -2), ..., A(1) are all optional routes. If the number of routes of type 1 is 1, then all initial routes are optional routes. In any of the above routes, the same road can be passed at most once.

[0054] S2.3. Determine the action type corresponding to each vehicle type. The action is represented as a vector containing multiple elements, and each action corresponds to a unique number for the same vehicle. The traffic simulation model includes two vehicle types: private cars and buses. The actions of private cars are...a i,b =[p i,j , v i, adj ], i For vehicle serial number, b Number the action. a i,b For vehicles i Selectable actions b p i,j For vehicles i Selected route feature values, j v is the optional route number. i, adj The target speed is determined by the speed limit of the road; the bus's movement is... a i,b = [a i, stop , a i, speed , p j ], a i, stop For docking items, a i, stop ∈{0, 1}, if the bus i Stop at the next station. i, stop = 1, without stopping, then a i, stop = 0, a i, speed For speed adjustment, -V d, max ≤ a i, speed ≤ V a, max V d, max For the maximum deceleration rate, V a, max For the maximum acceleration rate, V d, max With V a, max The values ​​are determined by the requirements for bus operation; vehicle i All selectable actions are grouped into a set, which is the action space A. i .

[0055] S2.4. The positions of all vehicles are described by point coordinates, and an agent state vector is established. s i,t The state vector of a private car is represented as s i,t = [ x i,t , y i,t , v speed i,t , q i,t , s k,t The state vector of the bus is represented as: s i,t = [ x i,t , yi,t , v speed i,t , q i,t , s k,t , cap rem i,t ], t For a moment, x i,t , y i,t vehicles i exist t Location at any given moment x axis, y Axis coordinates v speed i,t For vehicles i exist t The speed at any given moment q i,t For vehicles i exist t The remaining journey at that time, s k,t for t The road where the moment is k The characteristic parameters are the values ​​of the road state vector. S road k,t Second norm, k For vehicles i exist t The road number at which the time is located. cap rem i,t For buses i exist t The remaining passenger capacity at any given time. cap rem i,t = cap i - pass i,t , cap i For buses i Rated passenger capacity, pass i,t for t Bus schedule i The actual passenger capacity.

[0056] S2.5. Set the basic parameters for each road. The basic parameters for a given road include... or k With cap k , ork For roads k The congestion coefficient, cap k For roads k The rated traffic flow or k With cap k These are all parameters describing the inherent properties of the road, and their values ​​are obtained by combining the BPR function with traffic flow data. The traffic flow data is the historical road traffic data on which the traffic simulation model is based; a road state vector is established. S road k,t , S road k,t = [ v k,t , t k,t , cap k - v k,t ], v k,t for t Road at all times k Traffic flow on the road t k,t for t Vehicles pass through the road at any time k The estimated time required.

[0057] S3. Establish an individual objective function for each vehicle, which is used to calculate the individual reward value of the vehicle; at the same time, establish a global objective function for calculating the overall reward value of the road network, specifically including the following steps: S3.1. Establish the individual objective function for all vehicles, the specific process is as follows: S3.1.1. Establish the individual objective function for private cars. The individual objective function for private cars is expressed as shown in formula (1).

[0058] S3.1.2. Establish the individual objective function for the bus. The individual objective function for the bus is expressed as shown in formula (2).

[0059] S3.2. Establish a global objective function, which is expressed as shown in formula (3).

[0060] S4. Set the time step of the traffic simulation model and establish a sequential interaction model; in each time step, all vehicles execute actions in sequence based on the current strategy, and update the road state vector after each vehicle executes an action. In addition, update the agent state vectors corresponding to all vehicles in each time step, and then update the strategy of all vehicles based on the sequential interaction model, combined with the updated road state vectors and the updated agent state vectors.

[0061] The update of the agent's state vector and the road state vector specifically includes the following steps: S4.1. Set up a dynamic traffic scenario. Each vehicle in the traffic simulation model is trained through this dynamic traffic scenario, and the trained strategy becomes the vehicle's initial strategy. Set the number of time steps for the simulation calculation. Within each time step, the vehicle... i Before executing the action, a random number is generated within the range [0, 1] and compared with the critical value. i t In comparison, if the selected random number is less than i t Then the vehicle i Execute the action according to the current strategy, if the selected random number is not less than i t Then the vehicle i Randomly select one action from all currently available actions to execute. , T total This represents the number of time steps in the simulation calculation. Within the initial time step, the current policy is the initial policy.

[0062] S4.2. Each time step is divided into... N t Each step involves one vehicle performing an action; within any given step, all road traffic flow is updated using formula (4), where |Δ v i | = 1, bus Δ v i The calculation formula is , cap private This example represents the standard passenger capacity for a private car. cap private = 1, for entering the road k vehicles i Δ v i >0, for leaving the road k vehicles i Δ v i <0; The estimated time required for vehicles to travel on any road is updated using formula (5); t When 0 = 0, the traffic flow of any road is the initial traffic flow of that road at the current time step. The estimated time for vehicles to travel on any road is the initial estimated time for that road at the current time step. .

[0063] S4.3 Road State Vector Update at Each Time StepN t After that, based on the current time step vehicle i of s i,t With the action performed a i,b and vehicles i Current location k Initial traffic flow in the next time step v k,t+1 The initial estimated time required for vehicles to pass through this road. t k,t+1 The vehicle was obtained i Agent state vector at the next time step s i,t+1 .

[0064] The sequential interaction model is based on the Q-learning algorithm. The process of constructing the sequential interaction model specifically includes the following steps: S4.4. After a vehicle performs an action, its mixed reward is calculated using formula (6).

[0065] S4.5. Define the neighborhood of all vehicles. Within a certain time step, the neighborhood of a vehicle includes the road where the vehicle was before it performed the action, as well as the upstream and downstream roads of the road where the vehicle is located according to the current route. When no vehicle in the road network has performed any action, all vehicles in the neighborhood are related vehicles of the vehicle. The average action of the vehicle is calculated by formula (7).

[0066] S4.6. Establish a Q function for calculating the strategy Q value, and update the strategy Q value of all vehicles using formula (8).

[0067] The traffic simulation model includes normal and temporary states. In the temporary state, there are one or more special roads in the road network, and each special road has a temporary rated traffic flow cap. new k With temporary congestion coefficient or new k Under normal conditions, the formula for calculating the Q function is described as follows: The traffic simulation model under temporary conditions includes a temporary reward mechanism, in which cap new k , or new k Replace cap respectively k , or k Substituting into steps S2.5-S4.2, the temporary vehicle mixed reward calculated using formula (6) is denoted as... The temporary state Q function is calculated using formula (15).

[0068] S4.7. According to Q i,t+1 For vehicles i Update the current strategy, and use the updated strategy as the vehicle's strategy. i The current strategy for the next time step, the vehicle strategy update process is as follows: Figure 2 As shown.

[0069] S5. Set the termination condition of the traffic simulation model, simulate traffic phenomena based on the traffic simulation model, and determine whether the obtained simulation result meets the termination condition at each time step. If it meets the condition, stop the calculation and output the simulation result; otherwise, proceed to the next time step.

[0070] The specific termination conditions for the traffic simulation model are as follows: S5.1. Establish a congestion determination mechanism, which includes road network saturation conditions, congested road ratio conditions, and total delay convergence conditions, as detailed below: S5.1.1. Set the sample stage, wherein the sample stage includes the current time step. W For each consecutive time step, the average road saturation at that time step is calculated based on the simulation results obtained at a certain time step in the sample phase, and an overall saturation threshold is set. e ρ The average road saturation is calculated using formula (9); the average road saturation for each time step within the sample phase must be calculated using formula (9), and then the standard deviation of the average road saturation for all time steps is calculated. If the obtained standard deviation is less than e ρ If the simulation results meet the road network saturation condition, then the simulation results are considered to meet the condition; otherwise, they are not.

[0071] S5.1.2. Calculate the saturation of each road within a certain time step based on the simulation results obtained in the sample phase, and set an individual saturation threshold. e c and saturation ratio threshold e ratio The saturation of a certain road is calculated using formula (10); the saturation of all roads at a certain time step is then compared with... e c Compare and select those with saturation greater than 100%. e c The roads that are considered congested are determined after all roads have been compared, resulting in the number of congested roads within that time step. MThe ratio of these ratios is called the saturation ratio. The saturation ratio needs to be calculated for each time step within the sample phase, and then the standard deviation of the saturation ratios for all time steps in the sample phase is calculated. If the obtained standard deviation is less than... e ratio If the simulation results meet the congested road ratio condition, then the simulation results are considered to meet the condition; otherwise, they are not.

[0072] S5.1.3. Calculate the total delay of the road network within a certain time step based on the simulation results obtained in the sample phase, and set a total delay threshold. e D The total network delay is calculated using formula (11); the total network delay for each time step within the sample phase must be calculated using formula (11), and then the standard deviation of the total network delay for all time steps is calculated. If the obtained standard deviation is less than e D If the simulation results meet the total delay convergence condition, then the simulation results are considered to have met the condition; otherwise, they are not.

[0073] S5.1.4. If the network saturation condition, the congested road ratio condition, and the total delay convergence condition are all satisfied at a certain time step, then congestion is considered to have occurred in the simulation results obtained at that time step; otherwise, congestion is considered not to have occurred. After a certain number of time steps in the simulation calculation, the simulation results are judged by the aforementioned congestion determination mechanism at each time step. If congestion has occurred at a certain time step, or the number of time steps has reached a certain threshold, the congestion determination mechanism is used to determine the congestion determination. T total If the simulation calculation meets the termination condition, it is considered to have met the termination condition; otherwise, it is considered not to have met the termination condition.

[0074] Example 2: The traffic simulation method described in this example is based on the method described in Example 1. The specific termination conditions for the traffic simulation model set in the method are as follows: S5.2. Establish a convergence determination mechanism, which includes policy convergence conditions, traffic distribution convergence conditions, and global convergence conditions, as detailed below: S5.2.1. Calculate the global policy change degree based on the simulation results of the sample stage, and set the global policy threshold. e π The global policy change degree is calculated using formula (12); if Q average,t < e π If the simulation results meet the convergence conditions of the strategy, then the simulation results are considered to meet the convergence conditions of the strategy; otherwise, they are not.

[0075] S5.2.2. Based on the simulation results of the sample stage, establish the road network traffic vector for each time step. The road network traffic vector at a certain time step is represented as V. t = [ v 1,t, v 1,t ,…, v M,t It calculates the global traffic variability based on all road network traffic vectors and sets a global traffic threshold. e v The global flow change rate is calculated using formula (13); if V average,t < e v If the simulation results meet the convergence condition for the flow distribution, then the simulation results are considered to meet the convergence condition; otherwise, they are not.

[0076] S5.2.3. Calculate the global target change during the sample phase and set the global target change threshold. e F The change in the global target is calculated using formula (14); if F average,t < e F If the simulation results meet the global convergence condition, then the simulation results are considered to meet the global convergence condition; otherwise, they are not.

[0077] S5.2.4. If all three convergence conditions—strategy convergence condition, flow distribution convergence condition, and global convergence condition—are met at a certain time step, the traffic flow in the simulation results obtained at that time step is considered stable; otherwise, it is considered not yet stable. After a certain number of time steps in the simulation calculation, the simulation results are judged through the aforementioned convergence determination mechanism at each time step. If the traffic flow is stable at a certain time step, or the number of time steps has reached a certain threshold, the simulation results are considered stable. T total If the simulation is successful, the termination condition is considered met; otherwise, the termination condition is considered not yet met. The other parts of the traffic simulation method described in this embodiment are the same as those described in Embodiment 1.

Claims

1. A multi-agent sequential decision-making method for multi-modal traffic simulation, characterized in that, Specifically, the following steps are included: S1. A traffic simulation model is established based on the urban road network structure, road characteristics, and vehicle features. The traffic simulation model includes a road network and various vehicles. The road network includes multiple roads, each with a unique serial number. Each vehicle in the model is an intelligent agent, and all vehicles travel along the roads and each vehicle has a unique serial number. S2. Set the initial driving state and action space for each vehicle, wherein the action space is a set of selectable actions for a vehicle; establish an agent state vector to describe the vehicle driving state and a road state vector to describe the road traffic state. S3. Establish an individual objective function for each vehicle, which is used to calculate the individual reward value of the vehicle; at the same time, establish a global objective function to calculate the overall reward value of the road network; S4. Set the time step of the traffic simulation model and establish a sequential interaction model; in each time step, all vehicles execute actions in sequence based on the current strategy, and each vehicle updates the road state vector after executing an action. In addition, in each time step, all vehicles update the agent state vector, and then based on the sequential interaction model, the strategy of all vehicles is updated by combining the updated road state vector and the updated agent state vector. S5. Set the termination condition of the traffic simulation model, simulate traffic phenomena based on the traffic simulation model, and determine whether the obtained simulation result meets the termination condition at each time step. If it meets the condition, stop the calculation and output the simulation result; otherwise, proceed to the next time step.

2. The multi-modal traffic simulation method based on multi-agent sequential decision-making as described in claim 1, characterized in that, The road network is distributed on a plane. Step S2 specifically includes the following steps: S2.

1. Establish a Cartesian coordinate system for measuring the entire road network; if a location is the end of multiple roads, the location is taken as a node; if a location is the end of only one road, the location is taken as an endpoint; all roads are connected to other roads only through nodes; the road network also includes multiple fixed points, each of which is located at a non-end position on a certain road, and any two fixed points are located at different positions. S2.

2. The endpoints and fixed points are collectively referred to as route base points. The process of determining the route includes multiple stages, one of which is as follows: Take a route base point without a start point mark as the start point and another route base point as the end point. Then determine one or more optional routes from the start point to the end point. Each optional route includes one road or multiple connected roads. After determining all optional routes, mark the route base point that is the end point as the end point. Then take a route base point without an end point mark as the end point and continue to determine optional routes until all route base points except the start point have end point marks. Then mark the route base point that is the start point in the current stage as the start point and remove all end point marks. Then proceed to the next stage and repeat the above process until all route base points have start point marks. Each optional route has a unique number. S2.

3. Determine the action type corresponding to each vehicle type. The action is represented as a vector containing multiple elements, and each action corresponds to a unique number for the same vehicle. The traffic simulation model includes two vehicle types: private cars and buses. The actions of private cars are... a i,b =[p i,j , v i, adj ], i For vehicle serial number, b Number the action. a i,b For vehicles i Selectable actions b p ij For vehicles i Selected route feature values, j v is the optional route number. i, adj For vehicles i The target speed of the vehicle; the bus's movement is a i,b = [a i, stop , a i, speed , p i,j ], a i, stop For docking items, a i, stop ∈{ c 1, c 2}, c 1. c Both are constants and c 1≠ c 2, a i, speed For speed adjustment, -V d, max ≤ a i, speed ≤ V a, max V d, max For the maximum deceleration rate, V a, max Maximum acceleration rate; vehicle i All selectable actions are grouped into a set, which is the action space A. i ; S2.

4. The positions of all vehicles are described by point coordinates, and an agent state vector is established. s i,t The state vector of a private car is represented as s i,t = [ x i,t , y i,t , v speed i,t , q i,t , s k,t The state vector of the bus is represented as: s i,t =[ x i,t , y i,t , v speed i,t , q i,t , s k,t , cap rem i,t ], t For a moment, x i,t , y i,t vehicles i exist t Location at any given moment x axis, y Axis coordinates v speed i,t For vehicles i exist t The speed at any given moment q i,t For vehicles i exist t The remaining journey at that time, s k,t for t The road where the moment is k The characteristic parameters, whose values ​​are determined by the road state vector. S road k,t Decide, k For vehicles i exist t The road number at which the time is located. cap rem i,t For buses i exist t The remaining passenger capacity at any given time. cap rem i,t = cap i - pass i,t , cap i For buses i Rated passenger capacity, pass i,t for t Bus schedule i The actual passenger capacity; S2.

5. Set the basic parameters for each road. The basic parameters for a given road include... η k With cap k , η k For roads k The congestion coefficient, cap k For roads k Rated traffic flow; establish road state vector S road k,t , S road k,t = [ v k,t , t k,t ,cap k - v k,t ], v k,t for t Road at all times k Traffic flow on the road t k,t for t Vehicles pass through the road at any time k The estimated time required.

3. The multi-modal traffic simulation method based on multi-agent sequential decision-making as described in claim 1, characterized in that, Step S3 specifically includes the following steps: S3.

1. Establish the individual objective function for all vehicles, the specific process is as follows: S3.1.

1. Establish the individual objective function for private cars. The individual objective function for private cars is expressed as follows: (1) in, f i,t (·) represents the individual objective function. t i,t for t Time vehicle i Time already traveled within the road network, d i,t for t Moment Private Car i The difference in distance between the current route and the shortest route. t i,t , d i,t All are with s i,t Relevant parameters, α , β These are the first type of weighting coefficients. α , β > 0 and α + β = 1; S3.1.

2. Establish the individual objective function for the bus. The individual objective function for the bus is expressed by the following formula: (2) in, t i,plan For buses i The planned travel time, o i,t for t Bus schedule i The level of crowding inside the car , γ , δ , ε These are the second type of weighting coefficients. γ , δ , ε All are greater than 0 and γ + δ + ε = 1; S3.

2. Establish the global objective function, which is expressed as follows: (3) in, S t It is a set of road state vectors. S t = { S road 1,t , S road 2,t , …, S road M,t }, F (·) represents the global objective function. N t for t The total number of vehicles in the road network at any given time. M The total number of roads, ζ The congestion penalty coefficient, ζ > 0.

4. The multi-agent sequential decision-making method for traffic simulation as described in claim 1, characterized in that, The update of the agent's state vector and the road state vector in step S4 specifically includes the following steps: S4.

1. Determine the initial strategy for all vehicles and set the number of time steps for the simulation calculation; within each time step, the vehicles... i Before executing the action, a random number is generated within the range [0, 1] and compared with the critical value. θ t In comparison, if the selected random number is less than θ t Then the vehicle i Execute the action according to the current strategy, if the selected random number is not less than θ t Then the vehicle i Randomly select one action from all currently available actions to execute. , T total This is the number of time steps for the simulation calculation. Within the initial time step, the current policy is the initial policy. S4.

2. Each time step is divided into... N t Each step involves one vehicle performing an action; within any given step, all road traffic flow is updated using the following formula: (4) in, t 1+ ( t 0) and t 1+ ( t 0 - 1 represent the current step and the previous step, respectively. For the current steps within the road k Traffic flow changes l Entering or leaving the road within the current step k Vehicle serial number, For the set, all paths entering or leaving the current step. k All vehicle serial numbers are included in this set, Δ v i For vehicles i Road occupancy, for private cars |Δ v i | = C , C The bus Δ is a constant. v i The calculation formula is , cap private The standard passenger capacity for private cars is for vehicles entering the road. k vehicles i Δ v i > 0, for leaving the road k vehicles i Δ v i < 0; The estimated time required for vehicles to travel on any road is updated using the following formula: (5) t When 0 = 0, the traffic flow of any road is the initial traffic flow of that road at the current time step. The estimated time for vehicles to travel on any road is the initial estimated time for that road at the current time step. ; S4.3 Road State Vector Update at Each Time Step N t After that, based on the current time step vehicle i of s i,t With the action performed a i,b and vehicles i Current road k Initial traffic flow in the next time step v k,t+1 The initial estimated time required for vehicles to pass through this road. t k,t+1 The vehicle was obtained i Agent state vector at the next time step s i,t+1 .

5. The multi-modal traffic simulation method based on multi-agent sequential decision-making as described in claim 4, characterized in that, The sequential interaction model in step S4 is based on the Q-learning algorithm. The process of constructing the sequential interaction model specifically includes the following steps: S4.

4. After a vehicle performs an action, its mixed reward is calculated using the following formula: (6) in, , Vehicles at a certain time step i Mixed rewards and individual rewards, for t Global reward at any moment λ As a reward weighting coefficient, λ ∈[0, 1], F ( S t The objective function value is calculated based on the road traffic flow and estimated travel time within the target step, where the target step refers to the number of vehicles within that time step. i The steps involved in performing the action; S4.

5. Define the neighborhood of all vehicles. Within a certain time step, the neighborhood of a vehicle includes the road where the vehicle was located before performing the action, as well as the upstream and downstream roads of the road where the vehicle is located according to the current route. When no vehicles in the road network have performed any actions, all vehicles in the neighborhood are considered related vehicles of that vehicle. Calculate the average action of the vehicle using the following formula: (7) in, a ' i,t for t Time vehicle i Average movement, A i,t For a set, the set includes t Time vehicle i The actions performed by all relevant vehicles respectively n For the relevant vehicle serial number, for t Time vehicle i The total number of related vehicles; S4.

6. Establish a Q-function for calculating the policy Q-value, and update the policy Q-value for all vehicles using the following formula: (8) in, Q i,t and Q i,t+1 They are respectively t Time and t + 1 moment vehicle i The Q-value of the strategy, α 0 represents the learning rate. γ Let be the discount factor, and max be a function. Q (·) is the Q function. Q ( s i,t+1 , a i,b , a ' i,t )express t Time vehicle i In state s i,t+1 With average movement a ' i,t Next action a i,b Q value at that time a i,b ∈A i , Q i, 0 = 0; S4.

7. According to Q i,t+1 For vehicles i Update the current strategy, and use the updated strategy as the vehicle's strategy. i The current strategy for the next time step.

6. The multi-agent sequential decision-making method for traffic simulation as described in claim 5, characterized in that, The specific termination conditions for the traffic simulation model set in step S5 are as follows: S5.

1. Establish a congestion determination mechanism, which includes road network saturation conditions, congested road ratio conditions, and total delay convergence conditions, as detailed below: S5.1.

1. Set the sample stage, wherein the sample stage includes the current time step. W For each consecutive time step, the average road saturation at that time step is calculated based on the simulation results obtained at a certain time step in the sample phase, and an overall saturation threshold is set. ε ρ The average road saturation is calculated using the following formula: (9) in, ρ t for t The average road saturation at each time step; the average road saturation at each time step within the sample phase needs to be calculated using formula (9), and then the standard deviation of the average road saturation at all time steps is calculated. If the obtained standard deviation is less than ε ρ If the simulation results meet the road network saturation condition, then the condition is considered to be met; otherwise, it is not. S5.1.

2. Calculate the saturation of each road within a certain time step based on the simulation results obtained in the sample phase, and set an individual saturation threshold. ε c and saturation ratio threshold ε ratio The formula for calculating the saturation of a certain road is: (10) in, ρ k,t for t The Road of Time k The saturation level; the saturation level of all roads at a given time step is compared with... ε c Compare and select those with saturation greater than 100%. ε c The roads that are considered congested are determined after all roads have been compared, resulting in the number of congested roads in that time step. M The ratio of these ratios is called the saturation ratio. The saturation ratio needs to be calculated for each time step within the sample phase, and then the standard deviation of the saturation ratios for all time steps in the sample phase is calculated. If the obtained standard deviation is less than... ε ratio If the simulation results meet the congested road ratio condition, then the condition is considered met; otherwise, it is not met. S5.1.

3. Calculate the total delay of the road network within a certain time step based on the simulation results obtained in the sample phase, and set a total delay threshold. ε D The formula for calculating the total delay of the road network is: (11) in, D total,t for t Total network delay at each time step; the total network delay for each time step within the sample phase must be calculated using formula (11), and then the standard deviation of the total network delay for all time steps is calculated. If the obtained standard deviation is less than ε D If the simulation results meet the total delay convergence condition, then the simulation results are considered to have met the condition; otherwise, they are not. S5.1.

4. If the network saturation condition, the congested road ratio condition, and the total delay convergence condition are all satisfied at a certain time step, then congestion is considered to have occurred in the simulation results obtained at that time step; otherwise, congestion is considered not to have occurred. After a certain number of time steps in the simulation calculation, the simulation results are judged by the aforementioned congestion determination mechanism at each time step. If congestion has occurred at a certain time step, or the number of time steps has reached a certain threshold, the congestion determination mechanism is used to determine the congestion determination. T total If the simulation calculation meets the termination condition, it is considered to have met the termination condition; otherwise, it is considered not to have met the termination condition.

7. The multi-agent sequential decision-making method for traffic simulation as described in claim 5, characterized in that, The specific termination conditions for the traffic simulation model set in step S5 are as follows: S5.

2. Establish a convergence determination mechanism, which includes policy convergence conditions, traffic distribution convergence conditions, and global convergence conditions, as detailed below: S5.2.

1. Calculate the global policy change degree based on the simulation results of the sample stage, and set the global policy threshold. ε π The formula for calculating the global policy variability is as follows: (12) in, Q average,t For the degree of change in global strategy, τ Let be the ordinal number of the time step in the sample phase. t For the current moment; if Q average,t < ε π The simulation results are considered to meet the convergence conditions of the strategy if they meet the conditions, otherwise they are not. S5.2.

2. Based on the simulation results of the sample stage, establish the road network traffic vector for each time step. The road network traffic vector at a certain time step is represented as V. t = [ v 1,t , v 1,t ,…, v M,t It calculates the global traffic variability based on all road network traffic vectors and sets a global traffic threshold. ε v The global traffic variability is calculated using the following formula: (13) in, V average,t The degree of global traffic change; if V average,t < ε v The simulation results are considered to meet the flow distribution convergence condition; otherwise, they are not. S5.2.

3. Calculate the global target change during the sample phase and set the global target change threshold. ε F The formula for calculating the change in the global target is as follows: (14) in, F average,t This represents the change in the global objective; if F average,t < ε F The simulation results are considered to satisfy the global convergence condition; otherwise, they are not. S5.2.

4. If all three convergence conditions—strategy convergence condition, flow distribution convergence condition, and global convergence condition—are met at a certain time step, the traffic flow in the simulation results obtained at that time step is considered stable; otherwise, it is considered not yet stable. After a certain number of time steps in the simulation calculation, the simulation results are judged through the aforementioned convergence determination mechanism at each time step. If the traffic flow is stable at a certain time step, or the number of time steps has reached a certain threshold, the simulation results are considered stable. T total If the simulation calculation meets the termination condition, it is considered that the simulation calculation has met the termination condition; otherwise, it is considered that the termination condition has not yet been met.

8. A multi-agent sequential decision-making method for traffic simulation as described in claim 5, characterized in that, The traffic simulation model includes normal and temporary states. In the temporary state, there are one or more special roads in the road network, and each special road has a temporary rated traffic flow cap. new k With temporary congestion coefficient η new k Under normal conditions, the formula for calculating the Q function is described as follows: The traffic simulation model under temporary conditions includes a temporary reward mechanism, in which cap new k , η new k Replace cap respectively k , η k Substituting into steps S2.5-S4.2, the temporary vehicle mixed reward calculated using formula (6) is denoted as... The formula for calculating the temporary state Q function is: (15) in, ω For incremental learning rate, ω ∈(0, 0.5).

9. A multi-agent sequential decision-making method for traffic simulation as described in claim 2, characterized in that, The optional route in step S2.2 is determined based on the shortest path algorithm, and the specific process includes the following steps: S2.2.

1. Establish the road weight calculation formula, which is expressed in the following form: (16) in, C k For roads k The weight value, t k To pass through the road k Estimated time required l k For roads k Length, ω 1. ω 2. ω 3 are all road weight coefficients, and ω 1+ ω 2+ ω 3 = 1; S2.2.

2. Based on the road network topology, determine all routes connecting the starting point and the ending point, denoted as initial routes. Calculate the weight value of each initial route. The weight value of a given initial route is the sum of the weight values ​​of all roads along that route. Then, find the initial route with the smallest weight value and denote it as the baseline route. Denote the number of nodes in the baseline route as K. In order from the starting point to the ending point, all nodes in the baseline route are sequentially designated as node 1, node 2, ..., node K. ψ The portion between nodes is denoted as L( ψ ), ψ ∈[1, K] and ψ The integer is used to determine all connections based on the road network topology. ψ The route between the node and the destination is denoted as ψ any type of route ψ Class of routes and L( ψ All routes formed by connecting these points are categorized into set A. ψ ),like ψ If the number of routes of type 1 is 1, then A( ψ -1), A( ψ -2), ..., A(1) are all optional routes. If the number of routes of type 1 is 1, then all initial routes are optional routes. In any of the above routes, the same road can be passed at most once.

Images