Large-model-driven multi-subject trajectory generation and city-level behavior simulation method

By constructing a multi-agent Markov game environment and the MAGAIL framework, and using real-world vehicle trajectory data to learn traffic behavior, the problems of time-consuming complexity and insufficient vehicle interaction simulation in traditional traffic simulation models are solved, and realistic city-level traffic simulation and strategy evaluation are achieved.

CN122045808APending Publication Date: 2026-05-15PEKING UNIV SHENZHEN GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
PEKING UNIV SHENZHEN GRADUATE SCHOOL
Filing Date
2025-12-27
Publication Date
2026-05-15

AI Technical Summary

Technical Problem

Traditional traffic simulation models are time-consuming and complex, making it difficult to effectively simulate vehicle movement and traffic state evolution within a network. Furthermore, existing data-driven models fail to fully consider the interaction between vehicles and road segments.

Method used

Employing a multi-agent Markov game environment and the generative adversarial imitation learning (MAGAIL) framework, we construct vehicle and road segment subject models. By learning traffic behavior from real-world vehicle trajectory data, we generate realistic traffic scenarios and simulate city-level traffic dynamics.

Benefits of technology

It realizes data-driven mesoscopic traffic simulation, which can simulate the interaction between vehicles and road segments within the network, generate realistic traffic scenarios, overcome the limitations of traditional models, and provide a tool for evaluating traffic management strategies in a virtual environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122045808A_ABST
    Figure CN122045808A_ABST
Patent Text Reader

Abstract

The invention discloses a large-model-driven multi-subject trajectory generation and city-level behavior simulation method. According to the method, a multi-agent generation adversarial imitation learning framework is adopted, an MAGAIL-VL model is constructed, and synchronous simulation of vehicle movement and road section traffic state evolution is realized. An urban traffic system is abstracted into a Markov game, two types of agents including a vehicle agent and a road section agent are included, and expert demonstration is carried out by utilizing a real track and speed data. The model generator generates an action according to a local state, and the discriminator provides a reward signal by distinguishing generated data and expert data to guide strategy network updating. According to the framework, centralized training and distributed execution are adopted, the calculation complexity is reduced, meanwhile, the interaction influence between main bodies can be accurately described, and vivid urban-level traffic tracks and simulation scenes are generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of traffic simulation and artificial intelligence, and in particular to a method for generating multi-agent trajectories and simulating city-level behavior under a large model-driven approach. Background Technology

[0002] Traditional traffic simulation models play a crucial role in analyzing traffic systems and developing network traffic management strategies, alleviating traffic congestion by modeling complex traveler behaviors and traffic flow dynamics. Dynamic traffic assignment, a key component of traffic network modeling, aims to find equilibrium solutions through an iterative process to determine route selection and segment travel times. However, constructing traditional network traffic simulation models is a time-consuming task, requiring complex parameter estimation and calibration processes. Furthermore, traditional traffic models contain various strong theoretical and behavioral assumptions, and the inherent discrepancies between theory and real-world behavior complicate model validation.

[0003] To overcome these challenges and limitations of traditional traffic simulation methods, the use of data-driven models to learn traffic behavior directly from data has attracted increasing attention. In particular, with the increasing availability of high-resolution vehicle trajectory data and significant advancements in machine learning models, some studies have proposed learning following and / or lane-changing behaviors directly from microscopic vehicle trajectories, thereby providing data-driven microscopic traffic simulation models to model detailed vehicle motion within road segments. However, current efforts to develop data-driven mesoscopic models for network-wide traffic simulation, which describe vehicle (segment-to-segment) motion and the evolution of traffic states across the entire network, can estimate and predict network performance metrics such as segment speed and segment travel time. Existing trajectory pattern learning research typically focuses on independently modeling the motion of individual travelers without considering their interactions. Summary of the Invention

[0004] This invention aims to overcome the aforementioned deficiencies in the prior art and provides a method for multi-agent trajectory generation and city-level behavior simulation driven by a large model. It aims to model vehicle motion and traffic state evolution across a network by considering the interaction between vehicles and road segment traffic conditions. The specific technical solution is as follows.

[0005] A method for multi-agent trajectory generation and city-level behavior simulation driven by a large model includes the following steps: a. Construct a multi-agent Markov game environment, abstracting vehicles and road segments in the urban traffic network as agents in a multi-agent Markov game, including vehicle agents and road segment agents. The vehicle agent represents a subset of the vehicle group, and its position on the road segment in the network is modeled as a sequential decision of the vehicle agent at each time step. The road segment agent represents all road segments in the network, and its traffic congestion level at each time step is modeled as a sequential decision of the road segment agent. Define the local state space and action space of each agent in a partially observable environment, and assume that agents of the same type share the same strategy and action space. b. Acquire and transform expert teaching data. Obtain vehicle trajectory data from real-world vehicle trajectory data and convert the vehicle trajectories into time series of road segment indices and vehicle speeds. Based on this time series, create expert teaching data for the vehicle entity, represented as a state-action pair sequence. Based on the set of vehicle trajectories observed within the same time period, create expert teaching data for the road segment entity associated with a specific road segment to identify the average speed and corresponding speed range of the road segment at each time step, represented as a state-action pair sequence. c. Construct and train a multi-agent generative adversarial imitation learning (MAGAIL) model, including a generator and a discriminator; wherein, the generator includes a vehicle agent policy network and a road segment agent policy network, as well as a vehicle agent value estimator network and a road segment agent value estimator network; the discriminator sets up a discriminator network for each agent, and the training optimizes the objective function by alternately updating the discriminator parameters and policy parameters; d. Trajectory generation and simulation: Once the vehicle subject strategy and road segment subject strategy have been learned from the expert teaching data samples, the state-action pair sequence of each vehicle subject and each road segment subject can be generated within the required simulation time range. The generated sequence is then used to simulate vehicle motion and road segment traffic evolution to simulate real-world traffic dynamics.

[0006] Preferably, in step a, the state of the vehicle body i Where k represents the index of the road segment that vehicle i is currently traveling on. This indicates the congestion level of road segment k; the state of the main body i of the road segment. Where k is fixed as the index of the road segment modeled by the main body of this road segment. This indicates the current level of congestion.

[0007] Preferably, in step a, the movement space of the vehicle body This indicates the possible movements of a vehicle on a road segment, including moving to a downstream road segment, remaining on the current road segment, leaving the network, entering the network, remaining outside the network before entering, and remaining outside the network after leaving; the movement space of the main body of the road segment. This indicates possible changes in the level of congestion on a road segment.

[0008] Preferably, in step c, the policy network of the generator is modeled as a multilayer perceptron (MLP), which receives the current local state of the subject as input, outputs the probability distribution of each action, and uses the softmax function as the activation function of the output layer.

[0009] Preferably, in step c, the value estimator network of the generator is modeled as an MLP to calculate the state value function for each agent. ,in For expected returns, As a discount factor, The reward function for subject i is given by the value estimator network, which receives the states of all subjects and the actions of other subjects except the current subject as input, and outputs the estimated value or expected return.

[0010] Preferably, the output of the value estimator is used to calculate the advantage function. Used for policy updates.

[0011] Preferably, in step c, the discriminator network is modeled as an MLP, receiving the subject's state and actions as input, and outputting a binary classification result indicating whether the input comes from the generator or from expert teaching data; the output of the discriminator is used to provide a reward function for the generator. .

[0012] Preferably, in step c, the training algorithm employs a centralized training and distributed execution framework, by optimizing the objective function. Training was conducted, among which For expert teaching data, state-action pairs are used as samples. The samples are state-action pairs generated by the generator, where N is the number of subjects. The discriminator for subject i.

[0013] Preferably, in step c, the training algorithm employs an invalid action masking technique in the policy network output to set the probability of invalid actions to zero.

[0014] Preferably, real-world vehicle trajectory data includes the (x, y) coordinates of the vehicle's location and timestamp information.

[0015] The beneficial effects of the multi-agent trajectory generation and city-level behavior simulation method driven by a large model provided by this invention include:

[0016] (1) This invention provides a data-driven mesoscopic traffic simulation framework that can learn the distribution of vehicle motion and road segment states from data and generate realistic traffic scenarios, providing a new method for directly building simulation models from data.

[0017] (2) The present invention designs vehicle and road segment entities in a multi-entity system and develops MAGAIL-VL model based on MAGAIL framework to learn and reproduce the interaction between some vehicles and road segment traffic states in the network.

[0018] (3) This invention uses real-world vehicle trajectory and road segment speed data to train vehicles and road segments to imitate real-world behavior, and demonstrates the superior performance of the developed MAGAIL-VL model through experiments.

[0019] (4) This invention can create a large number of simulations of real-world environments. By imitating and replicating the behavior of the traffic network under study, it overcomes the limitation of traditional traffic prediction models that can only predict for specific inputs. Attached Figure Description

[0020] Figure 1 This is a flowchart of a method for multi-agent trajectory generation and city-level behavior simulation driven by a large model;

[0021] Figure 2 This is a flowchart of the MAGAIL model architecture and its training process;

[0022] Figure 3 This is a diagram showing the internal components of the generator. Detailed Implementation

[0023] This invention proposes a method for multi-agent trajectory generation and city-level behavior simulation driven by a large model. Its core lies in utilizing the multi-agent generative adversarial imitation learning (MAGAIL) framework to directly learn traffic behavior from large-scale urban vehicle trajectory data, thereby achieving realistic simulation of urban traffic networks. The invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown, a method for multi-agent trajectory generation and city-level behavior simulation driven by a large model includes the following steps: a. Construct a multi-agent Markov game environment, abstracting vehicles and road segments in the urban traffic network as agents in a multi-agent Markov game, including vehicle agents and road segment agents. The vehicle agent represents a subset of the vehicle group, and its position on the road segment in the network is modeled as a sequential decision of the vehicle agent at each time step. The road segment agent represents all road segments in the network, and its traffic congestion level at each time step is modeled as a sequential decision of the road segment agent. Define the local state space and action space of each agent in a partially observable environment, and assume that agents of the same type share the same strategy and action space.

[0025] Vehicle subjects represent a subset randomly selected from the entire vehicle population (e.g., 10% of all vehicles). Each vehicle subject's position on a network segment is modeled at each time step as a sequential decision made by that vehicle subject. This invention does not aim to model all vehicles in the network, as modeling a multi-subject system with thousands of subjects would be extremely computationally expensive. More importantly, the goal of this invention is not to build a traditional, high-fidelity Dynamic Traffic Assignment (DTA) simulator to model all individual vehicles to simulate their route assignments and associated segment traffic dynamics. Instead, this invention aims to model a representative set of vehicles as vehicle subjects to learn observed vehicle motion behavior from data, while considering surrounding segment traffic conditions captured by the segment subjects. Vehicle subjects also play a role in capturing vehicle-level information about network traffic (such as segment travel time) in a manner similar to that used in the real world to collect traffic data using probe vehicles. This provides additional environmental knowledge for segment subjects to learn realistic traffic dynamics.

[0026] Link Agents: Link agents represent all road segments in the network, where the traffic congestion level of a segment at each time step is modeled as a sequential decision by the link agent. This invention aims to utilize these link agents to describe the evolution of aggregated traffic states at the segment level throughout the network.

[0027] This invention considers a partially observable setting, where each agent i cannot access the complete global state of the environment containing all N agents. It can only access its own local state. Therefore, the strategy of each subject i The joint policy selects actions using only local state information. For simplicity, this invention models the vehicle body and the road segment body as two sets of homogeneous bodies: within each set, the bodies share the same action space ( and and share the same strategy. and (), where the superscripts v and l represent the vehicle subject and the road segment subject, respectively. This homogeneity is often assumed in multi-agent learning to reduce the dimensionality of modeling a large number of subjects.

[0028] In a preferred embodiment, the state space and action space include:

[0029] (1) The state of vehicle body i ) is represented as a two-dimensional vector Where k represents the index of the road segment that vehicle i is currently traveling on. (K is the set of indices of all road segments in the network). This represents the congestion level of road segment k (given c predefined speed ranges).

[0030] (2) The movement space of the vehicle body Indicates the possible movement of a vehicle on a road segment: moving to the j-th downstream road segment ( ), stays on the current road segment k, exits the network, enters the network, stays outside the network before entering, and stays outside the network after exiting. This represents the number of downstream road segments connected to road segment k. To represent the start and end points of a vehicle's journey, two virtual road segments are defined: the source and the sink. Vehicles in the network can choose to move to a downstream road segment, remain on their current road segment, or leave the network (move to the sink to end their journey). Vehicles that have not yet departed can move from the source to any road segment in the network or remain at the source. Vehicles that have completed their journey will remain at the sink.

[0031] (3) The state of road segment i ) is also represented as In this case, k is fixed as the index of the road segment modeled by the main body of that road segment. This indicates the current level of congestion (speed range).

[0032] (4) The main body of the road section's movement space This indicates possible changes in the congestion level of a road segment: {changing to speed range 1, ..., changing to speed range c}, which makes the state... It is possible The changes between them.

[0033] (5) For each subject i, the transition of the local state is deterministic, that is, given the current state, the transition is deterministic. and actions In any case, there is always only one possible next state. .

[0034] (6) Vehicle main strategy ( This determines the vehicle's segment-to-segment movement and travel speed selection, or travel time experience (intra-segment movement). Segment-based strategy ( This determines the evolution of congestion levels for a given road segment. Although each policy is executed independently, requiring only local state information of a given subject, these policies learn together during training by considering the dependencies between vehicles and road segments. This is achieved through a centralized training and distributed execution framework.

[0035] b. Acquire and transform expert teaching data. Obtain vehicle trajectory data from real-world vehicle trajectory data and convert the vehicle trajectories into time series of road segment indices and vehicle speeds. Based on this time series, create expert teaching data for the vehicle entity, represented as a state-action pair sequence. Based on the set of vehicle trajectories observed within the same time period, create expert teaching data for the road segment entity associated with a specific road segment to identify the average speed and corresponding speed range of the road segment at each time step, represented as a state-action pair sequence.

[0036] Specifically, vehicle trajectory data is set as follows: It is the vehicle trajectory, represented as a time-ordered sequence of vehicle positions, where These are (x, y) coordinates. It is the timestamp of the j-th point in the trajectory.

[0037] Specifically, the conversion into expert teaching data involves:

[0038] (1) Each vehicle trajectory can be converted into a time series of road segment indices and vehicle speeds generated at regular time intervals (e.g., every 5 seconds).

[0039] (2) This time series was then used to create expert teaching data for the vehicle body, represented as a specific time period. Internal state-action pair sequence: Using a road segment index sequence, the travel time on a specific road segment can be obtained by multiplying the number of consecutive occurrences of that road segment in the sequence by the width of the time interval. For example, with a time interval of 5 seconds, (A,A,A,B,B,B,B,B,C,C,...) means that a vehicle spends 15 seconds on road segment A, 25 seconds on road segment B, and so on.

[0040] (3) In order to be in the same time period The system creates expert teaching data for the main road segment associated with road segment k, using the set of vehicle trajectories observed during that time period to identify the average road segment speed and corresponding speed range of road segment k at each time step t. ,in This represents the expert teaching data representing the main body of the road section.

[0041] Specifically, the simulation generation is, once from and Learn the strategy from the samples and We can then perform simulations for each vehicle body (represented as) within the required simulation time frame. ) and the main body of each road segment (represented as Generate state-action pair sequences to simulate real-world traffic dynamics, including vehicle movement and road segment traffic evolution.

[0042] c. Construct and train a multi-agent generative adversarial imitation learning (MAGAIL) model, such as Figure 2 As shown, it includes a generator and a discriminator; where, as Figure 3 As shown, the generator includes a vehicle subject policy network and a road segment subject policy network, as well as a vehicle subject value estimator network and a road segment subject value estimator network; the discriminator sets up a discriminator network for each subject, and the training optimizes the objective function by alternately updating the discriminator parameters and policy parameters; specifically, the two policy networks in the generator component are one for vehicle subjects (… ), parameters are Another one is used for the main body of the road section ( ), parameters are Each policy network is modeled as a multilayer perceptron (MLP). An MLP network consists of an input layer, an output layer, and one or more hidden layers. The policy network receives the current local state of agent i. ) as input, and output each action ( The probability of actions is calculated. In this study, two hidden layers are used, each with 256 neurons, and the Modified Linear Unit (ReLU) is used as the activation function. The output layer uses the softmax function as the activation function to normalize the output of the policy network to a probability distribution of possible actions.

[0043] Specifically, the two value estimator networks in the generator component are one for the vehicle body ( ), parameters are Another one is used for the main body of the road section ( ), parameters are Each value estimator network is also modeled as an MLP network. The value estimator computes the state-value function for each agent i: in For expected returns, As a discount factor, Let i be the reward function for subject i; this function provides the expected reward, i.e., given the actions of other subjects. In the case of environmental conditions Follow the strategy The estimated total expected return. Where the subscript -i represents all entities other than i. Indicates the main body of the vehicle ( )of and the main body of the road section ( )of Therefore, the value estimator network receives the states of all N agents and the actions of N-1 agents as input, and outputs an estimated value or expected return. The output of the value estimator is used to calculate the advantage function. The advantage function measures how much better it is to take a specific action *a* in state *s* than to take the "average" action according to the current policy. The advantage function is used to update the policy in a reinforcement learning algorithm by increasing the probability of good actions (positive advantage) and decreasing the probability of bad actions (negative advantage). MAGAIL uses an advantage function of the following form, applying the k-step reward from the Asynchronous Advantage Actor-Critic (A3C) method to a multi-agent environment:

[0044] in, It is a discount factor. It is the reward function for subject i.

[0045] The discriminator component has N discriminator networks, one for each subject i, denoted as... The parameters are Each discriminator network is also modeled as an MLP network, which receives the state ( ) and actions ( The generator takes a state-action pair as input and produces a binary output indicating that the given input is a state-action pair (represented as...) generated by the generator (policy network). ) or state-action pairs constructed from expert instruction (real-world data) (represented as ). The value of represents the predicted probability of a given state-action pair from real data, and is used to provide the reward function for the generator. The generator, in turn, updates its policy to maximize this reward, thereby generating state-action pairs similar to the expert-taught data to "fool" the discriminator.

[0046] The training algorithm specifically involves initializing the parameters of the policy, value estimator, and discriminator networks, and then training them by... and Generate separately and Obtain a batch (batch size B) of state-action pair samples from the policy. Simultaneously, by constructing and sampling from real-world vehicle trajectory data... and Generate a batch of state-action pairs from expert teaching, with a batch size of B. The model updates the discriminator parameters alternately. ) and strategy parameters ( To train and optimize the following objective function:

[0047]

[0048] Where N is the number of subjects. It is the discriminator for subject i. This represents the joint policy of all agents. The discriminator update phase updates the discriminator parameters for each agent i. To increase the target:

[0049]

[0050] The goal of the discriminator is to correctly distinguish between real samples from expert teaching data and fake samples from generator strategies.

[0051] Generator Update Phase: To optimize generator parameters, MAGAIL uses the Multi-agent Actor-Critic with Kronecker-factors (MACK) algorithm, a natural policy gradient method adapted for multi-agent deep reinforcement learning. MACK employs a framework of centralized training and decentralized execution, where agents are trained using centralized information but executed in a decentralized manner. This allows the policy to use additional information (the states and actions of other agents) to enhance training, while allowing decentralized policies to reduce model complexity, i.e., each agent selects actions based only on its local state information during the execution of the learned policy. Centralized training is achieved by using a centralized value function (…). This function is implemented by taking the actions of other subjects as input, in addition to the state information.

[0052] First, update the value estimator parameters for each subject i. To reduce the true value function ( ) and the value estimator network estimate ( The difference of squares between:

[0053]

[0054] Since the true value function is unknown, in practice a certain objective is usually used as... Alternatives include, for example, bootstrapping k-step rewards used in time difference (TD) learning with value function approximation.

[0055] Next, using In Advantage estimation The policy parameters of each agent i are updated in small steps using the policy gradient. :

[0056]

[0057] The gradient update strategy aims to maximize the subject's expected return, that is, to maximize the reward provided by the discriminator for its generative behavior.

[0058] It's important to note that not all actions in the action space are valid in every subject state. In a preferred embodiment, when a vehicle moves to another road segment, it can only choose from its downstream road segments; a vehicle currently outside the network can only choose to remain outside the network or enter the network. To avoid repeatedly sampling invalid actions when sampling state-action pairs from the policy, this invention uses a technique called invalid action masking, which "masks" invalid actions and then samples only valid actions from a given state. This can be achieved by setting the probability of invalid actions in the policy network output to zero. This technique has proven effective in reducing the dimensionality of the action space and improving computational efficiency. The process of optimizing the above formula is tailored to different state-action pair samples ( and Repeatedly, the learned strategy ( ) and reward function ( This will be used as the output of the entire training algorithm.

[0059] d. Trajectory generation and simulation, once the vehicle's main strategy ( ) and main road segment strategies ( Learning from expert-led teaching data samples, the system can generate state-action pair sequences for each vehicle and road segment within the required simulation timeframe. These generated sequences are then used to simulate vehicle motion and road traffic evolution, thus mimicking real-world traffic dynamics. Specifically:

[0060] (1) Trajectory generation: Within the required simulation time range, the learned strategy is used. and It can generate each vehicle body ( ) and the main body of each road segment ( The generated sequences are state-action pairs that mimic real-world traffic dynamics, including vehicle movement paths, speed changes, and congestion conditions on road segments.

[0061] (2) City-level behavioral simulation: By generating a large number of these trajectory and road segment state sequences, this invention can create and reproduce the behavior of large-scale urban traffic networks. This simulation differs from traditional traffic prediction models, which only predict for specific inputs; the model of this invention can create a large number of simulation scenarios of real-world environments by imitating and replicating the behavior of the traffic network under study. This allows researchers and planners to evaluate and test different traffic management strategies, infrastructure changes, or emergencies in a data-driven virtual environment without having to conduct expensive and risky experiments in the real world. For example, it can simulate congestion patterns under different traffic demands, the impact of new road openings on traffic flow, or the rerouting behavior of vehicles when specific events (such as traffic accidents) occur.

[0062] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for multi-agent trajectory generation and city-level behavior simulation driven by a large model, characterized in that, Includes the following steps: a. Construct a multi-agent Markov game environment, abstracting vehicles and road segments in the urban traffic network into agents in a multi-agent Markov game, including vehicle agents and road segment agents, where the vehicle agent represents a subset of the vehicle group, and its position on the road segment in the network is modeled as a sequential decision of the vehicle agent at each time step. The road segment entity represents all road segments in the network, and its traffic congestion level at each time step is modeled as a sequential decision of the road segment entity; the local state space and action space of each entity in a partially observable environment are defined, and it is assumed that entities of the same type share the same strategy and action space; b. Acquire and transform expert teaching data. Obtain vehicle trajectory data from real-world vehicle trajectory data and convert the vehicle trajectories into time series of road segment indices and vehicle speeds. Based on this time series, create expert teaching data for the vehicle entity, represented as a state-action pair sequence. Based on the set of vehicle trajectories observed within the same time period, create expert teaching data for the road segment entity associated with a specific road segment to identify the average speed and corresponding speed range of the road segment at each time step, represented as a state-action pair sequence. c. Construct and train a multi-agent generative adversarial imitation learning (MAGAIL) model, including a generator and a discriminator; wherein, the generator includes a vehicle agent policy network and a road segment agent policy network, as well as a vehicle agent value estimator network and a road segment agent value estimator network; the discriminator sets up a discriminator network for each agent, and the training optimizes the objective function by alternately updating the discriminator parameters and policy parameters; d. Trajectory generation and simulation: Once the vehicle subject strategy and road segment subject strategy have been learned from the expert teaching data samples, the state-action pair sequence of each vehicle subject and each road segment subject can be generated within the required simulation time range. The generated sequence is then used to simulate vehicle motion and road segment traffic evolution to simulate real-world traffic dynamics.

2. The method according to claim 1, characterized in that, In step a, the state of the vehicle body i Where k represents the index of the road segment that vehicle i is currently traveling on. This indicates the congestion level of road segment k; the state of the main body i of the road segment. Where k is fixed as the index of the road segment modeled by the main body of this road segment. This indicates the current level of congestion.

3. The method according to claim 1, characterized in that, In step a, the movement space of the vehicle body This indicates the possible movements of a vehicle on a road segment, including moving to a downstream road segment, remaining on the current road segment, leaving the network, entering the network, remaining outside the network before entering, and remaining outside the network after leaving; the movement space of the main body of the road segment. This indicates possible changes in the level of congestion on a road segment.

4. The method according to claim 1, characterized in that, In step c, the policy network of the generator is modeled as a multilayer perceptron (MLP), which receives the current local state of the subject as input, outputs the probability distribution of each action, and uses the softmax function as the activation function of the output layer.

5. The method according to claim 1, characterized in that, In step c, the value estimator network of the generator is modeled as an MLP, and the state value function of each agent is calculated. ,in For expected returns, As a discount factor, The reward function for subject i is given by the value estimator network, which receives the states of all subjects and the actions of other subjects except the current subject as input, and outputs the estimated value or expected return.

6. The method according to claim 5, characterized in that, The output of the value estimator is used to calculate the advantage function. Used for policy updates.

7. The method according to claim 1, characterized in that, In step c, the discriminator network is modeled as an MLP, receiving the subject's state and actions as input, and outputting a binary classification result indicating whether the input comes from the generator or from expert teaching data; the output of the discriminator is used to provide a reward function for the generator. .

8. The method according to claim 1, characterized in that, In step c, the training algorithm employs a centralized training and distributed execution framework, optimizing the objective function. Training was conducted, among which For expert teaching data, state-action pairs are used as samples. The samples are state-action pairs generated by the generator, where N is the number of subjects. The discriminator for subject i.

9. The method according to claim 1, characterized in that, In step c, the training algorithm uses an invalid action masking technique in the policy network output to set the probability of invalid actions to zero.

10. The method according to claim 1, characterized in that, The real-world vehicle trajectory data includes the (x, y) coordinates of the vehicle's location and timestamp information.