Data-driven intersection vehicle complex interaction behavior strategy simulation method and system

The MA-AIRL approach addresses the challenges of simulating complex vehicle interactions at intersections by learning vehicle strategies from real data, enhancing model stability and accuracy through heterogeneous networks and target-oriented rewards, thereby improving the realism of intersection simulations.

CN120317096APending Publication Date: 2025-07-15TONGJI UNIV
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202510276008.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-10
Publication Date
2025-07-15

AI Technical Summary

Technical Problem

The existing data-driven model is difficult to accurately explain the dynamic competition and cooperative behavior of vehicles at intersections, and the model convergence and stability are poor, which cannot meet the needs of high-precision simulation.

Method used

The multi-agent anti-reverse reinforcement learning method is adopted to obtain the traffic trajectory data of real intersection vehicles, build a Gaussian strategy network, and use the goal-oriented gain reward and discriminator reward estimator to generate simulated vehicle trajectory data, update the reward estimator and potential function of the agent to realize the simulation of the complex interactive behavior of the vehicle.

Benefits of technology

The behavioral strategy of the vehicle at the intersection is accurately depicted, the convergence and stability of the model is improved, the heterogeneity of driving behavior is taken into account, and the accuracy of the trajectory is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120317096A_ABST
    Figure CN120317096A_ABST
Patent Text Reader

Abstract

The invention relates to a data-driven intersection vehicle complex interaction behavior strategy simulation method and system. The method comprises the following steps: acquiring real intersection vehicle traffic trajectory data, constructing a plurality of interaction scenes, extracting observation information of each time step, and obtaining real vehicle trajectory data; taking each vehicle as an intelligent agent, based on a Gaussian strategy network pre-constructed according to the vehicle type, calculating a reward by using a reward based on a target guiding gain and a reward estimator from a discriminator, and generating simulated vehicle trajectory data; based on the real vehicle trajectory data and the simulated vehicle trajectory data, respectively extracting observed value-action pair information, updating a discriminator of an intelligent agent by taking distinguishing different attribute information to the greatest extent as a target, and updating a Gaussian strategy network and a value estimator network in a generator by taking confusing the discriminator to the greatest extent as a target, so as to obtain the real vehicle trajectory data and the simulated vehicle trajectory data. And vehicle complex interaction behavior strategy simulation based on multi-agent countermeasure reverse reinforcement learning is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a data-driven simulation method and system for complex interactive behavior strategies of intersection vehicles. Background Art

[0002] In recent years, autonomous vehicles (AVs) have developed rapidly. To address the recognized problem of the need for hundreds of billions of kilometers of real vehicle road tests for autonomous driving tests, virtual simulation testing has become the core means of autonomous driving testing technology, and its effectiveness depends on the high-precision simulation of actual traffic flows. To better test the social interaction capabilities of AVs with other vehicles, it is necessary to generate background traffic flows that can finely simulate complex interactive social behaviors in actual traffic flows. Strategies are the upper-level guidance for vehicles to perform specific actions such as accelerating, decelerating, and turning. Only by accurately grasping the time-varying and diverse social interaction strategies can the motion states of vehicles be accurately simulated, and then the social interaction behaviors of traffic flows be finely characterized. Therefore, it is crucial to establish a traffic flow simulation model that can accurately characterize both the dynamic interaction strategies and motion states of vehicles to meet the needs of AV social interaction capability testing.

[0003] Existing research on methods for modeling microscopic driving behaviors can be mainly divided into two categories: mechanism-driven and data-driven. Mechanism-driven models usually use mathematical or physical models to represent driving behaviors. However, due to the limitations of the model's expression ability, it is necessary to simplify the interaction rules, resulting in relatively single simulation results and limiting the possibility of high-precision simulation of strategies and actions in complex interactive environments. Data-driven models to some extent make up for the shortcomings of mechanism-driven models and mainly use deep learning (DL) and reinforcement learning (RL) to construct complex models that can handle difficult problems. However, the black-box nature of DL makes it difficult to explain the decision-making process of the model and cannot meet the need to clearly describe the internal decision-making process during vehicle interaction. RL requires artificially setting a reward function to guide the agent to obtain optimal behavior. The reward function can represent the upper-level decision-making mechanism of the interaction task, but it is extremely difficult to accurately define the reward that conforms to the human interaction mechanism in a complex interactive environment.

[0004] Imitation learning (IL), as an extension of reinforcement learning, does not require artificially setting a reward function and directly learns tasks by imitating real data. Existing research has proposed Symphony based on Generative Adversarial Imitation Learning (GAIL). However, the GAIL model cannot explicitly obtain the reward coupled with the policy network, so it is impossible to explain the upper-level strategy that guides the vehicle to perform specific actions and difficult to determine whether the interaction strategy is accurately reproduced.

[0005] Generally speaking, data-driven models rely on powerful data feature mining capabilities and provide effective support for simulating complex interaction behaviors. However, current research is difficult to accurately interpret the upper-level interaction strategies that guide driving behavior generation and cannot meet the high-precision simulation requirements of strategies and actions. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide a data-driven simulation method and system for complex interaction behavior strategies of intersection vehicles, so as to solve or partially solve the problems of being unable to fully depict the dynamic competition and cooperation behaviors of different vehicles during the process of passing through intersections and the poor convergence stability of the model.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] In one aspect of the present invention, a data-driven simulation method for complex interaction behavior strategies of intersection vehicles is provided, including the following steps:

[0009] Obtain the traffic trajectory data of real intersection vehicles, construct multiple interaction scenarios, and for each vehicle in each interaction scenario, extract the observation information at each time step to obtain real vehicle trajectory data;

[0010] Taking each vehicle as an agent, based on the Gaussian policy network constructed in advance according to vehicle types, calculate the reward using the goal-oriented gain reward and the reward estimator from the discriminator, and generate simulated vehicle trajectory data;

[0011] Based on the real vehicle trajectory data and the simulated vehicle trajectory data, respectively extract the observation value-action pair information, and use the pre-constructed discriminator to judge whether the vehicle trajectory to which the observation value-action pair belongs is real. Among them, the discriminator aims to distinguish between generated and real trajectories, update the reward estimator and potential function of the agent, and the generator aims to confuse the discriminator to the greatest extent, update the Gaussian policy network and value estimator network of the agent, and realize the simulation of complex interaction behavior strategies of vehicles based on multi-agent adversarial inverse reinforcement learning.

[0012] As a preferred technical solution, during the process of extracting the observation values at each time step, the observation space includes:

[0013] o i ={s i , o i,j}

[0014]

[0015] where, o iThe observation space of vehicle i at time t, s i is the self-state of vehicle i, p x 、p y 、v x 、v y represent the position and velocity of the vehicle in the x-axis and y-axis directions respectively, d x 、d y represent the distances of the vehicle to the end point of the trajectory in the x-axis and y-axis directions respectively, h t-1 represents the heading angle of the vehicle at time t-1, d des represents the distance of the vehicle from the end point of the trajectory, represent the velocities of the vehicle in the x-axis and y-axis directions at time t-1 respectively, d x 、d y The positive or negative signs of represent the direction of the vehicle, o i,j is the observation state of vehicle i with respect to vehicle j. The observation information of agent i with respect to agent j is represented by the relative distances and velocities of the two agents in the x-axis and y-axis directions, represent the relative lateral and relative longitudinal distances between agents i and j respectively, represent the relative lateral and relative longitudinal velocities between agents i and j respectively.

[0016] As a preferred technical solution, the action space of the agent is modeled as:

[0017] A=(acc,Δh)

[0018] where acc is the magnitude of the vehicle's acceleration and Δh is the change rate of the vehicle's steering angle.

[0019] As a preferred technical solution, the discriminator includes a binary classification function corresponding to the number of agents, which is used to distinguish whether the observation-value pair comes from the real trajectory or the generator. The discriminator includes a reward estimator and a potential function, and the reward estimator is used to provide rewards for the generator.

[0020] As a preferred technical solution, based on the goal-oriented gain reward and the discriminator reward, the Gaussian policy network and the value estimator network of the agent are updated. Specifically, the generator includes a policy network and a value estimator network. The policy network is a Gaussian policy network based on a multi-layer perceptron for characterizing a high-dimensional continuous action space. The value estimator network is based on a multi-layer perceptron, and the value estimator network is used to estimate that when the state of the agent is o t , and other agents take actions a -i,t under the condition of, the policy π iThe total expected reward, where the subscript -i represents all agents except i. Based on the reward estimated by the discriminator-based reward estimator and the goal-oriented gain reward, update the Gaussian policy network of the agents in the generator. The goal-oriented gain reward is:

[0021] r e = k1r des + k2r lane

[0022]

[0023] where r e 、r des 、r lane represent the scalar-oriented gain reward, the goal-guided reward, and the direction constraint reward respectively. k represents the reward weight, u des is a preset value, and d min is the distance of the current trajectory from the center line of the lane.

[0024] As a preferred technical solution, the process of updating the discriminator and the generator includes:

[0025] For the discriminator, with the goal of maximizing the correctness of the discriminator's judgment of the vehicle trajectory to which the observation-action pair belongs, update the reward estimator parameters and potential function parameters of each agent;

[0026] For the generator, train the Gaussian policy network and the value estimator network, with the goal of confusing the discriminator to the greatest extent, and update the Gaussian policy network parameters and the value estimator network parameters.

[0027] As a preferred technical solution, it further includes:

[0028] Policy evaluation: For a time step in the scenario, obtain the state of the target vehicle and the observation information of the surrounding vehicles at this time step and the next time step. By adjusting the speed of the target vehicle in the x-axis and y-axis directions at the next time step, construct a heat map of the reward values corresponding to different actions taken at the current moment, and evaluate the strategic intentions of the target vehicle to compete and cooperate in a specific environment;

[0029] Trajectory accuracy evaluation: Evaluate the similarity between trajectories by calculating the root mean square error of positions at the micro level, and evaluate the distribution of speeds and / or angles between trajectories by calculating the JS distance at the macro level.

[0030] As a preferred technical solution, the discriminator includes a binary classification function that matches the number of agents.

[0031] As a preferred technical solution, the reward estimator and potential function, Gaussian policy network and value estimator network of the agent are updated by means of centralized training and distributed execution.

[0032] As a preferred technical solution, a heterogeneous policy network is established for vehicle flow to generate simulated vehicle trajectory data.

[0033] Another aspect of the present invention provides a data-driven simulation system for complex interactive behavior strategies of intersection vehicles, which is used to implement the foregoing data-driven simulation method for complex interactive behavior strategies of intersection vehicles. The simulation system includes a model construction module and an application simulation module.

[0034] The model construction module includes:

[0035] An expert data acquisition unit, which is used to determine the observation space and action space required for multi-agent imitation learning, acquire the traffic trajectory data of real intersection vehicles, and construct a feature matrix for further model training;

[0036] A policy simulation training unit, which is used to train the agent to simulate the behavior logic of the expert through multi-agent imitation learning to establish a social policy simulation model;

[0037] A policy simulation evaluation unit, which is used to construct a multi-dimensional evaluation system by combining the policy and trajectory accuracy, and evaluate the effectiveness and reliability of the social policy simulation model when performing real-world interaction tasks.

[0038] The application simulation module includes:

[0039] A data processing unit, which is used to provide the initial state of the vehicle and background information for generating the scenario;

[0040] A scenario generation unit, which is used to generate complex and diverse strong interaction scenarios based on the social policy simulation model.

[0041] As a preferred technical solution, the simulation model includes a generator and a discriminator. Among them, the generator includes a policy network and a value estimator network, and the discriminator includes a reward estimator and a potential function.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] (1) Fully characterizing the behavior of vehicles at intersections: Through the simulation of complex interactive behavior strategies of vehicles based on multi-agent adversarial inverse reinforcement learning, the present invention learns and updates the parameters of the generator and discriminator, restores the reward function containing the behavior mechanism, accurately characterizes the dynamic competition and cooperation strategies of the interaction, and effectively learns the diversity of interaction strategies, making the simulation results closer to the actual situation.

[0044] (2) Good model convergence stability: By introducing target-oriented gain rewards, the present invention overcomes the challenges in model convergence and stability, while providing more training signals and improving the learning effect of the model.

[0045] (3) Considering the heterogeneity of driving behaviors: In the process of generating simulated vehicle trajectory data, the present invention considers the heterogeneous characteristics of driving behaviors by adopting a heterogeneous policy network for vehicles in different flow directions. Compared with the homogeneous policy network, the trajectory accuracy is greatly improved. Description of the Drawings

[0046] Figure 1 It is a schematic flowchart of the data-driven simulation method for complex vehicle interaction behaviors at intersections in the embodiment;

[0047] Figure 2 It is a schematic layout diagram of the intersection in the embodiment;

[0048] Figure 3 It is a reward heat map of a typical scenario in the embodiment;

[0049] Figure 4 It is a schematic diagram of the data-driven simulation system for complex vehicle interaction behaviors at intersections in the embodiment;

[0050] Figure 5 For the simulation process diagram of the simulation system using Figure 4 in the embodiment;

[0051] Figure 6 It is a schematic diagram of the electronic device in the embodiment. Detailed Embodiment

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] Embodiment 1

[0054] To address the problems existing in the aforementioned prior art, this embodiment provides a data-driven simulation method for complex vehicle interaction behaviors at intersections, which uses multi-agent adversarial inverse reinforcement learning (MA-AIRL) for modeling. MA-AIRL is a multi-agent imitation learning method (MAIL) that directly learns the interaction mechanism from human-vehicle interaction data and uses an adversarial deep neural network discriminator to recover the multi-agent reward functions of road users without pre-supposing the interaction strategies of the agents considered. These reward functions form a concise representation of the vehicle interaction strategy, further guiding the generation of realistic driving actions. In addition, this method is scalable to high-dimensional state and action spaces as well as unknown dynamics, and can recover robust reward functions less affected by environmental dynamics, making the modeling framework have a certain degree of transferability to different scenarios. Based on MA-AIRL, this application constructs a heterogeneous policy network for vehicle flows, effectively considering the heterogeneity of the strategies of opposing vehicle flows; introduces a goal-oriented gain reward to improve the model stability and the learning efficiency of expert behaviors; and formulates a multi-dimensional evaluation system to evaluate the effectiveness of the model in capturing strategies and generating trajectories.

[0055] See Figure 1 , this method mainly includes input, training, output, and simulation stages. Taking the interaction behavior between left-turning and oncoming straight-going vehicles at an intersection without a left-turn protection phase as an example for illustration. This framework constructs policy networks for the heterogeneous driving behaviors of left-turning and straight-going traffic flows respectively, and through the adversarial generative learning of a generator and a discriminator, establishes a multi-directional vehicle flow simulation model at the intersection: a heterogeneous multi-agent adversarial inverse reinforcement learning model. Based on training with real interaction trajectory data, two main outputs are obtained: a policy network and a reward estimator. The policy network is used to generate the actions executed by the vehicles and update the two-dimensional trajectory positions; the reward estimator is used to calculate the rewards obtained for the executed actions, which are coupled with the gain rewards to represent the internal mechanism of the interaction and are used to analyze the dynamic decision-making process during the vehicle's travel, thereby realizing the simulation of dynamic interaction strategies and motion states. The specific processes of each stage are as follows:

[0056] Step S1, input stage. Obtain real vehicle trajectory data, including the vehicle states and vehicle actions at each time slice.

[0057] In this step, the observation space (i.e., vehicle state) and action space (i.e., vehicle action) required for multi-agent imitation learning are modeled. Each agent corresponds to a vehicle, and the observation space of the vehicle is as follows:

[0058] o i ={s i ,o i,j}

[0059] In the formula,

[0060]

[0061] o i is the observation space of vehicle i at time t, which is a combination of its own state and the observations of other vehicles. s i is its own state, p x , p y , v x , v y represent the position and speed of the vehicle in the x-axis and y-axis directions respectively. d x , d y represent the distances of the vehicle to the end point of the trajectory (the true trajectory position of the vehicle when leaving the intersection) in the x-axis and y-axis directions respectively. h t-1 represents the heading angle of the vehicle at time t - 1. d des represents the distance of the vehicle from the end point of the trajectory. represent the speeds of the vehicle in the x-axis and y-axis directions at time t - 1 respectively. The signs of d x , d y represent the direction of the vehicle. o i,j is the observation of other vehicles. The observation value of agent i for agent j is represented by the relative distances and speeds of the two agents in the x-axis and y-axis directions. represents the relative lateral distance between agents i and j.

[0062] The action space of the vehicle is as follows:

[0063] A = (acc, Δh)

[0064] where acc is the acceleration of the vehicle (excluding direction), and Δh is the change rate of the vehicle's steering angle.

[0065] The real vehicle trajectory data, that is, the expert data, uses the SinD public dataset, which records the high-resolution traffic trajectory data of an unprotected left-turn phase intersection in Tianjin, China. The data acquisition frequency is 10 Hz (0.1 s), and the intersection layout is as Figure 2As shown, the dashed line range is the research area. Vehicle trajectory data within the research area was collected, and a total of 171 interaction scenarios were extracted. Each scenario includes a left-turn vehicle (west approach lane) and a straight-through vehicle (east approach lane). The initial frames of the vehicles in the same scenario are the same, and the scenario duration is 3.3s - 13.9s. For each vehicle in each traffic scenario, its own vehicle state and vehicle actions will be extracted at each time step in the aforementioned manner. To improve the training efficiency, all data is normalized to (0, 1) or (-1, 1). The model is trained using the trajectories of 150 traffic scenarios as the training dataset, and the effectiveness of the model is tested on the trajectories of the remaining 21 traffic scenarios.

[0066] Collect the behavior data of the expert interaction scenarios within the currently selected target scenario. Based on the observation space and action space established above, process the expert data into a feature matrix for use in the model training of subsequent steps.

[0067] Step S2, training phase. Using the pre-constructed generator and discriminator, based on the real vehicle trajectory data in Step S1, perform multi-agent adversarial inverse reinforcement learning.

[0068] This step uses the feature matrix of the expert behavior obtained in Step S1 as the training data, and uses the Gaussian policy network in the generator to generate vehicle trajectories. Specifically, the generator uses the reward learned by the discriminator and the gain reward to learn to generate better vehicle trajectories.

[0069] Preferably, during the modeling process of the agent's generator, a heterogeneous policy network is established for the traffic flow, effectively considering the heterogeneity of the strategies of opposing traffic flows. Specifically, considering that the left-turning and straight-through motor vehicles interacting within an intersection with an unprotected left-turn phase come from different approach lanes and have significantly different driving behaviors, the vehicles are divided into two groups: left-turn vehicle agents and straight-through vehicle agents. Among them, the same type of vehicle agents (i.e., vehicles from the same approach lane) share a Gaussian policy network and a value estimation network, and the generation process is essentially a reinforcement learning process.

[0070] The discriminator contains N binary classification functions (N is the number of agents), which can distinguish whether a specific observation-action pair comes from the real trajectory or the generator, and provide a reward function for the generator.

[0071] Preferably, in addition to the reward provided by the discriminator to the generator, a target-oriented gain reward is introduced and jointly used for the training of the generator to improve the model stability and the learning efficiency of the expert behavior. The target-oriented gain reward includes target guidance and direction constraint, and the specific form is shown in the following formula:

[0072] r e = k1r des + k2r lane

[0073]

[0074] where k represents the reward weight; r des represents the target guidance. When the current position is within 0.5 m from the end point, a positive reward u des is given, and there is no reward in other cases; r lane represents the direction constraint. When the current trajectory is more than 1 m away from the center line of the lane, a negative reward exp(-(d min -1)) - 1 is given, and there is no reward in other cases, encouraging the vehicle to reduce unnecessary steering.

[0075] During the multi-agent adversarial inverse reinforcement learning process, it includes the parameter update for the discriminator and the parameter update process for the generator.

[0076] Parameter update process of the discriminator: By updating the reward estimator parameter (ω i ) and the potential function parameter (φ i ) of each agent i to train the model to maximize the following formula, that is, to maximize the probability that the prediction of the observation-action pair (χ E ) from the true trajectory is true, and at the same time minimize the probability that the prediction of the observation-action pair (χ π ) from the generator is true.

[0077]

[0078] where, is the binary classifier of agent i, f ω is the reward function, including the reward estimator g ω and the potential function h φ , both of which are double-layer MLP networks with 128 neurons in each layer. q θ is the adaptive sampler with parameter θ, representing the probability that the generator selects action a given the observation o.

[0079] Update process of the generator: First, update the value estimator parameter φ i of each agent i to reduce the squared difference between the true value function (V * ) and the estimated value of the value estimator network (V φ ). Further, use the advantage estimate value to update the policy parameter θ i of each agent i through a small step size of policy gradient. The policy gradient estimation is shown as follows. This method directly adjusts each policy parameter θ i to maximize its objective (expected cumulative return

[0080]

[0081] For the continuously sampled observation-action pairs (χ π and χ E ), repeat the above process. Finally, the trained model can generate vehicle trajectories to simulate the operation of vehicles within the intersection and can use the learned reward function to explain the interaction behavior.

[0082] Preferably, in this step, the interaction behavior between vehicles is captured by a training method of centralized training and decentralized execution (CTDE).

[0083] Step S3, output phase. Through the parameter update in step S2, a policy network for generating actions to update the two-dimensional trajectory position and a reward estimator for calculating rewards to analyze the dynamic decision-making process are obtained.

[0084] In step S2, repeat the process of step S2 for the continuously sampled observation-action pairs. Finally, the trained model can generate vehicle trajectories to simulate the operation of vehicles and can use the learned reward function to explain the interaction behavior.

[0085] Step S4, simulation phase. Based on the real vehicle trajectory data and the generated vehicle trajectory data, simulate the competitive (left-turning vehicles go first) and cooperative (straight-going vehicles go first) behaviors of vehicles when passing through the intersection.

[0086] The simulation evaluation can be divided into social policy evaluation and trajectory accuracy evaluation.

[0087] For the social policy evaluation, this step evaluates the overall decision-making accuracy and dynamic decision-making accuracy of the generated scenario to verify that the model can accurately reproduce the behavior logic and decision-making process during vehicle interaction. Specifically, the overall decision-making accuracy of the generated scenario is the degree of similarity between the order in which the model generates vehicles passing through the conflict point and the order of real-scenario vehicles passing through; the dynamic decision-making accuracy is evaluated through a heat map. For a time step in the real scenario, obtain the states of the target vehicle and the observation information of surrounding vehicles at this time step and the next time step, and keep the parameters except v x and v y relatively unchanged. By adjusting the values of v x and v y within a reasonable range, construct a heat map of the reward values corresponding to different actions taken at the current moment.

[0088] The brighter areas in the heat map indicate higher rewards, which reflect the desired speed and direction preferences of the target vehicle in a specific environment, i.e., the strategic intentions of competition and cooperation. By analyzing the dynamic decision-making process of vehicles in different scenarios and comparing it with the actual actions, the interpretability of the reward function restored by the model for the dynamic interaction strategy is demonstrated.

[0089] For the evaluation of trajectory accuracy, this step evaluates the similarity between the generated trajectory and the ground truth trajectory at the micro-trajectory level and the macro-feature distribution level. Specifically, at the micro-trajectory level, the root mean square error (RMSE) of the position is used to compare the similarity of the two trajectories, and at the macro-feature distribution level, the Jensen-Shannon (JS) divergence is used to evaluate the distributions of the speed and angle of the generated trajectory and the ground truth trajectory. The closer the JS divergence is to 0, the more similar the two distributions are. The calculation formulas for RMSE and JS divergence are as follows:

[0090]

[0091] where is the actual position of vehicle i at time t in the x-y coordinate system, is the position generated by the model for vehicle i at time t in the x-y coordinate system, N is the number of vehicles, and T is the total number of steps in a traffic scenario.

[0092]

[0093] where p and q respectively refer to the probability distributions of the evaluation metrics of the generated trajectory and the ground truth trajectory, m is the average distribution of the two probability distributions p and q, and D is the KL divergence:

[0094] To verify that the social strategy simulation model can accurately reproduce the behavior logic and decision-making process during vehicle interaction, the overall decision-making accuracy and dynamic decision-making accuracy of the generated scenarios are evaluated below using Figure 2 as an example.

[0095] (1) Overall interaction strategy.

[0096] The accuracy of the overall decision-making, i.e., whether the order in which the vehicles generated by the model pass through the conflict point is the same as that of the vehicles in the real scenario. If the left-turn vehicle passes through the conflict point earlier than the straight-ahead vehicle, it is considered that the left-turn vehicle goes first; otherwise, the left-turn vehicle yields. The evaluation results of the test scenarios with conflicts are shown in Table 1 below. The results show that the overall interaction decision-making accuracy of the model-generated scenarios is 100%, which reflects the accurate grasp of the basic intersection behavior of the MA-AIRL-2 model in traffic scenarios.

[0097] Table 1 Overall interaction strategy accuracy of the social strategy simulation model

[0098]

[0099] (2) Dynamic interaction strategy.

[0100] Select a scenario from the dataset. Four specific time steps are selected, and the reward heatmaps are plotted respectively to analyze the dynamic decision-making processes of left-turning and straight-going vehicles at intersections under different scenarios, and compare them with the actual actions, which proves the interpretability of the reward function recovered by the model for the interaction strategy, as Figure 3 shown. The first row is the acceleration curve graph of the left-turning vehicle and the straight-going vehicle in the real scenario at the current moment. The second row is the position and speed. The direction of the arrow represents the direction of the speed, and the length represents the absolute speed value. The third and fourth rows are the reward heatmaps of the straight-going vehicle and the left-turning vehicle respectively, and the arrows represent the tendency of higher reward functions.

[0101] At different times, the interaction strategies of the left-turning vehicle and the straight-going vehicle show dynamic changes. Through the heatmap of Scenario 1, it can be observed that initially, the two vehicles are far apart. The straight-going vehicle has a greater reward at a lower lateral speed and tends to decelerate. The longitudinal speed range covered by the peak reward of the left-turning vehicle is wider, and the reward is relatively large even at a lower longitudinal speed, indicating that the left-turning vehicle may tend to not turn or decelerate, and both parties expect to adopt a cooperative strategy. At 1.0 s, the two vehicles gradually approach. The reward of the straight-going vehicle gradually increases at a higher lateral speed and tends to adopt a competitive strategy, while the left-turning vehicle still may tend to stay away from the conflict point and shows a cooperative tendency. At 3.0 s, the two vehicles get closer. The straight-going vehicle still expects to accelerate through the intersection, and the lateral speed at a larger reward value is higher than that at 1.0 s, tending to adopt a competitive strategy. The longitudinal speed corresponding to the peak reward of the left-turning vehicle generally decreases compared with 1.0 s, indicating that the left-turning vehicle clearly tends to adopt a cooperative strategy. At 4.0 s, the straight-going vehicle still expects to drive away from the intersection at a high speed, while the left-turning vehicle has a greater reward at higher lateral and longitudinal speeds, indicating that the left-turning vehicle already tends to turn and drive towards the destination. In the real scenario, initially both parties adopt a cooperative strategy of decelerating, then the straight-going vehicle adopts an accelerating competitive strategy, and the left-turning vehicle adopts a cooperative strategy of not turning. Until the straight-going vehicle is about to drive away from the conflict point, the left-turning vehicle begins to have a turning tendency. Generally speaking, the interaction strategy between vehicles reflected by the reward heatmap is consistent with the acceleration and speed trends of the real trajectory.

[0102] Furthermore, the similarity between the model-generated trajectories and the real trajectories is evaluated at the micro-trajectory level and the macro-feature distribution level. It is compared with some benchmark models, including the heterogeneous MA-AIRL model without considering the gain reward (MA-AIRL-2’), the MA-AIRL model with homogeneous policy networks for left-turning and straight-going vehicles but without considering the gain reward (MA-AIRL-1), the deep learning models (SimNet, long short-term memory (LSTM) model), and the multi-agent reinforcement learning model (MA-A3C). The evaluation results are shown in Table 2 below.

[0103] Table 2 Comparison Results of Policy Simulation Model and Benchmark Models

[0104]

[0105] In summary, the policy simulation model has relatively good performance under the evaluation metrics, proving its effectiveness in learning to guide the generation of the optimal reward function and policy for real traffic scenarios. The features of this method are as follows:

[0106] (1) The social policy simulation model accurately depicts the dynamic competition and cooperation strategies of interactions by restoring the reward function containing behavioral mechanisms and effectively learns the diversity of interaction strategies.

[0107] (2) By adopting heterogeneous policy networks for vehicles in different flow directions, the heterogeneity characteristics of driving behaviors are considered, and the trajectory accuracy is greatly improved compared with the homogeneous policy network.

[0108] (3) By introducing the goal-oriented gain reward, the challenges in model convergence and stability are overcome, and at the same time, more training signals are provided to improve the learning effect of the model.

[0109] Embodiment 2

[0110] Based on the foregoing embodiments, referring to Figure 4 , this embodiment provides a data-driven intersection vehicle complex interaction behavior policy simulation system for implementing the data-driven intersection vehicle complex interaction behavior policy simulation method as in Embodiment 1. The simulation system includes a model construction module and an application simulation module.

[0111] The simulation system includes a model construction module and an application simulation module,

[0112] The model construction module includes:

[0113] An expert data acquisition unit, which is used to determine the observation space and action space required for multi-agent imitation learning, acquire the traffic trajectory data of real intersection vehicles, and construct a feature matrix for further model training;

[0114] A policy simulation training unit, which is used to train an agent to simulate the behavior logic of an expert through multi-agent imitation learning to establish a social policy simulation model;

[0115] A policy simulation evaluation unit, which is used to construct a multi-dimensional evaluation system by combining policies and trajectory accuracies to evaluate the effectiveness and reliability of the social policy simulation model when performing real-world interaction tasks.

[0116] The application simulation module includes:

[0117] A data processing unit, which is used to provide the initial state and background information of vehicles for generating scenarios;

[0118] A scenario generation unit, which is used to generate complex and diverse strong interaction scenarios based on the social policy simulation model.

[0119] See Figure 5 For the flow chart of using the established simulation system for simulation. First, use the model construction module to determine the imitation learning observation and action spaces, obtain and process expert data, construct a multi-agent imitation learning framework, train the social policy simulation model, construct a multi-dimensional evaluation system, and evaluate the effect of the policy simulation model. Finally, use the application simulation module to generate complex and diverse strong interaction scenarios based on the obtained simulation model.

[0120] Embodiment 3

[0121] This embodiment provides an electronic device, including: one or more processors and a memory. The memory stores one or more programs, and the one or more programs include instructions for executing the data-driven intersection vehicle complex interaction behavior policy simulation method as described in Embodiment 1.

[0122] As Figure 6 Described above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned policy simulation method. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or a logic device.

[0123] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0125] As described above, it is only the specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections, characterized in that The method includes the following steps: Obtain the traffic trajectory data of real intersection vehicles, construct multiple interaction scenarios, for each vehicle in each interaction scenario, extract the observation information at each time step to obtain the real vehicle trajectory data; Take each vehicle as an agent, and generate simulated vehicle trajectory data based on the Gaussian policy network constructed in advance according to vehicle types; Based on the real vehicle trajectory data and the simulated vehicle trajectory data, extract the observation - action pair information respectively, and use the pre - constructed discriminator to judge whether the vehicle trajectory to which the observation - action pair belongs is real. Among them, the discriminator aims to distinguish between generated and real trajectories, update the reward estimator and potential function of the agent, and the generator aims to confuse the discriminator to the greatest extent, update the Gaussian policy network and value estimator network of the agent, so as to realize the simulation of complex vehicle interaction behavior strategies based on multi - agent adversarial inverse reinforcement learning.

2. The data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, wherein During the process of extracting the observation values at each time step, the observation space includes: o i ={s i ,o i,j} Among them, o i is the observation space of vehicle i at time t, s i is the self-state of vehicle i, p x and p y and v x and v y represent the position and speed of the vehicle in the x-axis and y-axis directions respectively. d x and d y represent the distances of the vehicle to the end point of the trajectory in the x-axis and y-axis directions respectively. h t-1 represents the heading angle of the vehicle at time t - 1, and d des represents the distance of the vehicle from the end point of the trajectory. represent the speeds of the vehicle in the x-axis and y-axis directions at time t - 1 respectively. The signs of d x and d y represent the direction of the vehicle. o i,j is the observation state of vehicle i for vehicle j. The observation information of agent i for agent j is represented by the relative distances and speeds of the two agents in the x-axis and y-axis directions. represent the relative lateral and relative longitudinal distances between agents i and j respectively. represent the relative lateral and relative longitudinal speeds between agents i and j respectively.

3. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that The action space of the agent is modeled as: A=(acc,Δh) where acc is the magnitude of the vehicle's acceleration and Δh is the change rate of the vehicle's steering angle.

4. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that The discriminator includes binary classification functions corresponding to the number of agents, which are used to distinguish whether the observation - action pair comes from a real trajectory or the generator. The discriminator includes a reward estimator and a potential function, and the reward estimator is used to provide rewards for the generator.

5. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that The generator described above includes a policy network and a value estimator network. The policy network is a Gaussian policy network based on a multi-layer perceptron for representing a high-dimensional continuous action space. The value estimator network is based on a multi-layer perceptron and is used to estimate the total expected return of policy π t when the state of the agent is o -i,t and other agents take action a i . The subscript -i represents all agents except i. The Gaussian policy network of the agent in the generator is updated based on the reward estimated by the reward estimator of the discriminator and the target-oriented gain reward. The target-oriented gain reward is as follows: r e = k1r des + k2r lane Among them, r e , r des , r lane respectively represent the standard-oriented gain reward, the target-guided reward, and the direction constraint reward, k represents the reward weight, u des is a preset value, and d min is the distance of the current trajectory from the center line of the lane.

6. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that The process of updating the discriminator and the generator includes: For the discriminator, aiming to maximize the correctness of the discriminator's judgment of the vehicle trajectory to which the observation - action pair belongs, update the reward estimator parameters and potential function parameters of each agent; For the generator, train the Gaussian policy network and the value estimator network, aiming to confuse the discriminator to the greatest extent, and update the Gaussian policy network parameters and value estimator network parameters.

7. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that It also includes: Policy evaluation: For a time step in the scenario, obtain the state of the target vehicle and the observation information of surrounding vehicles at this time step and the next time step. By adjusting the speed of the target vehicle in the x - axis and y - axis directions at the next time step, construct a heat map of the reward values corresponding to different actions taken at the current moment, and evaluate the strategic intentions of the target vehicle to compete and cooperate in a specific environment; Trajectory accuracy evaluation: At the micro - level, evaluate the similarity between trajectories by calculating the root - mean - square error of positions, and at the macro - level, evaluate the distribution of speeds and / or angles between trajectories by calculating the JS distance.

8. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that Update the reward estimator and potential function, Gaussian policy network and value estimator network of the agent in a centralized training and distributed execution manner.

9. A data-driven simulation method for complex vehicle interaction behavior strategies at intersections according to claim 1, characterized in that, Generate simulated vehicle trajectory data by establishing a heterogeneous policy network for vehicle flow.

10. A data-driven simulation system for complex vehicle interaction behavior strategies at intersections, characterized in that, For implementing the data - driven simulation method for complex vehicle interaction behavior strategies at intersections as described in any one of claims 1 - 9, the simulation system includes a model construction module and an application simulation module, The model construction module includes: An expert data acquisition unit, which is used to determine the observation space and action space required for multi-agent imitation learning, acquire the traffic trajectory data of real intersection vehicles, and construct a feature matrix for further model training; A policy simulation training unit, which is used to train agents to simulate the behavior logic of experts through multi-agent imitation learning to establish a social policy simulation model; A policy simulation evaluation unit, which is used to construct a multi-dimensional evaluation system by combining policies and trajectory accuracy, and evaluate the effectiveness and reliability of the social policy simulation model when performing real-world interaction tasks, The application simulation module described above includes: A data processing unit, which is used to provide the initial state and background information of vehicles for generating scenarios; A scenario generation unit, which is used to generate complex and diverse strong interaction scenarios based on the social policy simulation model.

Citation Information

Cited By

  • Data-driven simulation method and system for complex interactive behaviors of mixed traffic flow

    CN121145678A

  • Data-driven simulation method and system for complex interaction behavior of mixed traffic flow

    CN121145678B

  • Prediction method for interaction track of right-turn vehicle and pedestrian at intersection and computer equipment

    CN121457756A

  • Multi-mode traffic simulation method based on multi-agent sequential decision

    CN121600715A

  • Traffic simulation parameter calibration method and device based on depth deterministic strategy gradient

    CN122221689A