Multi-agent complex dangerous scene generation method based on confrontation trajectory

Abstract

The embodiment of the invention discloses a multi-agent complex dangerous scene generation method based on an adversarial trajectory, and the method comprises the steps: obtaining a natural driving data set; determining a scene initial state based on the natural driving data set, and predicting a corresponding scene prediction state through a pre-trained multi-agent scene generation model according to the scene initial state, splicing the scene initial state and the scene prediction state to obtain a multi-agent complex safety scene, performing real multi-agent scene extraction based on the natural driving data set and the multi-agent complex safety scene, and constructing an aggressive driver model for training in the real multi-agent scene based on a preset PPO-MAE algorithm; based on the aggressive driver model, an adversarial trajectory is generated in a real multi-agent scene, a multi-agent complex dangerous scene is generated, and the method not only can increase the number of training samples and make the model more accurate, but also can increase dangerous scene samples, improve the dangerous scene extraction efficiency and reduce the complexity of algorithm calculation when dimensions are too many.

Description

Technical Field

[0001] This application relates to the field of autonomous driving simulation testing, specifically to a method for generating complex dangerous scenarios by multiple agents based on adversarial trajectories. Background Technology

[0002] With the rapid development of autonomous driving technology and the increasing prevalence of autonomous vehicles, the safety of autonomous driving has become a major concern. Simulation testing of autonomous vehicles before they are put on the road is an essential step. Scenario-based autonomous driving simulation testing is currently a mainstream method, requiring autonomous driving systems to be simulated under various road conditions, traffic situations, and abnormal circumstances. Current mainstream scenario generation methods, which generate simple entry, following, and lane-changing scenarios, are no longer sufficient to meet the increasingly complex simulation testing needs of autonomous driving systems. Generating reasonably predictable complex scenarios involving multiple agents can better simulate the complexity and diversity of the real world, thereby improving the accuracy and reliability of simulation testing. However, due to the interconnectedness of traffic participants, different traffic entities can significantly influence the decisions of other entities, making it difficult to generate reasonably predictable complex scenarios involving multiple traffic participants. Furthermore, the coverage of safety-critical scenarios and rare risk events is low in reality. To achieve comprehensive testing of autonomous driving, it is hoped that virtual simulation technology can be used with effective scenario generation methods to generate reasonably predictable dangerous scenarios and discover unknown performance defects in the system under test. However, existing scenario generation methods struggle to generate dangerous scenarios with multiple traffic participants. Based on the above points, it is clear that how to generate complex multi-agent scenarios with multiple traffic participants that pose risks is an urgent problem to be solved in the field of autonomous driving simulation testing. Summary of the Invention

[0003] To address the aforementioned problems, this application provides a method for generating complex and dangerous multi-agent scenarios based on adversarial trajectories. This method includes:

[0004] Step 1: Obtain the natural driving dataset;

[0005] Step 2: Determine the initial state of the scene based on the natural driving dataset, predict the corresponding scene prediction state based on the initial state of the scene using a pre-trained multi-agent scene generation model, and concatenate the initial state of the scene and the scene prediction state to obtain a multi-agent complex safety scene.

[0006] Step 3: Extract real multi-agent scenarios based on the natural driving dataset and the multi-agent complex safety scenario, and build an aggressive driver model for training based on the preset PPO-MAE algorithm in the real multi-agent scenario;

[0007] Step 4: Based on the radical driver model, generate adversarial trajectories in the real multi-agent scenario to generate a complex and dangerous multi-agent scenario.

[0008] The embodiments of this application have the following beneficial effects:

[0009] The method proposed in this application addresses the challenge of modeling the interactions between different traffic participants in complex scenarios, overcoming the limitations of using a single method in hazardous scene generation. These limitations include: 1. Insufficient training data on hazardous driving scenarios. 2. Extraction from natural driving data: The proportion of hazardous scenes in real-world driving is too small, resulting in low scene coverage and extraction efficiency. 3. Algorithm optimization: For complex scenarios with multiple traffic participants, the algorithm suffers from excessive dimensionality and computational complexity. Therefore, this application not only increases the number of training samples, making the model more accurate, but also increases the number of hazardous scene samples, improving hazardous scene extraction efficiency, and reduces the computational complexity of the algorithm when the dimensionality is too high. Attached Figure Description

[0010] To more clearly illustrate the technical solution of this application, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and therefore should not be regarded as a limitation on the scope of protection of this application. In the various drawings, similar components are numbered similarly.

[0011] Figure 1 A flowchart of the generation method proposed in this application is shown;

[0012] Figure 2 This diagram illustrates the generation method proposed in this application for generating complex security scenarios involving multiple agents.

[0013] Figure 3 This illustration shows a schematic diagram of the generation method proposed in this application for training an aggressive driver model based on the PPO-MAE algorithm.

[0014] Figure 4 The illustration shows a schematic diagram of the generation method proposed in the application embodiment for generating complex dangerous scenarios using multiple agents based on an aggressive driver model. Detailed Implementation

[0015] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0016] To more clearly demonstrate the implementation steps and advantages of this application, the specific implementation methods are described below with reference to the illustrations.

[0017] It should be noted that the "multi-agent scenario" in this article refers to a driving scenario that includes multiple traffic participants. This driving scenario is generally composed of several frames of data for each traffic participant. Each frame includes data related to the vehicle's motion state, such as the vehicle's speed, acceleration, two-dimensional coordinates, and orientation angle.

[0018] This application provides a method for generating complex and dangerous multi-agent scenarios based on adversarial trajectories. (See also...) Figure 1 The method for generating complex and dangerous multi-agent scenarios based on adversarial trajectories includes the following steps:

[0019] Step 1: Obtain the natural driving dataset.

[0020] Step 2: Determine the initial state of the scene based on the natural driving dataset, predict the corresponding scene prediction state based on the initial state of the scene using a pre-trained multi-agent scene generation model, and combine the initial state of the scene and the scene prediction state to obtain a multi-agent complex safety scene.

[0021] Step 3: Based on the multi-agent complex safety scenario, train the aggressive driver model using the preset PPO-MAE algorithm.

[0022] Step 4: Generate adversarial trajectories in complex multi-agent safety scenarios based on the aggressive driver model, thereby generating complex and dangerous multi-agent scenarios.

[0023] In this application, natural driving data will be acquired for subsequent generation of complex and dangerous multi-agent scenarios. Natural driving data refers to real vehicle driving data collected by devices such as drones or vehicle-mounted LiDAR and processed, presented in frames. Taking natural driving data based on drone video as an example, a single natural driving data set records data related to a vehicle's motion state, such as speed, acceleration, two-dimensional coordinates, and orientation angle, for each frame of the video. Furthermore, natural driving data also includes environmental information such as the vehicle's length and width, as well as lane width and lane line coordinates at the shooting location. A natural driving dataset will be acquired, comprising multiple natural driving data sets. Each natural driving data set can be used to simulate an agent, and each agent has a corresponding agent ID. In other words, the natural driving dataset corresponds to multiple agents, each with a corresponding agent ID, environmental information, and data related to the vehicle's motion state.

[0024] Based on the natural driving dataset, an agent is selected as the master agent in sequence according to the agent IDs of multiple agents. Starting from the nth frame (n is a positive integer) of the master agent, the process iterates to the last frame, searching for instances where the 2D Euclidean distance between the master agent and the current frame is less than a pre-set distance threshold X. s Other agents will be within a distance of less than the distance threshold X. s The other intelligent agents are denoted as environmental intelligent agents. The number of environmental intelligent agents should be greater than or equal to two. If only one environmental intelligent agent is found, the standard for a multi-agent scenario is not met, indicating that the master agent has not found an environmental intelligent agent; if two or more environmental intelligent agents are found, it is determined that the master agent has found the corresponding environmental intelligent agents.

[0025] Specifically, when selecting environmental agents, special judgments are needed based on road segment characteristics in the natural driving dataset. If vehicles are traveling in different directions and there are insurmountable obstacles between them, such as on highways, agents traveling in different directions cannot be considered environmental agents for this type of natural driving dataset. It can be understood that the above steps represent the preset agent selection principles in this example (the same applies below).

[0026] In this application, such as Figure 2 As shown, each primary agent and its corresponding environmental agent form an agent combination. The current agent combination includes the currently selected primary agent (i.e., the current primary agent) and the environmental agent corresponding to the current primary agent. If the current primary agent finds a corresponding environmental agent, it will determine whether the initial scene state of the current agent combination has been generated. If it has been generated, the current agent combination is skipped, and the next agent combination is evaluated; if it has not been generated, the initial scene state corresponding to the current agent combination is generated based on the aforementioned natural driving dataset. Specifically, the initial scene state corresponding to each agent combination can be extracted from the natural driving dataset, using the two-dimensional motion trajectory and motion state of an agent combination within a certain time period as the initial scene state.

[0027] Generate the initial state of the current agent group, specifically including:

[0028] Given a first preset number of frames N, where N is a positive integer, determine whether the natural driving dataset includes data from the current agent combination up to N frames prior to the current frame. If so, preprocess the data of all agents in the current agent combination up to the first preset number of frames prior to the current frame to obtain the corresponding Pytroch tensor, which satisfies the input of the multi-agent scene generation model. The preprocessed Pytroch tensor of the first preset number of frames will serve as the initial state of the scene for the current agent combination. Different methods can be used to preprocess different data to obtain the corresponding Pytroch tensor. Taking the roundD-datasets dataset as an example, the preprocessing steps are as follows (the preprocessing methods for other datasets may differ in detail due to differences in dataset structure, but the main steps are the same as those described below):

[0029] 1. Clean the data of the current agent combination in the natural driving dataset from the current frame back to the first preset number of frames in advance to remove specific vehicle trajectories (vehicle trajectories with abnormal data found by manual inspection).

[0030] 2. Add maneuver action labels to each vehicle trajectory data: Determine the vehicle's maneuver type based on the vehicle's starting and ending heading angles, and then select the corresponding maneuver action label. The specific steps are as follows:

[0031] For each vehicle, obtain its starting and ending heading angles; define possible steering angles -π / 2, 0, π / 2, and π, with the four steering angles corresponding to four vehicle behaviors: -π / 2 represents a left turn, 0 represents going straight, π / 2 represents a right turn, and π represents a U-turn. These four labels are used to assist training and inference; calculate the difference between the starting and ending heading angles of each vehicle and compare it with the defined possible steering angles; select the closest steering angle as the maneuver label for the vehicle's trajectory data.

[0032] 3. Add polar coordinate features to each vehicle trajectory data: Convert the vehicle's Cartesian coordinates to polar coordinates. The specific steps are as follows:

[0033] Obtain the Cartesian coordinates (x, y) of each vehicle; calculate the polar coordinates (ρ, θ) of each vehicle relative to the reference point (x0, y0), and add the polar coordinates to the vehicle trajectory data of each vehicle.

[0034] 4. Extract and downsample vehicle trajectory data from multiple consecutive specified time steps starting from the current frame, and perform data analysis based on the vehicle's heading angle ψ and velocity (v). x v y ), acceleration (a x a y Construct the input sequence;

[0035] The target sequence is constructed based on the vehicle's Cartesian coordinates, heading angle, velocity, and acceleration at multiple consecutive future time steps following the input sequence.

[0036] 5. Each vehicle (including the target vehicle and adjacent vehicles) corresponds to a node in an undirected graph, and there is an edge between any two vehicles, representing their interaction relationship (regardless of distance);

[0037] Considering NaN (Not a Number) values ​​in nodes (such as vehicle nodes with missing data will be excluded), construct an edge index separately for each time step, including fully connected nodes of all nodes (excluding NaN nodes), and construct a global edge index;

[0038] An undirected graph representing the interaction relationships between vehicles is constructed based on node and edge indices;

[0039] The Euclidean distance is calculated for each pair of vehicle coordinates using vehicle coordinate data at a single time step. The result is stored as a tensor. The sequence data is input, and the Euclidean distance for all vehicle pairs at each time step is calculated based on the tensor. Finally, a list of edge features for each time step is returned.

[0040] 6. Convert the above undirected graph, input sequence, target sequence, edge feature list, and edge index into Pytroch tensors.

[0041] Once all frames of the primary agent have been traversed, another agent that has not yet been selected is chosen as the primary agent, and the scene initial state generation process described above is executed. Scene initial state generation ends after all agents have executed the scene initial state generation process as primary agents.

[0042] The preprocessed initial scene state of each agent combination is input into the multi-agent scene generation model. The model outputs the predicted trajectories of all agents in each combination for the next second preset number of frames (M frames, where M is a positive integer, e.g., M = 25), denoted as the scene prediction state. By concatenating all the initial scene states and their corresponding prediction states, the complete trajectory of each agent in all agent combinations is obtained. The concatenated complete trajectory of all agent combinations constitutes the multi-agent complex security scene for that combination; in other words, multiple multi-agent complex security scenes are obtained. If it is necessary to generate multi-agent complex security scenes with a longer duration, the last N frames of the main agent's scene prediction state are preprocessed as described above to obtain the initial scene state corresponding to the last N frames of scene prediction state. This initial scene state is then input into the pre-trained multi-agent scene generation model to generate subsequent prediction trajectories, i.e., subsequent scene prediction states. This process can be repeated to generate multi-agent complex security scenes of arbitrary duration. Among them, the multi-agent scene generation model is pre-trained through a temporal graph neural network and is used to obtain the subsequent scene prediction state of the agent combination based on the initial scene state of the agent combination.

[0043] Furthermore, using the same selection principles for the primary agent and environment agent as in step 2, it is determined whether a real multi-agent scenario containing only the current primary agent and environment agent combination has been generated. If generated, the current agent combination is skipped, and the next agent combination is evaluated. If not generated, the complete trajectory of the current primary agent and environment agent is extracted as a real multi-agent scenario, i.e., the corresponding multi-agent complex security scenario of the current agent combination is extracted as a real multi-agent scenario for PPO-MAE (Proximal Policy Optimization based on Multi-Agent Environments) training. After traversing all frames of the current primary agent, another unselected agent is selected as the primary agent, and the above real multi-agent scenario extraction process is executed. The real multi-agent scenario extraction ends when all agents have performed the real multi-agent scenario extraction process as primary agents.

[0044] For adversarial trajectory generation tasks, the PPO-MAE algorithm in this application optimizes the advantage function and adaptively optimizes the pruning range based on the basic PPO (Proximal Policy Optimization) algorithm. The PPO-MAE algorithm is then used to train an aggressive driver model constructed in a real multi-agent scenario. Figure 3As shown, the PPO-MAE algorithm of this application initializes the policy network and value network, and selects a set of constituent environments in a real multi-agent scenario. In this environment, the current policy is executed and agent trajector trajector trajector D is collected: each trajectory D includes the state s, action a, and reward r for each step. The corresponding reward is calculated based on the state, action, and reward of each trajectory, and the advantage estimate is calculated using the self-attention mechanism of the PPO-MAE algorithm. Based on the corresponding advantage estimate, an aggressive driver model is trained until all agents have been completely traversed. The relevant configuration of the PPO-MAE algorithm is as follows:

[0045] ① Agent: A vehicle that needs to perform dangerous driving maneuvers to generate a counter-trajectory.

[0046] ② Environment: Includes the traffic roads within the lane coordinate range and the environmental vehicles traveling along the specified route.

[0047] ③Status: Agent's speed, orientation angle, two-dimensional coordinates, vehicle length, and vehicle width.

[0048] ④ Action: A two-dimensional continuous space consisting of the range of angle changes and the normalized range of velocity changes.

[0049] ⑤ Rewards: Environmental vehicles receive rewards when forced to stop or slow down by the agent; the closer the environmental vehicle is to the agent, the greater the reward. However, if the environmental vehicle collides with the agent, the close-range reward is reset to zero and a penalty is incurred. The more erratic the driving behavior, the greater the penalty. The specific reward function design is detailed later. The reward function design aims to construct an aggressive driver model, representing a driver who, based on their self-assessment, will engage in high-risk driving behaviors such as cutting in line if they believe they can ensure their own safety.

[0050] The PPO-MAE algorithm optimizes the basic PPO algorithm by adjusting the dominance function and adaptively optimizing the pruning range. The specific process is as follows:

[0051] The objective function of PPO-MAE is:

[0052]

[0053] In the formula, π represents the ratio of the probability of the new strategy to the probability of the old strategy. θ (a t |s t ) represents the current strategy. Indicates the strategy before the update, A t It is the dominant function. This indicates a pruning operation, used to prevent excessive changes to the policy network.

[0054] Introducing a self-attention mechanism into the policy network of PPO-MAE optimizes the computation of the PPO advantage function to capture long-term dependency information. This helps the agent consider longer-term impacts during decision-making, improving the rationality and effectiveness of decisions, especially when handling tasks with complex time-series relationships. The reward is fundamental to the objective function, in which the advantage function A... t Directly dependent on the reward function (e.g., calculated via Monte Carlo reward or TD error), the reward is obtained through the advantage function A. t This determines the direction and intensity of policy updates (i.e., positive advantages encourage actions, and negative advantages inhibit actions). In other words, the reward function sends reward signals to tell the agent what behaviors are good and what behaviors are bad, and the objective function uses mathematical methods to update the policy based on these reward signals.

[0055] When using the self-attention mechanism to optimize the advantage function in this application, firstly, assuming the current time step is t and the historical window length is L, the input sequence is constructed as follows:

[0056] X = [(s t-L ,s t-L ,s t-L ),...,(s t-1 ,s t-1 ,s t-1 ),(s t ,s t ,s t (2)

[0057] Each element is a tuple of state s, action a, and reward r.

[0058] Secondly, self-attention encoding: the input sequence X is converted into an embedding vector, and a context-aware representation is generated through a self-attention layer:

[0059] Embedding layer:

[0060] h i =Embedding(s i ,a i ,r i ) i∈{tL,…,t} (3)

[0061] Self-attention calculation:

[0062] Q = W Q H (4)

[0063] K = W K H (5)

[0064] V = W V H (6)

[0065]

[0066] Where H = [h] t-L ,...,h t ] is an embedded vector sequence, W Q W K W V It is a learnable parameter matrix, d k is the dimension of the key vector.

[0067] Furthermore, this application also improves the value function estimation by using self-attention output to generate an enhanced value function estimate V′(s). t ):

[0068] V′(s t )=MLP(Attention(Q,K,V)[t]) (8)

[0069] MLP stands for Multilayer Perceptron, which is used to map attention outputs to scalar value estimates.

[0070] Time difference error calculation includes:

[0071] The adjusted time difference error δ is calculated based on the improved V′. t ′, specifically as follows:

[0072] δ′ t =r t +γV′(s t+1 )-V′(s t (9)

[0073] In the formula, γ represents the hyperparameter, which can be set according to actual conditions.

[0074] Furthermore, this application also includes improvements to the generalized advantage estimation, using self-attention-weighted δ. t Calculate the advantage function A t :

[0075]

[0076] Wherein, weight w l Dynamically generated by a self-attention mechanism, it will replace the traditional fixed discount factor (γλ). l .

[0077] In PPO-MAE, the pruning operation is optimized for the original PPO. Adaptive pruning calculates the current pruning range and dynamically adjusts ∈ to improve training stability and convergence. In the early stages of training, a larger ∈ promotes agent exploration, enabling faster discovery of various states and reward scenarios in the environment and avoiding getting trapped in local optima. In the later stages of training, as the number of training steps increases, ∈ is reduced to decrease policy network oscillations, allowing the policy to gradually stabilize and thus improving stability. This allows the algorithm to converge to a better policy more quickly, improving training efficiency and the quality of the final policy. Adaptive pruning is achieved by combining Kullback-Leibler divergence and the number of training steps, as shown in the following formula:

[0078]

[0079] Where ∈0 is the initial pruning range (usually 0.2), used to limit the magnitude of policy updates, and the hyperparameter δ target Let KL divergence be the target. The KL divergence is updated for the current policy. The hyperparameter η is used to prevent the denominator from being 0. t represents the number of training epochs. λ is a hyperparameter that controls the decay rate of the pruning range.

[0080] like Figure 3 As shown, the PPO-MAE algorithm makes specific settings for the aggressive driver model in a multi-agent environment during the following training stages. The rest of the process is the same as the existing PPO technology, as follows:

[0081] ① Training environment setup:

[0082] The training environment includes: a two-dimensional coordinate range R of the lane defined by lane lines and lane boundaries, and an environmental vehicle V traveling along the complete trajectory. e And the agent vehicle V a .

[0083] The state space S of the agent vehicle is:

[0084]

[0085] In the formula, v t Indicates the speed of the agent vehicle. Indicates the orientation angle of the agent vehicle, (x t ,y t ) represents the two-dimensional coordinate position of the agent vehicle, l represents the longitudinal dimension of the agent vehicle, and w represents the lateral dimension of the agent vehicle.

[0086] The function of action space A is:

[0087]

[0088] In the formula, Represents the change in orientation angle, ranging from [-π, π]; Δv t Represents the velocity change, ranging from [-1, 1].

[0089] ②Trajectory generation settings:

[0090] In each round:

[0091] Agent vehicle V a From PPO sampling strategy π θ (a t |s t Select action a t ~π θ (a t |s t ) and execute.

[0092] Environmental Vehicle V e The trajectory following model, controlled by the Intelligent Driver Model (IDM), moves along the real trajectory τ. The trajectory following model based on IDM control consists of two parts: trajectory following and IDM control.

[0093] The trajectory following model updates the state of the vehicle in the environment based on real trajectory data (i.e., the complete trajectory in a real multi-agent scenario), as shown in the following formula:

[0094]

[0095] (4) In the formula, V t Let represent all environmental vehicles at time t. Let represent the state of all vehicles in the environment at time t, and let represent the corresponding vehicle state read from the real trajectory data (τ) at time t.

[0096] The response to the vehicle ahead will be combined with the IDM model to control the speed of vehicles in the current environment, as shown in the following formula:

[0097] v final =min(v trajectory ,v safe (15)

[0098] (5) In the formula, v trajectory It is the actual velocity of the trajectory point, v safe The IDM model calculates the safe speed based on the vehicle in front.

[0099] When v final equals v trajectory At the same time, maintain environmental vehicle status updates based on real trajectory data, when v final The first time does not equal vtrajectory When the IDM model triggers deceleration, the vehicle's state update is no longer based on real trajectory data. Instead, the PurePursuit controller is used in conjunction with the IDM model for trajectory following. Specifically:

[0100] The Pure Pursuit controller, combined with the IDM model, performs trajectory following consisting of two parts: lateral control and longitudinal control. Lateral control is implemented by the Pure Pursuit controller. First, a target point is selected based on the look-ahead distance *d*: starting from the real trajectory point closest to the current vehicle coordinates, the system iterates towards the endpoint of the real trajectory, finding the first point whose distance from the current vehicle coordinates is greater than the look-ahead distance as the target point. The formula for calculating the look-ahead distance *d* is as follows:

[0101] d = kv t +d min (16)

[0102] In equation (6), d is the look-ahead distance, k is the hyperparameter, and v is the scaling factor. t Given the current vehicle speed in the environment (the vehicle speed in the training environment of PPO), the hyperparameter d... min For the minimum look-ahead distance, hyperparameters k and d min All settings can be customized according to the actual situation.

[0103] The formula for calculating the orientation angle of vehicles in the current environment is as follows:

[0104]

[0105] Where, δ t Let L be the current orientation angle of the environmental vehicle, L be the wheelbase of the environmental vehicle (i.e., the width of the vehicle), and α be the angle between the environmental vehicle and the target point.

[0106] Vertical control is performed independently by the IDM model, as shown in the following formula:

[0107] v final =v safe (18)

[0108] Agent vehicle V a The state is updated according to the dynamic equations, as follows:

[0109]

[0110] v t+1 =v t +Δv t (twenty two)

[0111] Where, x t+1 ,y t+1 , v t+1 This indicates the state of the agent vehicle at time t+1.

[0112] Furthermore, the reward function design in this application is as follows:

[0113] The reward function is divided into three categories: regular rewards, collision penalties, and completion rewards. Regular rewards include concession rewards, approach rewards, driving smoothness penalties, and forward rewards.

[0114] Concession Reward: This concession reward is used to learn high-risk cutting behaviors that aggressive drivers may perform if the ambient vehicle is forced to stop by the agent vehicle (ambient vehicle speed). If the vehicle's speed falls below a threshold or it is forced to brake suddenly (when the ambient vehicle acceleration is less than the acceleration threshold), the concession reward increases. The concession reward R is calculated based on a preset concession reward function. brake The calculation method is as follows:

[0115]

[0116] in, The environment and vehicle V at time t e The acceleration of the vehicle in the environment, v represents the acceleration of the environment. e t represents the current training epoch, and the hyperparameter a threshold It is the acceleration threshold for a vehicle forced to brake suddenly in an environmental test, the hyperparameter λ. b and λ s Control the amount of rewards, The environment and vehicle V at time t e Speed, hyperparameter v threshold It is the speed threshold for detecting when a vehicle is forced to stop in the environment.

[0117] Approach Reward: The approach reward is used to learn behaviors that aggressive drivers might exhibit, such as failing to maintain a safe following distance. The closer the agent vehicle is to other vehicles in the environment, the higher the approach reward. Although an aggressive driver might engage in driving behaviors that could cause trouble for other vehicles, they will not do actions they perceive as dangerous to themselves; therefore, a penalty is imposed if the agent vehicle causes a collision. To prevent the agent vehicle from intentionally causing collisions in pursuit of high approach rewards, the approach reward is reset to zero upon a collision. Considering both the distance traveled and the approach reward generated by the collision, the approach reward R is calculated based on a preset approach reward function. close The calculation method is as follows:

[0118]

[0119] In equation (24), The environment and vehicle V at time t e With agent vehicle V aThe two-dimensional distance, hyperparameter ∈ close R is a sensitive factor used to control the degree to which the distance between the agent vehicle and environmental vehicles affects the reward. close It indicates that the reward is imminent.

[0120] Driving Smoothness Penalty: This penalty term ensures the rationality of the aggressive driver model and the reasonable predictability of the generated trajectory, making driving behavior more human-like. While aggressive drivers may exhibit reckless driving behavior, their driving is not entirely chaotic. This avoids unnecessary sudden braking or acceleration by the agent vehicle to deliberately create risks, while limiting the agent vehicle's steering angle changes within a reasonable range. The less smooth the agent vehicle's driving, the higher the penalty. The driving smoothness penalty is calculated based on a preset driving smoothness penalty function.

[0121] R smooth The calculation method is as follows:

[0122] j a,abs =|a t -a t-1 | (25)

[0123]

[0124] Wherein, the hyperparameter η j and j represents the range of control over the penalty. a,abs This represents the absolute rate of change of acceleration of the vehicle being transported. The hyperparameter j represents the absolute rate of change of the steering angle of the agent vehicle. a,threshold and These represent the acceleration change threshold and angular velocity change threshold of the proxy vehicle, respectively.

[0125] Forward Reward: The forward reward is used to ensure that the agent vehicle gradually approaches the finish line and completes the driving task. If the agent vehicle is closer to the finish line in the current time step than in the previous time step, it receives a reward; otherwise, it receives a penalty. The forward reward R is calculated based on a preset forward reward function. forward The calculation method is as follows:

[0126]

[0127] Where, λ f and λ b To control the reward magnitude using hyperparameters for rewards and penalties, d t-1 d represents the distance of the agent vehicle from the destination in the previous time step. t This represents the distance between the agent vehicle and the destination at the current time step.

[0128] Driving task completion reward: At the current time step, if the agent vehicle reaches the destination, it will not receive the regular reward for that time step, but will instead receive the driving task completion reward. The closer the agent vehicle's arrival time is to the actual trajectory, the higher the reward. If it exceeds the maximum allowed time, a penalty will be imposed according to the degree of time exceeded. The driving task completion reward R is calculated based on a preset driving task completion reward function. finish The calculation method is as follows:

[0129]

[0130] In equation (29), the hyperparameters ∈ finish Used to prevent the denominator from being 0, T ideal To complete the driving task in the ideal time, T actual The hyperparameter P represents the actual time required to complete the driving task. time To control the intensity of the timeout penalty, T max For the maximum allowed time, T ideal T max Hyperparameter P time and ∈ finish It can be set according to actual needs.

[0131] Collision Events and Collision Penalties: When the agent vehicle collides with an environment vehicle or lane boundary, the training round ends, and the agent does not receive the regular reward for that time step. Instead, it receives a collision penalty, calculated using a pre-defined collision penalty function. crash The calculation method is as follows:

[0132] R crash =r crash (30)

[0133] In equation (30), the hyperparameter r crash This is a fixed penalty value for collisions.

[0134] Preset final reward function R t The calculation method is as follows:

[0135]

[0136] Where ω1, ω2, ω3, and ω4 are hyperparameters controlling the reward weights, which can be set according to actual conditions. R smooth Indicates a penalty for poor driving smoothness, R close Indicates the imminent reward, R brake R represents a concession reward. forward This indicates a reward for progress.

[0137] Based on the above optimizations and settings for PPO, a PPO-MAE training and saving of the radical driver model is formed.

[0138] See Figure 4 Step 4 specifically includes the following steps:

[0139] Each obtained multi-agent complex safety scenario will serve as the basis for generating dangerous scenarios. Based on the complete multi-agent complex safety scenario generated in step 2, one multi-agent complex safety scenario is selected. In the selected multi-agent complex safety scenario, a vehicle that has not been selected before is chosen as the master agent, and the start and end points of the selected master agent's motion trajectory (i.e., the complete trajectory after splicing), as well as the vehicle's length and width, are recorded. The motion data includes the start and end points of the master agent's motion trajectory and the vehicle's length and width. The start and end points of the selected master agent are set as the start and end points of the aforementioned aggressive driver model. The lane information of this multi-agent complex safety scenario and the motion data of other vehicles in the environment besides the master agent are used as the environment of the aggressive driver model. The aggressive driver model is used to complete the driving task in this environment, and the motion state of each frame in the aggressive driver model is recorded, i.e., the generated dangerous trajectory. The generated dangerous trajectory replaces the original trajectory of the selected agent (i.e., replaces the safe trajectory of the master agent in the multi-agent complex safety scenario), resulting in a multi-agent complex dangerous scenario. If there are still agents that have not been selected as the master agent, the steps of recording the start and end points of the selected master vehicle's trajectory, as well as the vehicle's length and width, will be executed to generate the corresponding multi-agent complex dangerous scenario.

[0140] In summary, the above solutions can address the difficulty in modeling the interactions between different traffic participants in complex scenarios, and overcome the problems associated with using a single method in generating dangerous scenarios. These problems include: 1. Insufficient amount of dangerous driving data required for training in deep learning. 2. Extraction from natural driving data: The proportion of dangerous scenarios in real-world driving is too small, resulting in low scene coverage and extraction efficiency. 3. Algorithm optimization: For complex scenarios with multiple traffic participants, the dimensionality is too high, leading to excessive computational complexity. Therefore, this application not only increases the number of training samples, making the model more accurate, but also increases the number of dangerous scenario samples, improving the efficiency of dangerous scenario extraction, and reduces the computational complexity of the algorithm when the dimensionality is too high.

[0141] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for generating complex and dangerous multi-agent scenarios based on adversarial trajectories, characterized in that, Includes the following steps: Step 1: Obtain the natural driving dataset; Step 2: Determine the initial state of the scene based on the natural driving dataset, predict the corresponding scene prediction state based on the initial state of the scene using a pre-trained multi-agent scene generation model, and concatenate the initial state of the scene and the scene prediction state to obtain a multi-agent complex safety scene. Step 3: Extract real multi-agent scenarios based on the natural driving dataset and the multi-agent complex safety scenario, and build an aggressive driver model for training based on the preset PPO-MAE algorithm in the real multi-agent scenario; Step 4: Based on the radical driver model, generate adversarial trajectories in the real multi-agent scenario to generate a complex and dangerous multi-agent scenario.

2. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 1, characterized in that, The natural driving dataset includes multiple agents and a corresponding agent ID for each agent. Step 2 includes: Based on the order of agent IDs, a master agent and a corresponding environmental agent are selected using the natural driving dataset and preset agent selection principles. The master agent and the corresponding environmental agent constitute an agent combination. Based on each of the intelligent agent combinations, the data of the first preset number of frames in the natural driving dataset is preprocessed to obtain the Pytroch tensor corresponding to each group of intelligent agents as the initial state of the scene. All the initial states of the scene are sequentially input into a pre-trained multi-agent scene generation model, and the motion trajectories of all agents in each initial state of the scene are sequentially output as the scene prediction state in the subsequent second preset frame number. By concatenating the initial state of each scenario with the corresponding predicted state of the scenario, a corresponding multi-agent complex security scenario is obtained.

3. The method for generating complex dangerous scenarios using multi-agent systems based on adversarial trajectories according to claim 1, characterized in that, Step 3 includes: Based on the agent selection principle and the natural driving dataset, real multi-agent scenarios of various agent combinations are extracted from complex safety scenarios involving multiple agents. Based on the PPO algorithm, the cropping range is adaptively optimized for the adversarial trajectory generation task, and the advantage function is optimized using a self-attention mechanism to form the PPO-MAE algorithm. The radical driver model constructed in the real multi-agent scenario is trained using the PPO-MAE algorithm.

4. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 3, characterized in that, The adaptive optimization of the clipping range for the adversarial trajectory generation task includes: Adaptive cropping is achieved by combining KL divergence and training steps, as shown in the following formula: Where ∈0 is the initial clipping range (usually 0.2), and the hyperparameter δ target Let KL divergence be the target. The KL divergence is updated for the current policy. The hyperparameter η prevents the denominator from being 0. t is the number of training epochs. λ is a hyperparameter that controls the decay rate of the pruning range.

5. The method for generating complex dangerous scenarios using multi-agent systems based on adversarial trajectories according to claim 3, characterized in that, The optimization of the advantage function using a self-attention mechanism includes: Transform a pre-built input sequence into an embedded vector sequence; Based on the embedded vector sequence, a context-aware representation is generated through self-attention computation, and the output of self-attention is mapped to a scalar value function estimate through a multilayer perceptron. The adjusted time difference error is calculated based on the value function estimation, and the advantage function is calculated based on the adjusted time difference error.

6. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 5, characterized in that, The input sequence includes a reward tuple, and the reward function design for the reward tuple includes a regular reward, a collision penalty, and a completion reward. The regular reward includes a concession reward, a proximity reward, a driving smoothness penalty, and a forward reward, including: During the operation of the agent vehicle, the corresponding concession reward, the approach reward, the driving smoothness penalty and the forward reward will be calculated based on the preset concession reward function, the preset approach reward function, the preset driving smoothness penalty function and the preset forward reward function. When the agent vehicle reaches the destination, the driving task completion reward is calculated according to the preset driving task completion reward function; When the agent vehicle collides with environmental vehicles or lane boundaries, the corresponding collision penalty is calculated using a preset collision penalty function. The reward tuple is calculated based on the regular reward, the collision penalty, and the completion reward using a preset final reward function.

7. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 3, characterized in that, The training of the radical driver model includes: Based on the PPO-MAE algorithm, the aggressive driver model constructed in the real multi-agent scenario is configured accordingly. The configuration includes training environment settings, trajectory generation settings, and reward function design. The radical driver model was trained using the PPO-MAE algorithm.

8. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 7, characterized in that, The training environment settings include: The two-dimensional coordinate range of the lane, which is defined by lane lines and lane boundaries, and the environment, which consists of environmental vehicles traveling along the complete trajectory and agent vehicles; From velocity v t Orientation Angle Two-dimensional coordinate position (x) t ,y t The longitudinal dimension l and the lateral dimension w of the proxy vehicle constitute the state space S: Due to angle change and velocity change Δv t Forming action space A:

9. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 1, characterized in that, Step 4 includes: Each of the multi-agent complex security scenarios is used as the basis for generating dangerous scenarios. In the selected multi-agent complex security scenario, an agent that has not been selected before is selected as the master agent, and the corresponding motion data of the master agent is recorded. Based on the motion data, the aggressive driver model generates a dangerous trajectory, and the dangerous trajectory replaces the complete trajectory of the multi-agent complex safety scenario to obtain the multi-agent complex dangerous scenario.

10. The method for generating complex dangerous scenarios based on adversarial trajectories using multiple agents according to claim 9, characterized in that, The motion data includes the length and width of the main agent, the start and end points of the corresponding complete trajectory, and the generation of a dangerous trajectory using the aggressive driver model based on the motion data includes: Based on the starting and ending points of the main vehicle, an aggressive driver model is used to generate dangerous trajectories. The starting point and the ending point are set as the starting point and the ending point of the radical driver model, and the lane information of the complex safety scenario of the multi-agent and the motion data of the environmental vehicles corresponding to the main agent are used as the environment of the radical driver model. The aggressive driver model is used to complete driving tasks in the environment and its state is recorded for each frame to generate corresponding dangerous trajectories.

Images