Multi-agent adversarial testing method for autonomous driving systems based on large language models

Abstract

This invention belongs to the field of autonomous driving testing technology, and specifically relates to a multi-agent adversarial testing method for autonomous driving systems based on a large language model. This invention involves localizing and utilizing a pre-trained large language model, and fine-tuning the large language model based on domain knowledge. A basic adversarial testing environment is established, and the adversarial behavior of vehicles is modeled using multi-agent reinforcement learning methods, with a reward function set. The fine-tuned large language model is integrated into the adversarial testing process, using the global state of the scene at each time step as input. Semantic reasoning is used to identify vehicles capable of executing efficient adversarial behaviors, which are then labeled and executed. The results of this invention enable efficient testing of the performance of autonomous driving systems, rapidly uncovering system functional defects, and accelerating the industrialization of autonomous vehicles.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving testing technology, and specifically relates to a multi-agent adversarial testing method for autonomous driving systems based on a large language model. Background Technology

[0002] With the rapid development of autonomous vehicles, significant progress has been made in many core technologies such as perception, decision-making, and control, and the ability to cope with complex scenarios has been significantly enhanced. However, driving scenarios in the real world are endless, and thorough and reliable testing is a necessary prerequisite for the industrialization of autonomous vehicles. Adversarial testing is an important means of testing autonomous driving systems. By modeling the adversarial behavior of vehicles, the functional defects of autonomous driving systems can be fully discovered.

[0003] Reinforcement learning is a common method for modeling adversarial behavior of vehicles. By modeling vehicle agents and setting reward functions, efficient testing of autonomous driving systems can be achieved. However, existing methods mostly use reinforcement learning to independently model vehicles to explore the environment. The lack of cooperative adversarial strategies among vehicles limits the efficiency and scenario coverage of the adversarial testing process. Multi-agent reinforcement learning is a potential method to improve the above problems, but the strong randomness and poor policy convergence of multi-agent reinforcement learning exploration limit its application in adversarial testing of autonomous driving systems, making it difficult to efficiently discover functional defects of autonomous driving systems. Large language models have powerful global semantic understanding and reasoning capabilities, which can guide the exploration direction of multi-agent reinforcement learning and achieve efficient adversarial testing of autonomous driving systems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-agent adversarial testing method for autonomous driving systems based on a large language model. The method involves localizing and utilizing a pre-trained large language model, fine-tuning the model based on domain knowledge, establishing a basic adversarial testing environment, modeling the adversarial behavior of vehicles using multi-agent reinforcement learning, and setting a reward function. The fine-tuned large language model is integrated into the adversarial testing process, using the global state of the scene at each time step as input. Semantic reasoning is used to identify vehicles capable of executing efficient adversarial behaviors, which are then labeled and executed.

[0005] This invention provides a multi-agent adversarial testing method for autonomous driving systems based on a large language model, comprising the following steps:

[0006] Step 1: Deploy and invoke the large language model locally, and fine-tune it based on domain knowledge;

[0007] Based on the selection of a large language model, the localized deployment and invocation of the large language model are realized on the Ollam architecture, and a knowledge base for adversarial testing of autonomous driving systems is established. "Command-response" training sample pairs are constructed to realize the fine-tuning process of the large language model in professional fields.

[0008] Step 2: Set up the basic environment for adversarial testing and construct a multi-agent reinforcement learning algorithm;

[0009] Construct an adversarial test environment that includes a master vehicle equipped with an autonomous driving system and several other vehicles, and set the motion behavior patterns of the master vehicle and the other vehicles; define collision time and hazard metric in the test environment, and fuse the collision time and hazard metric as the risk metric score of the scenario; construct a multi-agent reinforcement learning algorithm, and construct a reward function that includes four parts: fused risk metric, smoothing term, rule soft constraint term, and cooperative consistency term, to guide the training and convergence of the algorithm and achieve the adversarial test objective;

[0010] Step 3: Integrate the large language model into the testing process to achieve efficient closed-loop adversarial testing of the system under test;

[0011] The global scene information at each time step during the adversarial test is integrated into the prompts of the large language model. The large language model is used to identify and label the set of vehicles that can perform efficient adversarial behavior, and multi-agent cooperative adversarial strategies are executed based on the labeled vehicles.

[0012] Furthermore, the method for step one is as follows:

[0013] 11) Localize the deployment and invocation of large language models;

[0014] Choose a large language model and deploy the large language model locally based on the Ollam large language model deployment architecture;

[0015] 12) Fine-tuning of large language models based on domain knowledge;

[0016] The fine-tuning of the large language model includes two parts: domain knowledge construction and lightweight parameter adaptation training. For domain knowledge construction, relevant data from adversarial tests of autonomous driving systems are collected, the information from the relevant data sources is cleaned, and three types of "instruction-response" training sample pairs are constructed based on this:

[0017] ① Identification type: Given global information of the scene, the model is required to identify "the vehicle with the greatest risk of causing confrontation and give a reason";

[0018] ② Ranking type: Given multiple candidate vehicles, the model is required to output a list and explanation of the vehicles ranked from highest to lowest risk.

[0019] ③ Explanation type: For known dangerous events, the model is required to use natural language to explain the "triggering causes";

[0020] By assigning high weights to adversarial examples while retaining a certain number of negative examples, the data is deduplicated and segmented to form training and validation sets. The model is then trained using lightweight parameter adaptation based on these sets.

[0021] Furthermore, the lightweight parameter adaptation training only trains a subset of incremental parameters, and the training objective employs cross-entropy and norm regularization, as shown below:

[0022]

[0023] In the formula, L is the training objective function, x i For the i-th instruction, y i The corresponding expected response, where N represents the number of training sample pairs, CE represents the cross-entropy loss, and w i λ represents the sample weights, Δθ is the trainable incremental weight tensor, and λ is the regularization coefficient. This represents the model after injecting lightweight incremental parameters.

[0024] Furthermore, the method for step two is as follows:

[0025] 21) Set up the basic environment for adversarial testing;

[0026] Construct a basic environment for adversarial testing that includes a master vehicle equipped with an autonomous driving system and several other vehicles. Model the problem as a Markov decision process. The master vehicle outputs driving behavior from the autonomous driving system under test, and the motion of the other vehicles adopts a discretized nonholonomic constrained motion model.

[0027] As a preferred embodiment, the vehicle motion model is shown below, with lateral acceleration constrained:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] In the formula, (x,y) represents the vehicle position, and t represents the time step. Let v be the heading angle. t For the longitudinal speed of the vehicle, For the front wheel steering angle, a t Let l be the longitudinal acceleration, l be the vehicle wheelbase, and a be the longitudinal acceleration. y,t Let Δt be the lateral acceleration and Δt be the time step of the adversarial test. This represents the maximum lateral acceleration.

[0034] This invention defines collision time (TTC) and hazard metric in a test environment, and integrates the two as a risk metric score for the scenario.

[0035] As a preferred method, the risk measurement score is calculated as follows:

[0036] i represents the vehicle index, ego represents the primary vehicle index, and p (i) p represents the location of the shuttle bus. ego Represents the location of the main vehicle, v (i) v represents the speed of the vehicle. ego The speed of the main vehicle is represented by r=p, and the relative positions of the two vehicles are r=p. (i) -p ego Relative velocity Δv=v (i) -v ego The unit vector between the two is n = r / ||r||2, and the speed at which they converge along the unit vector n is S. close =max(0,-<Δv,n>), introducing the distance buffer constant b>0 and the stability term ε>0, improves the numerical stability of the collision time TTC, resulting in the following TTC calculation method:

[0037]

[0038] A new hazard measurement method coupling distance, vehicle orientation, and speed is constructed, as shown below:

[0039]

[0040] In the formula, h is the coupled hazard metric, and the relative heading angle is... , For the heading angle of the vehicle, Main vehicle heading angle, distance scale >0, velocity normalization constant v ref >0, the fusion risk metric score R is shown below:

[0041]

[0042] In the formula, α is the balance coefficient between TTC and hazard measurement, which is between 0 and 1, the adjustment coefficient τ>0, and h is the coupled hazard measurement index.

[0043] 22) Construct a multi-agent reinforcement learning algorithm;

[0044] A multi-agent reinforcement learning algorithm is constructed to model the motion behavior of a transportation vehicle, and the problem is modeled as a partially observable Markov game, as shown below:

[0045]

[0046] In the formula, G is a partially observable Markov game, I is the set of agents, S is the global state space, and O is the global state space. i Let F be the observation space of the i-th agent. i Let T be the action space, T be the state transition, and r be the state transition. i Represents immediate reward, γ is the discount factor;

[0047] Define global scene information s t Define vehicle status The formula is shown below:

[0048]

[0049]

[0050] In the formula, M represents the lane map. Let g(s) be a semaphore, J be the set of vehicles, and g(s) be a semaphor t ) represents the constrained residuals, where p, v, ψ, and ω represent the vehicle's position, velocity, heading angle, and heading angular velocity, respectively. The subscript t represents time t, and the superscript ω represents the vehicle's position, velocity, heading angle, and heading angular velocity. Local observations are performed using a block aggregation method, as shown below:

[0051]

[0052] In the formula, This is a local observation of the i-th vehicle at time t. For vehicle movement and lane departure, The relative quantity of the nearest K neighboring cars. For forward curvature and velocity limiting sampling, For signal, To quantify risk characteristics, This is semantic prior information;

[0053] Constrain the vehicle's motion with the feasible region, and define the longitudinal acceleration of the i-th vehicle. Constraints The turning angle of the front wheel of the i-th vehicle Constraints , , , , These are the minimum acceleration, maximum acceleration, minimum front wheel steering angle, and maximum front wheel steering angle, respectively. Upper and lower bounds are assigned values ​​based on vehicle dynamics and comfort indicators. The multi-agent reinforcement learning algorithm strategy employs Gaussian reparameterization, as shown below:

[0054]

[0055] In the formula, Let represent the random action vector generated by the i-th agent at time t, μ and σ be the mean and standard deviation of the policy output, and η be a standard normal distribution. As an element-wise multiplication, this expression produces differentiable random actions, using the formula:

[0056]

[0057]

[0058] Compress the motion to the range [-1, 1], and apply the formula:

[0059]

[0060]

[0061] The normalization result is mapped to the physical execution space; where, , This represents two-channel sampling.

[0062] This invention employs a multi-agent framework with centralized training and distributed execution. The objectives of the critic and the actor are as follows:

[0063]

[0064]

[0065]

[0066] In the formula, For the critics' loss, Q w Let s be the value function. t For global scene information, f t For joint action, y t Let γ represent the temporal difference objective, which is an approximate objective used to train the commentator: "the current reward plus the discounted value of the next step," where γ is the discount factor and r is the approximation. t Represents an instant reward. This represents a local observation of the i-th vehicle at time t+1. For the target critic parameter, λ H Temperature is the entropy. Representing the joint action in the next moment, This represents the policy distribution of the i-th agent. For the i-th policy objective, For the two-dimensional actions of this intelligent agent, For splicing together the actions of the other intelligent agents, The parameters represent the policy network of the i-th agent. Represents the mean operator. For the local observation of the i-th vehicle at time t, s t Represents the global state;

[0067] 23) Set the reward function;

[0068] The algorithm training process is guided by setting a reward function, which includes four parts: risk metric, smoothing, rule soft constraint, and consensus. The real-time reward and cumulative total target of the vehicle are calculated. The master vehicle is the autonomous driving algorithm under test and does not participate in the training process.

[0069] Furthermore, the risk metric for fusion is given by the following formula when setting up the adversarial testing environment:

[0070]

[0071] In the formula, R is the fusion risk metric score, α is the balance coefficient between collision time and hazard metric, the adjustment coefficient τ>0, and h is the coupled hazard metric index.

[0072] The longitudinal acceleration rate of change is introduced as a smoothing term, as shown below:

[0073]

[0074] In the formula, Represents the rate of change of longitudinal acceleration. Let be the longitudinal acceleration of the i-th vehicle. For time step;

[0075] Traffic rules and feasible regions are introduced as soft constraints in the rules, as shown below:

[0076]

[0077] In the formula, B(g) is the value of the soft constraint term, c represents the constraint term, and g c Represents the residual under constraint c. As the slope coefficient, constraint term c is defined to include the following items: ① Vehicles must not exceed the vehicle dynamic limits; ② Vehicles must not leave the road boundary; ③ Vehicles must not overlap.

[0078] The following are the consensus items:

[0079]

[0080]

[0081] In the formula, m t Its mean, Representing the quantitative value of the vehicle coordination index, set according to the cooperative confrontation target, this invention is designed based on the risk created by the master vehicle during synchronization. A t K represents the set of vehicles labeled for a large model that can perform efficient adversarial behavior, where K is the number of vehicles. t For values ​​of coordinated and consistent items.

[0082] Based on the four indicators mentioned above—integrated risk measurement, smoothing, rule soft constraint, and coordination consistency—a real-time reward system for transportation vehicles is constructed. As shown below:

[0083]

[0084] In the formula, Let λ represent the executable physics commands for the i-th combat vehicle's adv, and λ1, λ2, λ3, and λ4 represent the weighting coefficients of each reward. This represents the risk metric for the i-th agent at time t. This represents the soft constraint value of the rule;

[0085] Real-time rewards for shuttle buses Participating in the temporal difference objective y t Iterative updates;

[0086] The cumulative total reward target for the shuttle bus is calculated as follows:

[0087]

[0088] In the formula, J adv This represents the cumulative reward for discounts within a round. This represents the discount factor.

[0089] Furthermore, the specific method for step three is as follows:

[0090] 31) Utilize a large language model to reason about global scene information, identify vehicles capable of performing efficient adversarial behaviors, and label them;

[0091] During adversarial testing, global scene information at each adversarial time step is incorporated into the prompts of the large language model. The large language model then analyzes the current scene information to identify the optimal set of adversarial vehicles, as shown below:

[0092]

[0093] In the formula, s t This represents global scene information, where k represents the vehicle identification number. The large model score represents the rating of the k-th vehicle at time t, with a value between 0 and 1, used to judge... Is it greater than the threshold τ?sem If the value is greater than the given value, then the "efficient adversarial" label is assigned, resulting in a set A of vehicles capable of performing efficient adversarial actions. t The semantic weight of the k-th vehicle is calculated as follows:

[0094]

[0095] In the formula, This represents the semantic weight of the k-th vehicle, with a value between 0 and 1. For numerically stable terms, >0;

[0096] 32) Based on the labeled vehicles, implement a multi-agent cooperative adversarial strategy;

[0097] During the adversarial testing, only the set of vehicles A was tested. t The vehicle calculation consensus term is used, and semantic weights are used. Weighting is applied, and risk-driven factors are amplified according to their weights to form a collaborative adversarial control strategy for the labeled vehicle set, as shown below:

[0098]

[0099] In the formula, For coordination indicators The weighted mean;

[0100] The weighted consensus term is calculated as follows:

[0101]

[0102] In the formula, For weighted coordination and consistency items, with Sync t Unlike other methods, it represents semantically weighted collaborative consistency terms, which are conducive to forming more aggressive collaborative adversarial strategies and obtaining real-time rewards for labeled traffic vehicles. As shown below:

[0103]

[0104] In the formula, Let λ represent the executable physics commands for the i-th combat vehicle's adv, and λ1, λ2, λ3, and λ4 represent the weighting coefficients of each reward. This represents the risk metric for the i-th agent at time t. Represents the rate of change of longitudinal acceleration;

[0105] The real-time reward update time-series differential objective y of the labeled vehicles obtained in this way t This enables a multi-agent adversarial testing process guided by a large model.

[0106] The beneficial effects of this invention are as follows:

[0107] This invention provides a multi-agent adversarial testing method for autonomous driving systems based on a large language model. It involves localizing and utilizing a pre-trained large language model, and fine-tuning the model based on domain knowledge. A basic adversarial testing environment is established, and multi-agent reinforcement learning is used to model the adversarial behavior of vehicles, with a reward function set to address the lack of collaborative adversarial strategies among vehicles. The fine-tuned large language model is integrated into the adversarial testing process. Using the global state of the scene at each time step as input, semantic reasoning identifies vehicles capable of executing efficient adversarial behaviors, which are then labeled and executed, improving the efficiency and scene coverage of the adversarial testing process. This overcomes the problems of strong randomness and poor policy convergence in multi-agent reinforcement learning exploration, promoting the application of this method in adversarial testing of autonomous driving systems and enabling efficient discovery of functional defects in autonomous driving systems. The large language model possesses powerful global semantic understanding and reasoning capabilities, guiding the exploration direction of multi-agent reinforcement learning, enabling efficient adversarial testing of autonomous driving systems, improving adversarial testing efficiency, quickly discovering system functional defects, and accelerating the industrialization of autonomous vehicles. Attached Figure Description

[0108] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0109] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0110] Figure 2 This is a schematic diagram of the multi-agent reinforcement learning algorithm structure of the present invention;

[0111] Figure 3 This is a schematic diagram of the centralized training of the multi-agent reinforcement learning algorithm of the present invention;

[0112] Figure 4 This is a schematic diagram illustrating the distributed execution of the multi-agent reinforcement learning algorithm of the present invention;

[0113] Figure 5 The diagram shows the results of generating adversarial scenarios according to the present invention; (a), (b), (c), and (d) are adversarial scenario generation results 1-4, respectively. Detailed Implementation

[0114] Example 1

[0115] See Figure 1This embodiment provides a multi-agent adversarial testing method for autonomous driving systems based on a large language model, comprising the following steps:

[0116] Step 1: Deploy and invoke the large language model locally, and fine-tune it based on domain knowledge;

[0117] Based on the selection of a large language model, the localized deployment and invocation of the large language model are realized on the Ollam architecture, and a knowledge base for adversarial testing of autonomous driving systems is established. "Command-response" training sample pairs are constructed to realize the fine-tuning process of the large language model in professional fields.

[0118] Furthermore, the specific method for step one is as follows:

[0119] 11) Localize the deployment and invocation of large language models;

[0120] This invention selects the large language model Qwen-7B-Chat and deploys the large language model locally based on the Ollam large language model deployment architecture.

[0121] Specifically, the GPU driver and dependencies are installed on the local host, Ollam is run to obtain the Qwen-7B-Chat large language model package from the model repository, the weights are downloaded and registered locally, the local inference service is started and the model running status is detected, and then the model is called, prompt words are input and the model inference is implemented based on Python scripts.

[0122] 12) Fine-tuning of large language models based on domain knowledge;

[0123] The fine-tuning of the large language model includes two parts: domain knowledge construction and lightweight parameter adaptation training. For domain knowledge construction, relevant data on adversarial testing of autonomous driving systems are collected, including adversarial scenario descriptions from published papers and reports, traffic rules and right-of-way statements, and known dangerous interaction fragments in adversarial logs. The above-mentioned information is cleaned and used to construct three types of "instruction-response" training sample pairs:

[0124] ① Identification type: Given global information of the scene, the model is required to identify "the vehicle with the greatest risk of causing confrontation and give a reason";

[0125] ② Ranking type: Given multiple candidate vehicles, the model is required to output a list and explanation of the vehicles ranked from highest to lowest risk.

[0126] ③ Explanation type: For known dangerous events, the model is required to use natural language to explain the "triggering causes";

[0127] To highlight the importance of key adversarial examples, adversarial examples are given higher weights, while a certain number of negative samples are retained to avoid insufficient model learning. This invention deduplicates and segments the data to form training and validation sets, and uses these as the basis for lightweight parameter adaptation training of the model. For lightweight parameter adaptation training, this invention only trains a small number of incremental parameters, reducing memory usage and catastrophic forgetting. The training objective adopts cross-entropy and norm regularization, as shown in Equation (1):

[0128] (1)

[0129] In the formula, L is the training objective function, x i For the i-th instruction, y i The corresponding expected response, where N represents the number of training sample pairs, CE represents the cross-entropy loss, and w i Δθ represents the weight of the adversarial examples, where higher weights are assigned to them; Δθ is the trainable incremental weight tensor; and λ is the regularization coefficient. This represents the model after injecting lightweight incremental parameters.

[0130] Step 2: Set up the basic environment for adversarial testing and construct a multi-agent reinforcement learning algorithm;

[0131] A multi-agent reinforcement learning adversarial testing environment was built, the motion behavior patterns of the master vehicle and the transport vehicle were set, and a multi-agent reinforcement learning algorithm was designed. See the algorithm structure diagram below. Figure 2 A reward function is constructed that includes four parts: a risk metric, a smoothing term, a rule-based soft constraint term, and a collaborative consistency term. This function guides the algorithm's training and convergence, achieving the goal of adversarial testing.

[0132] Furthermore, the specific method for step two is as follows:

[0133] 21) Set up the basic environment for adversarial testing;

[0134] A basic test environment for adversarial testing is constructed, consisting of a host vehicle equipped with an autonomous driving system and several other vehicles. The problem is modeled as a Markov decision process. The host vehicle's driving behavior is output by the autonomous driving system under test. The motion of the other vehicles adopts a discretized nonholonomic constrained motion model, as shown in equations (2)-(5). Equation (6) is used to constrain the lateral acceleration to ensure feasibility.

[0135] (2)

[0136] (3)

[0137] (4)

[0138] (5)

[0139] (6)

[0140] In the formula, (x,y) represents the vehicle position, and t represents the time step. Let v be the heading angle. t For the longitudinal speed of the vehicle, For the front wheel steering angle, a t Let l be the longitudinal acceleration, l be the vehicle wheelbase, and a be the longitudinal acceleration. y,t Let Δt be the lateral acceleration and Δt be the time step of the adversarial test. This represents the maximum lateral acceleration.

[0141] This invention defines time-to-collision (TTC) and hazard measurement in a test environment, and merges the two as a risk measurement score for the scene, as follows:

[0142] i represents the vehicle index, ego represents the primary vehicle index, and p (i) p represents the location of the shuttle bus. ego Represents the location of the main vehicle, v (i) v represents the speed of the vehicle. ego r represents the speed of the main vehicle, and r represents the relative position of the two vehicles, r=p (i) -p ego Relative velocity Δv=v (i) -v ego The unit vector between the two is n = r / ||r||2, and the speed at which they converge along the unit vector n is S. close =max(0,-<Δv,n>), to improve the numerical stability of collision time TTC, a distance buffer constant b>0 and a stability term ε>0 are introduced, and the TTC calculation method is obtained as shown in equation (7):

[0143] (7)

[0144] Equation (7) in S close →0 approaches the maximum value, when S close As the TTC increases, it monotonically decreases, which can intuitively reflect the trend of "how long it will take to cause a collision at the current speed". However, time scale indicators alone are not enough to distinguish different confrontation situations. This invention further constructs a hazard measurement method that couples the three elements of distance, vehicle orientation and speed, as shown in Equation (8):

[0145] (8)

[0146] In the formula, h is the coupled hazard metric, and the relative heading angle is... , For the heading angle of the vehicle, Main vehicle heading angle, distance scale >0, velocity normalization constant v ref >0, and thus the fusion risk metric score R is defined as shown in equation (9):

[0147] (9)

[0148] In the formula, α is the balance coefficient between TTC and hazard measurement, which is between 0 and 1, and the adjustment coefficient τ>0;

[0149] 22) Construct a multi-agent reinforcement learning algorithm;

[0150] This invention constructs a multi-agent reinforcement learning algorithm to model the motion behavior of a traffic vehicle, and models the problem as a partially observable Markov game, as shown in equation (10):

[0151] (10)

[0152] In the formula, G is a partially observable Markov game, I is the set of agents, S is the global state space, and O is the global state space. i Let F be the observation space of the i-th agent. i Let T be the action space, T be the state transition, and r be the state transition. i Represents immediate reward, γ is the discount factor;

[0153] Define global scene information s t As shown in equation (11), the vehicle state is defined. As shown in equation (12):

[0154] (11)

[0155] (12)

[0156] In the formula, M represents the lane map. Let g(s) be a semaphore, J be the set of vehicles, and g(s) be a semaphor t To constrain the residuals, p, v, ψ, and ω represent the vehicle position, velocity, heading angle, and heading angular velocity, respectively. Local observations are aggregated in blocks, as shown in equation (13).

[0157] (13)

[0158] In the formula, This is a local observation of the i-th vehicle at time t. For vehicle movement and lane departure, The relative quantity of the nearest K neighboring cars. For forward curvature and velocity limiting sampling, For signal, To quantify risk characteristics, This is semantic prior information;

[0159] This invention constrains the motion of the vehicles with the feasible region, and the longitudinal acceleration of the i-th vehicle... Constraints The turning angle of the front wheel of the i-th vehicle Constraints , , , , The minimum acceleration, maximum acceleration, minimum front wheel angle, and maximum front wheel angle are respectively defined. The upper and lower bounds of the constraints are assigned values ​​based on vehicle dynamics and comfort. The multi-agent reinforcement learning algorithm strategy adopts Gaussian reparameterization, as shown in Equation (14):

[0160] (14)

[0161] In the formula, Let represent the random action vector generated by the i-th agent at time t, μ and σ be the mean and standard deviation of the policy output, and η be a standard normal distribution. For element-wise multiplication, this formula generates differentiable random actions. Equations (15) and (16) are used to compress the actions to the range [-1, 1], and the normalized results are mapped to the physical execution space according to equations (17) and (18):

[0162] (15)

[0163] (16)

[0164] (17)

[0165] (18)

[0166] In the formula, , This represents two-channel sampling, see reference. Figure 3 , Figure 4 This invention employs a multi-agent framework with centralized training and distributed execution. The goals of the critic and the actor are shown in equations (19)-(21):

[0167] (19)

[0168] (20)

[0169] (twenty one)

[0170] In the formula, For the critics' loss, Q w Let s be the value function. t For global scene information, f t For joint action, y t Let γ represent the temporal difference objective, which is an approximate objective used to train the commentator: "the current reward plus the discounted value of the next step," where γ is the discount factor and r is the approximation. t Represents an instant reward. This represents a local observation of the i-th vehicle at time t+1. For the target critic parameter, λ H Temperature is the entropy. Representing the joint action in the next moment, This represents the policy distribution of the i-th agent. For the i-th policy objective, For the two-dimensional actions of this intelligent agent, For splicing together the actions of the other intelligent agents, The parameters represent the policy network of the i-th agent. Represents the mean operator. For the local observation of the i-th vehicle at time t, s t Represents the global state;

[0171] 23) Set the reward function;

[0172] This invention guides the algorithm training process by setting a reward function. The reward function includes four parts: a risk metric, a smoothing term, a rule soft constraint term, and a cooperative consistency term. The real-time reward and cumulative total target of the vehicle are calculated. The host vehicle is the autonomous driving algorithm under test and does not participate in the training process.

[0173] For the fusion risk metric, it is given by equation (9) when setting up the adversarial testing environment; to improve controllability, the longitudinal acceleration change rate is introduced as a smoothing term, as shown in equation (22):

[0174] (twenty two)

[0175] In the formula, Represents the rate of change of longitudinal acceleration. For time step;

[0176] This invention introduces traffic rules and feasible regions as soft constraint terms, as shown in equation (23):

[0177] (twenty three)

[0178] In the formula, B(g) is the value of the soft constraint term, c represents the constraint term, and g c Represents the residual under constraint c. As the slope coefficient, the constraint term c defined in this invention includes the following items: ① The vehicle shall not exceed the vehicle dynamic limit; ② The vehicle shall not leave the road boundary; ③ The vehicles shall not overlap.

[0179] The coherent terms are shown in equations (24) and (25):

[0180] (twenty four)

[0181] (25)

[0182] In the formula, m t Its mean, Representing the quantitative value of the vehicle coordination index, set according to the cooperative confrontation target, this invention is designed based on the risk created by the master vehicle during synchronization. A t K represents the set of vehicles labeled for a large model that can perform efficient adversarial behavior, where K is the number of vehicles. t For values ​​of coordinated and consistent items;

[0183] Based on the four indicators mentioned above—integrated risk measurement, smoothing, rule soft constraint, and coordination consistency—a real-time reward system for transportation vehicles is constructed. As shown in equation (26), it participates in the time-series difference objective y of equation (20). t Iterative updates:

[0184] (26)

[0185] In the formula, Let λ represent the executable physics commands for the i-th combat vehicle's adv, and λ1, λ2, λ3, and λ4 represent the weighting coefficients of each reward. This represents the risk metric for the i-th agent at time t. Represents the rate of change of longitudinal acceleration. The value of the soft constraint of the rule, s t For global scene information, The value of the coordinated item is used; the cumulative total reward target for the transportation vehicle is calculated as shown in equation (27):

[0186] (27)

[0187] In the formula, J adv This represents the cumulative reward for discounts within a round. This represents the discount factor.

[0188] Step 3: Integrate the large language model into the testing process to achieve efficient closed-loop adversarial testing of the system under test;

[0189] The global scene information at each time step during the adversarial test is integrated into the prompts of the large language model. The large language model is used to identify and label the set of vehicles that can perform efficient adversarial behavior, and multi-agent cooperative adversarial strategies are executed based on the labeled vehicles.

[0190] Furthermore, the specific method for step three is as follows:

[0191] 31) Utilize a large language model to reason about global scene information, identify vehicles capable of performing efficient adversarial behaviors, and label them;

[0192] During the adversarial testing process, this invention integrates the global scene information of each adversarial time step into the prompt words of the large language model. The large language model analyzes the current scene information to identify the best set of adversarial vehicles, as shown in Equation (28):

[0193] (28)

[0194] In the formula, s t This represents global scene information, where k represents the vehicle identification number. The large model score represents the rating of the k-th vehicle at time t, with a value between 0 and 1, used to judge... Is it greater than the threshold τ? sem If the value is greater than the given value, then the "efficient adversarial" label is assigned, resulting in a set A of vehicles capable of performing efficient adversarial actions. t The semantic weight of the k-th vehicle is calculated using equation (29):

[0195] (29)

[0196] In the formula, This represents the semantic weight of the k-th vehicle, with a value between 0 and 1. For numerically stable terms, >0; This semantic weight can fully leverage the role of the vehicle that can perform efficient adversarial behaviors during adversarial testing;

[0197] 32) Based on the labeled vehicles, implement a multi-agent cooperative adversarial strategy;

[0198] During the adversarial testing, only the set of vehicles A was tested. t The vehicle calculation coordination term is weighted by the semantic weight in equation (29), and the risk-driven term is amplified according to the weight to form a cooperative adversarial control strategy for the labeled vehicle set, as shown in equation (30):

[0199] (30)

[0200] In the formula, For coordination indicators The weighted mean;

[0201] The weighted consensus term is calculated as shown in equation (31):

[0202] (31)

[0203] In the formula, For the weighted, coordinated consistency items, the labeled transportation vehicles receive real-time rewards. As shown in equation (32):

[0204] (32)

[0205] In the formula, λ1, λ2, λ3, and λ4 represent the executable physics commands for the i-th combat vehicle adv, and λ1, λ2, λ3, and λ4 represent the weight coefficients of each reward.

[0206] The time-series differential objective of the real-time reward update formula (20) for the labeled traffic vehicle is used to realize the multi-agent adversarial testing process guided by the large model and improve the efficiency of adversarial testing.

[0207] Example 2

[0208] This embodiment selects the large language model Qwen-7B-Chat, uses the method in Embodiment 1 to deploy the pre-trained large language model locally, and realizes automated calling based on Python commands. At the same time, it builds a basic environment for adversarial testing of autonomous driving system, constructs a multi-agent reinforcement learning algorithm, integrates the large language model into the adversarial testing process, analyzes the global state information of the scene and performs semantic reasoning, and realizes multi-agent adversarial testing of autonomous driving system guided by large language model.

[0209] To verify the effectiveness of the proposed method, an adversarial testing environment was set up with one master vehicle and three other vehicles. Reinforcement learning, multi-agent reinforcement learning, and the large language model-guided multi-agent reinforcement learning method of this invention were used to model the adversarial behavior of the vehicles. Closed-loop adversarial testing was performed. The master vehicle's driving actions were output by the autonomous driving algorithm under test. After 1000 rounds of adversarial iteration testing, the number of high-risk adversarial test scenarios generated by the autonomous driving system under test is shown in Table 1. High-risk adversarial test scenarios are those where the minimum TTC for the master vehicle is less than 0.5s. The adversarial scenario generation results are shown in [reference needed]. Figure 5 .

[0210] Table 1. Statistics on the number of high-risk adversarial test scenarios under different adversarial strategies

[0211] Countermeasures Number of high-risk adversarial test scenarios reinforcement learning 203 Multi-agent reinforcement learning 354 Multi-agent reinforcement learning guided by large language models 524

[0212] As shown in Table 1, the multi-agent reinforcement learning adversarial testing method based on a large language model constructed in this invention can effectively improve the challenges of strong randomness and poor convergence of reinforcement learning methods. Compared with reinforcement learning and multi-agent reinforcement learning methods, the number of high-risk adversarial test scenarios is greatly increased under the same test resources, which greatly accelerates the testing efficiency of the system under test and the ability to fully discover functional defects, thus verifying the effectiveness of the method proposed in this invention.

Claims

1. A method for multi-agent adversarial testing of an autonomous driving system based on a large language model, characterized in that: Includes the following steps: Step 1: Deploy and invoke the large language model locally, and fine-tune it based on domain knowledge; Based on the selection of a large language model, the localized deployment and invocation of the large language model are realized using the Ollam architecture. A knowledge base for adversarial testing of autonomous driving systems is established, and "instruction-response" training sample pairs are constructed to realize the fine-tuning process of the large language model in professional fields. Step 2: Set up the basic environment for adversarial testing and construct a multi-agent reinforcement learning algorithm; Construct a confrontation test environment that includes a master vehicle equipped with an autonomous driving system and several vehicles, and set the motion behavior modes of the master vehicle and vehicles; define collision time and hazard measurement in the test environment, and integrate collision time and hazard measurement as the risk measurement score of the scenario; A multi-agent reinforcement learning algorithm is constructed, and a reward function comprising four parts—a risk metric, a smoothing term, a rule-based soft constraint term, and a cooperative consistency term—is built to guide the algorithm's training and convergence, achieving the goal of adversarial testing. The method for step two is as follows: 21) Set up the basic environment for adversarial testing; 22) Construct a multi-agent reinforcement learning algorithm; 23) Set the reward function; The algorithm training process is guided by setting a reward function, which includes four parts: risk metric, smoothing, rule soft constraint, and consensus. The real-time reward and cumulative total target of the vehicle are calculated. The master vehicle is the autonomous driving algorithm under test and does not participate in the training process. Based on four indicators—integrating risk measurement, smoothing, soft rule constraints, and consistency—a real-time reward system for transportation vehicles is constructed. As shown below: ； In the formula, λ1, λ2, λ3, and λ4 represent the executable physics commands for the i-th combat vehicle adv, and λ1, λ2, λ3, and λ4 represent the weight coefficients of each reward. This represents the risk metric for the i-th agent at time t. Represents the rate of change of longitudinal acceleration. The value of the soft constraint of the rule, s t For global scene information, For values ​​of coordinated and consistent items; Real-time rewards for shuttle buses Participating in the temporal difference objective y t Iterative updates; The cumulative total reward target for the shuttle bus is calculated as follows: ； In the formula, J adv The cumulative return from the discount within the round, Represents the discount factor; Step 3: Integrate the large language model into the testing process to achieve efficient closed-loop adversarial testing of the system under test; The global scene information at each time step during the adversarial test is integrated into the prompts of the large language model. The large language model is used to identify and label the set of vehicles that can perform efficient adversarial behavior, and multi-agent cooperative adversarial strategies are executed based on the labeled vehicles.

2. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 1, characterized in that: The method for step one is as follows: 11) Localize the deployment and invocation of large language models; Choose a large language model and deploy the large language model locally based on the Ollam large language model deployment architecture; 12) Fine-tuning of large language models based on domain knowledge; The fine-tuning of the large language model includes two parts: domain knowledge construction and lightweight parameter adaptation training. For domain knowledge construction, relevant data from adversarial tests of autonomous driving systems are collected, the information from the relevant data sources is cleaned, and three types of "instruction-response" training sample pairs are constructed based on this: ① Identification type: Given global information of the scene, the model is required to identify "the vehicle with the greatest risk of causing confrontation and give a reason"; ② Ranking type: Given multiple candidate vehicles, the model is required to output a list and explanation of the vehicles ranked from highest to lowest risk. ③ Explanation type: For known dangerous events, the model is required to use natural language to explain the "triggering causes"; By assigning high weights to adversarial examples while retaining a certain number of negative examples, the data is deduplicated and segmented to form training and validation sets. The model is then trained using lightweight parameter adaptation based on these sets.

3. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 2, characterized in that: The lightweight parameter adaptation training only trains a subset of incremental parameters, and the training objective uses cross-entropy and norm regularization, as shown below: ； In the formula, L is the training objective function, x i For the i-th instruction, y i The corresponding expected response, where N represents the number of training sample pairs, CE represents the cross-entropy loss, and w i λ represents the sample weights, Δθ is the trainable incremental weight tensor, and λ is the regularization coefficient. This represents the model after injecting lightweight incremental parameters.

4. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 1, characterized in that: The method for step two is as follows: 21) Set up the basic environment for adversarial testing; Construct a basic environment for adversarial testing that includes a master vehicle equipped with an autonomous driving system and several other vehicles. Model the problem as a Markov decision process. The master vehicle outputs driving behavior from the autonomous driving system under test, and the motion of the other vehicles adopts a discretized nonholonomic constrained motion model. 22) Construct a multi-agent reinforcement learning algorithm; A multi-agent reinforcement learning algorithm is constructed to model the motion behavior of a transportation vehicle, and the problem is modeled as a partially observable Markov game, as shown below: ； In the formula, G is a partially observable Markov game, I is the set of agents, S is the global state space, and O is the global state space. i Let F be the observation space of the i-th agent. i Let T be the action space, T be the state transition, and r be the state transition. i Represents immediate reward, γ is the discount factor; Define global scene information s t Define vehicle status The formula is shown below: ； ； In the formula, M represents the lane map. Let g(s) be a semaphore, J be the set of vehicles, and g(s) be a semaphor t () represents the constrained residuals, where p, v, ψ, and ω represent the vehicle's position, velocity, heading angle, and heading angular velocity, respectively. The subscript t represents time t, and the superscript ω represents the vehicle's position, velocity, heading angle, and heading angular velocity. Local observations are performed using a block aggregation method, as shown below: ； In the formula, This is a local observation of the i-th vehicle at time t. For vehicle movement and lane departure, The relative quantity of the nearest K neighboring cars. For forward curvature and velocity limiting sampling, For signal, To quantify risk characteristics, This is semantic prior information; Constrain the vehicle's motion against the feasible region, and define the longitudinal acceleration of the i-th vehicle. Constraints The turning angle of the front wheel of the i-th vehicle Constraints , , , , These are the minimum acceleration, maximum acceleration, minimum front wheel steering angle, and maximum front wheel steering angle, respectively. Upper and lower bounds are assigned values ​​based on vehicle dynamics and comfort indicators. The multi-agent reinforcement learning algorithm strategy employs Gaussian reparameterization, as shown below: ； In the formula, Let represent the random action vector generated by the i-th agent at time t, μ and σ be the mean and standard deviation of the policy output, and η be a standard normal distribution. As an element-wise multiplication, this expression produces differentiable random actions, using the formula: ； ； Compress the motion to the range [-1, 1], and apply the formula: ； ； The normalization result is mapped to the physical execution space; where, , This represents two-channel sampling; Employing a multi-agent framework with centralized training and distributed execution, the goals of the critics and actors are as follows: ； ； ； In the formula, For the critics' loss, Q w Let s be the value function. t For global scene information, f t For joint action, y t Let γ represent the temporal difference objective, which is an approximate objective used to train commentators: "the current reward plus the discounted value of the next step," where γ is the discount factor and r is the approximation. t Represents an immediate reward. This represents a local observation of the i-th vehicle at time t+1. For the target critic parameter, λ H Temperature is the entropy. Representing the joint action in the next moment, Represents the policy distribution of the i-th agent. For the i-th policy objective, For the two-dimensional actions of this intelligent agent, For splicing together the actions of the other intelligent agents, The parameters represent the policy network of the i-th agent. Represents the mean operator. This is a local observation of the i-th vehicle at time t.

5. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 4, characterized in that: In step 21), the vehicle motion model is shown below, with lateral acceleration constrained: ； ； ； ； ； In the formula, (x,y) represents the vehicle position, and t represents the time step. Let v be the heading angle. t For the longitudinal speed of the vehicle, For the front wheel steering angle, a t Let l be the longitudinal acceleration, l be the vehicle wheelbase, and a be the longitudinal acceleration. y,t Let Δt be the lateral acceleration and Δt be the time step of the adversarial test. This represents the maximum lateral acceleration.

6. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 1, characterized in that: The risk measurement score is calculated as follows: i represents the vehicle index, ego represents the primary vehicle index, and p (i) p represents the location of the shuttle bus. ego Represents the position of the main vehicle, v (i) v represents the speed of the vehicle. ego The speed of the main vehicle is represented by r=p, and the relative positions of the two vehicles are r=p. (i) -p ego Relative velocity Δv=v (i) -v ego The unit vector between the two is n = r / ||r||2, and the speed at which they converge along the unit vector n is S. close =max(0,-<Δv,n>), introducing the distance buffer constant b>0 and the stability term ε>0, improves the numerical stability of the collision time TTC, resulting in the following TTC calculation method: ； A new hazard measurement method coupling distance, vehicle orientation, and speed is constructed, as shown below: ； In the formula, h is the coupled hazard metric, and the relative heading angle is... , For the heading angle of the vehicle, Main vehicle heading angle, distance scale >0, velocity normalization constant v ref >0, the fusion risk metric score R is shown below: ； In the formula, α is the balance coefficient between TTC and hazard measurement, which is between 0 and 1, and the adjustment coefficient τ>0.

7. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 4, characterized in that: In step 23), the fusion risk metric is given by the following formula when setting up the adversarial testing environment: ； In the formula, R is the fusion risk metric score, α is the balance coefficient between collision time and hazard metric, the adjustment coefficient τ>0, h is the coupled hazard metric index, and TTC is the collision time; The longitudinal acceleration rate of change is introduced as a smoothing term, as shown below: ； In the formula, Represents the rate of change of longitudinal acceleration. Let be the longitudinal acceleration of the i-th vehicle. For time step; Traffic rules and feasible regions are introduced as soft constraints in the rules, as shown below: ； In the formula, B(g) is the value of the soft constraint term, c represents the constraint term, and g c Represents the residual under constraint c. As the slope coefficient, constraint term c is defined to include the following items: ① Vehicles must not exceed the vehicle dynamic limits; ② Vehicles must not leave the road boundary; ③ Vehicles must not overlap. The following are the consensus items: ； ； In the formula, m t Its mean, Representing the quantitative value of the vehicle coordination index, it is set according to the coordination and confrontation objectives, and designed based on the risk created by the master vehicle during synchronization. A t K is the set of vehicles labeled for a large model that can perform efficient adversarial behavior, where K is its number. t For values ​​of coordinated and consistent items.

8. The method for multi-agent adversarial testing of an autonomous driving system based on a large language model according to claim 1, characterized in that: The specific method for step three is as follows: 31) Utilize a large language model to reason about global scene information, identify vehicles capable of performing efficient adversarial behaviors, and label them; During adversarial testing, global scene information at each adversarial time step is incorporated into the prompts of the large language model. The large language model then analyzes the current scene information to identify the optimal set of adversarial vehicles, as shown below: ； In the formula, s t This represents global scene information, where k represents the vehicle identification number. The large model score represents the rating of the k-th vehicle at time t, with a value between 0 and 1, used for judgment. Is it greater than the threshold τ? sem If the value is greater than the given value, then the "efficient adversarial" label is assigned, resulting in a set A of vehicles capable of performing efficient adversarial actions. t The semantic weight of the k-th vehicle is calculated as follows: ； In the formula, This represents the semantic weight of the k-th vehicle, with a value between 0 and 1. For numerically stable terms, >0; 32) Based on the labeled vehicles, implement a multi-agent cooperative adversarial strategy; During the adversarial testing, only the set of vehicles A was tested. t The vehicle calculation consensus term is used, and semantic weights are used. Weighting is applied, and risk-driven factors are amplified according to their weights to form a collaborative adversarial control strategy for the labeled vehicle set, as shown below: ； In the formula, For coordination indicators The weighted mean; The weighted consensus term is calculated as follows: ； In the formula, For the weighted, coordinated consistency items, the labeled transportation vehicles receive real-time rewards. As shown below: ； In the formula, Let λ represent the executable physics commands for the i-th combat vehicle's adv, and λ1, λ2, λ3, and λ4 represent the weighting coefficients of each reward. This represents the risk metric for the i-th agent at time t. This represents the soft constraint value of the rule. Represents the rate of change of longitudinal acceleration; The real-time reward update time-series difference objective y of the labeled vehicles obtained in this way t This enables a multi-agent adversarial testing process guided by a large model.

Images