An automatic driving robustness adversarial training method based on an adaptive background vehicle model

Abstract

The application discloses an automatic driving robustness confrontation training method based on an adaptive background vehicle model and belongs to the technical field of intelligent traffic management and control. In order to solve the problem of insufficient universality in existing automatic driving confrontation training, the application comprises the following steps: constructing an automatic driving model robustness training environment; constructing a combined attenuation factor to analyze the risk between a background vehicle and a target vehicle, and adopting TTC to construct a mixed risk field of the background vehicle; constructing a probability risk assessment framework; introducing a scoring function to calculate the expected risk of the background vehicle and the expected risk of the surrounding environment, and constructing an activation function for screening the background vehicle and behavior which produce confrontation to the target vehicle; constructing an adaptive alternating training mechanism, the background vehicle model determines the behavior of the background vehicle according to the activation function, and the target vehicle model makes decisions based on a deep reinforcement learning strategy; and the adaptive alternating training mechanism dynamically adjusts the intensity of the confrontation behavior in the simulation training process, so that the automatic driving robustness confrontation training is completed.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic management technology, specifically a robust adversarial training method for autonomous driving based on an adaptive background vehicle model. Background Technology

[0002] In recent years, autonomous driving technology has developed rapidly and has become an important research direction in the field of intelligent transportation. With the gradual commercialization of autonomous vehicles, their safety and reliability in real-world traffic environments have received widespread attention. However, autonomous vehicles still face many safety challenges in traffic flows mixed with human-driven vehicles.

[0003] The behavior of manually driven vehicles is highly unpredictable. Drivers may make sudden lane changes or brake suddenly based on various subjective factors, actions that are often difficult for autonomous driving systems to accurately predict, significantly increasing the risk of traffic accidents. Furthermore, autonomous driving systems themselves also face several pressing issues. Existing autonomous driving models are mainly divided into two categories: traditional modular models and end-to-end models. While traditional modular models have clearer functional divisions, coordination between modules can falter in extreme scenarios, leading to a decline in overall system performance. End-to-end models, on the other hand, lack generalization capabilities in complex scenarios and struggle to effectively handle adversarial interference. When encountering malicious attacks or complex interference signals, the system may misjudge or even malfunction.

[0004] The training methods currently used in the training phase of autonomous driving systems have significant limitations. Most training environments are built based on static or rule-based scenarios, lacking the dynamic and adversarial elements of real traffic environments. This singular training environment means that although autonomous driving models can perform well in ideal scenarios after training, they often lack adaptability when facing complex and ever-changing real traffic scenarios, failing to effectively cope with various emergencies and thus affecting the safe driving of the vehicle.

[0005] To address these issues, some research has begun to explore adversarial training to improve the robustness of autonomous driving systems. However, these adversarial training methods have also revealed some shortcomings in practical applications. On the one hand, most of these methods rely on specific vehicle models, lacking sufficient versatility and making it difficult to directly extend to different autonomous driving systems, thus limiting their widespread application. On the other hand, the adversarial intensity is usually fixed during training, making the system prone to crashes or convergence difficulties, severely impacting training stability and efficiency. Furthermore, existing adversarial training methods lack a dynamic risk assessment mechanism, failing to accurately quantify the risks posed by adversarial behavior. This makes it difficult to effectively ensure system safety while increasing the challenge of adversarial training, resulting in a degree of blindness and risk in the entire adversarial training process. Summary of the Invention

[0006] The problem this invention aims to solve is the lack of universality, inaccurate risk quantification, and poor training stability in existing autonomous driving adversarial training methods. It proposes an autonomous driving robust adversarial training method based on an adaptive background vehicle model.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A robust adversarial training method for autonomous driving based on an adaptive background vehicle model includes the following steps:

[0009] S1. Define the autonomous vehicle under training as the target vehicle model, define the vehicle models around the target vehicle model as the background vehicle models, construct a robust training environment for the autonomous driving model, and collect driving parameters of the target vehicle model and the background vehicle models.

[0010] S2. Construct a combined attenuation factor analysis to analyze the risk between the background vehicle and the target vehicle. Use the pre-collision time TTC to construct a mixed risk field for the background vehicle, which is used to conduct a reverse risk assessment of the safety of the target vehicle.

[0011] S3. Construct a probabilistic risk assessment framework, classify risk levels into three levels: high, medium, and low, and calculate the conditional probability, posterior probability, and surrounding environment probability between the background vehicle and the target vehicle under different risk levels based on the mixed risk field obtained in step S2.

[0012] S4. Introduce a scoring function to map the risk level to a numerical score, calculate the expected risk of the background vehicle and the expected risk of the surrounding environment, and construct an activation function to filter background vehicles and behaviors that pose a threat to the target vehicle.

[0013] S5. Construct an adaptive alternating training mechanism, set the simulation training process to run for a total of M rounds, with T time steps in each round. In each simulation time step, the background vehicle model determines the background vehicle behavior according to the activation function in step S4, and the target vehicle model makes decisions based on a deep reinforcement learning strategy. During the simulation training process, the adaptive alternating training mechanism dynamically adjusts the intensity of adversarial behavior according to the safety performance of the target vehicle model, thereby completing the autonomous driving robust adversarial training based on the adaptive background vehicle model.

[0014] Furthermore, the specific implementation method of step S1 includes the following steps:

[0015] S1.1. Define the trained autonomous vehicle as the target vehicle, and the target vehicle model is π. α The vehicles surrounding the target vehicle are defined as background vehicles, and the background vehicle model is π. β ;

[0016] The target vehicle and the background vehicle are set as independent, competing entities. The target vehicle and the background vehicle operate without communication and lack white-box access to each other's input states. Parameter information is not shared in the weight configuration.

[0017] The interaction process between the target vehicle model and the background vehicle model is defined as a Markov process, expressed as:

[0018] M = <S,(O α O β ),(A α A β ),P,(R α ,R β ),γ> (1)

[0019] Where S represents the state space, O α and O β Let A be the state space of the target vehicle model and the background vehicle model, respectively. α and A β Let P be the action space of the target vehicle model and the background vehicle model, respectively, and let R be the transition probability function. α and R β Here, γ represents the reward functions for the target vehicle model and the background vehicle model, respectively, and γ is the discount factor.

[0020] P:S×A α ×A β →S′

[0021] Where S′ represents the probability distribution of the next state;

[0022] The goal is to maximize cumulative rewards R:S×A α ×A β×S′→R;

[0023] S1.2. The system is defined to use V2X communication technology to collect driving parameters of the target vehicle and six surrounding background vehicles at specific locations in real time. The background vehicle set Ω b Defined as:

[0024] b∈Ω b ={front, left front, right front, back, left back, right back} (2)

[0025] Where b represents the background vehicle;

[0026] The state space of the background vehicle is defined as including the target vehicle state, the background vehicle state, and lane features. The target vehicle state includes the lateral position x of the target vehicle. t Speed ​​v t acceleration a t and heading angle θ t The background vehicle status includes the relative distance Δd between nearby background vehicles and the target vehicle. b The speed v of the background vehicle b and acceleration a b Lane characteristics include the average speed of each lane. and density ρ lane The information is collected within a 300-meter radius around the target vehicle.

[0027] The motion space of the background vehicle is defined as including candidate background vehicles. Candidate adversarial behavior aa, which includes lane changing, rapid acceleration, and rapid deceleration; normal behavior na.

[0028] Furthermore, the specific implementation method of step S2 includes the following steps:

[0029] S2.1. Construct the longitudinal attenuation factor φ x,b (Δx,Δy,t), quantifies the risk decay along the vehicle's longitudinal axis, expressed as:

[0030]

[0031] Where Δx represents the longitudinal relative position, Δy represents the lateral relative position, x represents the longitudinal position coordinate, and L... b v is the length of the background vehicle x,b (t) represents the longitudinal velocity of the vehicle at time t, and the parameter μ x,b and ρ x,b These represent the longitudinal risk attenuation intensity based on vehicle type and the longitudinal risk attenuation intensity based on speed, respectively, with max being the maximum value function;

[0032] S2.2. Constructing the transverse attenuation factor φy,b (Δx,Δy,t), quantifies the risk decay along the vehicle's lateral axis, expressed as:

[0033]

[0034] Where y is the horizontal position coordinate, W b The width of the background vehicle, v y,b (t) represents the lateral velocity of the vehicle at time t, and the parameter μ y,b and ρ y,b These represent the lateral risk attenuation intensity based on vehicle type and the lateral risk attenuation intensity based on speed, respectively.

[0035] S2.3. Using the Euclidean norm, the longitudinal and transverse attenuation factors are combined to obtain the combined attenuation factor φ. k (Δx,Δy,t), the expression is:

[0036]

[0037] S2.4. Introducing TTC (Time to Collision) into the artificial risk domain, the calculation formulas for TTC on different axes are as follows:

[0038]

[0039] Among them, TTC x For longitudinal collision time, TTC y For the lateral collision time, Δv x Δv y These are the longitudinal relative velocity and the lateral relative velocity between the background vehicle and the target vehicle, respectively.

[0040] Construct the mixed risk field of the background vehicle, expressed as:

[0041]

[0042] Where, η b The risk value represents the risk field of the mixed risk field from background vehicle b to target vehicle b, and the risk value represents the risk at time t with relative velocity (Δv). x ,Δv y The risk value at position (Δx, Δy) around background vehicle b.

[0043] Furthermore, the specific implementation method of step S3 includes the following steps:

[0044] S3.1. Risk levels are divided into three categories: high, medium, and low. The risk level is defined as follows:

[0045] τ∈Ω={dangerous,attentive,safe}={D,A,S}(9)

[0046] Where τ represents the risk level, Ω represents the set of risk levels, D represents high risk, A represents medium risk, and S represents low risk;

[0047] S3.2. Define the conditional probability distribution function F(η) of background vehicle b under different risk levels. b |τ b ), which is defined as follows:

[0048]

[0049]

[0050] Where, F(η) b |τ b =D) is the conditional probability distribution function under the high-risk level, f(η) b |τ b =A) is the conditional probability distribution function under the medium-risk level, F(η) b |τ b =S) is the conditional probability distribution function under the low-risk level, η D η S σ S σ D These are predefined hyperparameters for high-risk threshold, low-risk threshold, low-risk standard deviation, and high-risk standard deviation, respectively.

[0051] S3.3. Calculate the given η using Bayesian inference. b Background vehicle b risk level τ b The posterior probability P(τ) b |η b The expression is:

[0052]

[0053] Wherein, P(τ) b ) indicates the risk level τ b The prior probability;

[0054] To simplify the calculation, we assume that the prior probability distributions for different risk levels are uniform and satisfy the normalization condition, as expressed in the following expression:

[0055]

[0056] S3.4. Taking into account the risk levels of all background vehicles in the surrounding environment, determine the overall risk distribution of the entire environment. The probability distribution of the risk levels of the surrounding environment is defined as follows:

[0057] P(τ env =D)=1-∏ b∈B (1-P(τ b =D|η b (12)

[0058] P(τ env =S)=∏ b∈B P(τ b =S|η b (13)

[0059] P(τ env =A)=1-P(τ) env =S)-P(τ) env =D) (14)

[0060] Wherein, P(τ) env =D), P(τ) env =A) and P(τ) env =S) represent the probabilities of the surrounding environment being high, medium, and low risk levels, respectively, represented by P(τ). b =D|η b ), P(τ) b =A|η b ), P(τ) b =S|η b ) represents the risk level probability of background vehicle b.

[0061] Furthermore, the specific implementation method of step S4 includes the following steps:

[0062] S4.1. Introduce the scoring function Sc(τ) to map the risk level to a numerical score, assigning values ​​of 2, 1, and 0 to D, A, and S respectively. Calculate the expected risk of the background vehicle and the expected risk of the surrounding environment, expressed as:

[0063]

[0064]

[0065] Where, ∈ b The expected risks of the background vehicle, ∈ env P(τ) represents the expected risk from the surrounding environment. b |n b ) represents the state n given by the given state n b Lower risk level τ b The conditional probability, Sc(τ) b Risk level τ b The score value, Sc(τ) env Risk level τ env The rating value;

[0066] S4.2. Based on the expected risks of candidate background vehicles Expected risk relative to the surrounding environment env Evaluate whether candidate background vehicles are suitable for performing adversarial actions and define activation functions. for:

[0067]

[0068] Where k is a positive constant that determines the steepness of the function, and M l and M u These are the upper and lower bounds of the activation function, respectively. If the activation function result is 1, the candidate background vehicle is determined to perform candidate adversarial behavior; if it is 0, the normal behavior is performed.

[0069] Furthermore, the specific implementation method of step S5 includes the following steps:

[0070] S5.1. Construct an adaptive alternating training mechanism based on the switching indicator C. R The training process adaptively switches between target vehicles and background vehicles, employing... As an indicator, C c This indicates the interval between two incidents in terms of the number of training rounds.

[0071] The switching is controlled by an adaptive alternating training threshold H, when C R When >H, background vehicle model training is turned off, target vehicle model training is started, and its decision-making strategy under adversarial conditions is enhanced.

[0072] S5.2. Setting the background vehicle model: The goal during the training phase is to optimize the background vehicle model π using the fixed strategy of the target vehicle model during the non-training period of the target vehicle model. β The strategy is to maximize the cumulative discount reward, expressed as:

[0073]

[0074] in, The optimal strategy for the background vehicle model β is the strategy that maximizes the long-term discount reward. For the background vehicle model π β Expectations under the strategy; R β Let be the reward function for the background vehicle model β, representing the state s. (t) Execute action Then transition to state s (t+1) The immediate reward obtained at that time; γ∈[0,1) is a discount factor used to weigh the importance of future rewards; s (t) and s (t+1)Let be the states at times t and t+1, respectively, and The action performed by background vehicle β at time t; To the target vehicle model π α Background vehicle model π under the influence of β The state transition probability function represents the probability of transitioning from state s to s. (t) Next action Later transferred to s (t+1) The probability; s0 = s represents the initial state as s;

[0075] S5.3. After the background vehicle model has been trained, adversarial behavior is introduced into the deep reinforcement learning strategy π of the target vehicle model. α When making decisions, if the adversarial behavior effectively creates failure cases for the target vehicle, the target vehicle model's weight parameters are updated through retraining, while the background vehicle model parameters are kept constant in the training environment. The target vehicle model's policy π... α It maximizes the cumulative discount reward, expressed as follows:

[0076]

[0077] in, The optimal strategy for the target vehicle model α is the strategy that maximizes the long-term discount reward. To the target vehicle model π α Expectations under the strategy; R α Let be the reward function for the target vehicle model α, representing the state s. (t) Execute action Then transition to state s (t+1) The immediate reward obtained at that time; γ∈[0,1) is a discount factor used to weigh the importance of future rewards; s (t) and s (t+1) Let be the states at times t and t+1, respectively, and The action performed by the target vehicle α at time t; For the background vehicle model π α Under the influence of the target vehicle model π α The state transition probability function represents the probability of transitioning from state s to s. (t) Next action Later transferred to s (t+1) The probability; s0 = s represents the initial state as s.

[0078] The beneficial effects of this invention are:

[0079] This invention discloses a robust adversarial training method for autonomous driving based on an adaptive background vehicle model. It constructs a hybrid risk field to finely quantify the interaction risks between the target vehicle and the background vehicle, and utilizes a reinforcement learning model to generate controllable adversarial actions. Simultaneously, it designs an adaptive activation function and an alternating training mechanism to dynamically balance adversarial intensity and training efficiency, avoiding model collapse caused by excessive interference. Ultimately, this method improves the robustness of autonomous driving systems in complex traffic environments, significantly reduces accident rates, and is compatible with multiple reinforcement learning algorithms, providing a general adversarial training framework for different autonomous driving models.

[0080] This invention presents a robust adversarial training method for autonomous driving based on an adaptive background vehicle model. It constructs an adaptive alternating training mechanism to mitigate the inherent instability of reinforcement learning frameworks. Furthermore, this invention allows for gradual adjustment of the adversarial intensity faced by the target vehicle, simulating a series of new environmental conditions after switching, thereby smoothing the change curve of adversarial intensity. This process enhances adaptability and learning efficiency at different stages while maintaining the universality of training, ensuring robustness and safety in various driving scenarios. Attached Figure Description

[0081] Figure 1 This is a flowchart of an autonomous driving robust adversarial training method based on an adaptive background vehicle model, as described in this invention.

[0082] Figure 2 This is a schematic diagram illustrating the acquisition of driving parameters of the target vehicle and six surrounding vehicles at specific locations in this invention. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0084] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0085] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 and attached Figure 2 Detailed explanation is as follows:

[0086] Example 1:

[0087] A robust adversarial training method for autonomous driving based on an adaptive background vehicle model includes the following steps:

[0088] S1. Define the autonomous vehicle under training as the target vehicle model, define the vehicle models around the target vehicle model as the background vehicle models, construct a robust training environment for the autonomous driving model, and collect driving parameters of the target vehicle model and the background vehicle models.

[0089] Furthermore, the specific implementation method of step S1 includes the following steps:

[0090] S1.1. Define the trained autonomous vehicle as the target vehicle, and the target vehicle model is π. α The vehicles surrounding the target vehicle are defined as background vehicles, and the background vehicle model is π. β ;

[0091] The target vehicle and the background vehicle are set as independent, competing entities. The target vehicle and the background vehicle operate without communication and lack white-box access to each other's input states. Parameter information is not shared in the weight configuration.

[0092] The interaction process between the target vehicle model and the background vehicle model is defined as a Markov process, expressed as:

[0093] M = <S,(O α O β ),(A α A β ),P,(R α ,R β ),γ> (1)

[0094] Where S represents the state space, O α and O β Let A be the state space of the target vehicle model and the background vehicle model, respectively. α and A β Let P be the action space of the target vehicle model and the background vehicle model, respectively, and let R be the transition probability function. α and R β Here, γ represents the reward functions for the target vehicle model and the background vehicle model, respectively, and γ is the discount factor.

[0095] P:S×A α ×A β →S′

[0096] Where S' represents the probability distribution of the next state;

[0097] The goal is to maximize cumulative rewards R:S×A α ×A β ×S′→R;

[0098] S1.2. The system is defined to use V2X communication technology to collect driving parameters of the target vehicle and six surrounding background vehicles at specific locations in real time. The background vehicle set Ω b Defined as:

[0099] b∈Ω b ={front, left front, right front, back, left back, right back} (2)

[0100] Where b represents the background vehicle;

[0101] The state space of the background vehicle is defined as including the target vehicle state, the background vehicle state, and lane features. The target vehicle state includes the lateral position x of the target vehicle. t Speed ​​v t acceleration a t and heading angle θ t The background vehicle status includes the relative distance Δd between nearby background vehicles and the target vehicle. b The speed v of the background vehicle b and acceleration a b Lane characteristics include the average speed of each lane. and density ρ lane The information is collected within a 300-meter radius around the target vehicle.

[0102] The motion space of the background vehicle is defined as including candidate background vehicles. Candidate adversarial behavior aa, which includes lane changing, rapid acceleration, and rapid deceleration; normal behavior na.

[0103] S2. Construct a combined attenuation factor analysis to analyze the risk between the background vehicle and the target vehicle. Use the pre-collision time TTC to construct a mixed risk field for the background vehicle, which is used to conduct a reverse risk assessment of the safety of the target vehicle.

[0104] Furthermore, the specific implementation method of step S2 includes the following steps:

[0105] S2.1. Construct the longitudinal attenuation factor φ x,b (Δx,Δy,t), quantifies the risk decay along the vehicle's longitudinal axis, expressed as:

[0106]

[0107] Where Δx represents the longitudinal relative position, Δy represents the lateral relative position, x represents the longitudinal position coordinate, and L... b v is the length of the background vehicle x,b (t) represents the longitudinal velocity of the vehicle at time t, and the parameter μ x,b and ρ x,b These represent the longitudinal risk attenuation intensity based on vehicle type and the longitudinal risk attenuation intensity based on speed, respectively, with max being the maximum value function;

[0108] S2.2. Constructing the transverse attenuation factor φ y,b (Δx,Δy,t), quantifies the risk decay along the vehicle's lateral axis, expressed as:

[0109]

[0110] Where y is the horizontal position coordinate, W b The width of the background vehicle, v y,b (t) represents the lateral velocity of the vehicle at time t, and the parameter μ y,b and ρ y,b These represent the lateral risk attenuation intensity based on vehicle type and the lateral risk attenuation intensity based on speed, respectively.

[0111] S2.3. Using the Euclidean norm, the longitudinal and transverse attenuation factors are combined to obtain the combined attenuation factor φ. k (Δx,Δy,t), the expression is:

[0112]

[0113] S2.4. Introducing TTC (Time to Collision) into the artificial risk domain, the calculation formulas for TTC on different axes are as follows:

[0114]

[0115] Among them, TTC x For longitudinal collision time, TTC y For the lateral collision time, Δv x Δv y These are the longitudinal relative velocity and the lateral relative velocity between the background vehicle and the target vehicle, respectively.

[0116] Construct the mixed risk field of the background vehicle, expressed as:

[0117]

[0118] Where, η b The risk value represents the risk field of the mixed risk field from background vehicle b to target vehicle b, and the risk value represents the risk at time t with relative velocity (Δv). x ,Δvy The risk value at position (Δx, Δy) around background vehicle b.

[0119] Furthermore, η b The risk is guaranteed to be maximized within the area occupied by the background vehicle and then smoothly decreases to near zero with increasing distance. The constant 1 in the denominator ensures that the risk function remains valid even if the background vehicle's speed is zero.

[0120] S3. Construct a probabilistic risk assessment framework, classify risk levels into three levels: high, medium, and low, and calculate the conditional probability, posterior probability, and surrounding environment probability between the background vehicle and the target vehicle under different risk levels based on the mixed risk field obtained in step S2.

[0121] Furthermore, the specific implementation method of step S3 includes the following steps:

[0122] S3.1. Risk levels are divided into three categories: high, medium, and low. The risk level is defined as follows:

[0123] τ∈Ω={dangerous,attentive,safe}={D,A,S}(9)

[0124] Where τ represents the risk level, Ω represents the set of risk levels, D represents high risk, A represents medium risk, and S represents low risk;

[0125] S3.2. Define the conditional probability distribution function F(η) of background vehicle b under different risk levels. b |τ b ), which is defined as follows:

[0126]

[0127]

[0128] Where, F(η) b |τ b =D) is the conditional probability distribution function under the high-risk level, F(η) b |τ b =A) is the conditional probability distribution function under the medium-risk level, F(η) b |τ b =S) is the conditional probability distribution function under the low-risk level, η D η S σ S σ D These are predefined hyperparameters for high-risk threshold, low-risk threshold, low-risk standard deviation, and high-risk standard deviation, respectively.

[0129] S3.3. Calculate the given η using Bayesian inference. b Background vehicle b risk level τ b The posterior probability P(τ) b |η b The expression is:

[0130]

[0131] Wherein, P(τ) b ) indicates the risk level τ b The prior probability;

[0132] To simplify the calculation, we assume that the prior probability distributions for different risk levels are uniform and satisfy the normalization condition, as expressed in the following expression:

[0133]

[0134] S3.4. Taking into account the risk levels of all background vehicles in the surrounding environment, determine the overall risk distribution of the entire environment. The probability distribution of the risk levels of the surrounding environment is defined as follows:

[0135] P(τ env =D)=1-∏ b∈B (1-P(τ b =D|η b (12)

[0136] P(τ env =S)=∏ b∈B P(τ b =S|η b (13)

[0137] P(τ env =A)=1-P(τ) env =S)-P(τ) env =D) (14)

[0138] Wherein, P(τ) env =D), P(τ) env =A) and P(τ) env =S) represent the probabilities of the surrounding environment being high, medium, and low risk levels, respectively, represented by P(τ). b =D|η b ), P(τ) b =A|η b ), P(τ) b =S|η b ) represents the risk level probability of background vehicle b.

[0139] S4. Introduce a scoring function to map the risk level to a numerical score, calculate the expected risk of the background vehicle and the expected risk of the surrounding environment, and construct an activation function to filter background vehicles and behaviors that pose a threat to the target vehicle.

[0140] Furthermore, the specific implementation method of step S4 includes the following steps:

[0141] S4.1. Introduce the scoring function Sc(τ) to map the risk level to a numerical score, assigning values ​​of 2, 1, and 0 to D, A, and S respectively. Calculate the expected risk of the background vehicle and the expected risk of the surrounding environment, expressed as:

[0142]

[0143] Where, ∈ b The expected risks of the background vehicle, ∈ env P(τ) represents the expected risk from the surrounding environment. b |n b ) represents the state n given by the given state n b Lower risk level τ b The conditional probability, Sc(τ) b Risk level τ b The score value, Sc(τ) env Risk level τ env The rating value;

[0144] S4.2. Based on the expected risks of candidate background vehicles Expected risk relative to the surrounding environment env Evaluate whether candidate background vehicles are suitable for performing adversarial actions and define activation functions. for:

[0145]

[0146] Where k is a positive constant that determines the steepness of the function, and M l and M u These are the upper and lower bounds of the activation function, respectively. If the activation function result is 1, the candidate background vehicle is determined to perform candidate adversarial behavior; if it is 0, the normal behavior is performed.

[0147] S5. Construct an adaptive alternating training mechanism, set the simulation training process to run for a total of M rounds, with T time steps in each round. In each simulation time step, the background vehicle model determines the background vehicle behavior according to the activation function in step S4, and the target vehicle model makes decisions based on a deep reinforcement learning strategy. During the simulation training process, the adaptive alternating training mechanism dynamically adjusts the intensity of adversarial behavior according to the safety performance of the target vehicle model, thereby completing the autonomous driving robust adversarial training based on the adaptive background vehicle model.

[0148] Furthermore, the specific implementation method of step S5 includes the following steps:

[0149] S5.1. Construct an adaptive alternating training mechanism based on the switching indicator C. R The training process adaptively switches between target vehicles and background vehicles, employing... As an indicator, C c This indicates the interval between two incidents in terms of the number of training rounds.

[0150] The switching is controlled by an adaptive alternating training threshold H, when C R When >H, background vehicle model training is turned off, target vehicle model training is started, and its decision-making strategy under adversarial conditions is enhanced.

[0151] S5.2. Setting the background vehicle model: The goal during the training phase is to optimize the background vehicle model π using the fixed strategy of the target vehicle model during the non-training period of the target vehicle model. β The strategy is to maximize the cumulative discount reward, expressed as:

[0152]

[0153] in, The optimal strategy for the background vehicle model β is the strategy that maximizes the long-term discount reward. For the background vehicle model π β Expectation under the strategy; π β Let be the reward function for the background vehicle model β, representing the state s. (t) Execute action Then transition to state s (t+1) The immediate reward obtained at that time; γ∈[0,1) is a discount factor used to weigh the importance of future rewards; s (t) and s (t+1) Let be the states at times t and t+1, respectively, and The action performed by background vehicle β at time t; To the target vehicle model π α Background vehicle model π under the influence of β The state transition probability function represents the probability of transitioning from state s to s. (t) Next action Later transferred to s (t+1) The probability; s0 = s represents the initial state as s;

[0154] S5.3. After the background vehicle model has been trained, adversarial behavior is introduced into the deep reinforcement learning strategy π of the target vehicle model. αWhen making decisions, if the adversarial behavior effectively creates failure cases for the target vehicle, the target vehicle model's weight parameters are updated through retraining, while the background vehicle model parameters are kept constant in the training environment. The target vehicle model's policy π... α It maximizes the cumulative discount reward, expressed as follows:

[0155]

[0156] in, The optimal strategy for the target vehicle model α is the strategy that maximizes the long-term discount reward. To the target vehicle model π α Expectations under the strategy; R α Let be the reward function for the target vehicle model α, representing the state s. (t) Execute action Then transition to state s (t+1) The immediate reward obtained at that time; γ∈[0,1) is a discount factor used to weigh the importance of future rewards; s (t) and s (t+1) Let be the states at times t and t+1, respectively, and The action performed by the target vehicle α at time t; For the background vehicle model π α Under the influence of the target vehicle model π α The state transition probability function represents the probability of transitioning from state s to s. (t) Next action Later transferred to s (t+1) The probability; s0 = s represents the initial state as s.

[0157] Furthermore, the adoption of an adaptive alternating training mechanism mitigates the inherent instability of reinforcement learning frameworks. In addition, this mechanism allows for gradual adjustment of the adversarial intensity faced by the target vehicle, simulating a series of new environmental conditions after the transition, thus smoothing the adversarial intensity variation curve. This process enhances adaptability and learning efficiency at different stages while maintaining training versatility, ensuring robustness and safety across various driving scenarios.

[0158] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A robust adversarial training method for autonomous driving based on an adaptive background vehicle model, characterized in that, Includes the following steps: S1. Define the autonomous vehicle being trained as the target vehicle model, define the vehicle models around the target vehicle model as the background vehicle models, construct a robust training environment for the autonomous driving model, and collect driving parameters of the target vehicle model and the background vehicle models. S2. Construct a combined attenuation factor analysis to analyze the risk between the background vehicle and the target vehicle. Use collision time TTC to construct a mixed risk field for the background vehicle, which is used to conduct a reverse risk assessment of the safety of the target vehicle. The specific implementation method of step S2 includes the following steps: S2.

1. Constructing the longitudinal attenuation factor The risk attenuation along the vehicle's longitudinal axis is quantified by the following expression: (3) in Indicates the relative position in the vertical direction. The relative horizontal position The vertical position coordinates are: The length of the background vehicle, Represents the longitudinal velocity of the vehicle at time t, parameter and These represent the longitudinal risk attenuation intensity based on vehicle type and the longitudinal risk attenuation intensity based on speed, respectively, with max being the maximum value function; S2.

2. Constructing the transverse attenuation factor The risk attenuation along the vehicle's lateral axis is quantified by the following expression: (4) in, The horizontal position coordinates are... The width of the background vehicle. Let be the lateral velocity of the vehicle at time t, and be the parameters. and These represent the lateral risk attenuation intensity based on vehicle type and the lateral risk attenuation intensity based on speed, respectively. S2.

3. Using the Euclidean norm, the longitudinal and transverse attenuation factors are combined to obtain the combined attenuation factor. The expression is: (5)； S2.

4. Introducing TTC (Time to Collision) into the artificial risk domain, the calculation formulas for TTC on different axes are as follows: (6) (7) in, For longitudinal collision time, For lateral collision time, , These are the longitudinal relative velocity and the lateral relative velocity between the background vehicle and the target vehicle, respectively. Construct the mixed risk field of the background vehicle, expressed as: (8) in, The risk value represents the mixed risk field from background vehicle b to target vehicle b, and the value represents the risk at time t with relative velocity b. Location around vehicle b in the background The risk value; S3. Construct a probabilistic risk assessment framework, classify risk levels into three levels: high, medium, and low, and calculate the conditional probability, posterior probability, and surrounding environment probability between the background vehicle and the target vehicle under different risk levels based on the mixed risk field obtained in step S2. S4. Introduce a scoring function to map risk levels to numerical scores, calculate the expected risk of background vehicles and the expected risk of the surrounding environment, and construct an activation function to filter background vehicles and behaviors that pose a threat to the target vehicle. S5. Construct an adaptive alternating training mechanism, set the simulation training process to run for a total of M' rounds, with T time steps executed in each round. In each simulation time step, the background vehicle model determines the background vehicle behavior according to the activation function in step S4, and the target vehicle model makes decisions based on a deep reinforcement learning strategy. During the simulation training process, the adaptive alternating training mechanism dynamically adjusts the intensity of adversarial behavior according to the safety performance of the target vehicle model, thereby completing the autonomous driving robust adversarial training based on the adaptive background vehicle model. The specific implementation method of step S5 includes the following steps: S5.

1. Construct an adaptive alternating training mechanism based on the switching indicator. The training process adaptively switches between target vehicles and background vehicles, employing... As an indicator, among This indicates the interval between two incidents in terms of the number of training rounds. The switching is controlled by an adaptive alternating training threshold H, when At this time, background vehicle model training is turned off, and target vehicle model training is started to enhance its decision-making strategy under adversarial conditions. S5.

2. Setting the background vehicle model: The goal during the training phase is to optimize the background vehicle model by utilizing the fixed strategy of the target vehicle model during the non-training period of the target vehicle. The strategy is to maximize the cumulative discount reward, expressed as: (17) in, Background vehicle model The optimal strategy is the strategy that maximizes the cumulative discount reward. For the background vehicle model Expectations under the strategy; Background vehicle model Execute action After transitioning to state Instant rewards obtained at that time; The discount factor is used to weigh the importance of future rewards. and They are time points and The state, and ; Vehicles in the background The action performed; In the target vehicle model Background vehicle model under the influence The state transition probability function represents the probability of transitioning from state to state. Next action Later transferred to The probability of; Indicates the initial state is ; S5.

3. After the background vehicle model has been trained, a deep reinforcement learning strategy is implemented for the target vehicle model by introducing adversarial behavior. To make decisions, when adversarial actions effectively create failure cases for the target vehicle, the target vehicle model's weight parameters are updated through retraining, while the background vehicle model parameters are kept constant in the training environment. This is a deep reinforcement learning strategy for the target vehicle model. To maximize the cumulative discount reward, the expression is: (18) in, For the target vehicle model The optimal strategy is the strategy that maximizes the cumulative discount reward. In the target vehicle model Expectations under the strategy; Let be the reward function for the target vehicle model, representing the state. Execute action After transitioning to state Instant rewards obtained at that time; The discount factor is used to weigh the importance of future rewards. and They are time points and The state, and ; For the target vehicle at time The action performed; For the background vehicle model Target vehicle model under the influence The state transition probability function represents the probability of transitioning from state to state. Next action Later transferred to The probability of; Indicates the initial state is .

2. The autonomous driving robustness adversarial training method based on an adaptive background vehicle model according to claim 1, characterized in that, The specific implementation method of step S1 includes the following steps: S1.

1. Define the trained autonomous vehicle as the target vehicle, and the target vehicle model is as follows: ; The vehicles surrounding the target vehicle are defined as background vehicles, and the background vehicle model is as follows: ; The target vehicle and the background vehicle are set as independent, competing entities. The target vehicle and the background vehicle operate without communication and lack white-box access to each other's input states. Parameter information is not shared in the weight configuration. The interaction process between the target vehicle model and the background vehicle model is defined as a Markov process, expressed as: （1） in, Representing the state space, and These are the state spaces of the target vehicle model and the background vehicle model, respectively. and Let be the action spaces of the target vehicle model and the background vehicle model, respectively, and let P be the transition probability function. and These are the reward functions for the target vehicle model and the background vehicle model, respectively. Discount factor; in, This represents the probability distribution of the next state. The goal is to maximize cumulative rewards. ; S1.

2. The system is defined to utilize V2X communication technology to collect real-time driving parameters of the target vehicle and six surrounding vehicles at specific locations. Defined as: （2） in, Vehicles in the background; The state space of the background vehicles is defined as including the target vehicle state, the background vehicle state, and lane features. The target vehicle state includes the lateral position of the target vehicle. ,speed acceleration and heading angle Background vehicle status includes the relative distance between nearby background vehicles and the target vehicle. The speed of the background vehicle and acceleration Lane characteristics include the average speed of each lane. and density The information is collected within a 300-meter radius around the target vehicle. The motion space of the background vehicle is defined as including candidate background vehicles. Candidate adversarial behaviors include lane changes, rapid acceleration, and rapid deceleration. Normal behaviors include na.

3. The autonomous driving robustness adversarial training method based on an adaptive background vehicle model according to claim 2, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. Risk levels are divided into three categories: high, medium, and low. The risk level is defined as follows: (9) in, Indicates the risk level. Represents a set of risk levels. Indicates high risk. Indicates medium risk. Indicates low risk; S3.

2. Define the conditional probability distribution function F of background vehicle b under different risk levels. Its definition is as follows: (10) in, Let be the conditional probability distribution function under the high-risk level. Let be the conditional probability distribution function under the medium-risk level. This is the conditional probability distribution function under the low-risk level. , , , These are predefined hyperparameters for high-risk threshold, low-risk threshold, low-risk standard deviation, and high-risk standard deviation, respectively. S3.

3. Calculate the given... using Bayesian inference. Risk level of vehicle b in the context of the situation posterior probability The expression is: (11) in, Indicates risk level The prior probability; To simplify the calculation, we assume that the prior probability distributions for different risk levels are uniform and satisfy the normalization condition, as expressed in the following expression: ； S3.

4. Taking into account the risk levels of all background vehicles in the surrounding environment, determine the overall risk distribution of the entire environment. The probability distribution of the risk levels of the surrounding environment is defined as follows: (12) (13) (14) in, , and These represent the probabilities of the surrounding environment being at high, medium, and low risk levels, respectively, and are represented by... , Indicates background vehicle The probability of risk level.

4. The autonomous driving robustness adversarial training method based on an adaptive background vehicle model according to claim 3, characterized in that, The specific implementation method of step S4 includes the following steps: S4.

1. Introducing a scoring function The risk levels are mapped to numerical scores, with D, A, and S assigned values ​​of 2, 1, and 0 respectively. The expected risks of the background vehicle and the surrounding environment are calculated using the following expressions: （15） in, The expected risks of the vehicle in the background. For the anticipated risks of the surrounding environment, Risk level Risk level The rating value; S4.

2. Based on the expected risks of candidate background vehicles Expected risks relative to the surrounding environment Evaluate whether candidate background vehicles are suitable for performing adversarial actions and define activation functions. for: (16) in, It is a positive constant that determines the steepness of the function. and These are the upper and lower bounds of the activation function, respectively. If the activation function result is 1, the candidate background vehicle is determined to perform candidate adversarial behavior; if it is 0, the normal behavior is performed.

Images