Automatic driving adversarial learning method based on recent development area guidance

Abstract

According to the automatic driving adversarial learning method based on recent development area guidance provided by the invention, personal factors and adversarial factors of an adversarial vehicle and an ideal return upper limit of the vehicle are comprehensively considered, so that a generated adversarial task is ensured to be solvable while having challenging performance, and the adversarial learning method accords with a traffic rule and is high in practicability. The learning and adaptation of the self-vehicle agent are effectively promoted; secondly, a concept of'potential vehicle threat 'is introduced, and dynamic adjustment of specific gravity coefficients among constraint terms of a loss function is realized based on a ZPD theory, so that task difficulty is adaptively changed in a training process, and a self-vehicle is promoted to learn faster; and finally, a collision analysis method is provided, the severity of collision is evaluated and a collision loss value is calculated through vehicle body collision points, collision angles and collision speed factors, and the collision loss value is incorporated into a loss function of the own vehicle and the confronting agent, so that the model is ensured to update parameters according to detailed collision risks.

Description

Technical Field

[0001] The present invention relates to autonomous driving, reinforcement learning and adversarial learning, and in particular to an autonomous driving adversarial learning method based on guidance of the zone of proximal development. Background Art

[0002] With the development of autonomous driving technology, its safety in diverse road conditions and emergencies has attracted much attention. Safety-critical scenarios, such as natural disasters, extreme climates, and complex traffic environments, pose extreme challenges to the decision-making and response of autonomous driving systems. Traditional methods often have difficulty maintaining efficient and safe operations in these dynamic environments. Adversarial learning, as a variant of reinforcement learning, improves the adaptability and robustness of autonomous driving systems by introducing adversarial environments. However, existing adversarial generation methods lack flexibility in designing challenges, which may lead to overly conservative strategies and make it difficult to balance safety and efficiency.

[0003] The Zone of Proximal Development (ZPD) is an important concept in psychology that describes the difference between tasks that an individual can complete under guidance and tasks that they can complete independently. Inspired by the ZPD theory, this paper proposes an adversarial learning method based on ZPD guidance to improve the robustness of autonomous vehicles in safety-critical situations. By adaptively and dynamically adjusting the challenge level and detailed collision analysis, the generated adversarial tasks are ensured to be both challenging and solvable, thereby promoting the learning and adaptation of the autonomous driving system. Summary of the invention

[0004] In view of the problem that existing methods often lack flexibility in designing adversarial challenges, resulting in the learned strategies being too conservative and unable to effectively balance safety and driving efficiency, the present invention proposes an autonomous driving adversarial learning method based on the guidance of the Zone of Proximal Development (ZPD), aiming to improve the robustness of autonomous driving vehicles in safety-critical situations.

[0005] An autonomous driving adversarial learning method based on the guidance of the zone of proximal development includes:

[0006] The loss function of the agent designed for training is:

[0007] C adv =α1×C Personal +α2×C adversary +α3×C IRC

[0008] Among them, C personal represents the adversarial constraint term, C adversary represents the personal constraint of the adversarial agent, CIRC represents the constraint item based on IRC, α1, α2, α3 are the corresponding weight factors;

[0009]

[0010] Among them, S collision，adv represents the severity of the collision of the adversarial agent, is the average speed of all adversarial agents within a certain range, v max，adv is the maximum speed of the adversarial agent;

[0011] C adversary =S collision，ego

[0012] Among them, S collision，ego represents the collision severity of the ego vehicle agent;

[0013] C IRC =1-max(1,1.5×f(d,α,θ adv , v ego ))

[0014]

[0015] Among them, v ego and v max，ego are the speed and maximum speed of the vehicle, d i , α i and θ adv，i are the relative distance, relative angle, and velocity direction of the i-th adversarial agent relative to the ego vehicle.

[0016] The optimal loss function designed for the ego-car agent is:

[0017] C ego =β1×C safety +β2×C efficiency

[0018] Among them, C safety represents the safety constraint, C efficiency represents the efficiency constraint term, β1 and β2 are the corresponding weight factors; C safety =S collision，ego ;

[0019] Preferably, the loss function of the agent is specifically:

[0020] C adv =C personal +(1-VPTL)×C adversary +VPTL×C IRC ;

[0021] Where, VPTL = 0.5 × VPTL′ + 0.5 × VPTL base ;

[0022] VPTL′=0.3×(1-SR pre )+0.4×CR pre +0.3×EN pre

[0023] Among them, SR pre is the success rate of the previous stage, CR pre is the collision rate in the previous stage, EN pre is the ratio of the number of steps in which emergency avoidance occurred in each round of the previous stage, and each n iterations constitutes a stage; where n is the set number of iterations;

[0024]

[0025] Among them, n epi is the number of iterations in the current stage, N epi is the total number of iterations.

[0026] Preferably, the collision severity S of the adversarial agent collision,adv =S collision ;

[0027] Among them, S collision Indicates the severity of the vehicle collision:

[0028]

[0029] Among them, P represents the collision position of the vehicle body, α rel represents the collision angle, V represents the collision speed, V car1 、V car2 Represent the speeds of the two vehicles that collided, k vel represents the speed-related scaling factor, k loc is a weighting factor related to the collision position, g(P, α rel ) represents the collision position P and collision angle α rel The corresponding collision position angle factor k loc The value of g(P, α rel ) is calculated as shown in the following table:

[0030]

[0031] The best one represents the collision severity of the ego vehicle agent as S collision,ego =S collision ;

[0032] Among them, S collisionIndicates the severity of the vehicle collision:

[0033]

[0034] Among them, P represents the collision position of the vehicle body, α rel represents the collision angle, V represents the collision speed, V car1 、V car2 Represent the speeds of the two vehicles that collided, k vel represents the speed-related scaling factor, k loc is a weighting factor related to the collision position, g(P, α rel ) indicates that the collision position P is below and at the collision angle α rel The corresponding values ​​are shown in the following table:

[0035]

[0036]

[0037] The present invention provides an autonomous driving adversarial learning method based on the guidance of the zone of proximal development, which has the following beneficial effects:

[0038] First, the present invention proposes an innovative loss function constraint design, which comprehensively considers the personal factors of the adversarial vehicle, the adversarial factors and the ideal reward upper limit of the ego vehicle, ensuring that the generated adversarial tasks are challenging and solvable, and comply with traffic rules, effectively promoting the learning and adaptation of the ego vehicle agent. Secondly, the present invention introduces the concept of "potential threat of vehicles" and realizes the dynamic adjustment of the weight coefficients between the constraints of the loss function based on the ZPD theory, so as to adaptively change the task difficulty during the training process and promote faster learning of the ego vehicle. Finally, the present invention proposes a detailed collision analysis method, which evaluates the severity of the collision through the factors of the collision point, collision angle and collision speed of the vehicle body, calculates the collision loss value, and incorporates it into the loss function of the ego vehicle and the adversarial agent, ensuring that the model updates its parameters according to the detailed collision risk.

[0039] The present invention can significantly improve the performance and robustness of the autonomous driving system in complex driving environments, achieving an optimal balance between safety and efficiency. DETAILED DESCRIPTION

[0040] The present invention proposes an adversarial learning method for autonomous driving based on the guidance of the zone of proximal development (ZPD). The implementation of the adversarial learning method involves the design of the agent's loss function constraint term, the potential threat of the vehicle, and the severity of the collision, as follows:

[0041] First, we design the loss function constraint for the agent. The loss function design of the adversarial agent aims to improve the performance of the adversarial agent through a reasonable incentive and penalty mechanism while maintaining the authenticity, challenge and solvability of the task. Specifically, the present invention adds constraints to the loss function, which consists of the following three core components:

[0042] (1) Adversarial loss constraint: This part of the loss is based on the performance of the adversarial agent (NPC) in causing the ego vehicle agent (player) to fail. Specifically, if the NPC cannot effectively guide the ego vehicle to collide or slow down, it will suffer a large loss. The original intention of this design is to simulate adversarial scenarios in the real world and ensure that the NPC can ensure the challenge and complexity of the task.

[0043] (2) Personal loss constraint: To prevent NPCs from taking only extreme countermeasures (such as directly hitting the player), the personal item is used to measure the NPC's performance in achieving its own goals, such as complying with traffic rules and safely reaching the intended destination. This design takes into account reasonable behavior in reality and ensures that NPCs conduct reasonable confrontation without violating mission rules. It allows NPCs to learn how to effectively interfere and prevents them from taking unnatural or extreme action strategies in order to make the player fail.

[0044] (3) Ideal Return Ceiling (IRC) of the ego-driving agent: This paper proposes the concept of an "ideal return ceiling" for the ego-driving agent as the last constraint of the loss function. The ideal return ceiling is estimated through expert knowledge and rules, and is used to characterize the maximum return that the ego-driving agent can theoretically obtain in this task scenario. The introduction of IRC is to ensure that the generated environment is challenging and solvable for the ego-driving agent, thereby avoiding unsolvable task scenarios that cause the ego-driving agent to learn overly conservative strategies.

[0045] Through the design of the above constraints, the present invention can enable the adversarial agent to effectively interact with the ego vehicle while achieving its personal goals. In addition, by reasonably setting the IRC of the ego vehicle agent, a balance is achieved between the challenge and achievability of the task, preventing the formation of overly aggressive or conservative game strategies. The complete constraints of the adversarial agent can be written as:

[0046] C adv =α1×C personal +α2×C adversary + × α 3 × C IRC

[0047] Among them, C personal represents the adversarial constraint term, C adversary represents the personal constraint of the adversarial agent, C IRCrepresents the IRC-based constraint, and α1, α2, and α3 are weight factors. This design ensures that the strategy of the adversarial agent is not only based on the maximization of a single adversarial objective, but a more comprehensive combination of objectives that is closer to the real-world scenario. personal The calculation formula is as follows:

[0048]

[0049] Among them, S collision，adv represents the severity of the collision of the adversarial agent, is the average speed of all adversarial agents within a certain range, v max，adv is the maximum speed of the adversarial agent.

[0050] C adversary The calculation formula is as follows:

[0051] C adversary =S collision，ego

[0052] Among them, S collision，ego Represents the collision severity of the ego vehicle agent.

[0053] C IRC The calculation formula is as follows:

[0054] C IRC =1-max(1,1.5×f(d,α,θ adv , v ego ))

[0055]

[0056] Among them, v geo and v max，ego are the speed and maximum speed of the vehicle, d i , α i and θ adv，i are the relative distance, relative angle and speed direction of the ith adversarial agent relative to the ego vehicle. collision，adv and S collision，ego The factors involved and the specific calculations are explained in detail later.

[0057] Similarly, the present invention also designs constraints for the self-driving agent, which enables the self-driving agent to effectively learn interactively in the face of adversarial tasks. The complete constraints of the self-driving agent can be written as:

[0058] C ego =β1×C safety +β2×C efficiency

[0059] Among them, C safetyrepresents the safety constraint, C efficiency represents the efficiency constraint term, β1 and β2 are weight factors. In the present invention, the values ​​of β1 and β2 are both set to 0.5. safety The calculation formula is as follows:

[0060] C safety＝ Sc ikkusuib，ego

[0061] C efficiency The calculation formula is as follows:

[0062]

[0063] Furthermore, in order to meet the demand for adaptive dynamic difficulty adjustment, the present invention proposes the concept of Vehicle Potential Threat Level (VPTL), and applies it as a new dynamic adjustment mechanism to the complete constraints of the adversarial agent.

[0064] Existing methods usually rely on fixed proportional coefficients or preset weight hyperparameters to determine the difficulty of the adversarial agent generation strategy. However, this static weight allocation mechanism ignores the need for dynamic adaptation to the environment. Inspired by the concept of "zone of proximal development" in psychology, this paper proposes a new dynamic adjustment mechanism, which aims to adaptively change the difficulty of the adversarial agent's task for the ego vehicle by adjusting the proportion between the various constraints in real time, thereby promoting the improvement of the learning and adaptability of the ego vehicle agent.

[0065] Specifically, the present invention introduces a new concept of "vehicle threat potential" to quantify the degree of challenge of the current task environment to the ego agent. The vehicle threat potential is determined by the performance of the ego agent and the intervention capability of the adversary agent. For example, if the intervention capability of the adversary agent increases while the capability of the ego agent remains unchanged, the vehicle threat potential will increase. On the contrary, if the capability of the ego agent increases while the intervention capability of the adversary agent remains unchanged, the vehicle threat potential will decrease.

[0066] In order to more accurately evaluate the potential threat of the vehicle, the present invention mainly considers the success rate, collision rate and emergency avoidance step ratio of the vehicle in the previous stage as key factors. Every 1000 iterations constitute a stage, and VPTL is updated once in each stage. The calculation formula of the main part of VPTL is as follows:

[0067] VPTL′=0.3×(1-SR pre )+0.4×CR pre +0.3×EN pre

[0068] Among them, SR preis the success rate of the previous stage, CR pre is the collision rate in the previous stage, EN pre is the ratio of the number of steps that occurred in each round of the previous stage. The present invention defines that when the acceleration a≤-0.8×a max Emergency avoidance occurs.

[0069] In addition, the present invention also sets a dynamically changing baseline to ensure that the potential threat of the adversarial agent to the ego vehicle will gradually increase over time, avoiding the stagnation of the training effect after the two reach a certain capability threshold. The calculation formula of the VPTL baseline is as follows:

[0070]

[0071] Among them, n epi is the current iteration number, N epi is the total number of iterations. The design of this baseline helps maintain continuous progress in training, pushing the ego-car agent and the adversarial agent to higher difficulty levels when their capabilities are comparable. The value of VPTL is scaled to [0,1].

[0072] The final VPTL calculation formula is as follows:

[0073] VPTL=0.5×VPTL′+0.5×VPTL base

[0074] The present invention directly incorporates the potential threat of the vehicle into the loss constraint term of the agent, replacing the previous fixed proportional coefficient. The complete constraint of the updated adversarial agent is shown in Formula 2:

[0075] C adv =C personal +(1-VPTL)×C adversary +VPTL×C IRC

[0076] Among them, VPTL is the potential threat of the vehicle. The updated constraints not only reflect the actual difficulty changes, but also prompt the adversarial agent to adjust its interference strategy in time during the training process to adapt to the improvement of the ego vehicle agent's ability. This dynamic adjustment mechanism ensures that the challenges faced by the ego vehicle agent during the training process continue to exist, promoting its ability to continue to learn and adapt.

[0077] Furthermore, in order to further reduce collision damage and improve safety in complex scenarios, the present invention proposes the concept of collision severity and applies it to the loss function constraints of the ego-vehicle agent and the adversarial agent.

[0078] The present invention proposes a severity analysis method based on detailed collision scenarios, aiming to optimize autonomous driving decisions and improve the robustness of the system by conducting in-depth discussions on the details of collision events.

[0079] The research of the present invention not only focuses on whether a collision occurs, but also deeply considers the specific details of the collision, and uses the severity of the collision to describe the detailed collision situation. The factors involved include the collision position of the vehicle body, the collision angle, the collision speed, etc., thereby providing a more accurate collision risk assessment and strategy adjustment basis.

[0080] The vehicle body collision position refers to the specific location of the collision point on a vehicle body. The present invention divides the vehicle body into four parts, namely the front position, the rear position, the side close to the driver, and the side away from the driver. According to common sense and practical experience, the position of the driver is directly related to vehicle control and personal safety. Therefore, the present invention believes that when the vehicle body collision point is located close to the driver's side, the collision severity is higher than the other three vehicle body position areas.

[0081] The collision angle indicates the angle between the driving directions of the two vehicles when the collision occurs, and the collision angle ranges from -180° to 180°. Since the collision angle directly affects the energy transfer of the collision and the force applied to the vehicle, the present invention believes that for the same collision position of the vehicle body, the collision severity changes linearly with the angle; when a head-on collision occurs, that is, when the driving direction of one vehicle is facing another vehicle, the collision severity is the highest.

[0082] The collision speed indicates the speed of each of the two vehicles when the collision occurs. The collision speed directly determines the kinetic energy of the collision. Therefore, the present invention believes that under the same collision position and collision angle, the severity of the collision is proportional to the collision speed. The higher the speed of the vehicle when the collision occurs, the greater the kinetic energy and the greater the damage caused by the collision.

[0083] The specific calculation method of collision severity is as follows:

[0084]

[0085] Among them, S collision represents the severity of the collision, P represents the collision position of the vehicle body, α rel represents the collision angle, V represents the collision speed, V car1 、V car2 Represent the speeds of the two vehicles that collided, k vel represents the speed-related scaling factor, k loc is a weighting factor related to the collision position, g(P, α rel ) represents the values ​​corresponding to different combinations of collision position and collision angle. g(P, α) corresponding to collision position and collision anglerel ) values ​​are as follows:

[0086]

[0087] Through the above formula, the present invention can quantitatively evaluate the severity of the collision and incorporate this parameter into the decision-making process of the autonomous driving system, thereby optimizing the driving strategy of the autonomous driving vehicle and reducing the risk and loss of collision.

[0088] Experimental results:

[0089] We conducted comparative tests with five baseline methods, including DDPG, PPO, SAC, TD3, and GAIL. The test indicators include success rate (SR), average collision severity (CS), and damage index (DI). The success rate represents the proportion of successful tests in the total number of tests; the average collision severity represents the average collision severity of all collisions that occurred in the test of the method, and the collision severity of a single collision is calculated by the formula in the previous article. The damage index is an indicator we define, which indicates the degree of damage to the agent. It is calculated by the formula DI = CR + CS, where CR is the proportion of collisions in the total number of tests, and CS is the average collision severity. Among the test indicators, the higher the success rate, the better, and the lower the average collision severity (CS) and damage index, the better.

[0090] The comparison results are shown in the following table. From the results, we can see that our method is significantly better than other baseline methods in terms of the three indicators of success rate (SR), average collision severity (CS) and average damage index (DI), which reflects the superiority of our method.

[0091]

Claims

1. An autonomous driving adversarial learning method based on the guidance of the zone of proximal development, wherein during the adversarial learning training process, the method is characterized in that: include: The loss function of the agent designed for training is: c adv =α1×C personal +α2×C adversary +α3×C IRC Among them, C personal represents the adversarial constraint term, C adversary represents the personal constraint of the adversarial agent, C IRC represents the constraint item based on IRC, α1, α2, α3 are the corresponding weight factors; Among them, S collision，adv represents the severity of the collision of the adversarial agent, is the average speed of all adversarial agents within a certain range, v max，adv is the maximum speed of the adversarial agent; C adversary =S collision，ego Among them, S collision，ego represents the collision severity of the ego vehicle agent; C IRC =1-max(1,1.5×f(d,α,θ) adv ,v ego )) Among them, v ego and v max，ego are the speed and maximum speed of the vehicle, d i , α i and θ adv，i are the relative distance, relative angle, and velocity direction of the i-th adversarial agent relative to the ego vehicle.

2. The autonomous driving adversarial learning method based on the guidance of the zone of proximal development according to claim 1, characterized in that: The loss function designed for the ego-car agent is: C ego =β1×C safety +β2×C efficiency Among them, C safety represents the safety constraint, C efficiency represents the efficiency constraint term, β1 and β2 are the corresponding weight factors; 3. The autonomous driving adversarial learning method based on the guidance of the zone of proximal development according to claim 1 or 2, characterized in that: The loss function of the agent is specifically: C adv =C personal +(1-VPTL)×C adversary +VPTL×C IRC ; Where, VPTL = 0.5 × VPTL′ + 0.5 × VPTL base ; VPTL′=0.3×(1-SR pre )+0.4×CR pre +0.3×EN pre Among them, SR pre is the success rate of the previous stage, CR pre is the collision rate in the previous stage, EN pre is the ratio of the number of steps in which emergency avoidance occurred in each round of the previous stage, and each n iterations constitutes a stage; where n is the set number of iterations; Among them, n epi is the number of iterations in the current stage, N epi is the total number of iterations.

4. The autonomous driving adversarial learning method based on the guidance of the zone of proximal development according to claim 1, characterized in that: The collision severity S of the adversarial agent collision,adv =S collision ; Among them, S collision Indicates the severity of the vehicle collision: Among them, P represents the collision position of the vehicle body, α rel represents the collision angle, V represents the collision speed, V car1 、V car2 Represent the speeds of the two vehicles that collided, k vel represents the speed-related scaling factor, k loc is a weighting factor related to the collision position, g(P, α rel ) represents the collision position P and collision angle α rel The corresponding collision position angle factor k loc The value of g(P, α rel ) is calculated as shown in the following table:

5. The autonomous driving adversarial learning method based on the guidance of the zone of proximal development according to claim 1, characterized in that: The collision severity of the ego vehicle agent is S collision,ego =S collision ; Among them, S collision Indicates the severity of the vehicle collision: Among them, P represents the collision position of the vehicle body, α rel represents the collision angle, V represents the collision speed, V car1 、V car2 Represent the speeds of the two vehicles that collided, k vel represents the speed-related scaling factor, k loc is a weighting factor related to the collision position, g(P, α rel ) indicates that the collision position P is below and at the collision angle α rel The corresponding values ​​are shown in the following table:

Images