Automatic driving test scene generation method based on constraint counter-reinforcement learning

Abstract

The application provides an automatic driving test scene generation method based on constraint adversarial reinforcement learning, comprising: constructing a constraint adversarial learning framework for target dynamic obstacles around a self-driving host vehicle guided by an attacker identification mechanism; taking a plurality of initial scene state vectors as training samples of the constraint adversarial learning framework, combining environmental interaction, a multi-dimensional constraint system for target dynamic obstacles and a composite reward function, iteratively training the constraint adversarial learning framework to obtain a target constraint adversarial learning framework; and adjusting actions of a given scene state vector through the target constraint adversarial learning framework to obtain a safe scene state vector for training the self-driving host vehicle, wherein the safe scene state vector corresponds to a safety critical scene. The application introduces the constraint adversarial learning framework guided by the attacker identification mechanism to improve the generation efficiency of the safety critical scene and improve the high fidelity and rationality of the scene.

Description

Technical Field

[0001] This application relates to the field of autonomous driving testing technology, and in particular to a method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning. Background Technology

[0002] As autonomous driving safety verification gradually shifts from real-vehicle testing to simulation testing, existing publicly available technologies have evolved from scenario construction based on rule scripts and fixed parameter configurations to the generation of safety-critical scenarios based on reinforcement learning, imitation learning, and adversarial optimization. However, autonomous driving scenarios inherently possess the core characteristics of multiple stakeholders, high-dimensional states, and strong interactions: multiple stakeholders mean that the scenario includes multiple independent decision-making entities such as the driver vehicle, background vehicles, pedestrians, and non-motorized vehicles, whose behaviors influence and interact with each other; high-dimensional states mean that the scenario state includes parameters in multiple dimensions such as vehicle position, speed, heading angle, road topology, and environmental conditions, resulting in a vast state space; strong interactions mean that the behaviors of each stakeholder are correlated in real time, and changes in the behavior of a single stakeholder can trigger dynamic adjustments to the entire scenario.

[0003] These characteristics make it difficult to effectively balance risk (the probability of triggering the failure of the autonomous driving system), rationality (compliance with traffic rules and physical laws), and generation efficiency (rapid batch generation of scenes) during the scene generation process. Existing technologies cannot fully balance the relationship between the three, which in turn leads to the inability to effectively meet the requirements of scene generation efficiency, scene generation rationality, and scene generation targeting. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide at least one method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning. By introducing a constraint adversarial learning framework guided by an attacker identification mechanism, the method can improve the generation efficiency of safety-critical scenarios and enhance the high fidelity and rationality of the scenarios.

[0005] This application mainly includes the following aspects: In a first aspect, embodiments of this application provide a method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning. The method includes: determining an initial scenario state vector for the autonomous vehicle at each moment based on multiple real autonomous driving data of the autonomous vehicle, wherein the scenario state vector represents the lane network features, traffic facility features, dynamic obstacle features, autonomous vehicle features, and environmental features at the corresponding moment; constructing a constraint adversarial learning framework for dynamic obstacles around the autonomous vehicle, guided by an attacker identification mechanism, wherein the input of the constraint adversarial learning framework is the scenario state vector, and the output of the constraint adversarial learning framework is a simulated scenario state vector; using multiple initial scenario state vectors indicated by multiple real driving data as training samples for the constraint adversarial learning framework, and combining environmental interaction, a multi-dimensional constraint system for dynamic obstacles, and a composite reward function to iteratively train the constraint adversarial learning framework to obtain a trained target constraint adversarial learning framework; inputting a given scenario state vector into the target constraint adversarial learning framework, and adjusting the actions of the given scenario state vector through the target constraint adversarial learning framework to obtain a safe scenario state vector for training the autonomous vehicle, wherein the safe scenario state vector corresponds to a safety-critical scenario.

[0006] In one possible implementation, the scene state vector corresponding to the autonomous vehicle is determined as follows: Road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental state data are extracted from real autonomous driving data; data preprocessing is performed on the road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental state data respectively to obtain preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental state data; data modeling is performed on the preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental state data to form structured scene data; multi-dimensional attribute information is extracted from the structured scene data to map the structured scene data to a high-dimensional state space, obtaining the high-dimensional scene state vector corresponding to each time step; the high-dimensional scene state vector is subjected to dimensionality reduction processing based on normalization according to the attribute information of each dimension to obtain the processed scene state vector.

[0007] In one possible implementation, the high-dimensional scene state vector for each moment is determined as follows: Geometric attributes and topological relationship attributes corresponding to the key nodes of the lane to which the autonomous vehicle belongs at that moment are extracted from the set of key lane nodes, the set of lane connection relationships, and the set of lane attributes described by the road network data, forming the lane network state vector for that moment; spatial distribution attributes and state attributes corresponding to target traffic facilities within a preset calibration range centered on the autonomous vehicle at that moment are extracted from the multiple traffic facilities described by the traffic facility data and the attribute information corresponding to each traffic facility, forming the traffic facility state vector for each target traffic facility at that moment; and the spatial distribution attributes and state attributes corresponding to target traffic facilities within a preset calibration range centered on the autonomous vehicle at that moment are extracted from the multiple target dynamic obstacles around the autonomous vehicle described by the dynamic obstacle operation data and the attribute information corresponding to each target obstacle are extracted from the dynamic obstacle operation data. From the attribute information corresponding to the dynamic obstacles, the state attributes corresponding to each target dynamic obstacle are extracted to form the dynamic obstacle state vector corresponding to each target dynamic obstacle at that moment; from the autonomous vehicle operation data, the state attributes of the autonomous vehicle at that moment are extracted to form the autonomous vehicle state vector at that moment; from the environmental state data, the target environmental attribute information of the external environment in which the autonomous vehicle is located at that moment that affects driving safety and trajectory decision is extracted to form the environmental state vector at that moment; from the lane network state vector corresponding to each lane key node, the traffic facility state vector corresponding to each traffic facility, the dynamic obstacle state vector corresponding to each dynamic obstacle, the autonomous vehicle state vector, and the environmental state vector at that moment, a high-dimensional scene state vector corresponding to that moment is formed.

[0008] In one possible implementation, the constraint adversarial learning framework includes an adversarial generative agent, a constraint evaluator, an environment interaction network, an experience replay pool, and a composite reward constraint function. The adversarial generative agent includes an action network and an evaluation network. The input to the action network is an initial scene state vector and a target attacker identified by an attacker identification mechanism. The target attacker is at least one of the target dynamic obstacles. The output of the action network is a simulated action vector corresponding to each target dynamic obstacle. The input to the evaluation network is the initial scene state vector and the simulated action vector corresponding to each target dynamic obstacle. The output of the evaluation network is an action value assessment of the simulated action vector of each target obstacle. The environment interaction network is used to simulate the interaction process between the target dynamic obstacles and the autonomous vehicle. The network input is a combination of the input and output of the action network. The output of the environment interaction network is the simulation scene state vector and the safety-critical parameters between each target dynamic obstacle and the autonomous vehicle. The constraint estimator adopts a multi-dimensional constraint system. The input of the constraint estimator is the simulation scene state vector and safety-critical parameters output by the environment interaction network. The output of the constraint estimator is the comprehensive constraint evaluation index corresponding to each target dynamic obstacle. The experience replay pool stores multiple training sample data generated during the training of the constraint adversarial learning framework. The input of the composite reward function is the simulation action vector corresponding to the target dynamic obstacle and its corresponding safety-critical parameters, comprehensive constraint evaluation index, and historical trajectory data. The output of the composite reward function is the composite reward index corresponding to each target dynamic obstacle.

[0009] In one possible implementation, the constraint evaluator includes a safety-critical constraint function, a physical constraint function, a traffic rule constraint function, and a comprehensive constraint evaluation function. The constraint evaluator obtains the comprehensive constraint evaluation index as follows: It acquires the safety-critical parameters corresponding to the target dynamic obstacle output by the environmental interaction network and inputs them into the safety-critical constraint function to obtain the safety-critical constraint index corresponding to the target dynamic obstacle; it inputs the velocity, acceleration, and steering parameters corresponding to the target dynamic obstacle indicated by the simulation scene state vector into the physical constraint function to obtain the physical constraint index corresponding to the target dynamic obstacle; it inputs the legal form result, traffic light indication compliance result, and yield rule compliance result of the target dynamic obstacle indicated by the simulation scene state vector into the traffic rule constraint function to obtain the traffic rule constraint index; and it performs a weighted calculation on the safety-critical constraint index, physical constraint index, and traffic rule constraint index into the comprehensive constraint evaluation function to obtain the comprehensive constraint index corresponding to each target dynamic obstacle. The comprehensive constraint evaluation index reflects the rationality of the target dynamic obstacle's action or trajectory in its respective scenario state.

[0010] In one possible implementation, the composite reward index corresponding to each target dynamic obstacle is determined as follows: The collision time and minimum collision distance corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent are input into the adversarial reward function to obtain the adversarial reward index of the target dynamic obstacle. This adversarial reward index is used to encourage the adversarial agent corresponding to the target dynamic obstacle to generate behaviors that are more threatening to the autonomous vehicle. The comprehensive constraint evaluation index corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent is input into the constraint reward function to obtain the constraint reward index of the target dynamic obstacle. This constraint reward index is used to encourage the actions or trajectories generated by the adversarial agent corresponding to the target dynamic obstacle to satisfy physical constraints and traffic rule constraints. The simulated action vector of the target dynamic obstacle output by the adversarial agent and multiple historical actions corresponding to the time of the simulated action vector are input into the diversity reward function to obtain the diversity reward index of the target dynamic obstacle. This diversity reward constraint index is used to suppress the homogenization of actions or trajectories generated by the adversarial agent. The adversarial reward index, constraint reward index, and diversity reward index are weighted and calculated to obtain the composite reward index.

[0011] In one possible implementation, the attacker identification mechanism includes: inputting the collision time, relative motion direction, and relative motion distance between the target dynamic obstacle and the autonomous vehicle into an attacker identification function; determining the attacker identification result of the target dynamic obstacle output by the attacker identification function; the attacker identification function is defined as follows: when the collision time is less than the collision time identification threshold, the target dynamic obstacle is located in a high-risk direction of the autonomous vehicle and has a tendency to approach the autonomous vehicle, the target dynamic obstacle is identified as a target attacker; based on the attacker identification result, at least one target attacker is identified and fed back to the action network of the adversarial generative agent.

[0012] In one possible implementation, the constrained adversarial learning framework is trained as follows: Multiple initial scene state vectors corresponding to the structured scene data are divided into multiple training batches; for each training batch, the following steps are performed: the adversarial agent is trained and optimized using the multiple initial scene state vectors corresponding to that training batch and the training samples stored in the experience replay pool, resulting in the optimized adversarial agent and the simulation scene state vectors corresponding to each initial scene state vector output by the environment interaction network during the adversarial agent optimization process; after the training of that batch of the adversarial agent is completed, the multiple simulation scene state vectors corresponding to that training batch and the simulation scene state vectors are collected. The constraint estimator is iteratively trained using the state vector of each simulation scenario as input and the constraint satisfaction status of each state vector as supervision label, based on the constraint satisfaction status of the constraint terms provided by the constraint estimator. This results in an optimized constraint estimator, which is then used to update the current constraint estimator applied by the constraint adversarial learning framework. The composite reward index, safety-critical scenario generation rate, and constraint satisfaction rate corresponding to the state vector of the simulation scenario are statistically analyzed. Based on the composite reward index corresponding to the state vector of the simulation scenario, the safety-critical scenario generation rate, and constraint satisfaction rate under the training batch, it is determined whether the training of the constraint adversarial learning framework is complete. In one possible implementation, the completion of training of the constraint adversarial learning framework is determined as follows: For each training batch, the following is executed: if the fluctuation range of each composite reward index under a consecutive preset number of simulation scenario state vectors is less than a preset value, the safety-critical scenario generation rate reaches a preset standard, and the constraint satisfaction rate reaches a preset standard, then the constraint adversarial learning framework is determined to have converged and training is stopped; if the number of updated scenario dynamic vectors whose fluctuation range of each composite reward index is less than a preset value does not meet the preset number requirement, the safety-critical scenario generation rate does not reach the preset standard, and / or the constraint satisfaction rate does not reach the preset standard, then the next training batch is started for iterative training.

[0013] In one possible implementation, the training steps of the adversarial generative agent (PGA) in each training batch include: for each initial scene state vector, performing the following steps: inputting the initial scene state vector into the GPA to obtain the initial output action vector of each target dynamic obstacle output by the GPA; obtaining the initial composite reward index corresponding to the initial scene state vector and its corresponding initial output action vector under the action of the environment interaction network, constraint evaluator, and composite reward function; filtering out at least one target attacker from the multiple target dynamic obstacles indicated by the initial scene state vector through an attacker identification mechanism and feeding it back to the GPA; and using the GPA to respond to the actions of the target attacker. The trajectory is modified to obtain the simulated action vector of each target dynamic obstacle. Under the action of the environment interaction network, constraint evaluator and composite reward function, the target composite reward index corresponding to the simulated scene state vector and its corresponding simulated action vector is obtained. The initial scene state vector and its corresponding initial output action vector and initial composite reward index of each target dynamic obstacle, as well as the simulated scene state vector and its corresponding simulated action vector and target composite reward index of each target dynamic obstacle, are integrated into training samples and stored in the experience replay pool. Multiple training samples are sampled from the experience replay pool to update the network parameters of the adversarial generative agent, thereby obtaining the updated adversarial generative agent.

[0014] In one possible implementation, the method further includes: inputting the safety critical constraint index corresponding to the safety scenario state vector into a safety critical verification function for evaluation to obtain the corresponding safety critical verification index; if the safety critical verification index indicates that the safety scenario state vector meets the safety critical constraints, then performing rationality optimization on the trajectory data of the target dynamic obstacle indicated by the safety scenario state vector to obtain an optimized safety scenario state vector; extracting data from the optimized scenario state vector according to a preset standard format to obtain target safety critical scenario data; if the safety critical verification index indicates that the safety scenario state vector does not meet the safety critical constraints, then returning to re-execute the training of the constraint adversarial learning framework.

[0015] In one possible implementation, the step of optimizing the trajectory data of the target dynamic obstacle indicated by the safety scene state vector includes: smoothing the action sequence indicated by the trajectory data of the target dynamic obstacle using a moving average filter; obtaining an optimized action sequence of the target dynamic obstacle; updating the safety scene state vector using the optimized action sequence to obtain an optimized safety scene state vector; and correcting the optimized action sequence of the optimized safety scene state vector according to rules to obtain the target safety scene state vector.

[0016] This application provides a method for generating autonomous driving test scenarios based on constraint adversarial learning, comprising: constructing a constraint adversarial learning framework guided by an attacker identification mechanism for dynamic obstacles around an autonomous vehicle; using multiple initial scene state vectors indicated by multiple real driving data as training samples for the constraint adversarial learning framework, and iteratively training the constraint adversarial learning framework by combining environmental interactions, a multi-dimensional constraint system for dynamic obstacles, and a composite reward function to obtain a trained target constraint adversarial learning framework; inputting a given scene state vector into the target constraint adversarial learning framework, and adjusting the actions of the given scene state vector through the target constraint adversarial learning framework to obtain a safety scene state vector used for training the autonomous vehicle, wherein the safety scene state vector corresponds to a safety-critical scenario. This application improves data generation efficiency and enhances the high fidelity and rationality of scenarios by introducing a constraint adversarial learning framework guided by an attacker identification mechanism for generating safety-critical scenarios for autonomous vehicle testing.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning, provided in an embodiment of this application, is shown. Figure 2 A flowchart illustrating a scene state vector generation method provided in an embodiment of this application is shown; Figure 3 A schematic diagram illustrating the training process of a constraint adversarial learning framework provided in an embodiment of this application is shown. Figure 4 This paper illustrates a functional block diagram of an autonomous driving test scenario generation device based on constraint adversarial reinforcement learning, according to an embodiment of this application. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0021] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] As autonomous driving safety verification gradually shifts from real-vehicle testing to simulation testing, existing publicly available technologies have evolved from scenario construction based on rule scripts and fixed parameter configurations to the generation of safety-critical scenarios based on reinforcement learning, imitation learning, and adversarial optimization. Among these, publicly available solutions closely related to the technical solution provided in this application include key scenario generation methods based on reinforcement learning to optimize dynamic environmental element parameters, and adversarial reinforcement learning generation methods combining natural driving priors or scenario editing. These solutions can simulate dangerous interaction scenarios in real roads by adjusting the behavior of background vehicles and other traffic participants in a simulation environment, thereby generating test scenarios with a certain degree of risk. However, existing technologies still have shortcomings and are insufficient to meet the actual needs of high-level autonomous driving safety verification. First, the identification and targeted intervention of key threat targets are insufficient. Most solutions only generate risk scenarios by optimizing global parameters, without clearly identifying the core threat targets in the scenarios (such as vehicles cutting in line or pedestrians suddenly appearing out of nowhere), and without designing targeted behavioral intervention strategies for such threat targets. As a result, although the generated scenarios have certain risks, they are difficult to accurately reach the vulnerabilities of the autonomous driving system.

[0023] Secondly, the rationality of the generated scenarios relies heavily on prior models or single-objective constraints. Some solutions over-rely on prior knowledge extracted from natural driving data and lack dynamic adaptation to real-time traffic rules and vehicle dynamics characteristics. Some solutions only take "maximizing risk" as the single objective, ignoring the continuity of the scenario trajectory, the consistency of traffic rules, and physical feasibility. This results in some generated scenarios having problems such as sudden changes in vehicle trajectory, violations of traffic rules, and exceeding vehicle dynamics limits.

[0024] Third, there are also issues such as strong data dependence, long training links, and most solutions requiring a large amount of high-quality natural driving data as a training basis, resulting in high data acquisition and preprocessing costs. At the same time, the training process of reinforcement learning or adversarial optimization is complex, with long iteration cycles, making it difficult to quickly generate test scenarios that meet the requirements.

[0025] Fourth, the engineering deployment is highly complex. The existing solutions have complex algorithm logic, are difficult to debug parameters, and have poor compatibility with mainstream simulation testing platforms, making it difficult to quickly deploy and apply them to actual engineering testing scenarios.

[0026] A deeper analysis of the root causes of these shortcomings reveals that autonomous driving scenarios inherently possess the core characteristics of multiple stakeholders, high-dimensional states, and strong interactions: Multiple stakeholders mean that the scenario includes multiple independent decision-making entities such as the driver vehicle, background vehicles, pedestrians, and non-motorized vehicles, whose behaviors influence and interact with each other; high-dimensional states mean that the scenario state includes parameters across multiple dimensions such as vehicle position, speed, heading angle, road topology, and environmental conditions, resulting in a vast state space; strong interactions mean that the behaviors of each stakeholder are correlated in real time, and changes in the behavior of a single entity can trigger dynamic adjustments to the entire scenario. These characteristics make it difficult to effectively balance risk (the probability of triggering autonomous driving system failure), rationality (compliance with traffic rules and physical laws), and generation efficiency (rapid batch generation of scenarios) during scenario generation. Existing technologies cannot adequately balance these three aspects, leading to the aforementioned problems. Therefore, a novel scenario generation technology solution is urgently needed to address the pain points of existing technologies and meet the practical needs of high-level autonomous driving safety verification.

[0027] Based on this, this application provides a method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning. By introducing a constraint adversarial learning framework guided by an attacker identification mechanism, the method improves the generation efficiency of safety-critical scenarios and enhances the high fidelity and rationality of the scenarios, as detailed below: Please see Figure 1 , Figure 1 A flowchart illustrating a method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning, as provided in an embodiment of this application, is shown. Figure 1 As shown, the method provided in this application embodiment includes the following steps: S100: Based on multiple real-world autonomous driving data, determine the scene state vector of the autonomous driving master vehicle at each moment.

[0028] The autonomous vehicle is an automated driving vehicle, and the scene state vector represents the lane network characteristics, traffic facility characteristics, dynamic obstacle characteristics, autonomous vehicle characteristics, and environmental characteristics at the corresponding moment.

[0029] S200. Construct a constraint adversarial learning framework for dynamic obstacles around the autonomous vehicle, guided by an attacker identification mechanism.

[0030] The input to the constrained adversarial learning framework is the scene state vector, and the output of the constrained adversarial learning framework is the simulation scene state vector.

[0031] S300: Multiple initial scene state vectors indicated by multiple real driving data are used as training samples for the constraint adversarial learning framework. Combined with environmental interaction, a multi-dimensional constraint system for target dynamic obstacles, and a composite reward function, the constraint adversarial learning framework is iteratively trained to obtain a trained target constraint adversarial learning framework.

[0032] S400. Input the given scene state vector into the target constraint adversarial learning framework, and adjust the action of the given scene state vector through the target constraint adversarial learning framework to obtain a safe scene state vector for training the autonomous vehicle.

[0033] The state vector of a safety scenario corresponds to a safety-critical scenario.

[0034] In specific implementation, in step S100, a small amount of real self-driving data is collected. Each piece of real self-driving data corresponds to a driving scenario. The real self-driving data includes, but is not limited to, at least one of the following: road network data L, traffic facility data A, dynamic obstacle operation data O, self-driving vehicle operation data E, and environmental status data C within a specified collection time period.

[0035] In a preferred embodiment, please refer to Figure 2 , Figure 2 A flowchart illustrating a scene state vector generation method provided in an embodiment of this application is shown. Figure 2 As shown, step S100 includes: S1100 extracts road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data from real-world autonomous driving data.

[0036] S1200 performs data preprocessing on road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data respectively, to obtain preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data.

[0037] Specifically, data preprocessing includes data cleaning, outlier and noise removal, and interpolation completion for missing values.

[0038] S1300 uses preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data to create structured scene data through data modeling.

[0039] The preprocessed road network data L, traffic facility data A, dynamic obstacle operation data O, autonomous vehicle operation data E, and environmental status data C are time-aligned. After alignment, data modeling is performed to form structured scene data.

[0040] Specifically, the structured scene data S is defined as:

[0041] The road network data L can be described using a lane graph structure, which specifically includes a set of key lane nodes, a set of lane connection relationships, and a set of lane attributes.

[0042] Traffic facility information A includes multiple traffic facilities and the attribute information corresponding to each traffic facility. Furthermore, the multiple traffic facilities include, but are not limited to, at least one of the following: traffic lights, stop lines, zebra crossings, traffic signs, streetlights, guardrails, medians, and construction barriers. The attribute information corresponding to the traffic facilities includes location, orientation, size, and facility type.

[0043] The dynamic obstacle operation data O includes a set of dynamic obstacles, which includes multiple dynamic obstacles other than the autonomous vehicle. Examples of dynamic obstacles are mobile traffic participants such as vehicles, motorcycles, and pedestrians. The attribute information of the dynamic obstacles includes, but is not limited to, at least one of the following: position, speed, acceleration, heading angle, and category parameter description.

[0044] The autonomous vehicle operation data E describes the vehicle status of the autonomous vehicle. The autonomous vehicle operation data includes, but is not limited to, at least one of the following: position, speed, acceleration, heading angle, initial state, target position, and planned task information.

[0045] Environmental status data describes the external environmental data of the scene in which the autonomous vehicle is located, including but not limited to at least one of the following: ambient temperature, weather category, and relevant roadside parameters.

[0046] S1400. Extract multi-dimensional attribute information from the structured scene data to map the structured scene data to a high-dimensional state space, and obtain the high-dimensional scene state vector corresponding to each time step.

[0047] In a preferred embodiment, step S1400 further includes: From the set of key lane nodes, lane connection relationships, and lane attributes described by the lane network data, the geometric and topological attributes corresponding to the key lane nodes of the autonomous vehicle at that moment are extracted to form the lane network state vector at that moment. From the multiple traffic facilities described by the traffic facility data and the attribute information corresponding to each traffic facility, the spatial distribution and state attributes of each traffic facility in the scene at that moment are extracted to form the traffic facility state vector of each traffic facility at that moment. From the multiple dynamic obstacles around the autonomous vehicle described by the dynamic obstacle operation data and the attribute information corresponding to each dynamic obstacle, the state attributes corresponding to each dynamic obstacle are extracted. This process generates a dynamic obstacle state vector for each dynamic obstacle at that moment. From the autonomous vehicle's operational data, the state attributes of the autonomous vehicle at that moment are extracted to form its state vector. From the environmental state data, the target environmental attribute information that influences driving safety and trajectory decision-making in the external environment at that moment is extracted to form its environmental state vector. Finally, from the lane network state vector corresponding to each key node in each lane, the traffic facility state vector corresponding to each traffic facility, the dynamic obstacle state vector corresponding to each dynamic obstacle, the autonomous vehicle state vector, and the environmental state vector, a high-dimensional scene state vector is formed for that moment.

[0048] Based on the above step S1400, the high-dimensional scene state vector corresponding to time t is... Defined as:

[0049] in, The vector represents the lane network state at time t, which further includes a geometric parameter subvector and a topological relation subvector. The elements of the geometric parameter subvector include the lane centerline coordinates, lane boundary coordinates, lane heading angle, lane curvature, lane width, lane length, lane type, and speed limit at the current position of the autonomous vehicle. The elements of the topological relation subvector include the connection relationships between the current lane of the autonomous vehicle and its preceding lane, succeeding lane, left adjacent lane, and right adjacent lane, as well as merging relationships, branching relationships, intersection affiliation relationships, and path connectivity relationships.

[0050] This represents the state matrix of the transportation facilities at time t. This includes the traffic facility state vector corresponding to each target traffic facility within a preset calibration range centered on the autonomous vehicle at time t. For example, if the target traffic facilities at time t include traffic lights, traffic signs, and traffic markings, then the traffic facility state vector at time t further includes traffic light state sub-vectors, traffic sign state sub-vectors, and traffic marking state sub-vectors. The elements of the traffic light state sub-vector include the signal light position coordinates, orientation angle, signal light type, lane in effect, current phase state, and remaining phase time. The traffic sign state sub-vector includes the sign position coordinates, orientation angle, sign type, speed limit value, priority rule attribute, and applicable lane. The traffic marking state sub-vector includes the marking start and end coordinates, marking type, marking attribute, and corresponding lane or area information.

[0051] This represents the dynamic obstacle state matrix at time t. This includes the dynamic obstacle state vector corresponding to each target dynamic obstacle located within the perimeter of the autonomous driving master vehicle at time t. The elements of the dynamic obstacle state vector include position, coordinates, velocity, acceleration, and heading angle.

[0052] This represents the autonomous vehicle's state vector at time t. The elements of the autonomous vehicle's state vector include its own position, velocity, acceleration, and target path information.

[0053] The environmental state vector represents the environmental elements that affect driving safety and trajectory decision-making in the external environment of the autonomous vehicle at time t. The environmental state vector includes meteorological environment vector, road surface environment vector, and visible environment vector. The elements of the meteorological environment vector include weather type, precipitation intensity, wind speed, and ambient temperature. The elements of the road surface environment vector include road adhesion coefficient, road surface condition, water depth, and road slope. The elements of the visible environment vector include visibility distance, light intensity, and degree of occlusion.

[0054] S1500. Perform dimensionality reduction processing on the high-dimensional scene state vector according to the attribute information of each dimension, and obtain the processed scene state vector.

[0055] In one specific embodiment, to improve the efficiency of subsequent model training, this application further normalizes the high-dimensional scene state vector, specifically referring to the following formula:

[0056] In one specific embodiment, This represents the scene state vector at time t after normalization. This includes the minimum values ​​of each dimension of attribute information within the sampling time corresponding to real self-driving data. Including the maximum values ​​of each dimension of attribute information within the sampling time corresponding to the real self-driving data, the scene state vector corresponding to time t is calculated according to the above formula (3). Mapping to a unified interval yields This is to avoid the impact of numerical differences in different dimensions on model training.

[0057] In a preferred embodiment, the constraint adversarial learning framework in step S200 includes an adversarial generative agent, a constraint evaluator, an environment interaction network, and an experience replay pool.

[0058] Preferably, step S200 further includes: The adversarial generative agent, constraint evaluator, environment interaction network, and experience replay pool are initialized, and the constraint adversarial learning framework is built from the adversarial generative agent, attacker identification module, constraint evaluator, environment interaction network, and experience replay pool.

[0059] The input to the attacker identification module is initialized to [value], and the output is initialized to [target attacker], where the target attacker is at least one of the target dynamic obstacles.

[0060] In a specific embodiment, the adversarial generative agent includes an action network and an evaluation network designed with a fully connected neural network architecture. The initialization process of the adversarial generative agent includes: initializing the input of the action network as a scene state vector and a target attacker; initializing the output of the action network as an action vector matrix, which includes simulated action vectors of each target dynamic obstacle; initializing the input of the evaluation network as a scene state vector and simulated action vectors of each target obstacle; initializing the output of the evaluation network as an action value assessment of the simulated action vectors of each target obstacle; and the output of the evaluation network serves as feedback for the decision network of each target dynamic agent in the action network.

[0061] In this application, the elements of the action vector include, but are not limited to, at least one of the following: change in velocity, change in acceleration, and change in heading angle. For example, the action vector of the target dynamic obstacle at time t is defined as follows:

[0062] in, Let O be the motion vector of the target dynamic obstacle O at time t. This represents the change in velocity of the target dynamic obstacle O at time t. This represents the change in acceleration of the target dynamic obstacle O at time t. This represents the change in the heading angle of the target dynamic obstacle O at time t.

[0063] In a specific embodiment, the network structure of the action network includes: a data input layer, a first hidden layer, a second hidden layer, and an action output layer. The data input layer has the same dimension as the normalized scene state vector and is used to receive the scene state vector. The first and second hidden layers extract features from the received scene state vector. The first hidden layer uses a high-dimensional representation and ReLU activation, while the second hidden layer uses a mid-dimensional representation and ReLU activation. The feature-extracted data is input into the action output layer. The action output layer includes multiple output processing units (each corresponding to a target dynamic obstacle, and each output processing unit corresponds to the action policy network of the target dynamic obstacle) and a Tanh activation function to map each action element of the action vector to the corresponding range, ensuring the physical rationality of the output action. Each output processing unit, as the action policy output network, also receives the attacker identification result output by the attacker identification module, so that the output processing unit corresponding to the target dynamic obstacle identified as the target attacker adjusts the output action.

[0064] In this application, the evaluation network is used to evaluate the action value corresponding to each simulated action vector of the target dynamic obstacle output by the action network, so as to provide guidance for updating the action network parameters (specifically, the action policy network of the target dynamic being the agent). Specifically, it includes: a data input layer, a first hidden layer and a second hidden layer for feature extraction, and a one-dimensional linear output layer. The input dimension of the data input layer is composed of the scene state vector dimension and the action vector dimension. The first hidden layer is a high-dimensional ReLU activation, and the second hidden layer is a mid-dimensional ReLU activation. The one-dimensional linear output layer receives the features passed by the second hidden layer and gives an estimate of the action value corresponding to each simulated action vector input by the action network.

[0065] In this application, the environmental interaction network is used to simulate the interaction process between the target dynamic obstacle and the autonomous vehicle. The environmental interaction network performs interaction simulation based on the scene state vector of the input action network and the simulation action vector of each target dynamic obstacle output by the action network, thereby updating the input scene state vector to the simulation scene state vector and the safety key parameters corresponding to each target dynamic obstacle. The safety key parameters include, but are not limited to, at least one of the following: collision time and minimum collision distance.

[0066] The constraint evaluator uses a fully connected neural network. The initialization process of the constraint evaluator includes: initializing the input of the constraint evaluator to the simulation scene state vector output by the environment interaction network, and initializing the output of the constraint evaluator to the comprehensive constraint evaluation index of the simulation scene state vector. The constraint evaluator adopts a multi-dimensional constraint system, which includes safety-critical constraint functions, physical constraint functions, traffic rule constraint functions, and comprehensive constraint evaluation functions.

[0067] In a specific embodiment, the constraint evaluator calculates the comprehensive constraint evaluation index of the target dynamic obstacle under the current scene state vector based on the input initial scene state vector, safety-critical constraint function, physical constraint function, traffic rule constraint function, and comprehensive constraint evaluation function.

[0068] In one possible implementation, the safety-critical constraint function is a constraint rule constructed based on safety-critical parameters. These safety-critical parameters include, but are not limited to, at least one of the following: the collision time (TTC) and minimum collision distance (MDC) between the autonomous vehicle and the target dynamic obstacle. The TTC and MDC are output by the environment interaction network. The safety-critical constraint function is used to determine the safety-critical constraint indicators of the target dynamic obstacle. The TTC and MDC are used to measure the risk level of the generated safety-critical scenario to the autonomous vehicle. By establishing corresponding safety-critical constraints, it can be ensured that the generated safety-critical scenario has a high risk.

[0069] Furthermore, the collision time between each target dynamic obstacle and the autonomous vehicle at time t. Defined as:

[0070] in, This indicates the position of the self-driving vehicle at time t. This indicates the position of the target dynamic obstacle at time t. This represents the Euclidean distance between the driver vehicle and the target dynamic obstacle. This represents the velocity vector of the target dynamic obstacle at time t. This represents the velocity vector of the autonomous vehicle at time t. This represents the unit vector along the line connecting the autonomous vehicle and the target dynamic obstacle.

[0071] The minimum collision distance between each target dynamic obstacle and the autonomous vehicle at time t. Defined as:

[0072] in, Indicates the current moment. This indicates the prediction time window.

[0073] This application exemplarily uses the above-mentioned collision time TTC and minimum collision distance MDC to construct the following safety-critical constraint function:

[0074] in, Indicates safety-critical constraints. Indicates the collision time threshold. This represents the minimum collision distance threshold, preferably, exemplary. and The determination can be achieved through the following two implementation methods: one of the implementation methods is: and The determination is based on the road type, relative closing speed, road surface adhesion coefficient, visibility, obstacle category, and outer contour dimensions of the target dynamic obstacle; different times correspond to different... and Another implementation method is: to Described as the sum of the base response time and the environmental correction time and obstacle type correction time, The value is described as the sum of the envelope collision boundary distance, dynamic safety distance, error compensation distance, and environmental correction distance. The base response time, environmental correction time, obstacle type correction time, envelope collision boundary distance, dynamic safety distance, error compensation distance, and environmental correction distance are pre-calibrated to represent the values ​​at different times in this scenario. and They are all the same.

[0075] For example, the collision time of the target dynamic obstacle at time t. and minimum collision distance Substituting the above safety-critical constraint function, we obtain the safety-critical constraint index corresponding to the target dynamic obstacle at time t.

[0076] In one possible implementation, the physical constraint function is a constraint rule constructed based on the velocity, acceleration, and turning of the target dynamic obstacle. It is used to determine the physical constraint index of the target dynamic obstacle and ensure that the trajectory of the target dynamic obstacle conforms to the laws of real physical kinematics.

[0077] Preferably, the physical constraint function further includes a velocity constraint function, an acceleration constraint function, and a steering constraint function. The velocity constraint function is used to determine the velocity constraint index of the target dynamic obstacle, the acceleration constraint function is used to determine the acceleration constraint index of the target dynamic obstacle, and the steering constraint function is used to determine the steering constraint index of the target dynamic obstacle.

[0078] Preferably, the physical constraint index of the target dynamic obstacle is the product of the target dynamic obstacle's velocity constraint index, acceleration constraint index, and steering constraint index, that is:

[0079] in, Indicates physical constraint index, Indicates speed constraint index, Indicates acceleration constraint index, This indicates the steering constraint indicator.

[0080] Preferably, the velocity constraint function is defined as:

[0081] in, Indicates speed constraint index, This represents the minimum speed threshold corresponding to the target dynamic obstacle. This represents the maximum speed threshold corresponding to the target dynamic obstacle. and The location of the target obstacle is dynamically determined based on the road type, obstacle type, traffic facility status, and external environment.

[0082] In one specific embodiment, different road types correspond to different base speed ranges. First, based on the road type where the target dynamic obstacle is located, the target base speed range (including the upper limit and lower limit of the base speed) corresponding to the target dynamic obstacle is determined. For example, the road type includes, but is not limited to, at least one of the following: parking lot, park road, residential area road, urban ordinary road, urban expressway, highway main line, ramp and intersection guide lane. Then, the determined upper limit of the base speed is corrected according to the obstacle type of the target dynamic obstacle to obtain the obstacle type corrected upper limit of the speed. Different obstacle types correspond to different correction coefficients.

[0083] The base speed limit is adjusted according to the condition of traffic facilities to obtain the traffic rule-corrected speed limit. The condition of the traffic facilities in the lane where the target dynamic obstacle is located is used as the basis for adjusting the base speed limit. For example, when the traffic facilities are red lights, yield signs or stop signs, the base speed limit is adjusted to the range of stopping or slow passage to obtain the traffic rule-corrected speed limit.

[0084] The base speed limit is adjusted according to the external environment conditions to obtain the environment-corrected speed limit. For example, the base speed limit is adjusted according to the road curvature, road surface adhesion coefficient and environmental visibility of the lane where the target dynamic obstacle is located. Specifically, the greater the road curvature, the lower the road surface adhesion coefficient and the worse the visibility, the smaller the base speed limit.

[0085] The minimum value among the base speed limit, obstacle type-corrected speed limit, traffic rule-corrected speed limit, and environment-corrected speed limit is selected as the final target speed limit in the speed constraint function.

[0086] The final target speed lower limit in the speed constraint function is determined by taking zero in normal foreground scenarios and a negative reversing speed threshold in parking scenarios where reversing is allowed.

[0087] In a preferred embodiment, the acceleration constraint function is defined as:

[0088] in, Indicates acceleration constraint index, This represents the minimum acceleration threshold corresponding to the target dynamic obstacle. This represents the maximum acceleration threshold corresponding to the target dynamic obstacle. and The dynamic settings are based on the road type, road surface adhesion conditions, and road curvature of the target dynamic obstacle.

[0089] In one specific embodiment, firstly, based on the road type where the target dynamic obstacle is located, the basic acceleration upper limit and the basic deceleration upper limit of the vehicle are determined. Then, based on the obstacle type, the basic driving acceleration upper limit and the basic deceleration upper limit are corrected to obtain the obstacle type corrected acceleration upper limit and the obstacle type corrected deceleration upper limit.

[0090] Furthermore, environmental corrections are made to the base acceleration and deceleration limits based on the road surface adhesion coefficient to obtain environmentally corrected acceleration and deceleration limits, which reflect the limitations of wet, slippery, waterlogged, snowy, or icy road surfaces on the dynamic obstacle target.

[0091] Then, by combining the current velocity of the target dynamic obstacle with the road curvature, the lateral acceleration is calculated. The lateral acceleration is then used to couple and correct the upper limit of the basic acceleration and the upper limit of the basic deceleration to obtain the upper limit of the road curvature-corrected acceleration and the lower limit of the road curvature-corrected deceleration.

[0092] The minimum value of the base acceleration upper limit, obstacle type-corrected acceleration upper limit, environment-corrected acceleration upper limit, and road curvature-corrected acceleration upper limit is taken as the acceleration upper limit threshold applied to the acceleration constraint function. The negative of the minimum value of the base deceleration upper limit, obstacle type modified deceleration upper limit, environment modified deceleration upper limit, and road curvature modified deceleration upper limit is taken as the lower limit threshold of the acceleration constraint function. This ensures that the trajectory of the generated target obstacle conforms to the motion patterns of real traffic participants in terms of longitudinal dynamics.

[0093] In a preferred embodiment, the steering constraint function is defined as:

[0094] in, Indicates the steering constraint index. This represents the maximum change in steering angle corresponding to the target dynamic obstacle. This represents the change in the steering angle of the target dynamic obstacle at time t.

[0095] In a preferred embodiment, the traffic rule constraint function is used to ensure that the generated safety-critical scenario conforms to basic traffic rules. The traffic rule constraint function outputs the traffic rule constraint index corresponding to the target dynamic obstacle. The traffic rule constraint function further includes a lane driving constraint function, a traffic light constraint function, and a yield constraint function. The lane driving constraint function is used to determine the lane driving constraint index corresponding to the target dynamic obstacle, the traffic light constraint function is used to determine the traffic light constraint index corresponding to the target dynamic obstacle, and the yield constraint function is used to determine the yield constraint index corresponding to the target dynamic obstacle.

[0096] Preferred traffic rule constraint indicators for target dynamic obstacles Lane driving constraint indicators Traffic light constraint indicators With yield constraint indicators The product between them, that is:

[0097] In a preferred embodiment, the lane driving constraint function is defined as:

[0098] in, This represents the legal driving lane range of the lane to which the target dynamic obstacle belongs at time t. It is a set of road areas that allow the dynamic obstacle to pass normally, defined by the traffic rules corresponding to the lane and the specific road design. Preferably, it represents the legal driving lane range. This includes spatial boundary constraints, functional rule constraints, and real-time state constraints.

[0099] The spatial boundary constraint includes the lane that is consistent with the driving direction of the lane to which the target dynamic obstacle belongs, and the spatial boundary constraint also requires that the target dynamic obstacle as a whole must be within the lane line area of ​​the lane it is driving on, and must not cross the line or intrude into the adjacent lane.

[0100] Functional rules stipulate that: the target dynamic obstacle must be compatible with the traffic function of the lane it occupies, and the target dynamic obstacle must comply with the temporary traffic control and lane restriction rules.

[0101] Real-time state constraints stipulate that the legal lane range must be determined by real-time data collected by high-precision map data and environmental perception module. The real-time data collected by high-precision map data and environmental perception module includes, but is not limited to, lane geometric boundaries, driving direction, and permitted steering behavior information.

[0102] In a preferred embodiment, the traffic light constraint function is defined as:

[0103] in, This indicates that the driving status of the target dynamic obstacle conforms to the indication of the traffic light it is currently facing.

[0104] Preferably, the yield constraint function is defined as:

[0105] in, This means that the target dynamic obstacle shall comply with the yielding rule in the specified scenario where the yielding requirement exists. The specified scenario includes, but is not limited to, at least one of the following: intersections, roundabout merging areas, pedestrian crossings, and other road scenarios where there is a path conflict. The yielding rule instructs the target dynamic obstacle to adopt a passing strategy of slowing down, stopping and waiting, or delaying entry into the conflict area for traffic participants with priority right-of-way, and shall not rush over, force the other party to stop, or create a dangerous conflict.

[0106] In a preferred embodiment, the comprehensive constraint evaluation function is defined as:

[0107] This represents the comprehensive constraint evaluation index corresponding to the target dynamic obstacle. , and The comprehensive constraint evaluation index, with predetermined weighting coefficients, reflects the rationality of the target dynamic obstacle's action or trajectory in the corresponding scene state at any given time.

[0108] Construct a composite reward function corresponding to the constraint adversarial learning framework.

[0109] The simulated action vector corresponding to each target dynamic obstacle output by the adversarial agent, along with its corresponding safety-critical parameters, comprehensive constraint evaluation index, and historical trajectory data, are input into the composite reward function to obtain the composite reward index corresponding to the target dynamic obstacle.

[0110] Preferably, the composite reward function includes at least an adversarial reward function, a constrained reward function, and a diversity reward function, wherein the composite reward index is obtained through the following method: The collision time and minimum collision distance corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent are input into the adversarial reward function to obtain the adversarial reward index of the target dynamic obstacle. The adversarial reward index is used to encourage the adversarial agent corresponding to the target dynamic obstacle to generate behaviors that are more threatening to the autonomous vehicle.

[0111] The comprehensive constraint evaluation index corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent is input into the constraint reward function to obtain the constraint reward index of the target dynamic obstacle. The constraint reward index is used to encourage the action or trajectory generated by the adversarial agent corresponding to the target dynamic obstacle to meet the physical constraints and traffic rule constraints.

[0112] The diversity reward index of the target dynamic obstacle is obtained by inputting the simulated action vector of the target dynamic obstacle output by the adversarial agent and the multiple historical actions corresponding to the time of the simulated action vector into the diversity reward function. The diversity reward constraint index is used to suppress the homogenization of the actions or trajectories generated by the adversarial agent.

[0113] The composite reward indicator is obtained by weighting the adversarial reward indicator, the constraint reward indicator, and the diversity reward indicator.

[0114] In a preferred embodiment, a corresponding adversarial reward function is constructed based on safety-critical parameters. The safety-critical parameters used to construct the adversarial reward function include, but are not limited to, any two of the following: collision time TTC, minimum collision distance MDC, and collision induction result.

[0115] For example, the adversarial reward function is designed using collision time TTC and minimum collision distance MDC as follows:

[0116] in, This represents the adversarial reward metric output by the adversarial reward function. The reward for the target dynamic obstacle's actions during collision time TTC. This refers to the reward term for the simulated motion vector against a target dynamic obstacle at the minimum collision distance. This indicates the weighting of the Time-to-Cross (TTC) reward in the overall competitive reward. This indicates the weight of the reward item for minimum collision distance (MDC) in the overall adversarial reward. Used to adjust the sensitivity of reward items to collision time. Used to adjust the sensitivity of reward items to minimum collision distance. , , and Pre-calibration offline, , , and The pre-calibrated values ​​enable the final generated reward adversarial index to characterize the danger of the scenario while taking into account the distinctiveness, stability and realism of reward changes and scenario generation.

[0117] In a preferred embodiment, the constraint reward function is designed based on a comprehensive constraint evaluation function, specifically as follows:

[0118] in, This indicates a constraint on reward indicators. To constrain the fulfillment of reward coefficients, which are used to adjust the intensity of positive rewards corresponding to the comprehensive constraint evaluation indicators. The penalty coefficient is used to constrain violations and adjust the intensity of negative penalties corresponding to the comprehensive constraint evaluation index. and Pre-calibration offline (details omitted here). and The pre-calibrated values ​​enable the constraint reward function to enhance constraint satisfaction while taking into account training stability and optimization efficiency. Specifically, when the comprehensive constraint evaluation index is 1, a positive reward is given, and when the comprehensive constraint evaluation index is 0, a penalty is given.

[0119] In a preferred embodiment, to avoid scenario homogenization, the diversity reward function is preferably designed based on the difference between the generated trajectory and the historical trajectory as follows:

[0120] in, This indicates the number of historical trajectory samples corresponding to the target dynamic obstacle. Let represent the simulated action vector corresponding to the trajectory output by the adversarial agent at time t. This represents the historical action vector of the k-th historical trajectory at time t. This represents the weighting coefficient, used to adjust the contribution of diversity reward items to the overall reward function. This represents the distance sensitivity coefficient, used to adjust the strength of the impact of action vector differences on the diversity reward metric. and It is pre-determined that in this application, the greater the difference between the simulated action vector output by the adversarial agent at a corresponding moment and the historical action vector of the historical trajectory at the corresponding moment, the higher the diversity reward index.

[0121] Action vectors are used to characterize the control behavior of a vehicle. The elements of an action vector include at least steering control and longitudinal acceleration control. In another preferred embodiment of this application, taking the action vector output by the adversarial agent as an example where the elements include steering control and longitudinal acceleration control, it is defined as follows:

[0122] This represents the steering control quantity at time t, which corresponds to the output of the adversarial agent. The steering control quantity can be expressed as the steering angle. describe, The acceleration control quantity at time t represents the output of the adversarial agent. Correspondingly, the action vector of the k-th historical trajectory at time t is represented as:

[0123] in, This represents the steering control amount of the k-th historical trajectory at time t. This represents the acceleration control amount of the k-th historical trajectory at time t.

[0124] Let be the distance metric between the simulated action vector output by the adversarial agent at time t and the historical action vector of the k-th historical trajectory. It is used to characterize the degree of difference between the action output by the adversarial agent and the action of the historical trajectory, and its expression is:

[0125] In this application, when the action vector contains more control components, the distance metric function is correspondingly expanded to multidimensional Euclidean distance.

[0126] By calculating the distance between the action vector of the generated trajectory of the adversary agent at a corresponding moment and the action vector of the historical trajectory at the corresponding moment, a diversity reward index is output. It can measure the degree of difference between the simulated action vectors output by the adversarial agent and the historical action vectors of the vehicle, thereby enabling a quantitative evaluation of scene diversity.

[0127] In a preferred embodiment, a target attacker with security-critical potential is selected from a plurality of target dynamic obstacles.

[0128] Preferably, the steps for screening target attackers with security-critical potential from multiple dynamic obstacles include: The collision time, relative motion direction, and relative motion distance between the target dynamic obstacle and the autonomous vehicle are input into the attacker identification function to determine the attacker identification result of the target dynamic obstacle output by the attacker identification function. Based on the attacker identification result, at least one target attacker is identified and fed back to the adversarial generative agent.

[0129] In one example, the attacker identification function is defined as:

[0130] in, This represents the collision time recognition threshold, used to determine whether a target dynamic obstacle has entered the safety-critical recognition zone. It is initially set based on the total risk response time of the autonomous vehicle under TTC (Traffic-Temporary Completion Time), and then corrected according to the temporal collision distribution characteristics of the target dynamic obstacle at the corresponding time, such as the road scene and obstacle type, to ultimately obtain the collision time recognition threshold. Preferably, the total risk response time of the autonomous vehicle to the Time-to-Cross (TTC) collision is described as the sum of the autonomous vehicle's perception processing time, decision-making and planning time, control execution time, and safety redundancy time, with the collision time identification threshold as the defining characteristic. The threshold, which is obtained through offline simulation verification, is the optimal threshold for both recall and false positive rates in identifying the target attacker. I will not go into too much detail here.

[0131] To improve the accuracy of target attacker screening for autonomous vehicles, it is necessary to further consider the relative motion relationship between the target dynamic obstacle and the autonomous vehicle. Therefore, a relative motion judgment function between the target dynamic obstacle and the autonomous vehicle is constructed. Relative motion judgment function This is used to characterize whether a dynamic obstacle at time t is located in a high-risk direction from the autonomous vehicle and shows a tendency to approach it; specifically, it is a relative motion judgment function. Represented as:

[0132] in, Indicates target dynamic obstacles The preferred relative position vector between the driver and the autonomous vehicle at time t is... Defined as:

[0133] This represents the position vector of the target dynamic obstacle at time t. This represents the position vector of the autonomous vehicle at time t.

[0134] Indicates target dynamic obstacles The preferred relative velocity vector between the driver and the autonomous vehicle at time t is... Defined as:

[0135] This represents the velocity vector of the target dynamic obstacle at time t. This represents the velocity vector of the autonomous vehicle at time t.

[0136] This indicates the orientation of the target dynamic obstacle relative to the autonomous vehicle. For high-risk directions, when When the target dynamic obstacle is located within the high-risk direction set, and the relative distance between the target dynamic obstacle and the autonomous vehicle decreases over time, This means determining whether the target dynamic obstacle and the self-driving vehicle satisfy a high-risk relative motion relationship.

[0137] In a preferred embodiment, please refer to Figure 3 , Figure 3 This diagram illustrates the training process of a constraint adversarial learning framework provided in an embodiment of this application. Figure 3 As shown, step S300 includes: S3001. Divide the multiple scene state vectors corresponding to the structured scene data into multiple training batches; For each training batch, the following steps are performed: S3002. Using the multiple initial scene state vectors corresponding to the training batch and the training samples stored in the experience replay pool, the adversarial agent is trained and optimized to obtain the optimized adversarial agent and the simulation scene state vector corresponding to each initial scene state vector output by the environment interaction network during the optimization process of the adversarial agent.

[0138] S3003. After the training of this batch of adversarial agents is completed, collect the state vectors of multiple simulation scenarios corresponding to this training batch and the constraint satisfaction status of each simulation scenario state vector for the constraint terms provided by the constraint evaluator.

[0139] Preferably, the dynamic obstacles corresponding to each target in the simulation scene's state vector are statistically analyzed. S3004. The constraint estimator is iteratively trained using the state vector of each simulation scenario as the input and the constraint satisfaction status of the state vector of each simulation scenario as the supervision label, to obtain the optimized constraint estimator.

[0140] S3005. Update the current constraint estimator applied to the constraint adversarial learning framework using the optimized constraint estimator.

[0141] S3006, Composite reward index corresponding to the state vector of statistical simulation scenario, generation rate of safety-critical scenarios, and constraint satisfaction rate.

[0142] S3007. Based on the composite reward index corresponding to the state vector of the simulation scenario, the generation rate of safety-critical scenarios, and the constraint satisfaction rate, determine whether the training of the constraint adversarial learning framework has been completed.

[0143] In a preferred embodiment, step S3007 includes: For each training batch, if the fluctuation range of each composite reward index under a consecutive preset number of simulation scenario state vectors is less than the preset value, the safety-critical scenario generation rate reaches the preset standard, and the constraint satisfaction rate reaches the preset standard, then the constraint adversarial learning framework is determined to have converged and training is stopped; otherwise, the next training batch is used for iterative training.

[0144] In one specific embodiment, the safety-critical scenario generation rate is determined by the comprehensive constraint index corresponding to the simulation scenario state vector. Specifically, if the comprehensive constraint index of each target dynamic obstacle under the simulation scenario state vector indicates that the obstacle meets the constraint degree, then the scenario corresponding to the simulation scenario state vector is determined to be a safety-critical scenario; otherwise, it is a non-safety-critical scenario. Based on this, the safety-critical scenario generation rate under the training batch is statistically analyzed.

[0145] The constraint satisfaction rate refers to the total percentage of constraint-satisfied items in the constraint evaluator for all dynamic obstacles under the state vectors of the entire simulation scenario in the training batch.

[0146] In a preferred embodiment, step S3002 further includes: For each initial scene state vector, the following steps are performed: The initial scene state vector is input into the adversarial generative agent (AGA), and the initial output action vector of each target dynamic obstacle is obtained from the AGA. Under the influence of the environment interaction network, constraint evaluator, and composite reward function, an initial composite reward index corresponding to the initial scene state vector and its corresponding initial output action vector is obtained. At least one target attacker is selected from the multiple target dynamic obstacles indicated by the initial scene state vector through an attacker identification mechanism and fed back to the AGA (specifically, fed back to the action network). The AGA modifies the target attacker's action trajectory to obtain the simulated action vector of each target dynamic obstacle. Under the influence of the environment interaction network, constraint evaluator, and composite reward function, a target composite reward index corresponding to the simulated scene state vector and its corresponding simulated action vector is obtained. The initial scene state vector and its corresponding initial output action vector and initial composite reward index for each target dynamic obstacle, as well as the simulated scene state vector and its corresponding simulated action vector and target composite reward index for each target dynamic obstacle, are integrated into training samples and stored in the experience replay pool. Multiple training samples are sampled from the experience replay pool to update the network parameters of the adversarial generative agent, resulting in an updated adversarial generative agent.

[0147] In a preferred embodiment, the step of obtaining the initial composite reward index corresponding to the initial scene state vector and its corresponding initial output action vector under the action of the environmental interaction network, constraint evaluator, and composite reward function further includes: The initial scene state vector and the initial output action vector corresponding to each target dynamic obstacle are input into the environment interaction network. Under the action of the environment interaction network, the initial safety key parameters corresponding to the initial scene state vector are obtained. The initial scene state vector and the initial safety key parameters corresponding to each target dynamic obstacle are input into the constraint evaluator to obtain the comprehensive constraint evaluation index corresponding to each target dynamic obstacle. The initial scene state vector and its corresponding safety key parameters, the comprehensive constraint evaluation index corresponding to each target dynamic obstacle, and the historical action vector corresponding to the historical trajectory data of the target dynamic obstacle are input into the composite reward function to obtain the initial composite reward index.

[0148] In one specific embodiment, the process of defining the target composite reward index corresponding to the simulation scene state vector and its corresponding simulation action vector is similar to that described above, and will not be elaborated further here.

[0149] In a preferred embodiment, step S400 further includes: The safety critical constraint index corresponding to the safety scenario state vector is input into the safety critical verification function for evaluation to obtain the corresponding safety critical verification index. If the safety critical verification index indicates that the safety scenario state vector meets the safety critical constraints, the trajectory data of the target dynamic obstacle indicated by the safety scenario state vector is optimized to obtain the optimized safety scenario state vector. Data is extracted from the optimized scenario state vector according to the preset standard format to obtain the target safety critical scenario data. If the safety critical verification index indicates that the safety scenario state vector does not meet the safety critical constraints, the process returns to step S200 to perform the secondary generation of the target constraint adversarial learning framework.

[0150] Preferably, the target safety critical scenario data includes at least complete road network information, trajectory information of target dynamic obstacles, and status information of the autonomous vehicle.

[0151] In a preferred embodiment, the step of optimizing the trajectory data of the target dynamic obstacle indicated by the safety scene state vector includes: The motion sequence indicated by the trajectory data of the target dynamic obstacle is smoothed by moving average filtering to obtain the optimized motion sequence of the target dynamic obstacle. The optimized motion sequence is used to update the safety scene state vector to obtain the optimized safety scene state vector. The optimized motion sequence of the optimized safety scene state vector is corrected for rule compliance to obtain the target safety scene state vector.

[0152] Preferably, the motion sequence indicated by the trajectory data of the target dynamic obstacle is smoothed to reduce the problem of trajectory discontinuity caused by the motion image. In this application, the smoothed motion of the target dynamic obstacle at time t is specifically defined as:

[0153] M represents the size of the sliding window. Let be the motion vector of the target dynamic obstacle at time t after smoothing. Let be the motion vector of the target dynamic obstacle at time ti. Through this smoothing process, the continuity and physical rationality of the target dynamic obstacle can be improved without changing the overall behavior trend of the target dynamic obstacle, thereby improving the usability and stability of the generated scene.

[0154] In one specific embodiment, rule compliance modification includes: Perform traffic rule compliance checks on the safety-critical scenarios indicated by the optimized safety scenario state vector, and adjust the state of the obstacle (Mrs. Dong) that slightly violates traffic rules, specifically including at least one of the following: If there is a lane departure, adjust the position coordinates of the target dynamic obstacle to the legal lane range. If there is a speed mismatch with traffic rules, adjust the speed change of the target dynamic obstacle. If there is a violation of the right-of-way rule, correct the movement sequence or path of the target dynamic obstacle.

[0155] In one specific embodiment, the following embodiment is provided for step S400: Provide a straight-line scenario where the autonomous vehicle is driving normally along its lane, with multiple dynamic obstacles in front or adjacent lanes. Based on this, construct the corresponding structured scenario data and form a given scenario state vector for each moment. Use an attacker identification mechanism to identify the most threatening target attacker from multiple dynamic obstacles, and use a trained adversarial learning framework to adjust its speed, acceleration, and heading angle to make the target attacker suddenly decelerate, cut in, or change lanes close to it. Output the corresponding safety scenario state vector. After verification and optimization, obtain the target safety critical scenario corresponding to the straight-line scenario.

[0156] In another embodiment, a city intersection scenario is provided. A self-driving vehicle passes through a signalized intersection according to a planned trajectory. There are left-turning vehicles, straight-going vehicles, or lateral crossing obstacles in the surrounding area. According to the steps provided in this application, the corresponding structured scene data is constructed and a given scene state vector is formed for each time. Using an attacker identification mechanism, the target obstacle that poses the greatest threat to the self-driving vehicle is extracted from the left-turning vehicles, straight-going vehicles, and lateral crossing obstacles. Its overtaking, rapid acceleration, or insufficient yielding behavior is generated, and the corresponding safety scene state vector is output. After verification and optimization, a key intersection safety scenario that combines high risk and rationality is obtained.

[0157] The advantages of this application are: First, this application requires only a small amount of real driving data to generate high-quality safety-critical scenarios, reducing the need for large amounts of data and solving the pain point of large data requirements in data-driven methods.

[0158] Secondly, through a step-by-step framework design and a multi-dimensional constraint system, the generated safety-critical scenarios have high fidelity and rationality. At the same time, by combining the composite reward function with the secondary generation strategy, the high-risk characteristics of the scenarios are guaranteed, while the diversified generation of scenarios is promoted, which can comprehensively cover the testing needs of autonomous driving systems.

[0159] Finally, the generated safety-critical scenarios can be directly used for virtual testing of autonomous driving systems, effectively exposing the system's defects in edge cases, improving the safety of autonomous driving systems, and having significant engineering application value.

[0160] Based on the same application concept, this application also provides an autonomous driving test scenario generation device based on constraint adversarial reinforcement learning, which corresponds to the autonomous driving test scenario generation method based on constraint adversarial reinforcement learning provided in the above embodiments. Since the principle of the device in this application is similar to the autonomous driving test scenario generation method based on constraint adversarial reinforcement learning in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0161] Please see Figure 4 , Figure 4 This diagram illustrates the functional block diagram of an autonomous driving test scenario generation device based on constraint adversarial reinforcement learning, as provided in an embodiment of this application. Figure 4 As shown, the device includes: The data construction module 500 is used to determine the initial scene state vector of the autonomous vehicle at each moment based on multiple real autonomous driving data of the autonomous vehicle. The scene state vector represents the lane network features, traffic facility features, dynamic obstacle features, autonomous vehicle features and environmental features at the corresponding moment. The learning framework construction module 510 is used to construct a constraint adversarial learning framework for dynamic obstacles around the autonomous vehicle, guided by an attacker identification mechanism. The input of the constraint adversarial learning framework is the scene state vector, and the output of the constraint adversarial learning framework is the simulation scene state vector. Training module 520 is used to take multiple initial scene state vectors indicated by multiple real driving data as training samples of the constraint adversarial learning framework, and combine environmental interaction, multi-dimensional constraint system for target dynamic obstacles and composite reward function to iteratively train the constraint adversarial learning framework to obtain the trained target constraint adversarial learning framework. The scene generation module 530 is used to input the given scene state vector into the target constraint adversarial learning framework, and to adjust the actions of the given scene state vector through the target constraint adversarial learning framework to obtain a safety scene state vector for training the autonomous vehicle. The safety scene state vector corresponds to a safety-critical scene.

[0162] Based on the same application concept, please refer to Figure 5 , Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. For example... Figure 5 As shown, the electronic device 60 includes a processor 601, a memory 602, and a bus 603. The memory 602 stores machine-readable instructions that can be executed by the processor 601. When the electronic device 60 is running, the processor 601 and the memory 602 communicate through the bus 603. The machine-readable instructions are executed by the processor 601 to perform the steps of the autonomous driving test scenario generation method based on constraint adversarial reinforcement learning provided in any of the above embodiments.

[0163] Based on the same concept, this application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the autonomous driving test scenario generation method based on constraint adversarial reinforcement learning provided in the above embodiments.

[0164] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0167] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating autonomous driving test scenarios based on constraint adversarial reinforcement learning, characterized in that, The method includes: Based on multiple real self-driving data from the autonomous vehicle, the initial scene state vector of the autonomous vehicle at each moment is determined. The scene state vector represents the lane network features, traffic facility features, dynamic obstacle features, autonomous vehicle features, and environmental features at the corresponding moment. A constraint adversarial learning framework for dynamic obstacles around an autonomous vehicle is constructed, guided by an attacker identification mechanism. The input of the constraint adversarial learning framework is a scene state vector, and the output of the constraint adversarial learning framework is a simulation scene state vector. Multiple initial scene state vectors indicated by multiple real driving data are used as training samples for the constraint adversarial learning framework. The constraint adversarial learning framework is iteratively trained by combining environmental interaction, a multi-dimensional constraint system for dynamic obstacles to the target, and a composite reward function to obtain a trained target constraint adversarial learning framework. The given scenario state vector is input into the target constraint adversarial learning framework. The target constraint adversarial learning framework adjusts the actions of the given scenario state vector to obtain a safe scenario state vector for training the autonomous vehicle. The safe scenario state vector corresponds to a safety-critical scenario.

2. The method according to claim 1, characterized in that, The scene state vector corresponding to the autonomous vehicle is determined in the following way: Extract road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data from real-world self-driving data; Data preprocessing was performed on road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data to obtain preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data. Data modeling is performed using preprocessed road network data, traffic facility data, dynamic obstacle operation data, autonomous vehicle operation data, and environmental status data to form structured scene data; Multi-dimensional attribute information is extracted from the structured scene data to map the structured scene data to a high-dimensional state space, thereby obtaining the high-dimensional scene state vector corresponding to each time step. The high-dimensional scene state vector is subjected to dimensionality reduction processing based on normalization according to the attribute information of each dimension to obtain the processed scene state vector.

3. The method according to claim 2, characterized in that, The high-dimensional scene state vector corresponding to each time step is determined in the following way: From the set of key lane nodes, the set of lane connection relationships, and the set of lane attributes described by the road network data, the geometric attributes and topological relationship attributes corresponding to the key lane nodes to which the autonomous vehicle belongs at that moment are extracted respectively to form the lane network state vector at that moment. From the multiple traffic facilities described in the traffic facility data and the attribute information corresponding to each traffic facility, extract the spatial distribution attributes and state attributes of the target traffic facilities within a preset calibration range centered on the self-driving vehicle at that time, and form the traffic facility state vector of each target traffic facility at that time. From the dynamic obstacle operation data describing multiple target dynamic obstacles around the autonomous vehicle and the attribute information corresponding to each target dynamic obstacle, extract the state attribute corresponding to each target dynamic obstacle to form the dynamic obstacle state vector corresponding to each target dynamic obstacle at that moment. Extract the state attributes of the autonomous vehicle at that moment from the autonomous vehicle operation data to form the autonomous vehicle state vector at that moment. From the environmental state data, target environmental attribute information that affects driving safety and trajectory decision-making based on the external environment of the autonomous vehicle at that moment is extracted to form an environmental state vector at that moment. The high-dimensional scene state vector at that moment is formed by the lane network state vector corresponding to each key node of the lane, the traffic facility state vector corresponding to each traffic facility, the dynamic obstacle state vector corresponding to each dynamic obstacle, the autonomous vehicle state vector, and the environment state vector.

4. The method according to claim 1, characterized in that, The constrained adversarial learning framework includes an adversarial generative agent, a constraint evaluator, an environment interaction network, an experience replay pool, and a composite reward constraint function. The adversarial generative agent includes an action network and an evaluation network. The input to the action network is an initial scene state vector and a target attacker determined by an attacker identification mechanism. The target attacker is at least one of the target dynamic obstacles. The output of the action network is a simulated action vector corresponding to each target dynamic obstacle. The input to the evaluation network is the initial scene state vector and the simulated action vector corresponding to each target dynamic obstacle. The output of the evaluation network is an action value assessment of the simulated action vector of each target obstacle. The environmental interaction network is used to simulate the interaction process between the target dynamic obstacle and the autonomous vehicle. The input of the environmental interaction network is a combination of the input and output of the action network. The output of the environmental interaction network is the simulation scene state vector and the safety key parameters between each target dynamic obstacle and the autonomous vehicle. The constraint evaluator adopts a multi-dimensional constraint system. The input of the constraint evaluator is the simulation scene state vector and safety key parameters output by the environmental interaction network. The output of the constraint evaluator is the comprehensive constraint evaluation index corresponding to each target dynamic obstacle. The experience replay pool stores multiple training sample data generated during the training of the constraint adversarial learning framework; The input to the composite reward function is the simulation action vector corresponding to the target dynamic obstacle and its corresponding safety key parameters, comprehensive constraint evaluation index and historical trajectory data. The output of the composite reward function is the composite reward index corresponding to each target dynamic obstacle.

5. The method according to claim 4, characterized in that, The constraint evaluator includes safety-critical constraint functions, physical constraint functions, traffic rule constraint functions, and comprehensive constraint evaluation functions. The constraint evaluator obtains the comprehensive constraint evaluation index in the following manner: Obtain the safety-critical parameters corresponding to the target dynamic obstacles output by the environmental interaction network and input them into the safety-critical constraint function to obtain the safety-critical constraint index corresponding to the target dynamic obstacles. Input the velocity, acceleration, and steering parameters corresponding to the target dynamic obstacle indicated by the state vector of the simulation scene into the physical constraint function to obtain the physical constraint index corresponding to the target dynamic obstacle; By substituting the legal form of the target dynamic obstacle indicated by the state vector of the simulation scene in the corresponding lane, the result of traffic light indication compliance, and the result of yield rule compliance into the traffic rule constraint function, the traffic rule constraint index is obtained. The safety-critical constraint indicators, physical constraint indicators, and traffic rule constraint indicators are input into the comprehensive constraint evaluation function for weighted calculation to obtain the comprehensive constraint indicator corresponding to each target dynamic obstacle. The comprehensive constraint evaluation indicator reflects the rationality of the target dynamic obstacle's action or driving trajectory in its respective scenario state.

6. The method according to claim 4, characterized in that, The composite reward metric for each target dynamic obstacle is determined using the following method: The collision time and minimum collision distance corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent are input into the adversarial reward function to obtain the adversarial reward index of the target dynamic obstacle. The adversarial reward index is used to encourage the adversarial agent corresponding to the target dynamic obstacle to generate behaviors that are more threatening to the autonomous vehicle. The comprehensive constraint evaluation index corresponding to the simulated action vector of the target dynamic obstacle output by the adversarial agent is input into the constraint reward function to obtain the constraint reward index of the target dynamic obstacle. The constraint reward index is used to encourage the action or trajectory generated by the adversarial agent corresponding to the target dynamic obstacle to meet the physical constraints and traffic rule constraints. The simulated action vector of the target dynamic obstacle output by the adversarial agent and multiple historical actions corresponding to the time of the simulated action vector are input into the diversity reward function to obtain the diversity reward index of the target dynamic obstacle. The diversity reward constraint index is used to suppress the homogenization of actions or trajectories generated by the adversarial agent. The composite reward indicator is obtained by weighting the adversarial reward indicator, the constraint reward indicator, and the diversity reward indicator.

7. The method according to claim 4, characterized in that, Attacker identification mechanisms include: The collision time, relative motion direction, and relative motion distance between the target dynamic obstacle and the autonomous vehicle are input into the attacker identification function to determine the attacker identification result of the target dynamic obstacle output by the attacker identification function. The attacker identification function is defined as follows: when the collision time is less than the collision time identification threshold, the target dynamic obstacle is located in the high-risk direction of the autonomous vehicle and has a tendency to approach the autonomous vehicle, the target dynamic obstacle is identified as the target attacker. Based on the attacker identification results, at least one target attacker is identified and fed back to the action network of the adversarial generative agent.

8. The method according to claim 4, characterized in that, The constraint adversarial learning framework is trained using the following method: The structured scene data is divided into multiple initial scene state vectors into multiple training batches; For each training batch, the following steps are performed: the adversarial agent is trained and optimized using multiple initial scene state vectors corresponding to the training batch and training samples stored in the experience replay pool, to obtain the optimized adversarial agent and the simulation scene state vector corresponding to each initial scene state vector output by the environment interaction network during the adversarial agent optimization process. After the training of this batch of adversarial agents is completed, multiple simulation scenario state vectors corresponding to the training batch and the constraint satisfaction status of each simulation scenario state vector for the constraint estimator are collected. The constraint estimator is iteratively trained with each simulation scenario state vector as the input of the constraint estimator and the constraint satisfaction status of each simulation scenario state vector as the supervision label to obtain an optimized constraint estimator. The optimized constraint estimator is then used to update the current constraint estimator applied by the constraint adversarial learning framework. The composite reward index, safety-critical scenario generation rate, and constraint satisfaction rate corresponding to the state vector of the simulation scenario are statistically analyzed. Based on the composite reward index corresponding to the state vector of the simulation scenario, the safety-critical scenario generation rate, and constraint satisfaction rate under the training batch, it is determined whether the training of the constraint adversarial learning framework has been completed.

9. The method according to claim 8, characterized in that, The completion of training for the constraint adversarial learning framework is determined by the following methods: For each training batch, execute: If the fluctuation range of each composite reward index under a consecutive preset number of simulation scenario state vectors is less than the preset value, the safety critical scenario generation rate reaches the preset standard, and the constraint term satisfaction rate reaches the preset standard, then the constraint adversarial learning framework is determined to have converged and training is stopped. If the fluctuation range of each composite reward indicator is less than the preset value, the number of updated scenario dynamic vectors does not meet the preset quantity requirement, the generation rate of safety-critical scenarios does not reach the preset standard, and / or the constraint satisfaction rate does not reach the preset standard, then the next training batch will be used for iterative training.

10. The method according to claim 4, characterized in that, The training steps for the adversarial generative agent in each training batch include: For each initial scene state vector, execute: Input the initial scene state vector into the adversarial generative agent to obtain the initial output action vector of each target dynamic obstacle output by the adversarial generative agent; Under the influence of the environmental interaction network, constraint evaluator and composite reward function, the initial composite reward index corresponding to the initial scene state vector and its corresponding initial output action vector is obtained. At least one target attacker is selected from multiple dynamic obstacles indicated by the initial scene state vector through an attacker identification mechanism and fed back to the adversarial generative agent. By using adversarial generative agents to modify the action trajectory of the target attacker, a simulated action vector for each target dynamic obstacle is obtained; Under the influence of the environmental interaction network, constraint evaluator and composite reward function, the target composite reward index corresponding to the simulation scene state vector and its corresponding simulation action vector is obtained; The initial scene state vector and its corresponding initial output action vector and initial composite reward index for each target dynamic obstacle, as well as the simulation scene state vector and its corresponding simulation action vector and target composite reward index for each target dynamic obstacle, are integrated into training samples and stored in the experience replay pool. Multiple training samples are sampled from the experience replay pool to update the network parameters of the adversarial generative agent, resulting in an updated adversarial generative agent.

11. The method according to claim 1, characterized in that, The method further includes: The safety key constraint indicators corresponding to the safety scenario state vector are input into the safety key verification function for evaluation, and the corresponding safety key verification indicators are obtained. If the safety critical verification indicators indicate that the safety scenario state vector meets the safety critical constraints, then the trajectory data of the target dynamic obstacle indicated by the safety scenario state vector is rationally optimized to obtain the optimized safety scenario state vector. Data is extracted from the optimized scenario state vector according to a preset standard format to obtain target safety critical scenario data; If the safety critical verification metrics indicate that the safety scenario state vector does not meet the safety critical constraints, then the training of the constraint adversarial learning framework will be re-executed.

12. The method according to claim 11, characterized in that, The steps for rationally optimizing the trajectory data of dynamic obstacles indicating the target as shown by the state vector of the safety scenario include: A moving average filter is used to smooth the action sequence indicated by the trajectory data of the target dynamic obstacle; The optimized action sequence of the target dynamic obstacle is obtained, and the optimized action sequence is used to update the safety scene state vector to obtain the optimized safety scene state vector. The optimized action sequence of the security scenario state vector is corrected for rule compliance to obtain the target security scenario state vector.

Images