A method for generating edge scenes of intelligent vehicles based on hierarchical reinforcement architecture

Abstract

The present invention belongs to the field of autonomous driving test technology, specifically, a method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture. The method includes the following steps: step 1, processing natural driving data and constructing an initial test environment; step 2, establishing a risk conflict prediction method; step 3, establishing a test scene evaluation method, including collision rationality evaluation and spatiotemporal similarity evaluation; step 4, constructing a scene library based on index management; step 5, constructing a basic reinforcement learning environment, building a static environment and basic interfaces; step 6, dividing the scene hierarchical scheduling period, and dividing the test period based on scene characteristics; step 7, optimizing the reward mechanism and action strategy, and adjusting according to the scene period and vehicle behavior; step 8, model training application and scene data output. Finally, the diversity generation of edge scenes is achieved, breaking through the local convergence problem and improving the efficiency of edge scene excavation.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving testing, and specifically is a method for generating edge scenes of smart cars based on a layered reinforcement architecture. Background Art

[0002] Smart car technology is the direction of scientific research and development in the automotive industry and the progress of the times. The large number of edge scenarios in the natural driving world poses severe challenges to the safe operation of smart cars. How to find a large number of dangerous edge scenarios is the key to smart car testing and evaluation technology.

[0003] In the scene generation method of intelligent vehicles, compared with the basic combination test, the optimization generation method of dangerous scenes and the parameter sampling test, the test scene adversarial generation method based on reinforcement learning can dynamically generate new scenes with driving behavior edges and dangerous test processes, which has higher test value and provides more valuable data for algorithm optimization and improvement. It has become a research hotspot for relevant scholars at home and abroad. For example, patents CN117236194A and CN116680979A have announced scene generation methods based on reinforcement learning. However, this method also has several challenges that need to be solved. On the one hand, the reward signals in reinforcement learning are usually sparse and delayed, which leads to the convergence problem of the training process, making it difficult for the intelligent agent to efficiently learn the ideal scene generation strategy. On the other hand, the reinforcement learning algorithm is prone to fall into the local optimal solution, and the test scenes generated by exploration lack sufficient global representativeness, thereby weakening the scene's ability to expose algorithm defects. If the design of diverse rewards is forced, it will affect the convergence of the reinforcement learning algorithm and it will be difficult to train edge scenes that can explore algorithm performance defects. In addition, when defining diversity, existing reinforcement learning scenario generation frameworks often focus too much on similarity metrics at the trajectory morphology level, and fail to take into account key factors such as robustness to trajectory uncertainty noise, real-time response to road spatial layout in reality, and dynamic changes in the environment.

[0004] Therefore, there is an urgent need for a reinforcement learning algorithm that comprehensively considers algorithm convergence and diversity exploration to achieve targeted and continuous generation of edge scenarios. Summary of the invention

[0005] To solve the above problems, the present invention provides a method for generating edge scenes of intelligent vehicles based on a hierarchical reinforcement architecture, targeting the sparse reward phenomenon in the reinforcement learning exploration process. Through the active period management of the test process and the adjustment of the reward mechanism and action strategy in each period, the continuous generation of diverse test scenes is achieved. A risk area identification method based on short-term conflict prediction is proposed, which guides the traffic vehicle agent to generate dangerous actions with probability, solving the problem of slow exploration of traditional reinforcement learning algorithms. A method for evaluating the spatiotemporal similarity of test scenes is proposed, which comprehensively defines the agent diversity reward mechanism through the similarity of relative trajectory, time series characteristics and spatial structure. And the rationality of the generated scene is defined by the playback test model and the perception-constrained model. It breaks through the limitations of the existing test scene evaluation system that rarely considers the rationality of the scenes generated by reinforcement learning and only evaluates the scene similarity from the perspective of trajectory.

[0006] According to different state designs, a targeted reward mechanism for traffic vehicles is designed, so as to convert the overall complex test problem into simple sub-tasks, and finally realize the diverse generation of edge scenes, break through the local convergence problem, and improve the efficiency of edge scene discovery.

[0007] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:

[0008] A method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture comprises the following steps:

[0009] Step 1: Process natural driving data and build the initial test environment;

[0010] Step 2: Establish a risk conflict prediction method;

[0011] Step 3: Establish a test scenario evaluation method, including collision rationality evaluation and spatiotemporal similarity evaluation;

[0012] Step 4: Build a scene library based on index management;

[0013] Step 5: Build a basic reinforcement learning environment, set up a static environment and basic interface;

[0014] Step 6: Divide the test period by scenario hierarchical scheduling, and divide the test period based on scenario characteristics;

[0015] Step 7: Optimize the reward mechanism and action strategy, and adjust according to the scene period and vehicle behavior;

[0016] Step 8: Model training application and scene data output.

[0017] Furthermore, the specific method of step one is as follows:

[0018] 11) Combine the existing natural driving database to extract vehicle trajectory information; the vehicle trajectory information includes a data subset of time period, lane, and vehicle ID; for each vehicle ID, sort the data based on the timestamp, and form a continuous time series based on the position information on the two-dimensional plane, that is, the vehicle's driving trajectory, and convert the vehicle's driving trajectory into a list;

[0019] 12) Statistical range of various kinematic parameters, including relative speed and relative distance; statistical distribution of each variable;

[0020] 13) Select simulation software that can dynamically control the movement of vehicles in real time.

[0021] Furthermore, the specific method of step 2 is as follows:

[0022] 21) Design a conflict prediction module to perceive conflict points in advance by predicting future trajectories, and convert the upper-level semantics output by the action strategy into lower-level operations that can be input by the agent; the conflict prediction module mainly consists of two parts: trajectory prediction and risk area calculation;

[0023] 22) The trajectory prediction part proposes an interactive area conflict prediction method based on IMM-KF-Bayes, combines the motion characteristics of the adversarial test vehicle to build a kinematic model of different behavior patterns, and builds a future moment risk calculation method based on reinforcement learning actions, as follows:

[0024] The vehicle behavior in the environment is described by the uniform speed model, uniform speed change model, and uniform angular velocity steering model; the observation likelihood function of each model is defined. Model, observation likelihood function Indicates at time Observed data In the model Next, given the state The probability of occurrence, It is a model In time The state vector of ; using the Bayesian formula, the posterior probability is calculated based on the prior probability and the observation likelihood function. The calculation method is as follows:

[0025] In the formula, For Model From time Time The prior probability of

[0026] Based on the updated model weights, the probability distribution of the predicted trajectory is obtained by weighting the prediction results of all models. The predicted state distribution of The calculation method is as follows:

[0027] in, From time Time All observation data of For Model The state transfer function is used to output the vehicle trajectory data within the preset time period in the future, including the position (x, y) and posture (θ) at each time node, and the uncertainty of trajectory prediction is quantified based on the Bayesian distribution.

[0028] 23) Risk area calculation part: For the conflict area prediction process, the historical motion information is used to predict the next three states, that is, the time step is 0.1s, and the cumulative motion state is 0.3s; the area around the smart car is divided into 8 areas, namely the front, rear, left, right, left front, left rear, right front, and right rear, which are respectively recorded as the first area, the second area, the third area, the fourth area, the fifth area, the sixth area, the seventh area, and the eighth area, which are expanded outward at 0.4 times the width of the vehicle to be tested, and the center point of the outer contour of the area is used as the risk information calculation point; first, the kinematic information of the reinforcement learning traffic vehicle and the vehicle to be tested in each state is obtained using the information of the current moment and the next three states. The regional risk calculation method is as follows:

[0029] In the formula, For the Risk of conflict in each region; is the prediction uncertainty of the state; For the The calculation point in the region is The relative distance between the prediction time step and the traffic vehicle; For the The calculation point in the region is The component of the relative speed of the prediction time step and the traffic vehicle on the line connecting the two; For the The collision time for each prediction time step; and In the The contribution weights of the collision time and relative distance of each prediction time step to the cumulative risk; within each prediction step, the contribution weights are set as shown in Table 1:

[0030] Table 1 Contribution weights of conflict risk components

[0031]

[0032] Therefore, the risk indicators of different areas are obtained by calculating the collision time and safety distance information.

[0033] Furthermore, the specific method of step three is as follows:

[0034] 31) Establish a scenario criticality evaluation method; construct a collision rationality evaluation model that includes a perception-constrained model and a playback test model. For the test scenario where a collision occurs, the requirements are as follows: On the one hand, the system under test perceives the target object, requiring that the traffic vehicle must be detected by the perception model 3 seconds before the collision. The perception model is designed in the description of the system under test; on the other hand, through playback testing, the intelligent driving main vehicle can execute the maximum risk avoidance strategy at the perception node to avoid the collision. If both requirements are met at the same time, it means that the scenario is a reasonable collision scenario and can effectively touch the performance boundary of the intelligent driving vehicle under test;

[0035] 32) Establish a scene diversity evaluation method, take two types of key scenes as anchor points, extract the trajectory data of the main vehicle and the traffic vehicle 5 seconds before and after the trigger moment for evaluation; design a trajectory sequence similarity evaluation method based on dynamic time warping, and finally output whether the key scene is a new scene, as follows:

[0036] 321) Define the high-dimensional distance metric function with Euclidean distance to represent the position difference between trajectories; define the multi-dimensional space-time conflict vector as follows: ① spatial feature: the lateral distance between the main vehicle and the center of the lane; ② distance conflict: the lateral and longitudinal distance between the mass centers of the two vehicles; ③ speed conflict: the lateral and longitudinal relative speeds of the two vehicles; ④ acceleration conflict: the lateral and longitudinal relative accelerations of the two vehicles; ⑤ lane conflict: the adjacent lane coefficient is 1, otherwise 0, the lane change behavior is close to 1, away from 0, and unchanged 0.5; ⑥ time conflict: the collision time information of the two vehicles in the same lane, when the traffic vehicle is in front of the intelligent driving car, the relative distance is calculated divided by the relative speed, and the upper limit is set to 10s, reflecting the situation of extremely low danger; for all collision and avoidable test scenarios, all instantaneous conflict indicators are normalized;

[0037] 322) Initialize the DTW matrix; create a (n+1)×(m+1) matrix D, where n and m are the time steps of the current key scene trajectory and each known scene trajectory in the scene library; set the first row and first column of the matrix to infinity;

[0038] 323) Fill the DTW matrix; Use the recursive formula to fill the entire matrix, for all i = 1 to n and j = 1 to m:

[0039] In the formula, and They are the currently generated key scene trajectory and the known scene trajectory in the scene library;

[0040] 324) Determine the similarity and find the minimum cumulative distance, that is, the minimum The preset threshold is set to 10. The DTW distance value is compared with the preset threshold. If the DTW distance value is small, the two trajectory sequences are considered similar; otherwise, they are considered dissimilar.

[0041] Furthermore, the specific method of step 4 is as follows:

[0042] 41) Develop key scene management tools based on Python automation scripts; design a scene library including scene ID, criticality, generation time, and algorithm version;

[0043] 42) For each generated scenario, design and create a corresponding index, and record relevant information in the scenario library.

[0044] Furthermore, the specific method of step five is as follows:

[0045] 51) Select simulation software that can dynamically control the movement of vehicles in real time;

[0046] 52) A basic longitudinal three-lane test scenario is built in the simulation software. A main vehicle equipped with an autonomous driving function and a traffic vehicle for reinforcement learning training are placed in the scenario. The three lanes are numbered 0, 1, and 2 from top to bottom. The lane number is determined by the centroid of the vehicle.

[0047] 53) Obtain the vehicle control interface in the simulation software, including throttle, brake and steering.

[0048] Furthermore, the specific method of step six is ​​as follows:

[0049] 61) According to the vehicle motion information in the scene, the test process is divided into three parts: guidance period, confrontation period, and exploration period;

[0050] 62) The characteristics of the guidance period are low scene generation efficiency and difficulty in triggering collision failure. The triggering condition is continuous There is no collision in the cycle; the characteristics of the confrontation period are that most of the unreasonable collisions are generated, and it is difficult to explore the performance boundary. The trigger condition is continuous collision, but the number of cycles of unreasonable collisions reaches The exploration period is characterized by a single type of generated scenes, falling into local convergence, and the triggering condition is that reasonable collisions occur continuously, but the number of cycles that do not meet the diversity conditions reaches ;in, , and are the discrimination thresholds for each period.

[0051] Furthermore, the specific method of step seven is as follows:

[0052] 71) Design a reward mechanism; the core task of the reward mechanism is to find more diverse edge test scenarios, and divide them into three periods to enhance the algorithm's diversity exploration capability; the reinforcement learning strategy architecture uses the basic DQN model; the core purpose of the guidance period is to guide the traffic vehicle to collide with the intelligent driving main vehicle, so the target action is to bring the two vehicles closer; entering the confrontation period, the reward function is set to the Euclidean distance correction value of the two vehicles and whether the traffic vehicle touches the road boundary; for the confrontation period, the traffic vehicle is required to trigger the performance boundary of the autonomous driving vehicle, so the reward function is set to the key evaluation results in step three in addition to whether the traffic vehicle touches the road boundary; for the exploration period, the traffic vehicle is required to explore more diverse scenarios and jump out of the local optimum, so the reward function is set as follows: ① Collision reward , weight ,②Key rewards , weight ,③Diversity Rewards , weight ④ Driving process rewards , weight ; If a collision occurs, a collision reward of 20 points will be obtained; if the collision can be sensed by the system under test and the playback test passes, it means that the key test scenario has been found, and a criticality reward of 50 points will be obtained; if a new scenario that is not in the scenario library is found, a diversity reward of 50 points will be obtained; if the vehicle goes out of the road boundary or exceeds the speed limit, a single-step reward of -2 points will be obtained; the weights of various rewards are set to be dynamically adjustable, as follows:

[0053]

[0054] The reinforcement learning optimization part outputs different lower-level action strategies and reward mechanisms based on the period division results. In the guidance period, it is expected that the traffic vehicle will approach the main vehicle, collide with it, and obtain more collision rewards. In the confrontation period, it is expected that the traffic vehicle will have a reasonable collision and obtain more reasonable collision rewards. In the guidance period, it is expected that the traffic vehicle will jump out of the local optimum and obtain more diverse rewards.

[0055] Therefore, the calculation method of each reward weight in the mth round is as follows:

[0056] In the formula, , and The basic weights for collision reward, criticality reward and diversity reward; , and is the adjustment factor of the corresponding weight; is the number of consecutive cycles without collision; is the number of cycles in which unreasonable collisions occur continuously; The number of cycles to find repeated plausible collision scenarios;

[0057] 72) Guiding action strategy: Introduce action guidance strategy based on risk area; Control the action of traffic vehicles according to the risk area identification result in step 2;

[0058] The recommended actions for each period are calculated according to the risk situation of each risk area; the period is divided into three parts: the guidance period, the confrontation period and the exploration period; for the guidance period, strive to cause a collision. If the risk sum of the predicted step size = 2 and = 3 is greater than the risk of the step size = 0 and = 1, no action is performed, otherwise the action is performed to approach the vehicle to be tested, and the action is solved by building a state machine based on the position and lane; for the confrontation period, on the basis of the occurrence of a collision, it is expected that a reasonable and avoidable collision will occur. Calculate whether the cumulative risk of the first, fifth and sixth interactive areas in step 23) exists in the area with the highest risk of all areas. If so, no action is performed, otherwise the action is performed to avoid collision with the vehicle to be tested; for the exploration period, on the premise of a reasonable and avoidable collision, more diverse collision scenarios are expected to occur. Based on the confrontation period strategy, a random action is generated every 4 execution steps; through targeted action strategies in different periods, a guidance path is designed for the reinforcement learning agent from the perspective of conflict risk.

[0059] Furthermore, the specific method of step eight is as follows:

[0060] 81) Based on the designed risk prediction model and reinforcement learning strategy, the model is trained using the collected natural driving data and the constructed test scenarios, and the agent strategy is updated by continuously interacting with the simulated environment; the reward function is adjusted to reflect the risk status of different scenarios and vehicle behaviors, so as to encourage the agent to learn safer and more effective driving strategies;

[0061] 82) During the reinforcement learning process, the system will automatically generate a series of challenging driving scenarios; these scenarios include the state transition sequence of the intelligent agent under different decision-making stages, the actions taken and the corresponding reward values ​​received, and the driving trajectory parameters; the verified and evaluated valid scenario data will be stored in a scenario library based on index management to realize a complete test scenario generation process.

[0062] Furthermore, a method for generating an edge scene of an intelligent vehicle based on a hierarchical reinforcement architecture is implemented by a system for generating an edge scene of an intelligent vehicle based on a hierarchical reinforcement architecture, and the system includes:

[0063] The data processing module is used to process natural driving data and build the initial test environment;

[0064] Conflict prediction module, used to establish risk conflict prediction method;

[0065] The scenario evaluation module is used to establish the test scenario evaluation method, including the collision rationality evaluation and the time-space similarity evaluation;

[0066] The first building module is used to build a scene library based on index management;

[0067] The second building block is used to build a basic reinforcement learning environment, a static environment and basic interfaces;

[0068] Hierarchical scheduling module, which is used for hierarchical scheduling of scenarios, divides the test period based on scenario characteristics, and optimizes the reward mechanism and action strategy, adjusting them according to the scenario period and vehicle behavior;

[0069] Output module, used for model training application and scene data output.

[0070] The beneficial effects of the present invention are:

[0071] 1) The present invention can quickly achieve confrontation effects and generate confrontation scenarios by combining the conflict prediction module with the reinforcement learning agent;

[0072] 2) The present invention can comprehensively evaluate the value of generating test scenarios through the critical evaluation and diversity evaluation in the scenario evaluation module, and provide a complete scenario preservation tool chain in combination with scenario library management;

[0073] 3) The core content of this invention is the intelligent agent hierarchical scheduling architecture, which divides the reinforcement learning test cycle into two periods according to the spatiotemporal characteristics of the scene, designs targeted reward functions and action strategies for different periods, and finally realizes the diverse generation of edge scenes, breaking through the local convergence problem and improving the efficiency of edge scene discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0075] Figure 1 A schematic diagram of a process of generating a smart car edge scene based on a layered reinforcement architecture according to the present invention;

[0076] Figure 2A schematic diagram of the structure of a smart car edge scene generation system based on a layered reinforcement architecture according to the present invention;

[0077] Figure 3 This is a schematic diagram of a conflict prediction method based on IMM-KF-Bayes according to the present invention;

[0078] Figure 4 The autonomous driving perception area model described in the present invention;

[0079] Figure 5 It is the reward curve of the algorithm training process described in the present invention. DETAILED DESCRIPTION

[0080] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0081] Example 1: See Figure 1 and Figure 2 The present invention provides a method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture, comprising the following steps:

[0082] Step 1: Process natural driving data. The specific method is as follows:

[0083] 11) Combine the existing natural driving database to extract vehicle trajectory information;

[0084] The existing natural driving database may adopt a HighD dataset.

[0085] The vehicle trajectory information includes a data subset of time period, lane, and vehicle ID; for each vehicle ID, the data is sorted based on the timestamp, and a continuous time series is formed according to the position information on the two-dimensional plane, that is, the vehicle's driving trajectory, and the vehicle's driving trajectory is converted into a list;

[0086] 12) Count the ranges of various kinematic parameters, including speed, acceleration, and relative distance. For the extracted vehicle trajectory information, calculate the maximum and minimum speeds of all vehicles, and estimate the acceleration based on the speed difference at consecutive time points. For relative distance, group the vehicle IDs and calculate the two-dimensional distance between adjacent groups.

[0087] 13) Select simulation software that can dynamically control the movement of vehicles in real time. You can choose Highway-env, Carla, etc.

[0088] Step 2: Establish a risk conflict prediction model, as follows:

[0089] 21) Design a conflict prediction module to perceive conflict points in advance by predicting future trajectories, and convert the upper-level semantics output by the action strategy into lower-level operations that can be input by the agent; the conflict prediction module mainly consists of two parts: trajectory prediction and risk area calculation;

[0090] 22) Regarding trajectory prediction;

[0091] A conflict prediction method for interactive areas based on IMM-KF-Bayes is proposed. Kinematic models of different behavior patterns are constructed by combining the motion characteristics of adversarial test vehicles. A future moment risk calculation method is constructed based on reinforcement learning actions. Figure 3 shown.

[0092] The vehicle behavior in the environment is described by the uniform speed model, uniform speed change model, and uniform angular velocity steering model. The observation likelihood function of each model is defined. Model, observation likelihood function Indicates at time Observed data In the model Next, given the state The probability of occurrence, It is a model In time The state vector of ; using the Bayesian formula, the posterior probability is calculated based on the prior probability and the observation likelihood function. The calculation method is as follows:

[0093] In the formula, It is a model From time Time The prior probability of .

[0094] Based on the updated model weights, the probability distribution of the predicted trajectory is obtained by weighting the prediction results of all models. The predicted state distribution of The calculation method is as follows:

[0095] in, From time Time All observation data of For Model The state transfer function of the vehicle is used to output the vehicle trajectory data within the preset time period in the future, including the position (x, y) and posture (θ) at each time node, and the uncertainty of trajectory prediction is quantified based on the Bayesian distribution.

[0096] 23) Regarding the calculation of risk areas;

[0097] For the conflict area prediction process, the historical motion information is used to predict the motion state of the next three states (time step 0.1s, cumulative 0.3s). The area around the smart car is divided into 8 areas: front, back, left, right, left front, left back, right front, and right back, which are respectively recorded as the first area, the second area, the third area, the fourth area, the fifth area, the sixth area, the seventh area, and the eighth area. The specific information of the area is as follows: Figure 2 As shown, it expands outwards by 0.4 times the width of the vehicle to be tested, and the center point of the outer contour of the area is used as the risk information calculation point. In order to associate the conflict prediction with the vehicle action during the test process, it is used for the subsequent period action guidance of the reinforcement learning algorithm; first, the kinematic information of the reinforcement learning traffic vehicle and the vehicle to be tested in each state is obtained using the information of the current moment and the future three states. The regional risk calculation method is as follows:

[0098] In the formula, For the Risk of conflict in each region; is the prediction uncertainty of the state; For the The calculation point in the region is The relative distance between the prediction time step and the traffic vehicle; For the The calculation point in the region is The component of the relative speed of the prediction time step and the traffic vehicle on the line connecting the two; For the The collision time for each prediction time step; and In the The contribution weights of the collision time and relative distance of each prediction time step to the cumulative risk. As the prediction step increases, the prediction accuracy will inevitably decrease. The safety assessment of the collision time at a longer distance is more accurate. Therefore, within each prediction step, the contribution weights are set as shown in Table 1.

[0099] Table 1 Contribution weights of conflict risk components

[0100]

[0101] Therefore, the risk indicators of different areas are obtained by calculating the collision time and safety distance information.

[0102] Step 3: Establish a test scenario evaluation method, which includes two parts: scenario criticality evaluation and scenario diversity evaluation. The details are as follows:

[0103] 31) Establish a scenario criticality evaluation method. A collision occurred during the scenario test, indicating that the scenario is dangerous to the system under test, but not all dangerous collision scenarios are valuable critical scenarios. Non-end-to-end intelligent driving algorithms strictly follow the "perception-decision-planning-control" process, and the perception of interactive objects is the prerequisite for triggering subsequent decisions. At the same time, if the interactive object is detected, even if the smart car adopts the maximum risk avoidance strategy, it still cannot avoid the collision, which means that the scenario exceeds the dynamic limit of the smart car. Such scenarios are unreasonable collision scenarios and cannot effectively optimize the performance of the system under test.

[0104] In summary, a collision rationality evaluation model consisting of a limited perception model and a playback test model is constructed. For the test scenario where a collision occurs, there are two requirements: On the one hand, the system under test perceives the target object. The present invention requires that the traffic vehicle must be detected by the perception model 3s before the collision. The perception model is designed in the description of the system under test, such as Figure 4 On the other hand, by replaying the test, the intelligent driving vehicle can be made to execute the maximum risk avoidance strategy (deceleration at the vehicle's limit braking deceleration in the present invention) at the sensing node to avoid the collision. If both requirements are met at the same time, it means that the scenario is a reasonable collision scenario and can effectively touch the performance boundary of the intelligent driving vehicle to be tested.

[0105] 32) Establish a scene diversity evaluation method, take two types of key scenes as anchor points, extract the trajectory data of the main vehicle and the traffic vehicle 5 seconds before and after the trigger moment for evaluation; design a trajectory sequence similarity evaluation method based on dynamic time warping, and finally output whether the key scene is a new scene, as follows:

[0106] 321) Define the high-dimensional distance measurement function with Euclidean distance to represent the position difference between trajectories; define the multi-dimensional space-time conflict vector as follows: ① Spatial features: the lateral distance between the main vehicle and the center of the lane. ② Distance conflict: the lateral and longitudinal distances between the centers of mass of the two vehicles. ③ Speed ​​conflict: the lateral and longitudinal relative speeds of the two vehicles. ④ Acceleration conflict: the lateral and longitudinal relative accelerations of the two vehicles. ⑤ Lane conflict: adjacent lane coefficient (yes 1, no 0), lane change behavior (close 1, far away 0, unchanged 0.5). ⑥ Time conflict: collision time information of two vehicles in the same lane. When the traffic vehicle is in front of the smart driving car, the relative distance is calculated divided by the relative speed. The upper limit is set to 10s, reflecting the situation with extremely low danger. For all collision and avoidable test scenarios, all instantaneous conflict indicators are normalized;

[0107] 322) Initialize the DTW matrix; create a (n+1)×(m+1) matrix D, where n and m are the time steps of the current key scene trajectory and each known scene trajectory in the scene library; set the first row and first column of the matrix to infinity;

[0108] 323) Fill the DTW matrix; Use the recursive formula to fill the entire matrix, for all i = 1 to n and j = 1 to m:

[0109] In the formula, and They are the currently generated key scene trajectory and the known scene trajectory in the scene library;

[0110] 324) Determine the similarity and find the minimum cumulative distance, that is, the minimum The preset threshold is set to 10. The DTW distance value is compared with the preset threshold. If the DTW distance value is small, the two trajectory sequences are considered similar; otherwise, they are considered dissimilar.

[0111] Step 4: Build a scene library based on index management, as follows:

[0112] 41) Develop key scene management tools based on Python automation scripts; design a scene library including scene ID, criticality, generation time, and algorithm version;

[0113] 42) For each generated scenario, design and create a corresponding index, and record relevant information in the scenario library.

[0114] Step 5: Build a basic reinforcement learning environment, set up a static environment and basic interfaces, as follows:

[0115] 51) Choose simulation software that can dynamically control the movement of vehicles in real time, such as the Highway library in Python, Carla software, or PreScan software;

[0116] 52) A basic longitudinal three-lane test scenario is built in the simulation software. A main vehicle equipped with an autonomous driving function and a traffic vehicle for reinforcement learning training are placed in the scenario. The three lanes are numbered 0, 1, and 2 from top to bottom. The lane number is determined by the centroid of the vehicle.

[0117] 53) Obtain the vehicle control interface in the simulation software, including throttle, brake and steering.

[0118] Step 6: The scene hierarchical scheduling phase is divided as follows:

[0119] 61) According to the vehicle motion information in the scene, the test process is divided into three parts: guidance period, confrontation period, and exploration period;

[0120] 62) The characteristics of the guidance period are low scene generation efficiency and difficulty in triggering collision failure. The triggering condition is continuous There is no collision in the cycle; the characteristics of the confrontation period are that most of the unreasonable collisions are generated, and it is difficult to explore the performance boundary. The trigger condition is continuous collision, but the number of cycles of unreasonable collisions reaches The exploration period is characterized by a single type of generated scenes, falling into local convergence, and the triggering condition is that reasonable collisions occur continuously, but the number of cycles that do not meet the diversity conditions reaches ;in, , and are the discrimination thresholds for each period.

[0121] Step 7: Reward mechanism design and strategy optimization. Adjust the design according to the scenario period and vehicle behavior. The details are as follows:

[0122] 71) Design a reward mechanism; the core task of the reward mechanism is to find more diverse edge test scenarios and divide them into three periods to enhance the algorithm’s diverse exploration capabilities; the reinforcement learning strategy architecture uses the existing basic DQN model; the core purpose of the guidance period is to guide the traffic vehicle to collide with the intelligent driving main vehicle, so the target action is to bring the two vehicles closer; entering the confrontation period, the reward function is set to the Euclidean distance correction value of the two vehicles and whether the traffic vehicle touches the road boundary; for the confrontation period, the traffic vehicle is required to trigger the performance boundary of the autonomous driving vehicle, so the reward function is set to the key evaluation results in step three in addition to whether the traffic vehicle touches the road boundary; for the exploration period, the traffic vehicle is required to explore more diverse scenarios and jump out of the local optimum, so the reward function is set as follows: ① Collision reward , weight ,②Key rewards , weight ,③Diversity Rewards , weight ④ Driving process rewards , weight . If a collision occurs, a collision reward of 20 points will be obtained; if the collision can be sensed by the system under test and the playback test passes, it means that the key test scenario has been found, and a criticality reward of 50 points will be obtained. If a new scenario that is not in the scenario library is found, a diversity reward of 50 points will be obtained. If the vehicle drives out of the road boundary or exceeds the speed limit, a single-step reward of -2 points will be obtained. The weights of each reward are set to be dynamically adjustable, as follows.

[0123] The weights of various rewards are set to be dynamically adjustable, as follows:

[0124]

[0125] The reinforcement learning optimization part outputs different lower-level action strategies and reward mechanisms based on the period division results. In the induction period, it is expected that the traffic car will approach the main car, collide with it, and obtain more collision rewards. In the confrontation period, it is expected that the traffic car will have a reasonable collision and obtain more reasonable collision rewards. In the induction period, it is expected that the traffic car will jump out of the local optimum and obtain more diverse rewards.

[0126] Therefore, the calculation method of each reward weight in the mth round is as follows:

[0127] In the formula, , and It is the basic weight of collision reward, criticality reward and diversity reward. , and is the adjustment factor of the corresponding weight, represents the number of consecutive cycles without collision, Indicates the number of cycles in which unreasonable collisions occur continuously. Indicates the number of cycles of finding repeated reasonable collision scenarios. The specific reward value is calculated through the scenario evaluation module.

[0128] 72) Guiding action strategy; for the confrontation period, introduce an action guidance strategy based on risk areas; according to the risk area identification results in step 2, control the movement of traffic vehicles.

[0129] The recommended actions for each period are calculated according to the risk situation of each risk area. The present invention divides the period into three parts: the guidance period, the confrontation period and the exploration period. For the guidance period, strive to cause a collision. If the risk of the predicted step size = 2 and = 3 is greater than the risk of the step size = 0 and = 1, no action is performed, otherwise the action is performed to approach the vehicle to be tested, and the action is solved by building a state machine based on the position and lane. For the confrontation period, on the basis of the occurrence of a collision, it is expected that a reasonable and avoidable collision will occur. The cumulative risk of the first, fifth and sixth interactive areas is calculated to see whether there is an area with the highest risk among all areas. If so, no action is performed, otherwise the action is performed to avoid collision with the vehicle to be tested; for the exploration period, on the premise of a reasonable and avoidable collision, more diverse collision scenarios are expected to occur. Based on the confrontation period strategy, a random action is generated every 4 execution steps. Through targeted action strategies in different periods, a guidance path is designed for the reinforcement learning agent from the perspective of conflict risk.

[0130] Step 8: Model training application and scenario data output, as follows:

[0131] 81) Based on the designed risk prediction model and reinforcement learning strategy, the model is trained using the collected natural driving data and the constructed test scenarios, and the agent strategy is updated by continuously interacting with the simulated environment; the reward function is adjusted to reflect the risk status of different scenarios and vehicle behaviors, so as to encourage the agent to learn safer and more effective driving strategies;

[0132] 82) During the reinforcement learning process, the system automatically generates a series of challenging driving scenarios; these scenarios include the state transition sequence of the agent under different decision-making stages, the actions taken and the corresponding reward values ​​received, and the driving trajectory parameters; the verified and evaluated effective scenario data is stored in the scenario library based on index management to achieve a complete test scenario generation process. The reward curve of the algorithm training process is as follows Figure 5 shown.

[0133] Example 2: See Figure 2 This embodiment provides a smart car edge scene generation system based on a layered reinforcement architecture, which is used to implement a smart car edge scene generation method based on a layered reinforcement architecture. The system includes:

[0134] The data processing module is used to process natural driving data and build the initial test environment;

[0135] Conflict prediction module, used to establish risk conflict prediction method;

[0136] The scenario evaluation module is used to establish the test scenario evaluation method, including the collision rationality evaluation and the time-space similarity evaluation;

[0137] The first building module is used to build a scene library based on index management;

[0138] The second building block is used to build a basic reinforcement learning environment, a static environment and basic interfaces;

[0139] Hierarchical scheduling module, which is used for hierarchical scheduling of scenarios, divides the test period based on scenario characteristics, and optimizes the reward mechanism and action strategy, adjusting them according to the scenario period and vehicle behavior;

[0140] Output module, used for model training application and scene data output.

[0141] In summary, the present invention aims at the reward sparsity phenomenon in the exploration process of reinforcement learning, constructs a scene prediction and guidance module, uses the historical trajectory of the vehicle to predict the risk area, guides the intelligent body to generate actions, and constructs an evaluation and preservation module and a hierarchical scheduling module to address the local convergence problem of reinforcement learning. A scene generation architecture based on the hierarchical idea is designed, which includes an upper scheduling module and a lower reinforcement module. The reinforcement learning edge scene generation process is divided into three parts: a guidance period, a confrontation period, and an exploration period, and hierarchical intervals are divided according to the scene state. According to different state designs, a targeted reward mechanism for traffic vehicles is designed, thereby converting the overall complex test problem into a simple subtask, and finally realizing the diverse generation of edge scenes, breaking through the local convergence problem and improving the efficiency of edge scene exploration.

[0142] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture, characterized in that: The following steps are involved: Step 1: Process natural driving data and build the initial test environment; Step 2: Establish a risk conflict prediction method; the details are as follows: 21) Design a conflict prediction module to perceive conflict points in advance by predicting future trajectories, and convert the upper-level semantics output by the action strategy into lower-level operations that can be input by the agent; the conflict prediction module mainly consists of two parts: trajectory prediction and risk area calculation; 22) The trajectory prediction part proposes an interactive area conflict prediction method based on IMM-KF-Bayes, combines the motion characteristics of the adversarial test vehicle to build a kinematic model of different behavior patterns, and builds a future moment risk calculation method based on reinforcement learning actions, as follows: The vehicle behavior in the environment is described by the uniform speed model, uniform speed change model, and uniform angular velocity steering model; the observation likelihood function of each model is defined. For the i-th model, the observation likelihood function represents the data z observed at time t t In model m i Next, given the state The probability of occurrence, is the model m i The state vector at time t; using the Bayesian formula, the posterior probability is calculated based on the prior probability and the observation likelihood function, and the calculation method is as follows: In the formula, π i (t|t-1) is the model m i The prior probability from time t-1 to time t; Based on the updated model weights, the probability distribution of the predicted trajectory is obtained by weighting the prediction results of all models, and the predicted state distribution x at time t is t The calculation method is as follows: Among them, Z 1:t is all the observation data from time 1 to time t; For model m i The state transfer function of the vehicle is used to output the vehicle trajectory data within the preset time period in the future, including the position and posture information at each time node, and the uncertainty of trajectory prediction is quantified based on the Bayesian distribution. 23) Risk area calculation part: For the conflict area prediction process, the historical motion information is used to predict the next three states, i.e., the time step is 0.1s, and the cumulative motion state is 0.3s; the area around the smart car is divided into 8 areas, namely, the front, rear, left, right, left front, left rear, right front, and right rear, which are respectively recorded as the first area, the second area, the third area, the fourth area, the fifth area, the sixth area, the seventh area, and the eighth area, which are expanded outward by 0.4 times the width of the vehicle to be tested, and the center point of the outer contour of the area is used as the risk information calculation point; the kinematic information of the reinforcement learning traffic vehicle and the vehicle to be tested in each state is obtained using the information of the current moment and the next three states. The regional risk calculation method is as follows: In the formula, r k is the conflict risk of the kth region; p(x t ) is the prediction uncertainty of the state; d kt Calculate the relative distance between the point in the kth region and the traffic vehicle at the tth prediction time step; v kt The component of the relative speed between the k-th region calculation point and the traffic vehicle on the line connecting the two at the t-th prediction time step; d kt / v kt is the collision time at the tth prediction time step; ω 1t and ω 2t are the contribution weights of the collision time and relative distance to the cumulative risk at the tth prediction time step, respectively; Thus, the risk indicators of different areas are obtained by calculating the collision time and safety distance information; Step 3: Establish a test scenario evaluation method, including collision rationality evaluation and spatiotemporal similarity evaluation, as follows: 31) Establish a scenario criticality evaluation method; construct a collision rationality evaluation model that includes a perception-constrained model and a playback test model. For the test scenario where a collision occurs, the requirements are as follows: On the one hand, the system to be tested perceives the target object, requiring that the traffic vehicle must be detected by the perception model 3 seconds before the collision; on the other hand, through playback testing, the intelligent driving main vehicle is required to execute the maximum risk avoidance strategy at the perception node; if both requirements are met at the same time, it means that the scenario is a reasonable collision scenario and can effectively touch the performance boundary of the intelligent driving vehicle to be tested; 32) A scene diversity evaluation method is established. Taking two types of key scenes as anchor points, the trajectory data of the main vehicle and the traffic vehicle 5 seconds before and after the triggering moment are extracted for evaluation; a trajectory sequence similarity evaluation method based on dynamic time warping is designed, and finally the key scene is output as follows: 321) Define the high-dimensional distance metric function with Euclidean distance to represent the position difference between trajectories; define the multi-dimensional spatiotemporal conflict vector as follows: ① spatial feature: the lateral distance between the main vehicle and the center of the lane; ② distance conflict: the lateral and longitudinal distance between the mass centers of the two vehicles; ③ speed conflict: the lateral and longitudinal relative speeds of the two vehicles; ④ acceleration conflict: the lateral and longitudinal relative accelerations of the two vehicles; ⑤ lane conflict: adjacent lane coefficient: if the two vehicles are in adjacent lanes, it is 1, otherwise it is 0; lane change behavior: if the two vehicles are close, it is 1, if they are far away, it is 0, and if they are unchanged, it is 0.5; ⑥ time conflict: the collision time information of the two vehicles in the same lane. When the traffic vehicle is in front of the intelligent driving car, the relative distance is calculated divided by the relative speed, and the upper limit is set to 10s, reflecting the situation of extremely low danger; for all collision and avoidable test scenarios, all instantaneous conflict indicators are normalized; 322) Initialize the DTW matrix; create a (n+1)×(m+1) matrix D, where n and m are the time steps of the current key scene trajectory and each known scene trajectory in the scene library; set the first row and the first column of the matrix to infinity; 323) Fill the DTW matrix; Use the recursive formula to fill the entire matrix, for all i = 1 to n and j = 1 to m: D ij =d(traj i (i-1),traj j (j-1))+min(D (i-1)(j-1) ,D (i-1)(j) ,D (i)(j-1) ) In the formula, traj i and traj j They are the currently generated key scene trajectory and the known scene trajectory in the scene library; 324) Determine the similarity and find the minimum cumulative distance, that is, the minimum D ij The preset threshold is set to 10. The DTW distance value is compared with the preset threshold. If the DTW distance value is small, the two trajectory sequences are considered similar; otherwise, they are considered dissimilar. Step 4: Build a scene library based on index management; Step 5: Build a basic reinforcement learning environment, set up a static environment and basic interface; Step 6: Divide the test period by scenario hierarchical scheduling. Divide the test period based on scenario characteristics. The details are as follows: 61) According to the vehicle motion information in the scene, the test process is divided into three parts: guidance period, confrontation period, and exploration period; 62) The characteristics of the guidance period are low scene generation efficiency and difficulty in triggering collision failure. The triggering condition is continuous τ guide There is no collision in the cycle; the characteristics of the confrontation period are that most of the unreasonable collisions are generated, and it is difficult to explore the performance boundary. The trigger condition is continuous collision, but the number of cycles of unreasonable collisions reaches τ oppose The exploration period is characterized by a single type of generated scenes, falling into local convergence, and the triggering condition is that reasonable collisions occur continuously, but the number of cycles that do not meet the diversity conditions reaches τ explore ; where τ guide , τ oppose and τ explore are the discrimination thresholds for each period; Step 7: Optimize the reward mechanism and action strategy, and adjust according to the scene period and vehicle behavior, as follows: 71) Design a reward mechanism; the core task of the reward mechanism is to find more diverse edge test scenarios, and divide them into three periods to enhance the algorithm's diversity exploration capability; the reinforcement learning strategy architecture adopts the basic DQN model; the core purpose of the guidance period is to guide the traffic vehicle to collide with the intelligent driving main vehicle, so the target action is to bring the two vehicles closer; enter the confrontation period, so the reward function is set to the Euclidean distance correction value of the two vehicles and whether the traffic vehicle touches the road boundary; for the confrontation period, the traffic vehicle is required to trigger the performance boundary of the autonomous driving vehicle, so the reward function is set to the key evaluation result in step 3 in addition to whether the traffic vehicle touches the road boundary; for the exploration period, the traffic vehicle is required to explore more diverse scenarios and jump out of the local optimum, so the reward function is set as follows: ① Collision reward r collision , weight ω collision ,② Critical reward r critical , weight ω critical ,③Diversity reward r diversity , weight ω diversity , ④ Driving process reward r process , weight ω process ; If a collision occurs, a collision reward of 20 points will be obtained; if the collision can be sensed by the system under test and the playback test passes, it means that the key test scenario has been found, and a criticality reward of 50 points will be obtained; if a new scenario that is not in the scenario library is found, a diversity reward of 50 points will be obtained; if the vehicle goes out of the road boundary or exceeds the speed limit, a single-step reward of -2 points will be obtained; the weights of various rewards are set to be dynamically adjustable, as follows: r=ω collision ×r collision +oh critical ×r critical +oh diversity ×r diversity +oh process ×r process The reinforcement learning optimization part outputs different lower-level action strategies and reward mechanisms based on the period division results. In the guidance period, it is expected that the traffic vehicle will approach the main vehicle, collide with it, and obtain more collision rewards. In the confrontation period, it is expected that the traffic vehicle will have a reasonable collision and obtain more reasonable collision rewards. In the guidance period, it is expected that the traffic vehicle will jump out of the local optimum and obtain more diverse rewards. Therefore, the calculation method of each reward weight in the mth round is as follows: oh collision (m)=o collision ×exp(α×min(max(0,p1(m)-τ guide ),1)) oh critical (m)=o critical ×exp(β×min(max(0,p2(m)-τ oppose ),1)) ω diversity (m) = o diversity ×exp(δ×min(max(0,p3(m)-τ explore ),1))In the formula, o collision , o critical and diversity are the basic weights of collision reward, critical reward and diversity reward; α, β and δ are the adjustment factors of the corresponding weights; p1(m) is the number of consecutive cycles without collision; p2(m) is the number of consecutive cycles with unreasonable collision; p3(m) is the number of consecutive cycles with repeated reasonable collision scenarios. The specific reward value is calculated by the scenario evaluation module; 72) Guiding action strategy: For the confrontation period, introduce the action guidance strategy based on the risk area; according to the risk area identification result in step 2, control the action of the traffic vehicle; The recommended actions for each period are calculated according to the risk situation of each risk area; the period is divided into three parts: the guidance period, the confrontation period and the exploration period; for the guidance period, it is strived to cause a collision. If the risk of the predicted step length = 2 and = 3 is greater than the risk of the step length = 0 and = 1, no action is performed, otherwise the action is performed to approach the vehicle to be tested, and the action is solved by building a state machine based on the position and lane; for the confrontation period, on the basis of the collision, it is expected that a reasonable and avoidable collision will occur, and the cumulative risk of the first area, the fifth area and the sixth area is calculated to see whether there is an area with the highest risk among all areas. If so, no action is performed, otherwise the action is performed to avoid collision with the vehicle to be tested; for the exploration period, on the premise of a reasonable and avoidable collision, more diverse collision scenarios are expected to occur, and a random action is generated every 4 execution steps based on the confrontation period strategy; through targeted action strategies in different periods, a guidance path is designed for the reinforcement learning agent from the perspective of conflict risk; Step 8: Model training application and scenario data output, as follows: 81) Based on the designed risk prediction model and reinforcement learning strategy, the collected natural driving data and constructed test scenarios are used for model training, and the agent strategy is updated by continuously interacting with the simulated environment; the reward function is adjusted to reflect the risk status of different scenarios and vehicle behaviors, so as to encourage the agent to learn safer and more effective driving strategies; 82) During the reinforcement learning process, the system will automatically generate a series of challenging driving scenarios; these scenarios include the state transition sequence of the agent under different decision-making stages, the actions taken and the corresponding reward values ​​received, and the driving trajectory parameters; the verified and evaluated valid scenario data will be stored in the index-based management scenario library to realize the complete test scenario generation process.

2. According to claim 1, a method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture is characterized in that: The specific method of step one is as follows: 11) Extracting vehicle trajectory information in combination with an existing natural driving database; the vehicle trajectory information includes a data subset of a time period, a lane, and a vehicle ID; for each vehicle ID, sorting the data based on a timestamp, and forming a continuous time series based on the position information on a two-dimensional plane, i.e., the vehicle's driving trajectory, and converting the vehicle's driving trajectory into a list; 12) Calculate the range of various kinematic parameters, including relative speed and relative distance; calculate the distribution of each variable; 13) Select simulation software that can dynamically control the movement of vehicles in real time.

3. According to claim 1, a method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture is characterized in that: The specific method of step 4 is as follows: 41) Develop key scene management tools based on Python automation scripts; design a scene library including scene ID, criticality, generation time, and algorithm version; 42) For each generated scene, design and create a corresponding index, and record the relevant information in the scene library.

4. According to the method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture as described in claim 1, it is characterized in that: The specific method of step five is as follows: 51) Select simulation software that can dynamically control the movement of vehicles in real time; 52) A basic longitudinal three-lane test scenario is built in the simulation software. A main vehicle equipped with an autonomous driving function and a traffic vehicle for reinforcement learning training are placed in the scenario. The three lanes are numbered 0, 1, and 2 from top to bottom. The lane number is determined by the centroid of the vehicle. 53) Obtain the vehicle control interface in the simulation software, including throttle, brake and steering.

5. The method for generating edge scenes of smart cars based on a hierarchical reinforcement architecture according to claim 1, characterized in that: The system is implemented by a smart car edge scene generation system based on a hierarchical reinforcement architecture, and the system includes: The data processing module is used to process natural driving data and build the initial test environment; Conflict prediction module, used to establish risk conflict prediction method; The scenario evaluation module is used to establish the test scenario evaluation method, including the collision rationality evaluation and the time-space similarity evaluation; The first building module is used to build a scene library based on index management; The second building block is used to build a basic reinforcement learning environment, a static environment and basic interfaces; Hierarchical scheduling module, which is used for hierarchical scheduling of scenarios, divides the test period based on scenario characteristics, and optimizes the reward mechanism and action strategy, adjusting them according to the scenario period and vehicle behavior; Output module, used for model training application and scene data output.

Images