Autonomous driving scenario generation method based on the combination of random generation and adjustment strategy

Abstract

The present invention relates to a method and system for generating an autonomous driving scene based on a combination of random generation and adjustment strategies, the method comprising: S1: processing the map information of the simulator, randomly selecting a suitable road area to place the own vehicle EGO; S2: determining the visible range of the own vehicle EGO, randomly generating other vehicles on the road in the visible range area, and adjusting out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain an initial scene; S3: according to the user's demand for the final scene, through advanced adjustments, the vehicle position in the initial scene is adjusted based on the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box of the vehicle in the front view image of the own vehicle EGO to obtain the final scene. The method provided by the present invention can save manpower and material resources, and can test dangerous scenes that are difficult to set up in reality, which helps to quickly discover safety hazards in the autonomous driving system.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method and system for generating an autonomous driving scene based on a combination of random generation and adjustment strategies. Background Art

[0002] As autonomous driving technology becomes more and more popular, people's requirements for the safety of autonomous driving are gradually increasing. Autonomous driving systems need to be able to respond promptly and quickly to complex and diverse driving scenarios to avoid accidents. Therefore, there is an urgent need to fully verify the autonomous driving system model, both in terms of performance and robustness, to ensure the safety of autonomous driving. The perception system is a key module in the autonomous driving system, which is used to perceive static or dynamic objects in the surrounding environment. With the great achievements of deep learning in tasks such as visual perception, perception models based on deep learning (DNN) have become the main way for autonomous driving systems to obtain environmental information. Therefore, a comprehensive evaluation of the performance and robustness of these perception models is of great significance for analyzing and improving the safety of autonomous driving.

[0003] DNN perception models need to be trained on training data, so the quality of the training data determines the practicality of these models. The usual verification method is to test the model on a validation set independent of the training set to determine whether the perception model is overfitting or robust. Commonly used data include public datasets and collected data. However, public datasets usually lack sufficient training data and test data, and collecting real-world data can be slow and expensive because the data must be preprocessed and properly labeled before use. On the other hand, autonomous driving systems are more likely to have problems in rare and dangerous scenarios. However, due to the nature of these scenarios, collecting data describing such unusual situations can be expensive and difficult to scale. It is impractical to build a physical fleet that can run for billions of hours in all possible test environments. Therefore, how to quickly and massively generate driving scenarios that meet user requirements and reflect real-world driving situations has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an autonomous driving scene generation method and system based on a combination of random generation and adjustment strategies.

[0005] The technical solution of the present invention is: a method for generating an autonomous driving scene based on a combination of random generation and adjustment strategies, comprising:

[0006] Step S1: Process the map information of the simulator and randomly select a suitable road area to place the own vehicle EGO;

[0007] Step S2: determining the visible range of the own vehicle EGO, randomly generating other vehicles on the road in the visible range, and adjusting out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain an initial scene;

[0008] Step S3: According to the user's requirements for the final scene, through advanced adjustments, the vehicle position in the initial scene is adjusted based on the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box of the vehicle in the front view image of the own vehicle EGO to obtain the final scene.

[0009] Compared with the prior art, the present invention has the following advantages:

[0010] The present invention discloses a method for generating autonomous driving scenarios based on a combination of random generation and adjustment strategies. By quickly and massively generating simulated driving scenarios in a simulator and collecting synthetic data from simulated sensors, developers can test the performance of autonomous driving systems in different scenarios. The method provided by the present invention can effectively save manpower and material resources, and can test dangerous scenarios that are difficult to set up in reality, which helps to quickly discover safety hazards in autonomous driving systems, thereby helping algorithm designers to improve the design of autonomous driving systems and improve overall safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flow chart of a method for generating an autonomous driving scene based on a combination of random generation and adjustment strategies in an embodiment of the present invention;

[0012] Figure 2 This is a schematic diagram of map information in an embodiment of the present invention;

[0013] Figure 3 Schematic diagram of the EGO of the vehicle itself in an embodiment of the present invention;

[0014] Figure 4 Schematic diagram of other vehicles in embodiments of the present invention;

[0015] Figure 5 Schematic diagram of a two-dimensional bounding box of a vehicle in an EGO front view image in an embodiment of the present invention;

[0016] Figure 6 A schematic diagram of a truncated ratio of a vehicle in an EGO front view image according to an embodiment of the present invention;

[0017] Figure 7 Schematic diagram of the occlusion ratio of a vehicle in an EGO front view image in an embodiment of the present invention;

[0018] Figure 8 This is a structural block diagram of an autonomous driving scenario generation system based on a combination of random generation and adjustment strategies in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention provides an autonomous driving scenario generation method based on a combination of random generation and adjustment strategies, which can effectively save manpower and material resources, and can test dangerous scenarios that are difficult to set up in reality, which helps to quickly discover safety hazards in the autonomous driving system, thereby helping algorithm designers to improve the design of the autonomous driving system and improve overall safety.

[0020] Embodiment 1

[0021] like Figure 1 As shown, an embodiment of the present invention provides an automatic driving scene generation method based on a combination of random generation and adjustment strategy, comprising the following steps:

[0022] Step S1: Process the map information of the simulator and randomly select a suitable road area to place the own vehicle EGO;

[0023] Step S2: Determine the visible range of the vehicle EGO, randomly generate other vehicles on the road in the visible range, and adjust the out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain the initial scene;

[0024] Step S3: According to the user's requirements for the final scene, through advanced adjustments, the vehicle position in the initial scene is adjusted based on the pixel area height, truncation ratio and occlusion ratio of the vehicle's two-dimensional bounding box in the front view image of the own vehicle EGO to obtain the final scene.

[0025] In one embodiment, the above step S1: processing the map information of the simulator and randomly selecting a suitable road area to place the own vehicle EGO specifically includes:

[0026] Step S11: extracting data information of a road R, a lane L of the road R, and a center line C of the lane L from the map information;

[0027] The embodiment of the present invention uses but is not limited to using the open source autonomous driving simulator Carla, in which the map data format is the OpenDrive format, extracts the map information in the simulator, obtains the road R, the lane L of the road R, and the center line C of the lane L, such as Figure 2 shown.

[0028] Step S12: Randomly select a point p on the center line C as the position of the EGO of the own vehicle, and set the direction h of the EGO to the tangent direction of the point p at C.

[0029] like Figure 3As shown, a point p on the center line of a lane at the intersection is selected to place the EGO. Since the EGO is turning at this time, its direction h is set to the tangent direction of the point p at C.

[0030] In one embodiment, the above step S2: determining the visible range of the own vehicle, randomly generating other vehicles on the road in the visible range, and adjusting out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain an initial scene, specifically includes:

[0031] Step S21: Define the EGO visible range area as P EGO , the road area in the map is defined as P R , and define the overlapping area between the two as P I ;

[0032] like Figure 2 As shown, the fan-shaped area in front of EGO is its visible range area P EGO , the overlapping area with the road is the diagonal area P I , which is the area where the driving scene is generated later.

[0033] Step S22: In P I Randomly select a point p to place a new vehicle O, repeat n times, and you can get n newly generated vehicles. represents the set of n vehicles generated;

[0034] like Figure 4 As shown, in P I In the area, select point p on the intersection to place vehicle O. Since O is on a curve at this time, p c is the point on the lane centerline closest to p, and the orientation of vehicle O is c The tangent lines on the curve have the same direction;

[0035] Step S23: For a vehicle O whose vehicle boundary exceeds the road boundary i , let its position be p i , adjust its position according to formula (1);

[0036]

[0037] Among them, (x i ,y i ) is O i Coordinates in the coordinate system of the two-dimensional top-down plane; (x c ,y c ) is O i The center line of the lane is i The nearest point p c Coordinates in a two-dimensional top-down plane coordinate system;

[0038] Because in P I The vehicle is placed at a random position in the area, so the edge of the vehicle may exceed the road boundary, so the vehicle position needs to be adjusted several times to make it closer to the center line of the lane and finally place the vehicle completely on the lane;

[0039] Step S24: Repeat step S23 until i The boundary of the lane no longer exceeds the boundary of the lane, and maintains O i The direction and p c The tangent lines at the points have the same direction;

[0040] Step S25: grouping all overlapping vehicles so that any vehicle in the same group overlaps with at least one other vehicle in the same group, and any two vehicles in different groups do not overlap;

[0041] Step S26: For the parent group of vehicles Randomly select a vehicle in the group as the reserved vehicle and remove the vehicles that intersect with it; determine whether there is any overlap between the remaining vehicles. If there is overlap, continue to remove the remaining vehicles until there is no overlap between the reserved vehicles in the group.

[0042] After the basic adjustments in the above steps, an initial scene that conforms to basic physical rules and general traffic rules can be obtained. Based on this initial scene, advanced adjustments can be made according to user needs to obtain driving scenes of different difficulty levels.

[0043] In one embodiment, the above step S3: according to the user's demand for the final scene, the position of the vehicle in the initial scene is adjusted through advanced adjustment based on the height of the pixel area of ​​the two-dimensional bounding box of the vehicle in the front view image of the own vehicle EGO, the truncation ratio and the occlusion ratio to obtain the final scene, specifically including:

[0044] Step S31: Obtain the coordinates of the three-dimensional bounding box of the vehicle in the simulator world. Let the coordinates of point p of the three-dimensional bounding box be (x, y, z), and then project the coordinates of the point to the two-dimensional image captured by the image sensor of the EGO, that is, the coordinates of the front view image of the EGO are as shown in formula (2). Thus, the two-dimensional bounding box of each vehicle can be obtained, and the distance between the vehicle and the EGO can be calculated according to the height of the two-dimensional bounding box;

[0045]

[0046] Where, intrinsic and extrinsic are the intrinsic parameter matrix and extrinsic parameter matrix of the EGO image sensor respectively, and (x', y') is the coordinate of point p in the two-dimensional image coordinate system collected by the EGO image sensor;

[0047] like Figure 5 As shown, it is a two-dimensional bounding box of the vehicle in the front view image of EGO; according to the height of the two-dimensional bounding box, the distance between the vehicle and EGO can be calculated;

[0048] Step S32: Calculate the cutoff ratio of the two-dimensional bounding box in the front view image and the ratio of the vehicle appearing in the visual range of the EGO according to the coordinates of the two-dimensional bounding box of the vehicle and the coordinates of the front view image;

[0049] like Figure 6 As shown in Figure 1, the upper left corner of the EGO front view image is the origin of the front view image coordinate system (0, 0), and the image area is the area formed by the coordinates from the origin of the image coordinate system to the lower right corner of the image. When only part of the vehicle appears in the front view image, the two-dimensional bounding box of the vehicle calculated using formula (2) may exceed the area range of the front view image, for example Figure 6 The left part of the frame 1 in the figure exceeds the range of the EGO front view image, so the y-axis coordinates of its upper left corner and lower left corner are calculated to be negative values ​​less than 0. At this time, the two-dimensional bounding box is truncated, and only the part within the front view image area is retained, that is, frame 2. At this time, the truncation ratio of the vehicle is the ratio of the area of ​​frame 2 to the area of ​​frame 1.

[0050] Step S33: obtaining an instance segmentation image of each vehicle when it is blocked, and an instance segmentation image of each vehicle when it is not blocked at all, and calculating the area ratio of the segmented regions of the two to obtain the blocking ratio of the vehicle;

[0051] like Figure 7 Therefore, the simulator is used to extract the instance segmentation image of the vehicle in the EGO front view image when it is occluded. Then, by removing other vehicles, only the instance segmentation image of the vehicle when it is completely unoccluded is retained, and this step is repeated until the instance segmentation image of each vehicle in the EGO front view image when it is completely unoccluded is extracted. The area ratio of the instance segmentation image of each vehicle that is occluded and completely unoccluded is calculated to obtain the occlusion ratio of each vehicle;

[0052] Step S34: according to the pixel area height of the two-dimensional bounding box of the vehicle, the cutoff ratio of the two-dimensional bounding box of the vehicle and the occlusion ratio of the vehicle, the difficulty level of the vehicle in the target detection task is set; the user selects the difficulty level to achieve the setting of the final scene;

[0053] The embodiment of the present invention sets three difficulty levels for the vehicle in the autonomous driving scene: simple, medium and difficult, based on the pixel area height, truncation ratio and occlusion ratio of the vehicle's two-dimensional bounding box. Users can select the corresponding difficulty level according to their needs.

[0054] When the pixel area height of the vehicle's two-dimensional bounding box is greater than 40: when the occlusion ratio is 0 and the truncation ratio is in the interval [0,0.15), the detection difficulty is easy; when the occlusion ratio is in the interval (0,0.5) and the truncation ratio is in the interval [0,0.3), or when the occlusion ratio is 0 and the truncation ratio is in the interval [0.15,0.3), the detection difficulty is medium, and other cases are difficult;

[0055] When the pixel area height of the vehicle's two-dimensional bounding box is greater than 25 and less than 40: the occlusion ratio is in the interval [0,0.5) and the truncation ratio is in the interval [0,0.3), the detection difficulty is medium, and other cases are difficult;

[0056] When the pixel area height of the vehicle's two-dimensional bounding box is less than 25: the detection difficulty is difficult.

[0057] Step S35: Through advanced adjustments, the position and orientation of the vehicle in the initial scene are iteratively adjusted, and the pixel area height, truncation ratio, and occlusion ratio of the two-dimensional bounding box are recalculated after each adjustment until a final scene that meets the requirements is generated.

[0058] The pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box of each vehicle are calculated for the generated initial scene, and the detection difficulty of each vehicle is evaluated according to step S34, and it is determined whether the scene meets the requirements. For scenes that do not meet the requirements, for example, the user requires the generation of a scene with difficult vehicles, but the generated scene only has simple or medium vehicles, the following operations are performed: a fixed initial velocity v is assigned to each vehicle in the scene, and each vehicle is made to move forward in the direction of the road it is currently on for a time t. During this period, every The newly generated scene is re-evaluated according to step S34 at the time t. If a new scene meets the requirements within the time t, the adjustment is stopped and the new scene is output. If no scene that meets the requirements is found after the time t, the vehicle position is adjusted and recalculated.

[0059] The present invention discloses a method for generating autonomous driving scenarios based on a combination of random generation and adjustment strategies. By quickly and massively generating simulated driving scenarios in a simulator and collecting synthetic data from simulated sensors, developers can test the performance of autonomous driving systems in different scenarios. The method provided by the present invention can effectively save manpower and material resources, and can test dangerous scenarios that are difficult to set up in reality, which helps to quickly discover safety hazards in autonomous driving systems, thereby helping algorithm designers to improve the design of autonomous driving systems and improve overall safety.

[0060] Embodiment 2

[0061] like Figure 8As shown, an embodiment of the present invention provides an autonomous driving scene generation system based on a combination of random generation and adjustment strategies, including the following modules:

[0062] A scene initialization module 41 is used to process the map information of the simulator and randomly select a suitable road area to place the own vehicle EGO;

[0063] The initial scene generation module 42 is used to determine the visible range of the own vehicle, randomly generate other vehicles on the road in the visible range area, and adjust the out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain the initial scene;

[0064] The final scene generation module 43 is used to adjust the vehicle position in the initial scene based on the user's demand for the final scene through advanced adjustments based on the pixel area height, truncation ratio and occlusion ratio of the vehicle's two-dimensional bounding box in the front view image of the own vehicle EGO to obtain the final scene.

[0065] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A method for generating an autonomous driving scenario based on a combination of random generation and adjustment strategies, characterized in that: include: Step S1: Process the map information of the simulator and randomly select a suitable road area to place the own vehicle EGO; Step S2: Determine the visible range of the own vehicle EGO, randomly generate other vehicles on the road in the visible range, and adjust out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain an initial scene, specifically including: Step S21: Define the EGO visual range area as , the road area in the map is defined as , and define the overlapping area between the two as ; Step S22: Randomly select a point Placing a new vehicle , repeat n times, and you can get n newly generated vehicles. represents the set of n cars generated; Step S23: For vehicles whose vehicle boundary exceeds the road boundary , and set its position to , adjust its position according to formula (1); (1) in, for Coordinates in the coordinate system of the two-dimensional top-down plane; for The center line of the lane where you are located The nearest point Coordinates in a two-dimensional top-down plane coordinate system; Step S24: Repeat step S23 until The boundary of the lane no longer exceeds the boundary of the lane and remains The direction and The tangent lines at the points have the same direction; Step S25: grouping all overlapping vehicles so that any vehicle in the same group overlaps with at least one other vehicle in the same group, and any two vehicles in different groups do not overlap; Step S26: For each group of vehicles , randomly select a vehicle in the group as the reserved vehicle and remove the vehicles that intersect with it; determine whether there is any overlap between the remaining vehicles. If there is overlap, continue to remove the remaining vehicles until there is no overlap between the reserved vehicles in the group; Step S3: According to the user's requirements for the final scene, the position of the vehicle in the initial scene is adjusted through advanced adjustments based on the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box of the vehicle in the front view image of the own vehicle EGO to obtain the final scene, which specifically includes: Step S31: Obtain the coordinates of the three-dimensional bounding box of the vehicle in the simulator world, and set the points of the three-dimensional bounding box The coordinates of , then the coordinates of the point are projected onto the two-dimensional image captured by the image sensor of the EGO, that is, the coordinates of the front view image of the EGO are shown in formula (2), thereby obtaining the two-dimensional bounding box of each vehicle, and the distance between the vehicle and the EGO can be calculated according to the height of the two-dimensional bounding box; (2) Among them, intrinsic and extrinsic are the intrinsic parameter matrix and extrinsic parameter matrix of the EGO image sensor respectively. For point Coordinates in the two-dimensional image coordinate system acquired by the EGO image sensor; Step S32: calculating the cutoff ratio of the two-dimensional bounding box in the front view image and the ratio of the vehicle appearing in the visual range of the EGO according to the coordinates of the two-dimensional bounding box of the vehicle and the coordinates of the front view image; Step S33: obtaining an instance segmentation image of each of the vehicles when being blocked, and an instance segmentation image of each of the vehicles when being completely unblocked, and calculating the area ratio of the segmented regions of the two, so as to obtain the blocking ratio of the vehicle; Step S34: setting the difficulty of the vehicle in the target detection task according to the pixel area height of the two-dimensional bounding box of the vehicle, the truncation ratio of the two-dimensional bounding box of the vehicle, and the occlusion ratio of the vehicle; the user selects the difficulty to achieve the setting of the final scene; Step S35: Through advanced adjustments, the position and orientation of the vehicle in the initial scene are iteratively adjusted, and the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box are recalculated after each adjustment until a final scene that meets the requirements is generated.

2. The method for generating an autonomous driving scenario based on a combination of random generation and adjustment strategy according to claim 1, characterized in that: The step S1: processing the map information of the simulator and randomly selecting a suitable road area to place the vehicle EGO, specifically includes: Step S11: Extracting roads from map information ,the way Lane , and lanes Center line data information; Step S12: Randomly select a center line Previous point , as the position of the EGO of the own vehicle, the direction of the EGO Set as Point exist The tangent direction at .

3. An autonomous driving scenario generation system based on a combination of random generation and adjustment strategies, characterized in that: Includes the following modules: The scene initialization module is used to process the map information of the simulator and randomly select a suitable road area to place the vehicle EGO; The initial scene generation module is used to determine the visible range of the own vehicle, randomly generate other vehicles on the road in the visible range area, and adjust the out-of-bounds vehicles and overlapping vehicles through basic adjustments to obtain the initial scene, which specifically includes: Step S21: Define the EGO visual range area as , the road area in the map is defined as , and define the overlapping area between the two as ; Step S22: Randomly select a point Placing a new vehicle , repeat n times, and you can get n newly generated vehicles. represents the set of n cars generated; Step S23: For vehicles whose vehicle boundary exceeds the road boundary , and set its position to , adjust its position according to formula (1); (1) in, for Coordinates in the coordinate system of the two-dimensional top-down plane; for The center line of the lane where you are located The nearest point Coordinates in a two-dimensional top-down plane coordinate system; Step S24: Repeat step S23 until The boundary of the lane no longer exceeds the boundary of the lane and remains The direction and The tangent lines at the points have the same direction; Step S25: grouping all overlapping vehicles so that any vehicle in the same group overlaps with at least one other vehicle in the same group, and any two vehicles in different groups do not overlap; Step S26: For each group of vehicles , randomly select a vehicle in the group as the reserved vehicle and remove the vehicles that intersect with it; determine whether there is any overlap between the remaining vehicles. If there is overlap, continue to remove the remaining vehicles until there is no overlap between the reserved vehicles in the group; The final scene generation module is used to adjust the vehicle position in the initial scene based on the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box of the vehicle in the front view image of the own vehicle EGO through advanced adjustment according to the user's demand for the final scene, so as to obtain the final scene, specifically including: Step S31: Obtain the coordinates of the three-dimensional bounding box of the vehicle in the simulator world, and set the points of the three-dimensional bounding box The coordinates of , then the coordinates of the point are projected onto the two-dimensional image captured by the image sensor of the EGO, that is, the coordinates of the front view image of the EGO are shown in formula (2), thereby obtaining the two-dimensional bounding box of each vehicle, and the distance between the vehicle and the EGO can be calculated according to the height of the two-dimensional bounding box; (2) Among them, intrinsic and extrinsic are the intrinsic parameter matrix and extrinsic parameter matrix of the EGO image sensor respectively. For point Coordinates in the two-dimensional image coordinate system acquired by the EGO image sensor; Step S32: calculating the cutoff ratio of the two-dimensional bounding box in the front view image and the ratio of the vehicle appearing in the visual range of the EGO according to the coordinates of the two-dimensional bounding box of the vehicle and the coordinates of the front view image; Step S33: obtaining an instance segmentation image of each of the vehicles when being blocked, and an instance segmentation image of each of the vehicles when being completely unblocked, and calculating the area ratio of the segmented regions of the two, so as to obtain the blocking ratio of the vehicle; Step S34: setting the difficulty of the vehicle in the target detection task according to the pixel area height of the two-dimensional bounding box of the vehicle, the truncation ratio of the two-dimensional bounding box of the vehicle, and the occlusion ratio of the vehicle; the user selects the difficulty to achieve the setting of the final scene; Step S35: Through advanced adjustments, the position and orientation of the vehicle in the initial scene are iteratively adjusted, and the pixel area height, truncation ratio and occlusion ratio of the two-dimensional bounding box are recalculated after each adjustment until a final scene that meets the requirements is generated.

Images