Automatic driving simulation evaluation method and related equipment

Abstract

The invention provides an automatic driving simulation evaluation method and related equipment, and relates to the technical field of automatic driving. The method comprises the following steps: performing scene classification on an automatic driving simulation environment to obtain a simulation scene; determining a main vehicle and a target obstacle in the simulation scene; starting a simulation test, and extracting first driving data of the main vehicle and second driving data of the target obstacle; if the first driving data and the second driving data do not meet the evaluation rule corresponding to the simulation scene, it is judged that the simulation scene is invalid, a simulation scene failure early warning system is developed based on the simulation test set established by Logsim, the failure simulation scene is recognized, simulation test result distortion caused by the failure simulation scene is effectively avoided, and the simulation test efficiency is improved. The testing personnel can be notified to pay attention to and process the invalid simulation scene, the accuracy and reliability of the simulation test are improved, and the edition issuing quality is improved.

Description

Background Art

[0002] Autonomous driving simulation testing plays a vital role in the development of autonomous driving technology and can accelerate the iteration and verification process of autonomous driving technology.

[0003] In related technologies, Logsim is usually used to extract logs for autonomous driving simulation testing. However, the scenarios of the above autonomous driving simulation methods are relatively fixed. Once a new autonomous driving algorithm is launched, the scenarios of Logsim are prone to failure, resulting in simulation test failure that is difficult to detect, reducing the quality of the release.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides an autonomous driving simulation evaluation method and related equipment, which at least to a certain extent overcome the problem in the autonomous driving simulation evaluation method in the related art that the scene is fixed and failure leads to simulation test failure which is not easy to be discovered.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, there is provided an autonomous driving simulation evaluation method, comprising: performing scene classification on an autonomous driving simulation environment to obtain a simulation scene; determining a main vehicle and a target obstacle in the simulation scene; starting a simulation test to extract first driving data of the main vehicle and second driving data of the target obstacle; if the first driving data and the second driving data do not satisfy an evaluation rule corresponding to the simulation scene, determining that the simulation scene is invalid.

[0008] In one embodiment of the present disclosure, the evaluation rules corresponding to the simulation scenario include a first evaluation rule and a second evaluation rule; wherein, if the first driving data and the second driving data do not satisfy the evaluation rule corresponding to the simulation scenario, the simulation scenario is determined to be invalid, including: judging whether the target obstacle and the main vehicle satisfy the first evaluation rule based on the first driving data and the second driving data, the first evaluation rule characterizing that the target obstacle has an action corresponding to the simulation scenario on the main vehicle; if satisfied, obtaining the relative distance between the main vehicle and the target obstacle; if the relative distance between the main vehicle and the target obstacle does not satisfy the second evaluation rule within a preset time period, the simulation scenario is determined to be invalid, the second evaluation rule characterizing that the target obstacle interacts with the main vehicle after executing the action corresponding to the simulation scenario.

[0009] In one embodiment of the present disclosure, the method further includes: if the target obstacle and the host vehicle do not satisfy the first evaluation rule, determining that the simulation scenario is invalid.

[0010] In one embodiment of the present disclosure, the method further includes: if the relative distance between the main vehicle and the target obstacle satisfies a second evaluation rule within a preset time period, determining that the simulation scenario is valid, and generating a simulation evaluation report.

[0011] In one embodiment of the present disclosure, classifying the scenes of the autonomous driving simulation environment to obtain the simulation scenes includes: classifying the scenes of the autonomous driving simulation environment based on a deep learning model and / or a manual verification method to obtain the simulation scenes, wherein the simulation scenes include at least one of a cut-in scene, an overtaking scene, and a reverse driving scene.

[0012] In one embodiment of the present disclosure, after determining that the simulation scenario is failed if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scenario, the method further includes: if the simulation scenario fails, marking the simulation scenario as a failure scenario package, removing the evaluation results of the simulation scenario from the simulation evaluation test report, and generating a failure scenario report.

[0013] According to another aspect of the present disclosure, an autonomous driving simulation evaluation device is also provided, including: a scene classification module, used to perform scene classification on an autonomous driving simulation environment to obtain a simulation scene; a target determination module, used to determine a main vehicle and a target obstacle in the simulation scene; a data extraction module, used to start a simulation test, and extract first driving data of the main vehicle and second driving data of the target obstacle; and a simulation evaluation module, used to determine that the simulation scene is invalid if the first driving data and the second driving data do not meet the evaluation rules corresponding to the simulation scene.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned autonomous driving simulation evaluation method by executing the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned autonomous driving simulation evaluation method is implemented.

[0016] According to another aspect of the present disclosure, a computer program product is provided, which includes a computer program or computer instructions, and the computer program or the computer instructions are loaded and executed by a processor to enable a computer to implement the above-mentioned autonomous driving simulation evaluation method.

[0017] In the disclosed embodiment, the scene classification of the autonomous driving simulation environment is performed to obtain a simulation scene; the main vehicle and the target obstacle in the simulation scene are determined; the simulation test is started, and the first driving data of the main vehicle and the second driving data of the target obstacle are extracted; if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scene, the simulation scene is determined to be invalid. The disclosed system has developed a simulation scene failure warning system based on the simulation test set built by Logsim, which identifies failed simulation scenes and effectively avoids distortion of simulation test results caused by failed simulation scenes. The system can notify testers to pay attention to and deal with failed simulation scenes, thereby improving the accuracy and reliability of simulation tests and improving the quality of releases.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A flow chart of an autonomous driving simulation evaluation method provided by an embodiment of the present disclosure is shown;

[0021] Figure 2 A flow chart of another autonomous driving simulation evaluation method provided by an embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram showing a structure of a cut-in scenario provided by an embodiment of the present disclosure is shown;

[0023] Figure 4 A failure evaluation principle diagram of a cut-in scenario provided by an embodiment of the present disclosure is shown;

[0024] Figure 5 A structural schematic diagram showing another cut-in scenario provided by an embodiment of the present disclosure;

[0025] Figure 6 An example flow chart of an autonomous driving simulation evaluation method provided by an embodiment of the present disclosure is shown;

[0026] Figure 7 Another example flow chart of an autonomous driving simulation evaluation method provided by an embodiment of the present disclosure is shown;

[0027] Figure 8 A schematic diagram showing the structure of an automatic driving simulation evaluation device provided by an embodiment of the present disclosure is shown;

[0028] Fig. 9 A structural block diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0031] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0032] During the autonomous driving simulation test process, developers can quickly verify and optimize the performance of the autonomous driving system by continuously improving the simulation environment and test cases, which helps shorten the development cycle, improve development efficiency, and put autonomous driving technology into practical application more quickly. Simulation testing can also help developers detect and fix potential safety issues. By simulating various possible dangerous situations and traffic accidents in the simulation environment, the performance of the autonomous driving system in different situations can be evaluated, and improvements and optimizations can be made in a timely manner to improve the reliability and safety of the autonomous driving system.

[0033] In the related art, Logsim is usually used to extract logs for simulation testing. Logsim is the abbreviation of Log Simulation. It is a data playback simulation based on existing real data to build scenarios, including data collection, data storage and processing, data playback and simulation result analysis. Among them, data collection is to collect real road data through various sensors on the vehicle; data storage and processing is to store the real road data collected by various sensors into the database, and clean the data in the database to remove unnecessary interference and errors. Data playback is to simulate different traffic scenarios in the simulation system to test and verify autonomous driving. Simulation result analysis is the process of evaluating the performance of the intelligent driving system based on the test results, and correcting and optimizing the problems. Logsim is a scenario provided by road test data, which truly reflects the complex and changeable obstacles and traffic conditions in the actual traffic environment. It occupies an extremely important position in the simulation test process.

[0034] However, the test set scenarios collected based on Logsim are relatively fixed. After the algorithm is updated, the interaction between the main vehicle and the original obstacle vehicle is likely to fail, and the original Logsim scenarios are likely to fail, which in turn leads to distortion of the simulation test results. This situation is not easy to be discovered, which reduces the quality of the release and easily causes repeated simulations to occur from time to time, reducing simulation efficiency.

[0035] In order to at least partially solve the above technical problems, the present disclosure obtains simulation scenes by classifying scenes in an autonomous driving simulation environment; determines a main vehicle and a target obstacle in the simulation scene; starts a simulation test, and extracts first driving data of the main vehicle and second driving data of the target obstacle; if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scene, the simulation scene is determined to be invalid. The present disclosure develops a simulation scene failure warning system based on a simulation test set built by Logsim, which identifies failed simulation scenes and effectively avoids distortion of simulation test results caused by failed simulation scenes. The system can notify testers to pay attention to and handle failed simulation scenes, thereby improving the accuracy and reliability of simulation tests and the quality of releases.

[0036] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0037] The present exemplary implementation is described in detail below with reference to the accompanying drawings and embodiments.

[0038] First, an automatic driving simulation evaluation method is provided in the embodiments of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities. The automatic driving simulation evaluation method provided in the embodiments of the present disclosure can be executed by a server, for example, it can be executed by an automatic driving simulation evaluation device configured on the server, and the above device can be a programmable logic controller (PLC) or a single-chip microcomputer, etc., as long as it can realize the corresponding functions.

[0039] Figure 1 A flow chart of an autonomous driving simulation evaluation method provided in an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the autonomous driving simulation evaluation method provided in the embodiment of the present disclosure includes the following steps:

[0040] S102: Classify the scenes of the autonomous driving simulation environment to obtain simulation scenes.

[0041] The autonomous driving simulation environment may include the establishment of a scene material library, the development of simulation models, and the selection and use of simulation software. The establishment of a scene material library is the basis for building an autonomous driving simulation environment. The scene material library includes a variety of scene materials, such as roads, buildings, vegetation, etc.

[0042] In one embodiment, the above S102 classifies the scene of the autonomous driving simulation environment to obtain the simulation scene, including: classifying the scene of the autonomous driving simulation environment based on a deep learning model and / or a manual verification method to obtain the simulation scene. The deep learning model may include but is not limited to Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Deep Belief Network (DBN), Long Short-Term Memory (LSTM), Generative Adversarial Networks (GAN) and Attention Mechanism Transformer model, etc.

[0043] In an autonomous driving simulation test task, multiple scenarios can be simulated at the same time. The simulation scenario can include at least one of a cut-in scenario, an overtaking scenario, and a reverse driving scenario.

[0044] Cut-in scenarios refer to the various cut-in scenarios that autonomous vehicles encounter in actual road environments simulated by computer simulation technology, including but not limited to interactions with other vehicles, changes in road conditions, etc.

[0045] Overtaking scenario refers to a series of steps and conditions of overtaking operations in actual driving simulated by computer simulation technology. It is used to test the decision-making, planning and control capabilities of the autonomous driving system during the overtaking process, ensuring that overtaking actions can be completed safely and effectively under different conditions.

[0046] The reverse driving scenario refers to a simulation of a self-driving car encountering a situation where it needs to drive in the opposite direction during driving. It is used to test the reaction and response strategy of the self-driving system in a specific environment to ensure safety and reliability in complex traffic conditions.

[0047] After classifying the scenes in the autonomous driving simulation environment, each simulation scene can be labeled and stored in the database.

[0048] S104: Determine the host vehicle and target obstacles in the simulation scene.

[0049] The main vehicle is the autonomous vehicle in the simulation scene, namely Autonomous Vehicle (AV), which is the core object in the simulation system and is used to simulate real driving scenes and autonomous driving functions.

[0050] In the autonomous driving simulation, obstacles can include moving objects or stationary objects, such as roadblocks, ducklings used to simulate pedestrians, and cars. Roadblocks are stationary, while ducklings and cars can be stationary or moving.

[0051] In the simulation scene, there may be a variety of obstacles near the main vehicle, and the various obstacles are marked, and the marking information includes but is not limited to the number, type, name, etc. of the obstacle.

[0052] In S104, the target obstacle is an obstacle that may interact with the host vehicle in the simulation scene, thereby narrowing the tracking range of the obstacle, reducing the data collection requirements in the simulation scene, and improving the computing power.

[0053] S106 , starting a simulation test, and extracting first driving data of the host vehicle and second driving data of the target obstacle.

[0054] After determining the main vehicle and target obstacles, the simulation test can be started.

[0055] During the simulation test, the driving data of the main vehicle and the target obstacle can be extracted in real time. The first driving data includes at least one of the position, posture, speed, lane information, and map information of the main vehicle; the second driving data includes at least one of the position, posture, speed, lane information, map information, and obstacle type of the target obstacle.

[0056] The position of the host vehicle and the position of the target obstacle can be represented by the coordinates of the host vehicle in the simulation system. The posture of the host vehicle and the target obstacle can be represented by the driving direction of the vehicle, for example, the direction of the front and rear of the vehicle. Lane information refers to the lane position of the vehicle on the road, for example, the target obstacle is in the same lane as the host vehicle, the target obstacle is in an adjacent lane to the host vehicle, etc. Obstacle types can include obstacle vehicles, roadblocks, pedestrians, etc.

[0057] S108: If the first driving data and the second driving data do not satisfy the evaluation rule corresponding to the simulation scenario, determine that the simulation scenario is invalid.

[0058] In one embodiment, one simulation scene corresponds to one evaluation rule, and different simulation scenes have different evaluation rules. Accordingly, for different simulation scenes, the types of the first driving data of the host vehicle and the second driving data of the target obstacle required are also different.

[0059] For example, for a cut-in scenario, the first driving data and the second driving data may select data such as the vehicle's posture, position, and distance; for an overtaking scenario, the first driving data and the second driving data may also select data such as posture, position, distance, and speed.

[0060] In another embodiment, for the same simulation scenario, multiple evaluation rules and corresponding types of driving data may also be developed.

[0061] The evaluation rules serve as the evaluation criteria for whether the autonomous driving simulation test enters the simulation scene. The evaluation rules can be pre-stored in the database. When the simulation scene is obtained, the evaluation rules corresponding to the simulation scene can be bound from the database to evaluate the simulation scene and obtain a simulation test report.

[0062] When the first driving data and the second driving data meet the evaluation rules corresponding to the simulation scene, it indicates that the main vehicle has entered the simulation scene, the simulation scene is valid, and the simulation results in the simulation test report are valid; when the first driving data and the second driving data do not meet the evaluation rules corresponding to the simulation scene, it indicates that the main vehicle has not entered the simulation scene, and the simulation scene is invalid. At this time, the simulation results in the simulation test report are distorted and cannot be used as a basis for evaluating autonomous driving.

[0063] In the disclosed embodiment, the scene classification of the autonomous driving simulation environment is performed to obtain a simulation scene; the main vehicle and the target obstacle in the simulation scene are determined; the simulation test is started, and the first driving data of the main vehicle and the second driving data of the target obstacle are extracted; if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scene, the simulation scene is determined to be invalid. The disclosed system has developed a simulation scene failure warning system based on the simulation test set built by Logsim, which identifies failed simulation scenes and effectively avoids distortion of simulation test results caused by failed simulation scenes. The system can notify testers to pay attention to and deal with failed simulation scenes, thereby improving the accuracy and reliability of simulation tests and improving the quality of releases.

[0064] Figure 2 FIG. 2 is a flowchart of another automatic driving simulation evaluation method provided by an embodiment of the present disclosure. Figure 2 As shown, in one embodiment, the evaluation rules corresponding to the simulation scenario include a first evaluation rule and a second evaluation rule; wherein, if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scenario in S108, then determining that the simulation scenario is invalid includes:

[0065] S202, judging whether the target obstacle and the host vehicle satisfy a first evaluation rule based on the first driving data and the second driving data, where the first evaluation rule indicates that the target obstacle has an action on the host vehicle corresponding to the simulation scenario;

[0066] S204: If the conditions are met, the relative distance between the host vehicle and the target obstacle is obtained;

[0067] S206, determining whether the relative distance between the host vehicle and the target obstacle within a preset time period satisfies a second evaluation rule, if not, executing S208;

[0068] S208. If the relative distance between the host vehicle and the target obstacle does not satisfy a second evaluation rule within a preset period of time, the simulation scenario is determined to be invalid, and the second evaluation rule indicates that the target obstacle interacts with the host vehicle after executing an action corresponding to the simulation scenario.

[0069] When the simulation scene is a cut-in scene, the action corresponding to the simulation scene is a cut-in action, and the cut-in action includes the target obstacle cutting in from the left side or the right side of the main vehicle. The first evaluation rule indicates that the target obstacle has a cut-in action on the main vehicle. Figure 3 As shown in the figure, in the autonomous driving simulation environment, there are two obstacle vehicles on the road near the main vehicle, namely, obstacle vehicle ID1 and obstacle vehicle ID2. Among them, obstacle vehicle ID1 is an obstacle vehicle that may interact with the main vehicle, and obstacle vehicle ID1 is determined as the target obstacle. Obstacle vehicle ID1 is located on the left side of the main vehicle. At this time, the simulation scene is a left-side cut-in scene.

[0070] When the simulation scene is an overtaking scene, the action corresponding to the simulation scene is an overtaking action, and the overtaking action may include the host vehicle overtaking from the left side or the right side of the target obstacle. The first evaluation rule indicates that the host vehicle has an overtaking action on the target obstacle.

[0071] When the simulation scenario is a reverse driving scenario, the action corresponding to the reverse driving scenario is a reverse driving action, and the first evaluation rule indicates that the host vehicle has a reverse driving action on the target obstacle in the opposite lane.

[0072] In one embodiment, the first evaluation rule can be expressed in the form of a regular expression. For example, in the left cut-in scene, Figure 3 As shown, the position change of the target obstacle vehicle satisfies the following: the target obstacle vehicle starts from the last frame on the left side of the main vehicle, passes the left front of the main vehicle, and ends with the target obstacle vehicle’s front in front of the main vehicle and the target obstacle vehicle’s rear in front of the main vehicle in the first frame. Only then can it be determined that the first driving data and the second driving data meet the first evaluation rule. The position of the target obstacle vehicle’s front is obtained in sequence according to the time series as: A+mB+nC, where A indicates that the target obstacle vehicle’s front is on the left side of the main vehicle, B indicates that the target obstacle vehicle’s front is in front of the left side of the main vehicle, C indicates that the target obstacle vehicle’s front is in front of the main vehicle, m is the number of frames where the target obstacle vehicle’s front is in front of the left side of the main vehicle, and n is the number of frames where the target obstacle vehicle’s front is in front of the main vehicle.

[0073] If the frame number m of the target obstacle vehicle's front end in front of the left side of the main vehicle is less than or equal to the first preset frame number threshold and the frame number n of the target obstacle vehicle's front end in front of the main vehicle is less than or equal to the second preset frame number threshold, it is determined that the first evaluation rule is met. Among them, the first preset frame number threshold and the second preset frame number threshold are determined according to actual needs, and the present disclosure does not make specific limitations. For example, the first preset frame number threshold is 15, and the second preset frame number threshold is 5. When the expression of the front position of the target obstacle vehicle is obtained according to the time series as A+12B+3C, m=12<15, and n=3<5, it is determined that the first evaluation rule is met.

[0074] It should be noted that for overtaking scenarios and reverse driving scenarios, corresponding rule expressions can be set according to specific scenarios, and this disclosure will not go into details.

[0075] When the target obstacle and the main vehicle meet the first evaluation rule, it indicates that the action corresponding to the simulation scene has been completed. Figure 4 As shown, when the target obstacle vehicle passes from the left side of the main vehicle to the left front of the main vehicle and arrives in front of the main vehicle, the target obstacle vehicle completes the cutting action on the main vehicle.

[0076] The second evaluation rule can be used as an evaluation criterion for whether there is interaction between the target obstacle and the main vehicle after the target obstacle and the main vehicle complete the action corresponding to the simulation scene. The second evaluation rule can be pre-stored in a database, and when the simulation scene is obtained, the simulation scene can be associated with the second evaluation rule to evaluate the simulation scene.

[0077] The second evaluation rule may be a pre-stored distance threshold, for example, Figure 4 The distance threshold line in .

[0078] When the relative distance between the main vehicle and the target obstacle is greater than the distance threshold, it indicates that the second evaluation rule is not satisfied, there is no interaction between the main vehicle and the target obstacle, and the simulation scenario fails.

[0079] In one embodiment, the preset time period can be determined according to actual needs, for example, for a cut-in scene, 250 frames, 10 seconds, etc. after the target obstacle completes the cut-in action on the main vehicle.

[0080] like Figure 5 As shown in the figure, after the ID1 obstacle vehicle completes the cut-in action on the main vehicle, the main vehicle slows down due to certain reasons, causing the distance between the main vehicle and the ID1 obstacle vehicle to be greater than the distance threshold, so that there is no interaction between the main vehicle and the ID1 faulty vehicle. At this time, the simulation scenario fails.

[0081] In the disclosed embodiment, the first driving data and the second driving data are used to determine whether the target obstacle and the main vehicle satisfy the first evaluation rule and the second evaluation rule. When the first evaluation rule is satisfied but the second evaluation rule is not satisfied, the simulation scenario is determined to be invalid, that is, the target obstacle and the main vehicle complete the actions corresponding to the simulation scenario without interacting, thereby promptly reminding the user to check the failed simulation scenario and improving the reliability of the autonomous driving simulation.

[0082] Continue to refer Figure 2 In one embodiment, the autonomous driving simulation evaluation method of the disclosed embodiment further includes: if the target obstacle and the main vehicle do not satisfy the first evaluation rule, the simulation scenario is determined to be invalid.

[0083] When the target obstacle and the host vehicle do not satisfy the first evaluation rule, it indicates that the target obstacle and the host vehicle do not perform actions corresponding to the simulation scenario, thereby improving the efficiency of simulation scenario failure determination.

[0084] Continue to refer Figure 2 In one embodiment, the method further includes: S210, if the relative distance between the main vehicle and the target obstacle within a preset time period satisfies a second evaluation rule, the simulation scenario is determined to be valid, and a simulation evaluation report is generated.

[0085] When the host vehicle and the target obstacle satisfy the first evaluation rule and the second evaluation rule, it indicates that the host vehicle and the target obstacle have not only completed the actions corresponding to the simulation scene, but also interacted with each other, that is, the simulation scene is valid.

[0086] In one embodiment, in S108, if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scenario, then after determining that the simulation scenario has failed, the method further includes: if the simulation scenario has failed, marking the simulation scenario as a failure scenario package, removing the evaluation results of the simulation scenario from the simulation evaluation test report, and generating a failure scenario report.

[0087] The failure scenario report can include information such as the main vehicle, target obstacle, and failure type. Failure types can include action failure, interaction failure, etc. Action failure means that the main vehicle and the target obstacle do not perform actions corresponding to the simulation scenario, and interaction failure means that the main vehicle and the target obstacle complete actions corresponding to the simulation scenario but do not interact. When a simulation scenario fails, the corresponding evaluation results are removed in time to prevent users from misjudgment.

[0088] In order to deepen the understanding of the disclosed autonomous driving simulation evaluation method, the following Figure 3-7 Provide explanation.

[0089] like Figure 6 As shown, the automatic driving evaluation method disclosed in the present invention comprises the following steps:

[0090] S601, classify and store the evaluation scenarios. Classify the scenarios based on the deep learning model scenario mining and manual verification to obtain multiple simulation scenarios, such as cut-in scenarios, overtaking scenarios, and reverse driving scenarios. Different simulation scenarios have different evaluation rules and failure models. It is necessary to map the simulation scenarios with the corresponding evaluation rules and failure models, and store the mapping relationship in the database, which can be a distributed database, cloud database, etc.

[0091] S602, scene target obstacle labeling. Different simulation scenes correspond to different target obstacles, and it is necessary to record the target obstacle number (such as ID number) and the action type corresponding to the simulation scene, and store the relevant information of the target obstacle (such as the above ID number and action, etc.) in association with the simulation scene in the database.

[0092] After completing S601 and S602, the simulation test may begin.

[0093] S603, failure model evaluation. After the simulation test is started, a failure model evaluation can be performed to evaluate whether the target obstacle and the host vehicle perform an interactive action corresponding to the simulation scenario, that is, whether the first evaluation rule is satisfied, and if there is an interactive action corresponding to the simulation scenario, whether the second evaluation rule is satisfied.

[0094] S604: Determine whether the simulation scenario is invalid. If yes, execute S605; if no, execute S607. When the first evaluation rule is not satisfied or the first evaluation rule is satisfied but the second evaluation rule is not satisfied, the simulation scenario is determined to be invalid. When both the first evaluation rule and the second evaluation rule are satisfied, the simulation scenario is determined to be valid.

[0095] S605: Eliminate the simulation scene evaluation result. If the target obstacle does not meet the first evaluation rule or meets the first evaluation rule but does not meet the second evaluation rule, the simulation scene is determined to be invalid, and the simulation evaluation result eliminates the evaluation result under the simulation scene.

[0096] S606. Mark the scenario package of the simulation scenario in S605 as an invalid scenario package.

[0097] S607: Generate a simulation evaluation report and a scenario failure report.

[0098] like Figure 7 As shown, for the failure model evaluation in S603, different simulation scenarios correspond to different scenario failure models. The algorithm of the scenario failure model is Figure 7 S701~S704 in the implementation:

[0099] S701. Extract the position, posture, and speed of the host vehicle in real time.

[0100] S702: extract the position, posture, and speed of the target obstacle in real time.

[0101] S703: Calculate the relative position of the target obstacle and the host vehicle based on the Euclidean distance.

[0102] S704: Perform failure determination based on the rule discriminant.

[0103] Below Figure 3 and Figure 4 The left-side cut-in scene in the figure is used as an example for detailed description.

[0104] The following aspects are used to determine whether the target obstacle vehicle has cut into the main vehicle:

[0105] 1. The position (or relative position) of the target obstacle vehicle relative to the main vehicle, including but not limited to the relative position of the front of the target obstacle vehicle and the main vehicle and the relative position of the rear of the target obstacle vehicle and the main vehicle;

[0106] 2. The relative distance between the target obstacle vehicle and the main vehicle (e.g. polygon distance);

[0107] 3. The maximum relative distance threshold between the target obstacle vehicle and the main vehicle.

[0108] The evaluation rules include a first evaluation rule and a second evaluation rule. The first evaluation rule is expressed in the form of a rule discriminant. The second evaluation rule can be a distance threshold, a range, etc.

[0109] like Figure 4 As shown, A indicates that the front of the target obstacle vehicle is on the left side of the main vehicle, B indicates that the front of the target obstacle vehicle is in front of the left of the main vehicle, and C indicates that the front of the target faulty vehicle is in front of the main vehicle.

[0110] The first evaluation rule is satisfied if the position change of the target obstacle vehicle satisfies: the target obstacle vehicle starts from the last frame where the front of the target obstacle vehicle is on the left side of the main vehicle, passes the left front of the main vehicle, and ends at the first frame where the front of the target obstacle vehicle is in front of the main vehicle and the rear of the target obstacle vehicle is in front of the main vehicle. The specific steps are to obtain the position of the front of the target obstacle vehicle in sequence according to the time series and satisfy the expression: A+mB+nC. Among them, m and n are positive integers, m represents the number of frames where the front of the target obstacle vehicle is in front of the left front of the main vehicle, and n represents the number of frames where the front of the target obstacle vehicle is in front of the main vehicle and the rear of the target obstacle vehicle is in front of the main vehicle.

[0111] When the first evaluation rule is satisfied, that is, based on the position change of the target obstacle vehicle satisfying the cut-in action, the relative distance between the target obstacle vehicle and the host vehicle is obtained.

[0112] Meeting the second evaluation rule includes: the relative distance between the target obstacle vehicle and the main vehicle is less than or equal to the above-mentioned maximum relative distance threshold between the target obstacle vehicle and the main vehicle. For example, Figure 4 In each frame, the target obstacle vehicle appears within the elliptical distance threshold line, and the distance between the target obstacle vehicle and the main vehicle in each frame is determined to be less than or equal to the maximum relative distance threshold between the target obstacle vehicle and the main vehicle.

[0113] When the first evaluation rule and the second evaluation rule are satisfied, the simulation scenario is determined to be valid.

[0114] If the newly submitted test algorithm causes the main vehicle to slow down due to some reasons, the target obstacle vehicle and the main vehicle will no longer have an in-cut interaction, such as Figure 5 As shown, the simulation scenario is determined to be a failure simulation scenario.

[0115] It should be noted that different evaluation rules may be set for different simulation scenarios, and the evaluation rules for other simulation scenarios may be set with reference to the left-side cut-in scenario, which will not be elaborated in the present disclosure.

[0116] Based on the same inventive concept, the embodiments of the present disclosure also provide an automatic driving simulation evaluation device, such as the following embodiments. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the embodiment of the device can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0117] Figure 8 FIG. 1 is a schematic diagram of an automatic driving simulation evaluation device provided by an embodiment of the present disclosure. Figure 8 As shown, the autonomous driving simulation evaluation device of this embodiment includes a scene classification module 810, a target determination module 820, a data extraction module 830 and a simulation evaluation module 840.

[0118] The scene classification module 810 is used to classify the scene of the autonomous driving simulation environment to obtain a simulation scene;

[0119] A target determination module 820 is used to determine the host vehicle and target obstacles in the simulation scene;

[0120] A data extraction module 830 is used to start a simulation test and extract first driving data of the host vehicle and second driving data of the target obstacle;

[0121] The simulation evaluation module 840 is used to determine that the simulation scenario is invalid if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scenario.

[0122] It should be noted that the above-mentioned scenario classification module 810, target determination module 820, data extraction module 830 and simulation evaluation module 840 correspond to S102 to S108 in the method embodiment, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0123] In one embodiment, the evaluation rules corresponding to the simulation scenario include a first evaluation rule and a second evaluation rule; the simulation evaluation module 840 is used to determine whether the target obstacle and the main vehicle satisfy the first evaluation rule based on the first driving data and the second driving data, and the first evaluation rule indicates that the target obstacle has an action corresponding to the simulation scenario on the main vehicle; if satisfied, the relative distance between the main vehicle and the target obstacle is obtained; if the relative distance between the main vehicle and the target obstacle does not satisfy the second evaluation rule within a preset time period, the simulation scenario is determined to be invalid, and the second evaluation rule indicates that the target obstacle interacts with the main vehicle after executing the action corresponding to the simulation scenario.

[0124] In one embodiment, the simulation evaluation module 840 is further configured to determine that the simulation scenario is invalid if the target obstacle and the host vehicle do not satisfy the first evaluation rule.

[0125] In one embodiment, the simulation evaluation module 840 is further configured to determine that the simulation scenario is valid and generate a simulation evaluation report if the relative distance between the host vehicle and the target obstacle satisfies a second evaluation rule within a preset time period.

[0126] In one embodiment, the scene classification module 810 is used to classify the scenes in the autonomous driving simulation environment based on a deep learning model and / or manual verification method to obtain a simulation scene, wherein the simulation scene includes at least one of a cut-in scene, an overtaking scene, and a reverse driving scene.

[0127] In one embodiment, the device also includes a report generation module not shown in the accompanying drawings, and the report generation module is used to mark the simulation scenario as a failure scenario package if the simulation scenario fails, remove the evaluation results of the simulation scenario from the simulation evaluation test report, and generate a failure scenario report.

[0128] In the disclosed embodiment, the scene classification of the autonomous driving simulation environment is performed to obtain a simulation scene; the main vehicle and the target obstacle in the simulation scene are determined; the simulation test is started, and the first driving data of the main vehicle and the second driving data of the target obstacle are extracted; if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scene, the simulation scene is determined to be invalid. The disclosed system has developed a simulation scene failure warning system based on the simulation test set built by Logsim, which identifies failed simulation scenes and effectively avoids distortion of simulation test results caused by failed simulation scenes. The system can notify testers to pay attention to and deal with failed simulation scenes, thereby improving the accuracy and reliability of simulation tests and improving the quality of releases.

[0129] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0130] Refer to the following Fig. 9 An electronic device 900 according to this embodiment of the present invention will be described. Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0131] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0132] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps of various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 1 As shown in the embodiment of the present disclosure, the scene classification of the autonomous driving simulation environment is performed to obtain a simulation scene; the main vehicle and the target obstacle in the simulation scene are determined; the simulation test is started to extract the first driving data of the main vehicle and the second driving data of the target obstacle; if the first driving data and the second driving data do not meet the evaluation rules corresponding to the simulation scene, the simulation scene is determined to be failed, and a failure scene report is generated.

[0133] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0134] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0135] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0136] The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the system, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 960. Furthermore, the system may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 960. Fig. 9 As shown, the network adapter 960 communicates with other modules of the electronic device 900 via the bus 930. It should be understood that although Fig. 9 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0138] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0139] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0140] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0141] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0142] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0143] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0144] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0145] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0146] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An automatic driving simulation evaluation method, characterized in that: include: Classify the scenes of the autonomous driving simulation environment to obtain simulation scenes; Determining a host vehicle and a target obstacle in the simulation scene; Starting a simulation test, extracting first driving data of the host vehicle and second driving data of the target obstacle; If the first driving data and the second driving data do not satisfy the evaluation rule corresponding to the simulation scenario, it is determined that the simulation scenario is invalid.

2. The automatic driving simulation evaluation method according to claim 1, characterized in that: The evaluation rules corresponding to the simulation scenario include a first evaluation rule and a second evaluation rule; Wherein, if the first driving data and the second driving data do not satisfy the evaluation rule corresponding to the simulation scenario, determining that the simulation scenario is invalid includes: determining, based on the first driving data and the second driving data, whether the target obstacle and the host vehicle satisfy the first evaluation rule, wherein the first evaluation rule indicates that the target obstacle has an action on the host vehicle corresponding to the simulation scenario; If satisfied, the relative distance between the host vehicle and the target obstacle is obtained; If the relative distance between the host vehicle and the target obstacle does not satisfy a second evaluation rule within a preset period of time, the simulation scenario is determined to be invalid, and the second evaluation rule indicates that the target obstacle interacts with the host vehicle after performing an action corresponding to the simulation scenario.

3. The automatic driving simulation evaluation method according to claim 2, characterized in that: The method further comprises: If the target obstacle and the host vehicle do not satisfy the first evaluation rule, the simulation scenario is determined to be invalid.

4. The automatic driving simulation evaluation method according to claim 2, characterized in that: The method further comprises: If the relative distance between the host vehicle and the target obstacle satisfies a second evaluation rule within a preset time period, the simulation scenario is determined to be valid and a simulation evaluation report is generated.

5. The automatic driving simulation evaluation method according to claim 1, characterized in that: The scene classification of the autonomous driving simulation environment to obtain the simulation scene includes: The autonomous driving simulation environment is subjected to scene classification based on a deep learning model and / or a manual verification method to obtain the simulation scene, wherein the simulation scene includes at least one of a cut-in scene, an overtaking scene, and a reverse driving scene.

6. The autonomous driving simulation evaluation method according to any one of claims 1 to 5, characterized in that: After determining that the simulation scenario is invalid if the first driving data and the second driving data do not satisfy the evaluation rule corresponding to the simulation scenario, the method further includes: If the simulation scenario fails, the simulation scenario is marked as a failure scenario package, and the evaluation result of the simulation scenario is removed from the simulation evaluation test report to generate a failure scenario report.

7. An automatic driving simulation evaluation device, characterized in that: include: A scene classification module is used to classify scenes in the autonomous driving simulation environment to obtain simulation scenes; A target determination module, used to determine the main vehicle and target obstacles in the simulation scene; A data extraction module, used to start a simulation test and extract first driving data of the host vehicle and second driving data of the target obstacle; A simulation evaluation module is used to determine that the simulation scenario is invalid if the first driving data and the second driving data do not satisfy the evaluation rules corresponding to the simulation scenario.

8. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the autonomous driving simulation evaluation method as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the autonomous driving simulation evaluation method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program or computer instructions, and the computer program or the computer instructions are loaded and executed by a processor so that a computer implements the autonomous driving simulation evaluation method as described in any one of claims 1 to 6.

Images