Method for evaluating simulation consistency of automatic driving perception algorithm

Abstract

The invention discloses a method for evaluating simulation consistency of an automatic driving perception algorithm. The method comprises the following steps: acquiring a target object recognition result and a target object bounding box corresponding to the perception algorithm in a real scene and a simulation scene; setting a difference degree calculation rule of comparison between a simulation scene and a real scene, and calculating a target object recognition rate in a set time period according to the target object recognition result and the difference degree calculation rule; obtaining a target object bounding box obtained by identifying each target object of the simulation scene and the real scene through a perception algorithm; judging the coincidence degree of the target object bounding boxes corresponding to each target object in the simulation scene and the real scene at the set moment, and further calculating to obtain the target recognition accuracy of the scene in the set time period; and performing consistency evaluation according to the target object recognition rate and the target recognition accuracy. According to the method, the confidence of the whole simulation test can be improved, and powerful support is provided for the automatic driving perception test.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving perception testing, and more specifically, to a method for evaluating the simulation consistency of an autonomous driving perception algorithm. Background Art

[0002] The output of existing perception algorithms, whether pure vision or multi-sensor fusion, is the surrounding environment data. For perception algorithm simulation testing, the quality of the algorithm is usually evaluated by observing whether it can accurately identify key targets. Although existing perception simulation can build high-definition simulation scenes through powerful tools and platforms in many aspects, there are still some significant differences between it and the real world. These inconsistencies will be reflected in the recognition results of the perception algorithm, affecting the effectiveness and credibility of the perception simulation. For autonomous driving perception algorithms, it is often difficult to evaluate how big the difference is between the simulation environment and the real world recognition results in the same scene. Therefore, how to quantify this difference by evaluating the consistency of the autonomous driving perception algorithm simulation, and thus measure the credibility of the perception simulation to a certain extent, is of great significance for autonomous driving perception testing. Summary of the invention

[0003] The present invention provides a method for evaluating the simulation consistency of an autonomous driving perception algorithm, which solves the problem that the consistency between the existing autonomous driving perception algorithm simulation and the real world is difficult to evaluate. The method can be used to measure the credibility of the perception simulation, improve the confidence of the entire simulation test, and provide strong support for autonomous driving perception testing.

[0004] To achieve the above objectives, the present invention provides the following technical solutions:

[0005] A method for evaluating the simulation consistency of an autonomous driving perception algorithm, comprising:

[0006] Obtain the corresponding target recognition results and target bounding boxes of the perception algorithm in real scenes and simulation scenes;

[0007] Setting a difference calculation rule for comparing the simulation scene with the real scene, and calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule;

[0008] Obtain the bounding boxes of the objects obtained by the perception algorithm for identifying the objects in the simulation scene and the real scene;

[0009] Determine the degree of overlap between the target object bounding boxes of each target object in the simulation scene and the real scene at the set time, and then calculate the target recognition accuracy of the scene within the set time period;

[0010] A consistency evaluation is performed based on the target object recognition rate and the target recognition accuracy.

[0011] Preferably, the setting of the difference calculation rule between the simulation scene and the real scene includes:

[0012] If the recognition results in the real scene are exactly the same as those in the simulation scene, it is considered that there is no difference and the difference degree is 0;

[0013] If the target object is identified but the category is incorrectly identified, it is considered that there is a certain difference, and the difference degree is 0.5;

[0014] If the target object is not identified, it is considered that there is a large difference and the difference degree is 1.

[0015] Preferably, the step of calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule includes:

[0016] Taking the target object recognition result of the perception algorithm in the real scene as a benchmark, it is judged whether the recognition result of the perception algorithm in the simulation scene is consistent with the recognition result in the real scene. If inconsistent, it is considered that there is a difference, and the target object recognition rate is calculated by calculating the difference between the perception algorithm in the entire simulation scene and the real scene.

[0017] Preferably, the calculation of the target object recognition rate by calculating the difference between the perception algorithm in the entire simulation scene and the real scene includes:

[0018] According to the calculation formula , the target recognition rate is calculated, where t is the total duration of the scene, n is the number of detection tasks performed per second, rec is the target recognition rate, and o i represents the total number of objects recognized in each perception task in the real scene, d i Represents the total difference of the target object perceived each time.

[0019] Preferably, the calculation to obtain the target recognition accuracy of the scene within the set time period includes:

[0020] According to the calculation formula , calculate the target recognition accuracy acc, where o m Indicates the total number of targets identified in each perception task, acc_single n Indicates the recognition accuracy of a single target at a certain moment.

[0021] Preferably, calculating the single target recognition accuracy includes:

[0022] According to the calculation formula , calculate the recognition accuracy of a single target, where I s Represents the bounding box of the target object in the simulation scene, I rRepresents the target object bounding box of the real scene, I s -I r Indicates the difference part, I s ∩I r Indicates the overlapping part.

[0023] Preferably, it also includes:

[0024] The smaller the difference between the target object bounding box in the simulation scene and the target object bounding box in the real scene, and the larger the overlap, the higher the target recognition accuracy is.

[0025] Preferably, the consistency evaluation based on the target object recognition rate and the target recognition accuracy includes:

[0026] According to the formula , the evaluation value con of the perception algorithm simulation consistency is calculated.

[0027] The present invention provides a method for evaluating the simulation consistency of an autonomous driving perception algorithm, and evaluates the simulation consistency by measuring the recognition rate and accuracy of the target object in the perception algorithm in a simulation scenario. This method solves the problem that it is difficult to evaluate the consistency between the existing autonomous driving perception algorithm simulation and the real world, and can be used to measure the credibility of the perception simulation, improve the confidence of the entire simulation test, and provide strong support for autonomous driving perception testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below.

[0029] Figure 1 It is a schematic diagram of a vehicle vision sharing system provided by the present invention.

[0030] Figure 2 It is a flow chart of interaction between a vehicle and a cloud platform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable persons skilled in the art to better understand the solutions of the embodiments of the present invention, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and implementation modes.

[0032] In view of the current problem that it is difficult to evaluate the consistency between the simulation of autonomous driving perception algorithms and the real world, the present invention provides a method for evaluating the consistency of autonomous driving perception algorithm simulation, which solves the problem that it is difficult to evaluate the consistency between the existing autonomous driving perception algorithm simulation and the real world. It can be used to measure the credibility of the perception simulation, improve the confidence of the entire simulation test, and provide strong support for autonomous driving perception testing.

[0033] like Figure 1As shown, a method for evaluating the simulation consistency of an autonomous driving perception algorithm includes:

[0034] S1: Obtain the corresponding target recognition results and target bounding boxes of the perception algorithm in real scenes and simulation scenes.

[0035] S2: setting a difference calculation rule for comparing the simulation scene with the real scene, and calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule.

[0036] S3: Obtain the target object bounding box obtained by the perception algorithm for identifying each target object in the simulation scene and the real scene.

[0037] S4: Determine the degree of overlap between the target object boundary boxes corresponding to each target object in the simulation scene and the real scene at a set time, and then calculate the target recognition accuracy of the scene within the set time period.

[0038] S5: Perform consistency evaluation based on the target object recognition rate and the target recognition accuracy.

[0039] Specifically, the simulation consistency is evaluated in two dimensions: the recognition rate (rec) and the recognition accuracy (acc) of the perception algorithm in the simulation scenario. It is mainly divided into the following steps:

[0040] (1) Definition of perception simulation scenario. Generally speaking, the scenario consists of static environmental elements and dynamic traffic flow information. The purpose of perception algorithm simulation is to verify the algorithm's perception of obstacles or traffic participants in the scenario, such as surrounding pedestrians, vehicles, and obstacles. Therefore, the simulation scenario used for perception algorithm simulation consistency evaluation should restore the static environmental elements and dynamic traffic participants in the real scene as much as possible, and be as close to the 1:1 real scene as possible. Then, by comparing the differences between the perception algorithms in the simulation scenario and the real scenario, the purpose of evaluating the simulation consistency can be achieved.

[0041] (2) Definition of target recognition rate and recognition accuracy. The target recognition rate refers to the recognition rate in the corresponding simulation scene based on the target recognition result of the perception algorithm in the real scene. The target recognition accuracy (acc) refers to the degree of overlap between the target bounding box of the perception algorithm in the real scene and the target bounding box in the corresponding simulation scene.

[0042] (3) Calculation of perception algorithm simulation consistency. Perception algorithm simulation consistency refers to the overall consistency of the algorithm recognition results in the simulation scenario and the corresponding real scenario, which is measured by the target object recognition rate and recognition accuracy.

[0043] This method evaluates the simulation consistency based on the two dimensions of target recognition rate and recognition accuracy of the perception algorithm in the simulation scenario. It can be used to measure the credibility of the perception simulation, thereby improving the confidence of the entire simulation test and providing strong support for autonomous driving access.

[0044] Furthermore, the difference calculation rules for comparing the simulation scene with the real scene are set, including:

[0045] If the recognition results in the real scene are exactly the same as those in the simulation scene, it is considered that there is no difference and the difference degree is 0;

[0046] If the target object is identified but the category is incorrectly identified, it is considered that there is a certain difference, and the difference degree is 0.5;

[0047] If the target object is not identified, it is considered that there is a large difference and the difference degree is 1.

[0048] Further, the calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule includes:

[0049] Taking the target object recognition result of the perception algorithm in the real scene as a benchmark, it is judged whether the recognition result of the perception algorithm in the simulation scene is consistent with the recognition result in the real scene. If inconsistent, it is considered that there is a difference, and the target object recognition rate is calculated by calculating the difference between the perception algorithm in the entire simulation scene and the real scene.

[0050] In one embodiment, if Figure 2 As shown, the recognition result of target object A at time T in the simulation scene is compared with the recognition result of target object A at time T in the real scene. If the results are consistent, the difference is 0; if not, it is judged that the simulation scene recognizes the target object but the type is wrong. If so, it is considered that there is a certain difference, and the difference is increased by 0.5; if the target object is not recognized, it is considered that there is a large difference, and the difference is increased by 1.

[0051] Furthermore, the calculation of the target object recognition rate by calculating the difference between the perception algorithm in the entire simulation scene and the real scene includes:

[0052] The target object recognition rate of the scene is calculated by averaging the target object recognition rate of each perception task result, according to the calculation formula , the target recognition rate is calculated, where t is the total duration of the scene, n is the number of detection tasks performed per second, nt is the total number of perception tasks performed in the entire scene, rec is the target recognition rate, and o i represents the total number of objects recognized in each perception task in the real scene, d i Represents the total difference of the target object perceived each time.

[0053] Further, the calculation to obtain the target recognition accuracy of the scene within the set time period includes:

[0054] According to the calculation formula , calculate the target recognition accuracy acc, where o m Indicates the total number of targets identified in each perception task, acc_single n Indicates the recognition accuracy of a single target at a certain moment.

[0055] Furthermore, the accuracy of single target recognition is calculated, including:

[0056] According to the calculation formula , calculate the recognition accuracy of a single target, where I s Represents the bounding box of the target object in the simulation scene, I r Represents the target object bounding box of the real scene, I s -I r Indicates the difference part, I s ∩I r Indicates the overlapping part.

[0057] The sigmoid function is a common mathematical function, and its formula is .

[0058] The method also includes: when the difference between the target object boundary box in the simulation scene and the target object boundary box in the real scene of the perception algorithm is smaller and the overlap is larger, the target recognition accuracy is considered to be higher.

[0059] Further, the consistency evaluation is performed according to the target object recognition rate and the target recognition accuracy, including:

[0060] According to the formula , the evaluation value con of the perception algorithm simulation consistency is calculated.

[0061] It can be seen that the present invention provides a method for evaluating the simulation consistency of an autonomous driving perception algorithm, and evaluates the simulation consistency by the recognition rate and recognition accuracy of the target object in the simulation scene of the perception algorithm. This solves the problem that it is difficult to evaluate the consistency between the existing autonomous driving perception algorithm simulation and the real world, and can be used to measure the credibility of the perception simulation, improve the confidence of the entire simulation test, and provide strong support for autonomous driving perception testing.

[0062] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which still do not exceed the spirit covered by the description and drawings, should be within the protection scope of the present invention.

Claims

1. A method for evaluating the simulation consistency of an autonomous driving perception algorithm, characterized in that: include: Obtain the corresponding target recognition results and target bounding boxes of the perception algorithm in real scenes and simulation scenes; Setting a difference calculation rule for comparing the simulation scene with the real scene, and calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule; Obtain the bounding boxes of the objects obtained by the perception algorithm for identifying the objects in the simulation scene and the real scene; Determine the degree of overlap between the target object bounding boxes of each target object in the simulation scene and the real scene at the set time, and then calculate the target recognition accuracy of the scene within the set time period; A consistency evaluation is performed based on the target object recognition rate and the target recognition accuracy.

2. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 1, characterized in that: The setting of the difference calculation rule between the simulation scene and the real scene includes: If the recognition results in the real scene are exactly the same as those in the simulation scene, it is considered that there is no difference and the difference degree is 0; If the target object is identified but the category is incorrectly identified, it is considered that there is a certain difference, and the difference degree is 0.5; If the target object is not identified, it is considered that there is a large difference and the difference degree is 1.

3. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 2, characterized in that: The calculating the target object recognition rate within a set time period according to the target object recognition result and the difference calculation rule includes: The target object recognition result of the perception algorithm in the real scene is used as a benchmark to determine whether the recognition result of the perception algorithm in the simulation scene is consistent with the recognition result in the real scene. If not, it is considered that there is a difference, and the target object recognition rate is calculated by calculating the difference between the perception algorithm in the entire simulation scene and the real scene.

4. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 3, characterized in that: The calculation of the target object recognition rate by calculating the difference between the perception algorithm in the entire simulation scene and the real scene includes: According to the calculation formula , the target recognition rate is calculated, where t is the total duration of the scene, n is the number of detection tasks performed per second, rec is the target recognition rate, and o i represents the total number of objects recognized in each perception task in the real scene, d i Represents the total difference of the target object perceived each time.

5. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 4, characterized in that: The calculation to obtain the target recognition accuracy of the scene within the set time period includes: According to the calculation formula , calculate the target recognition accuracy acc, where o m Indicates the total number of targets identified in each perception task, acc_single n Indicates the recognition accuracy of a single target at a certain moment.

6. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 5, characterized in that: Calculate the recognition accuracy of a single target, including: According to the calculation formula , calculate the recognition accuracy of a single target, where I s Represents the bounding box of the target object in the simulation scene, I r Represents the target bounding box of the real scene, I s -I r Indicates the difference part, I s ∩I r Indicates the overlapping part.

7. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 6, characterized in that: Also includes: The smaller the difference between the target object bounding box in the simulation scene and the target object bounding box in the real scene, and the larger the overlap, the higher the target recognition accuracy is.

8. The method for evaluating the simulation consistency of an autonomous driving perception algorithm according to claim 7, characterized in that: The performing consistency evaluation according to the target object recognition rate and the target recognition accuracy includes: According to the formula , the evaluation value con of the perception algorithm simulation consistency is calculated.

Images