A multi-index evaluation method for evaluating the fidelity of street models

Abstract

The present invention discloses a multi-index evaluation method for evaluating the fidelity of a street model, including: training a picture classification model based on real images and simulated pictures, classifying the simulated pictures to be evaluated based on the trained picture classification model, and obtaining a depth evaluation index based on the classification result; in a simulated street, obtaining a consistency index by comparing the consistency between the simulated tracking trajectory and the real trajectory; unifying the units of the depth evaluation index and the consistency index, and mapping the values ​​in the 0-1 space, and obtaining an evaluation result based on the mapping result. The present invention evaluates the street model through prediction confidence at the image layer, and evaluates the fidelity of the street model through the target tracking indicator OSPA at the perception layer. The structure of the present invention is very flexible, and can be adjusted according to actual needs, such as different perception methods, and can efficiently evaluate the effect of the street model, with economic and social benefits.

Description

Technical Field

[0001] The invention belongs to the field of simulation evaluation, and in particular relates to a multi-index evaluation method for evaluating the fidelity of a street model. Background Art

[0002] With the development of autonomous driving technology, autonomous driving simulation technology is becoming more and more indispensable. Simulation uses software simulation to discover and reproduce problems without the need for real environment and hardware, which can greatly save costs and time. The autonomous driving simulation platform can greatly improve the training time and speed up the model iteration by collecting data through simulation. To simulate the environment street is to simulate the environment where the car is located, such as the need to simulate the houses, vehicles, trees, pedestrians, traffic lights, etc. in the real world. At the same time, the physical laws of the real world also need to be simulated, such as weather, sunlight, cloud cover, etc. At present, mainstream simulation software is developed based on game engines. Therefore, it is necessary to evaluate the gap between the street model of the simulation software and the real world. This method proposes a multi-index evaluation method for evaluating the fidelity of the street model. Summary of the invention

[0003] The purpose of the present invention is to provide a multi-index evaluation method for evaluating the fidelity of a street model, which is used to perform an overall quantitative evaluation of the fidelity of the street model to solve the problems existing in the above-mentioned prior art.

[0004] To achieve the above object, the present invention provides a multi-index evaluation method for evaluating the fidelity of a street model, comprising:

[0005] Training an image classification model based on real images and simulated images, classifying the simulated images to be evaluated based on the trained image classification model, and obtaining a depth evaluation index based on the classification result;

[0006] In the simulated street, the consistency index is obtained by comparing the consistency between the simulated tracking trajectory and the real trajectory;

[0007] The units of the depth evaluation index and the consistency index are unified, and the values ​​are mapped in a 0-1 space, and an evaluation result is obtained based on the mapping result.

[0008] Optionally, before training the image classification model based on the real image and the simulated image, the method further includes:

[0009] Setting a test scenario, wherein the test scenario includes a traffic scenario of following a preceding vehicle;

[0010] generating a simulated environment including streets, traffic lights and weather based on the test scenario;

[0011] Taking a screenshot of the simulation environment to obtain the simulation picture;

[0012] The real image is a picture of a real autonomous driving environment.

[0013] Optionally, the process of generating a simulated environment including streets, traffic lights and weather based on the test scenario includes:

[0014] According to the test scenario, a simulation environment is generated by a street model of an autonomous driving simulation software, wherein the simulation environment refers to a high-fidelity driving scenario suitable for testing perception algorithms, and the street model of the autonomous driving simulation software is usually constructed using the virtual engine Unreal Engine developed by Epic Games.

[0015] Optionally, in the process of training the picture classification model based on the real image and the simulated image, the picture classification model uses a convolutional neural network, the structure of the convolutional neural network uses a deep residual network ResNet, and the training process includes:

[0016] Mixing the real image and the simulated image, and inputting the mixed image into the image classification model as a training set;

[0017] After the training accuracy of the image classification model reaches a preset value, the training is terminated.

[0018] Optionally, the simulated image to be evaluated is classified based on the trained image classification model, and the process of obtaining the depth evaluation index based on the classification result includes:

[0019] The simulated image to be evaluated is classified based on the trained image classification model to obtain a prediction confidence score, and the prediction confidence score is used as the depth evaluation indicator.

[0020] Optionally, in the simulated street, the process of obtaining the consistency index by comparing the consistency between the simulated tracking trajectory and the real trajectory includes:

[0021] Constructing the traffic scene of the preceding vehicle and the vehicle to be tracked;

[0022] The vehicle to be tracked moves on a preset path, and a movement trajectory of the vehicle to be tracked is obtained as a real trajectory;

[0023] Track the vehicle to be tracked using a tracking algorithm to generate a tracking trajectory, and use the tracking trajectory as the simulated tracking trajectory;

[0024] The consistency of the real trajectory and the simulated tracking trajectory is compared to obtain the consistency index.

[0025] Optionally, the process of comparing the consistency of the real trajectory and the simulated tracking trajectory to obtain the consistency index includes:

[0026] The consistency between the real trajectory and the simulated tracking trajectory is calculated based on the optimal sub-mode allocation distance, and the obtained OSPA index is used as the consistency index.

[0027] Optionally, the process of unifying the units of the depth evaluation index and the consistency index includes:

[0028] The depth evaluation index and the consistency index are z-score standardized to unify the units.

[0029] Optionally, the numerical value is mapped in a 0-1 space, and the process of obtaining the evaluation result based on the mapping result also includes:

[0030] The weights of the consistency index and the depth evaluation index are adjusted respectively as needed.

[0031] The technical effects of the present invention are:

[0032] This method is a multi-index evaluation method for evaluating the fidelity of street models. It evaluates the street model fidelity through prediction confidence at the image level and through the target tracking indicator OSPA at the perception level. The structure of this method is very flexible and can be adjusted according to actual needs, such as different perception methods. It can efficiently evaluate the effect of street models and has high economic and social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0034] Figure 1 Schematic diagram of the depth evaluation metric structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0036] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] In order to determine the reliability of the street model, it is necessary to detect whether the gap between simulation and reality is applied to the purpose. This method not only compares the simulation effect, but also evaluates the fidelity of the street model from the perception level.

[0038] The simulation effect refers to the intuitive image effect of the simulation, and the comparison object is the real data set of autonomous driving, such as the KITTI public data set. The image classification network model is trained with a mixture of real and simulated images, and then the trained classification model is used to classify the street model images to be evaluated, and the prediction confidence score of the "real image" class is used as the deep evaluation metric (DEM).

[0039] The evaluation at the perception level is to pass the real street scene through the perception module, such as object tracking, and evaluate the quality of the street model by comparing the consistency between the tracking trajectory and the real trajectory. The evaluation indicator is the optimal subpattern assignment distance (OSPA), which is a commonly used method to evaluate the overall performance consistency of the target tracking system. p is the distance sensitivity parameter, c is the level adjustment number, is an estimate.

[0040]

[0041]

[0042]

[0043] like Figure 1 As shown, this embodiment provides a multi-index evaluation method for evaluating the fidelity of a street model, including:

[0044] Step 1: Select a test scenario, such as a traffic scenario following the vehicle in front.

[0045] Step 2: Based on the test scenario, use the street model of the autonomous driving simulation software, such as roadrunner, blender and other scene generation software, to automatically generate simulated environments such as streets, traffic lights, and weather.

[0046] Step 3: Mix the simulated environment screenshots and the real autonomous driving environment images to train the image classification model. The trained classification model is used to classify the environment screenshots generated by the street model to be tested, and the prediction confidence score of the "real image" class is used as the depth evaluation metric DEM.

[0047] Step 4: Generate a corresponding test environment using the street model of the simulation software. The target vehicle moves on a preset path (real trajectory) as the vehicle to be tracked. Use a tracking algorithm to track, generate a tracking trajectory, and calculate the OSPA index compared with the real trajectory. In this embodiment, the tracking algorithm uses, for example, the GOTURN algorithm.

[0048] Step 5: Standardize the results of each metric. The units of the directly calculated depth metric and the OSPA metric are different, so they need to be scaled. After the results are z-score standardized, the values ​​are mapped in the 0-1 space. The importance coefficients of the traditional metric and the depth metric can also be adjusted as needed.

[0049] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-index evaluation method for evaluating the fidelity of street models, characterized in that: The following steps are involved: Training an image classification model based on real images and simulated images, classifying the simulated images to be evaluated based on the trained image classification model, and obtaining a depth evaluation index based on the classification result; In the simulated street, the consistency index is obtained by comparing the consistency between the simulated tracking trajectory and the real trajectory; Unifying the units of the depth evaluation index and the consistency index, and mapping the values ​​in a 0-1 space, and obtaining an evaluation result based on the mapping result; The process of classifying the simulated image to be evaluated based on the trained image classification model and obtaining the depth evaluation index based on the classification result includes: Classifying the simulation image to be evaluated based on the trained image classification model to obtain a prediction confidence score, and using the prediction confidence score as the depth evaluation indicator; In the simulated street, the process of obtaining the consistency index by comparing the consistency between the simulated tracking trajectory and the real trajectory includes: Construct the traffic scene following the preceding vehicle and the vehicle to be tracked; The vehicle to be tracked moves on a preset path, and a movement trajectory of the vehicle to be tracked is obtained as a real trajectory; Track the vehicle to be tracked using a tracking algorithm to generate a tracking trajectory, and use the tracking trajectory as the simulated tracking trajectory; Comparing the consistency between the real trajectory and the simulated tracking trajectory to obtain the consistency index; The process of comparing the consistency of the real trajectory and the simulated tracking trajectory to obtain the consistency index includes: The consistency between the real trajectory and the simulated tracking trajectory is calculated based on the optimal sub-mode allocation distance, and the obtained OSPA index is used as the consistency index.

2. The method according to claim 1, characterized in that Before training the picture classification model based on the real image and the simulated image, the method further includes: Setting a test scenario, wherein the test scenario includes a traffic scenario of following a preceding vehicle; generating a simulated environment including streets, traffic lights and weather based on the test scenario; Taking a screenshot of the simulation environment to obtain the simulation picture; The real image is a picture of a real autonomous driving environment.

3. The method according to claim 2, characterized in that The process of generating a simulated environment including streets, traffic lights and weather based on the test scenario includes: According to the test scenario, a simulation environment is generated by a street model of an autonomous driving simulation software, wherein the simulation environment refers to a high-fidelity driving scenario suitable for testing perception algorithms, and the street model of the autonomous driving simulation software is usually constructed using the virtual engine UnrealEngine developed by Epic Games.

4. The method according to claim 1, characterized in that In the process of training the picture classification model based on real images and simulated scene pictures, the picture classification model adopts a convolutional neural network, and the structure of the convolutional neural network adopts a deep residual network ResNet. The training process includes: Mixing the real image and the simulated image, and inputting the mixed image into the image classification model as a training set; After the training accuracy of the image classification model reaches a preset value, the training is terminated.

5. The method according to claim 1, characterized in that The process of unifying the units of the depth evaluation index and the consistency index includes: The depth evaluation index and the consistency index are z-score standardized to unify the units.

6. The method according to claim 5, characterized in that The process of mapping the numerical value in the 0-1 space and obtaining the evaluation result based on the mapping result also includes: The weights of the consistency index and the depth evaluation index are adjusted respectively as needed.

Images