A method, device and related equipment for comparing simulation task results

Abstract

The present application discloses a method, apparatus and related equipment for comparing simulation task results. The method includes: obtaining index data of at least two target simulation tasks in the same scenario; respectively processing the index data of each target simulation task to obtain corresponding index data packets; filtering the index data in each index data packet according to the target index input by the user to obtain corresponding target index data sets; comparing each target index data set according to the weight of the target index to obtain the measurement results of each target simulation task. By processing the index data, the present application makes tens of thousands of index data have a relatively simple data structure and a unified data type. On the one hand, it improves the efficiency of information display and is more convenient for engineering technicians to read; on the other hand, it increases the flexibility of index data in screening and comparison, simplifies the calculation process, and can efficiently implement cross-task comparison of index data.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and more specifically, to a method and device for comparing simulation task results and related equipment. Background Art

[0002] Simulation testing is an important part of the research and development of autonomous driving technology. Among them, a test task usually involves simulation testing of thousands of scenarios, and the simulation test results under each scenario can include tens of thousands of metric data. After the simulation test task is completed, the task publisher often does not only analyze the simulation test results of a single task, but conducts comparative analysis based on two or more historical tasks to evaluate the quality of the algorithm model. Especially in the current stage of rapid evolution of autonomous driving systems, a large number of simulation test tasks need to be carried out in numerous scenarios during the research and development process of autonomous driving algorithms, resulting in a large amount of data often involved in the analysis of autonomous driving simulation test results.

[0003] In addition, usually in the simulation testing of autonomous driving, different tasks also set different output metrics. Therefore, in the process of comparative analysis of multiple tasks, it is necessary to achieve consistency matching of scenarios and metrics. Only after pairing according to scenarios and metrics can the differences of a certain metric of different tasks under a certain scenario be calculated.

[0004] Therefore, there is an urgent need for a fast cross-task metric data analysis and comparison method to improve the research and development efficiency of unmanned driving technology. Summary of the Invention

[0005] In view of this, this application provides a method and device for comparing simulation task results and related equipment to achieve the analysis and comparison of a large amount of simulation test data in the simulation test of autonomous driving algorithms.

[0006] To achieve the above object, the first aspect of this application provides a method for comparing simulation task results, including:

[0007] Obtain the metric data of at least two target simulation tasks under the same scenario, where the metric data includes the metric data of at least one metric;

[0008] Process the metric data of each target simulation task respectively to obtain a metric data packet corresponding to each target simulation task, and the metric data in the metric data packet has a unified data type;

[0009] Filter the metric data in each metric data packet according to the target metric input by the user to obtain a target metric data set corresponding to each metric data packet;

[0010] Compare the data sets of each target metric according to the weights of each target metric input by the user to obtain the measurement results of each target simulation task.

[0011] Preferably, the process of comparing the data sets of each target metric according to the weights of each target metric input by the user to obtain the measurement results of each target simulation task includes:

[0012] Compare the data sets of each target metric to obtain the differential results of each target metric of each target simulation task in each scenario;

[0013] Calculate the measurement results of each target simulation task according to the weights of each target metric and the differential results.

[0014] Preferably, the process of comparing the data sets of each target metric to obtain the differential results of each target metric of each target simulation task in each scenario includes:

[0015] Match and combine the metric data in the data sets of each target metric according to the scenario and metric to which the metric data belongs to obtain a metric data table; wherein, the metric data table is a hash table, indexed by the identifier of the scenario and the identifier of the target metric, and the metric data corresponding to the index under each target simulation task in the data sets of each target metric is used as the value;

[0016] For each target simulation task, perform differential calculation on the value of the index under the target simulation task and the values of the index under other target simulation tasks to obtain the differential result of the target simulation task under each index.

[0017] Preferably, the process of calculating the measurement results of each target simulation task according to the weights of each target metric and the differential results includes:

[0018] For each scenario, calculate the score of each target simulation task under each target metric according to the weight of each target metric;

[0019] Calculate the first total score of each target simulation task in each scenario according to the scores of each target simulation task under each scenario and each target metric;

[0020] Calculate the second total score of each target simulation task under each target metric according to the differential results of each target simulation task under each target metric and each scenario;

[0021] Calculate the measurement results of each target simulation task according to the first total score and the second total score.

[0022] Preferably, the process of processing the index data of each target simulation task to obtain an index data packet corresponding to each target simulation task includes:

[0023] For each target simulation task:

[0024] Flatten the index data of the target simulation task in a streaming batch processing manner to obtain flattened index data;

[0025] Convert the flattened index data into floating-point index data in a streaming batch processing manner to obtain the index data packet.

[0026] Preferably, the process of flattening the index data of the target simulation task includes:

[0027] Convert the data structure of the index data of the target simulation task into a list structure.

[0028] Preferably, the process of converting the flattened index data into floating-point index data includes:

[0029] Obtain the data type of the flattened index data;

[0030] If the data type is boolean, convert the index data with the value of TRUE into floating-point data with the value of 1.0, and convert the index data with the value of FALSE into floating-point data with the value of 0.0;

[0031] If the data type is integer, convert the index data into floating-point data with the same numerical value.

[0032] Preferably, the process of obtaining the index data of at least two target simulation tasks includes:

[0033] According to the target simulation tasks input by the user, obtain the index data of the target simulation tasks from a preset data lake;

[0034] Wherein, the index data includes structured data and / or unstructured data.

[0035] A second aspect of this application provides a simulation task result comparison device, including:

[0036] An acquisition unit, configured to acquire the index data of at least two target simulation tasks in the same scenario, where the index data includes the index data of at least one index;

[0037] A preprocessing unit, configured to process the index data of each target simulation task respectively to obtain an index data packet corresponding to each target simulation task, and the index data in the index data packet has a unified data type;

[0038] A filtering unit, configured to filter the metric data in each metric data packet according to the target metrics input by the user, so as to obtain a set of target metric data corresponding to each metric data packet;

[0039] A comparison unit, configured to compare each set of target metric data according to the weights of the respective target metrics input by the user, so as to obtain the measurement results of each target simulation task.

[0040] A third aspect of the present application provides a simulation task result comparison device, including: a memory and a processor;

[0041] The memory is configured to store programs;

[0042] The processor is configured to execute the program to implement each step of the above-mentioned simulation task result comparison method.

[0043] A fourth aspect of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, each step of the simulation task result comparison method as described above is implemented.

[0044] As can be seen from the above technical solutions, the present application first obtains the metric data of at least two target simulation tasks, where the metric data includes data of at least one metric in at least one scenario. Then, the metric data of each target simulation task is processed in a streaming batch processing manner to obtain a metric data packet corresponding to each target simulation task. The metric data in the metric data packet has a flat data structure and a unified data type. By processing the metric data of each target simulation task in a streaming batch processing manner, the data processing speed can be accelerated; by processing the metric data into a flat data structure with a unified data type, subsequent calculations are facilitated and the calculation process is simplified. Next, according to the target metrics input by the user, the metric data in each metric data packet is filtered respectively to obtain a set of target metric data corresponding to each metric data packet. Finally, according to the weights of the respective target metrics input by the user, each set of target metric data is compared to obtain the measurement results of each target simulation task. Since the set of target metric data is processed and filtered data, its data structure is simple and the data type is unified, making the comparison process computationally simple. In summary, through the processing of the metric data, tens of thousands of metric data have a relatively simple data structure and a unified data type. On the one hand, the efficiency of information display is improved, which is more convenient for engineers and technicians to read; on the other hand, the flexibility of the metric data in screening and comparison is increased, the calculation process is simplified, and the cross-task metric data comparison can be efficiently realized. Description of the Drawings

[0045] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0046] Figure 1 Schematic diagram of the simulation task result comparison method disclosed in the embodiments of the present application;

[0047] Figure 2 Workflow schematic diagram in the simulation task result comparison method disclosed in the embodiments of the present application;

[0048] Figure 3 Schematic diagram of the simulation task result comparison device disclosed in the embodiments of the present application;

[0049] Figure 4 Schematic diagram of the simulation task result comparison device disclosed in the embodiments of the present application. Specific implementation manners

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope protected by the present application.

[0051] The following introduces the simulation task result comparison method provided by the embodiments of the present application. Please refer to Figure 1 , the simulation task result comparison method provided by the embodiments of the present application may include the following steps:

[0052] Step S101, obtain the index data of at least two target simulation tasks in the same scenario.

[0053] Generally, the simulation tasks of autonomous driving algorithms involve thousands of scenarios, and each scenario contains thousands of metrics. Among them, the scenario refers to the road scenario during autonomous driving, such as passing through a crosswalk, passing through a supervised intersection, passing through an unsupervised intersection, passing through a blind spot, etc.; the metric refers to the events concerned in autonomous driving, such as the number of collisions, the number of brakes, planning delay, maximum acceleration, system failure, etc.

[0054] It can be understood that the above-mentioned index data packets can include data of at least one index. In addition, there can be only two target simulation tasks, and the analysis and comparison process analyzes and compares the index data of these two target simulation tasks; there can also be three or more target simulation tasks, and one of the target simulation tasks can be set as the main consideration object, and it is compared with other target simulation tasks one by one.

[0055] Step S102: Process the index data of each target simulation task respectively to obtain an index data packet corresponding to each target simulation task.

[0056] Generally, these index data can include multiple data types, such as boolean, integer, and floating-point types. Among them, boolean index data can be used to represent whether a certain test passes, and the optional values are TRUE or FALSE. For example, whether a systematic failure occurs, whether a collision occurs, etc.; integer index data can be used to represent the counting situation in a certain test, such as the number of lane changes, the number of brakes, etc.; floating-point index data can be used to represent the scoring situation in a certain test, such as the maximum acceleration, the driving comfort score, the planning delay, etc.

[0057] By processing the index data of each target simulation task respectively, the index data in each index data packet has a unified data type, which is convenient for subsequent analysis and calculation.

[0058] Step S103: Filter the index data in each index data packet according to the target index input by the user to obtain a set of target index data corresponding to each index data packet.

[0059] In one analysis and comparison, the user may only be concerned about some of the indexes. The indexes concerned by the user can be input, and these indexes constitute the target index. Filter the index data in the index data packet according to the target index, and extract the index data corresponding to the target index, so as to form a set of target index data.

[0060] Step S104: Compare each set of target index data according to the weights of each target index input by the user to obtain the measurement results of each target simulation task.

[0061] Since each target simulation task corresponds to a set of target index data, combining the weights of each target index and comparing each set of target index data can obtain the measurement results of each target simulation task. Among them, the measurement results can include qualitative and quantitative evaluation results.

[0062] This application first obtains the metric data of at least two target simulation tasks. Among them, the metric data includes the data of at least one metric in at least one scenario. Then, the metric data of each target simulation task is processed in a streaming batch processing manner to obtain a metric data packet corresponding to each target simulation task. Among them, the metric data in the metric data packet has a flat data structure and a unified data type. By processing the metric data of each target simulation task in a streaming batch processing manner, the data processing speed can be accelerated; by processing the metric data into a flat data structure and a unified data type, it is convenient for subsequent calculations and simplifies the calculation process. Then, according to the target metrics input by the user, the metric data in each metric data packet is filtered respectively to obtain a set of target metric data corresponding to each metric data packet. Finally, according to the weights of the respective target metrics input by the user, the sets of target metric data are compared to obtain the measurement results of each target simulation task. Since the set of target metric data is processed and filtered data, its data structure is simple and the data type is unified, making the comparison process computationally simple. In summary, through the processing of the metric data, this application makes tens of thousands of metric data have a relatively simple data structure and a unified data type. On the one hand, it improves the efficiency of information display and is more convenient for engineering and technical personnel to read; on the other hand, it increases the flexibility of the metric data in screening and comparison, simplifies the calculation process, and can efficiently implement cross-task metric data comparison.

[0063] In some embodiments of this application, the process of obtaining the metric data of at least two target simulation tasks in step S101 may include:

[0064] According to the target simulation tasks input by the user, obtain the metric data of the target simulation tasks from a preset data lake.

[0065] Among them, a data lake (Data Lake, DL) is a repository or system that stores data in its original format, stores the data as it is, without prior structured processing of the data. A data lake allows storing structured data (such as tables in a relational database), semi-structured data (such as CSV, logs, XML, JSON), unstructured data (such as emails, documents), and binary data (such as graphics, audio, video) at any scale. It can be understood that the above metric data may include at least one of structured data and unstructured data.

[0066] Since the data lake has no explicit requirements for data formats and there is no need to format the data, during simulation testing, various types of metric data output by the simulation test platform can be directly synchronized to the data lake without conversion. When it is necessary to analyze and compare each simulation task, the metric data of the target simulation task can be obtained from the data lake.

[0067] In some embodiments of the present application, the process of processing the metric data of each target simulation task in step S102 to obtain a metric data packet corresponding to each target simulation task may include:

[0068] For each target simulation task:

[0069] S1, perform flattening processing on the metric data of the target simulation task in a streaming batch processing manner to obtain flattened metric data.

[0070] S2, convert the flattened metric data into floating-point metric data in a streaming batch processing manner to obtain the metric data packet.

[0071] Here, streaming batch processing refers to combining stream processing and batch processing during the data processing process. The downstream nodes in the workflow actively establish a streaming request connection with the upstream nodes, and then start receiving the data processed by the upstream nodes. When the received data reaches a certain number (such as 1000 pieces), or considering the timeout mechanism at the same time (such as reaching 1 minute), the downstream nodes start to calculate the task.

[0072] Flattening processing refers to converting the data structure of the metric data into a flat structure form. Specifically, since the original metric data may have multi-level nested or tree-structured metrics, for example, metric data a may contain metric data of three sub-metrics b, c, and d, and a, b, c, and d form a set of data with a two-layer structure {a, {b, c, d}}. After flattening processing, all metrics will be extracted to form data with only one layer of structure, that is, {a, b, c, d}. In this way, the metric data of each metric is arranged in a flat data structure, which is convenient for subsequent unified processing of the metric data of each metric.

[0073] In some embodiments of the present application, the process of performing flattening processing on the metric data of the target simulation task in S1 above may include:

[0074] Convert the data structure of the metric data of the target simulation task into a list structure.

[0075] For example, for the data {a, {b, c, d}} in the above two-layer structure, after flattening, all metrics are extracted to form a list, that is, [a, b, c, d]. The elements in the list are located through pointers, facilitating operations such as element insertion and deletion.

[0076] In some embodiments of the present application, the process of S2 converting the flattened metric data into floating-point metric data may include:

[0077] S21, obtaining the data type of the flattened metric data.

[0078] S22, if the data type is boolean, converting the metric data with the value of TRUE into floating-point data with the value of 1.0, and converting the metric data with the value of FALSE into floating-point data with the value of 0.0.

[0079] It can be understood that since boolean data only includes two values, "true" and "false", when converting it into floating-point data, only need to set "true" and "false" into floating-point data with different values respectively. The above conversion of boolean metric data of TRUE and FALSE into 1.0 and 0.0 respectively is for the consideration of the programming habits of R & D personnel.

[0080] S23, if the data type is integer, converting the metric data into floating-point data with the same value.

[0081] For example, for the main task T1 and the reference task T2, the metric data after their flattening are shown in Table 1 and Table 2 respectively. Among them, SID (Scenario ID) refers to the scenario ID, Metric Name refers to the metric name, Metric Result refers to the metric value, indicating that the name and the metric value constitute the metric data.

[0082] Table 1: Metric Data of Task T1

[0083] TID SID Metric Name Metric Result T1 S1 NO_COLLISION TRUE T1 S1 BRAKE_NUM 3 T1 S2 NO_COLLISION TRUE T1 S2 BRAKE_NUM 5 T1 S2 PLANNING_LATENCY 0.3 T1 S3 NO_SYSTEM_FAILING FALSE

[0084] Table 2: Metric Data of Task T2

[0085] TID SID Metric Name Metric Result T2 S1 NO_COLLISION TRUE T2 S1 BRAKE_NUM 5 T2 S2 NO_COLLISION FALSE T2 S2 BRAKE_NUM 10 T2 S2 PLANNING_LATENCY 0.4

[0086] For the metric data NO_SYSTEM_FAILING with the value of FALSE, after the above S2 converts the data type, its value will become 0.0; for the metric data LANE_CHANGE_NUM with the value of 50, after being processed by the above S2, its value will become 50.0. The metric data in Table 1 and Table 2 above after being processed by the above S2 are shown in Table 3 and Table 4 respectively.

[0087] Table 3: Index data after data type conversion for task T1

[0088] TID SID Metric Name Metric Result T1 S1 NO_COLLISION 1.0 T1 S1 BRAKE_NUM 3.0 T1 S2 NO_COLLISION 1.0 T1 S2 BRAKE_NUM 5.0 T1 S2 PLANNING_LATENCY 0.3 T1 S3 NO_SYSTEM_FAILING 0.0

[0089] Table 4: Index data after data type conversion for task T2

[0090] TID SID Metric Name Metric Result T2 S1 NO_COLLISION 1.0 T2 S1 BRAKE_NUM 5.0 T2 S2 NO_COLLISION 0.0 T2 S2 BRAKE_NUM 10.0 T2 S2 PLANNING_LATENCY 0.4

[0091] Next, for the index data in Table 3 and Table 4, assuming that the target indexes input by the user include NO_COLLISION, BRAKE_NUM, and PLANNING_LATENCY, then according to the method in step S103 above, filter them to obtain the target index data sets corresponding to each index data packet, as shown in Table 5 and Table 6 respectively.

[0092] Table 5: Index data after filtering for task T1

[0093] TID SID Metric Name Metric Result T1 S1 NO_COLLISION 1.0 T1 S1 BRAKE_NUM 3.0 T1 S2 NO_COLLISION 1.0 T1 S2 BRAKE_NUM 5.0 T1 S2 PLANNING_LATENCY 0.3

[0094] Table 6: Index data after filtering for task T2

[0095] TID SID Metric Name Metric Result T2 S1 NO_COLLISION 1.0 T2 S1 BRAKE_NUM 5.0 T2 S2 NO_COLLISION 0.0 T2 S2 BRAKE_NUM 10.0 T2 S2 PLANNING_LATENCY 0.4

[0096] In some embodiments of the present application, the process in step S104 of comparing the target index data sets according to the weights of each target index input by the user to obtain the measurement results of each target simulation task may include:

[0097] S1. Compare each target index data set to obtain the difference result of each target index of each target simulation task in each scenario.

[0098] S2. Calculate the measurement results of each target simulation task according to the weights of each target index and the difference results.

[0099] In some embodiments of the present application, the process in S1 of comparing each target index data set to obtain the difference result of each target index of each target simulation task in each scenario may include:

[0100] S11. Match and combine the index data in each target index data set according to the scenario and index to which the index data belongs to obtain an index data table.

[0101] Among them, the index data table is a hash table, indexed by the identifier of the scenario and the identifier of the target index, and the index data corresponding to the index in each target simulation task in each target index data set is used as the value.

[0102] S12. For each target simulation task, calculate the difference between the value of this index under this target simulation task and the values of this index under other target simulation tasks, to obtain the difference result of this target simulation task under each index.

[0103] For example, for the index data in Table 5 and Table 6 above, the elements in the table can be first matched through the JOIN mode, and the matching result is shown in Table 7.

[0104] Table 7: Index data of Task T1 and Task T2 after matching through the JOIN mode

[0105] SID Metric Name Result(T1) Result(T2) S1 NO_COLLISION 1.0 1.0 S1 BRAKE_NUM 3.0 5.0 S2 NO_COLLISION 1.0 0.0 S2 BRAKE_NUM 5.0 10.0 S2 PLANNING_LATENCY 0.3 0.4

[0106] According to the above S1, compare the index data in Table 7, and the obtained difference result is shown in Table 8.

[0107] Table 8: Difference results obtained after comparison of Task T1 and Task T2

[0108] SID Metric Name ResultT1 ResultT2 DIFF(T1,T2) S1 NO_COLLISION 1.0 1.0 0.0 S1 BRAKE_NUM 3.0 5.0 -2.0 S2 NO_COLLISION 1.0 0.0 1.0 S2 BRAKE_NUM 5.0 10.0 -5.0 S2 PLANNING_LATENCY 0.3 0.4 -0.1

[0109] Assume that the weights input by the user for each target index are respectively (both the number of brakes BRAKE_NUM and the planning latency PLANNING_LATENCY are negative impacts, so the weights are negative):

[0110] {NO_COLLISION: 1, BRAKE_NUM: -0.1, PLANNING_LATENCY: -1}

[0111] Next, according to the weights of each target index and the difference results, calculate the index data of each task in each scenario, and finally calculate the metric results of each target simulation task. Among them, the metric results may include: for each scenario, the metric results of each target simulation task under each target index; and, for each target index, the metric results of each target simulation task under each scenario.

[0112] Based on this, in some embodiments of the present application, the process of the above S2 calculating the metric results of each target simulation task according to the weights of each target index and the difference results may include:

[0113] S21. For each scenario, calculate the score of each target simulation task under each target index according to the weight of each target index.

[0114] S22. Calculate the first total score of each target simulation task in each scenario according to the scores of each target simulation task under each scenario and each target index.

[0115] Among them, the first total score of a certain target task in a certain scenario is used as the measurement result of the target simulation task in this scenario.

[0116] S23. According to the differential results of each target simulation task under each target metric and each scenario, calculate the second total score of each target simulation task under each target metric.

[0117] Among them, the second total score of a certain target simulation task under a certain target metric is used as the measurement result of the target simulation task under this target metric.

[0118] S24. According to the first total score and the second total score, calculate the measurement results of each target simulation task.

[0119] For example, the score of target simulation task T1 in S2 is:

[0120] 1.0 * 1 + 5.0 * (-0.1) + 0.3 * (-1) = 0.2

[0121] The score of target simulation task T2 in S2 is:

[0122] 0.0 * 1 + 1.0 * (-0.1) + 0.4 * (-1) = -1.4

[0123] Therefore, in scenario S2, the score of T1 is better than that of T2. The specific measurement results are shown in Table 9.

[0124] Table  9: Measurement Results of Task T1 and Task T2

[0125] Metrics Scenario T1win 2 2 T2win 1 0 T1win ratio 66.67％ 100.00％

[0126] As can be seen from Table 9, from the perspective of the comprehensive performance of the metrics, target simulation task T1 has 2 metrics better than target simulation task T2 and 1 metric worse than target simulation task T2; from the perspective of the comprehensive performance of the scenarios, target simulation task T1 performs better than target simulation task T2 in all two scenarios. Compared with target simulation task T2, 66.67% of the metric results of the model used by target simulation task T1 are better.

[0127] The above example illustrates the task comparison in the case of one main task and one reference task. The following combines Figure 2 An example is given to illustrate the comparison and analysis work for one main task T1 and two reference tasks T2 and T3.

[0128] Please refer to Figure 2, the metric data in the simulation task results is synchronized from the simulation test platform to the data lake (DL), and then the metric data is preprocessed by the preprocessing unit. This preprocessing process includes flattening the data and converting the data type to floating point. For the main task T1 and the reference tasks T2 and T3, the metric data after flattening is shown in Tables 10, 11, and 12 respectively.

[0129] Table 10: Metric Data of Task T1

[0130] TID SID Metric Name Metric Result T1 S1 NO_COLLISION TRUE T1 S1 BRAKE_NUM 3 T1 S2 NO_COLLISION TRUE T1 S2 BRAKE_NUM 5 T1 S2 PLANNING_LATENCY 0.3 T1 S3 NO_SYSTEM_FAILING FALSE

[0131] Table 11: Metric Data of Task T2

[0132] TID SID Metric Name Metric Result T2 S1 NO_COLLISION TRUE T2 S1 BRAKE_NUM 5 T2 S2 NO_COLLISION FALSE T2 S2 BRAKE_NUM 10 T2 S2 PLANNING_LATENCY 0.4

[0133] Table 12: Metric Data of Task T3

[0134] TID SID Metric Name Metric Result T3 S1 NO_COLLISION TRUE T3 S1 BRAKE_NUM 4 T3 S2 NO_COLLISION FALSE T3 S2 BRAKE_NUM 8 T3 S2 PLANNING_LATENCY 0.5 T3 S3 NO_SYSTEM_FAILING TRUE

[0135] Next, the metric data is uniformly converted into floating point data. The converted metric data of the main task T1 and the reference tasks T2 and T3 are shown in Tables 13, 14, and 15 respectively.

[0136] Table 13: Metric Data of Task T1 after Data Type Conversion

[0137] TID SID Metric Name Metric Result T1 S1 NO_COLLISION 1.0 T1 S1 BRAKE_NUM 3.0 T1 S2 NO_COLLISION 1.0 T1 S2 BRAKE_NUM 5.0 T1 S2 PLANNING_LATENCY 0.3 T1 S3 NO_SYSTEM_FAILING 0.0

[0138] Table 14: Metric Data of Task T2 after Data Type Conversion

[0139] TID SID Metric Name Metric Result T2 S1 NO_COLLISION 1.0 T2 S1 BRAKE_NUM 5.0 T2 S2 NO_COLLISION 0.0 T2 S2 BRAKE_NUM 10.0 T2 S2 PLANNING_LATENCY 0.4

[0140] Table 15: Metric Data of Task T3 after Data Type Conversion

[0141] TID SID Metric Name Metric Result T3 S1 NO_COLLISION 1.0 T3 S1 BRAKE_NUM 4 T3 S2 NO_COLLISION 0.0 T3 S2 BRAKE_NUM 8 T3 S2 PLANNING_LATENCY 0.5 T3 S3 NO_SYSTEM_FAILING 1.0

[0142] Next, for the metric data in Tables 3 and 4, assuming that the target metrics input by the user include NO_COLLISION, PLANNING_LATENCY, and NO_SYSTEM_FAILING, then according to the method of step S103 above, through Figure 2 the filtering unit in it filters them, and the target metric data sets corresponding to each metric data packet are shown in Tables 16, 17, and 18 respectively.

[0143] Table 16: Filtered Metric Data of Task T1

[0144] TID SID Metric Name Metric Result T1 S1 NO_COLLISION 1.0 T1 S2 NO_COLLISION 1.0 T1 S2 PLANNING_LATENCY 0.3 T1 S3 NO_SYSTEM_FAILING 0.0

[0145] Table 17: Filtered metric data for Task T2

[0146] TID SID Metric Name Metric Result T2 S1 NO_COLLISION 1.0 T2 S2 NO_COLLISION 0.0

[0147] Table 18: Filtered metric data for Task T3

[0148]

[0149]

[0150] For the metric data in Tables 16, 17 and 18 above, cross-task metric matching can be first performed on the elements in the tables in UNION mode, and the matching results are shown in Table 19.

[0151] Table 19: Metric data after matching for Tasks T1, T2 and T3 in UNION mode

[0152] SID Metric Name Result(T1) Result(T2) Result(T3) S1 NO_COLLISION 1.0 1.0 1.0 S2 NO_COLLISION 1.0 0.0 0.0 S2 PLANNING_LATENCY 0.3 0.4 0.5 S3 NO_SYSTEM_FAILING 0.0 NULL 1.0

[0153] Perform metric comparison on the metric data in Table 19, and the differential results are shown in Table 20.

[0154] Table 20: Differential results obtained after comparing Task T1 with Tasks T2 and T3 respectively

[0155]

[0156] Assume that the weights input by the user for each target metric are as follows:

[0157] {NO_COLLISION: 0.6, PLANNING_LATENCY: -1, NO_SYSTEM_FAILING: 1}

[0158] Next, calculate the metric data of each task in each scenario according to the weights of each target metric and the differential results. The specific task measurement results are shown in Table 21.

[0159] Table 21: Measurement results for Tasks T1, T2 and T2

[0160] Metrics Scenario T1win 2 2 T2win 0 0 T3win 1 1 T1win ratio 66.67％ 66.67％

[0161] As can be seen from Table 21, in terms of the comprehensive performance of the indicators, there are 2 indicators of the target simulation task T1 that are better than those of the target simulation tasks T2 and T3, and 1 indicator that is worse than that of the target simulation task T3; in terms of the comprehensive performance of the scenarios, the performance of the target simulation task T1 in two scenarios is better than that of the target simulation tasks T2 and T3, and the performance in one scenario is worse than that of the target simulation task T3. Compared with the target simulation task T2, all the indicator results of the model used in the target simulation task T1 and its comprehensive performance in all scenarios are better. Compared with the target simulation task T3, 66.67% of the indicator results of the model used in the target simulation task T1 are better, and its comprehensive performance is better in 2 / 3 of the scenarios.

[0162] The simulation task result comparison device provided by the embodiments of the present application will be described below. The simulation task result comparison device described below can be correspondingly referred to the simulation task result comparison method described above.

[0163] Please refer to Figure 3 , the simulation task result comparison device provided by the embodiments of the present application may include:

[0164] An acquisition unit 21, configured to acquire indicator data of at least two target simulation tasks in the same scenario, where the indicator data includes data of at least one type of indicator in at least one scenario;

[0165] A preprocessing unit 22, configured to process the indicator data of each target simulation task respectively to obtain an indicator data packet corresponding to each target simulation task, and the indicator data in the indicator data packet has a unified data type;

[0166] A filtering unit 23, configured to filter the indicator data in each indicator data packet according to the target indicators input by the user to obtain a set of target indicator data corresponding to each indicator data packet;

[0167] A comparison unit 24, configured to compare the sets of target indicator data according to the weights of the respective target indicators input by the user to obtain the measurement results of each target simulation task.

[0168] In some embodiments of the present application, the process of the acquisition unit 21 acquiring the indicator data of at least two target simulation tasks includes:

[0169] Acquiring the indicator data of the target simulation task from a preset data lake according to the target simulation task input by the user;

[0170] Wherein, the indicator data includes structured data and / or unstructured data.

[0171] In some embodiments of the present application, the process in which the above-mentioned preprocessing unit 22 processes the index data of each target simulation task to obtain an index data packet corresponding to each target simulation task may include:

[0172] For each target simulation task:

[0173] Flatten the index data of the target simulation task in a streaming batch processing manner to obtain flattened index data;

[0174] Convert the flattened index data into floating-point index data in a streaming batch processing manner to obtain the index data packet.

[0175] In some embodiments of the present application, the process in which the above-mentioned preprocessing unit 22 flattens the index data of the target simulation task may include:

[0176] Convert the data structure of the index data of the target simulation task into a list structure.

[0177] In some embodiments of the present application, the process in which the above-mentioned preprocessing unit 22 converts the flattened index data into floating-point index data may include:

[0178] Obtain the data type of the flattened index data;

[0179] If the data type is boolean, convert the index data with the value of TRUE into floating-point data with the value of 1.0, and convert the index data with the value of FALSE into floating-point data with the value of 0.0;

[0180] If the data type is integer, convert the index data into floating-point data with the same numerical value.

[0181] In some embodiments of the present application, the process in which the above-mentioned comparison unit 24 compares each target index data set according to the weights of each target index input by the user to obtain the measurement result of each target simulation task may include:

[0182] Compare each target index data set to obtain the difference result of each target index of each target simulation task in each scenario;

[0183] Calculate the measurement result of each target simulation task according to the weights of each target index and the difference result.

[0184] In some embodiments of the present application, the process in which the above-mentioned comparison unit 24 compares each target index data set to obtain the difference result of each target index of each target simulation task in each scenario may include:

[0185] Match and combine the metric data in each target metric data set according to the scenarios and metrics to which the metric data belongs, and obtain a metric data table; wherein, the metric data table is a hash table, indexed by the identifier of the scenario and the identifier of the target metric, and the metric data corresponding to the index under each target simulation task in each target metric data set is used as the value;

[0186] For each target simulation task, perform a difference calculation on the value of the index under the target simulation task and the values of the index under other target simulation tasks, and obtain the difference result of the target simulation task under each index.

[0187] The simulation task result comparison device provided by the embodiments of the present application can be applied to simulation task result comparison devices, such as computers, etc. Optionally, Figure 4 shows a hardware structure block diagram of a simulation task result comparison device. Refer to Figure 4 , the hardware structure of the simulation task result comparison device may include: at least one processor 31, at least one communication interface 32, at least one memory 33, and at least one communication bus 34.

[0188] In the embodiments of the present application, the number of the processor 31, the communication interface 32, the memory 33, and the communication bus 34 is at least one, and the processor 31, the communication interface 32, and the memory 33 complete mutual communication through the communication bus 34;

[0189] The processor 31 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application, etc.;

[0190] The memory 32 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory;

[0191] Among them, the memory 33 stores a program, and the processor 31 can call the program stored in the memory 33, and the program is used for:

[0192] Obtain the metric data of at least two target simulation tasks in the same scenario, and the metric data includes the metric data of at least one metric;

[0193] Process the metric data of each target simulation task respectively, and obtain a metric data packet corresponding to each target simulation task, and the metric data in the metric data packet has a unified data type;

[0194] Filter the metric data in each metric data packet according to the target metrics input by the user to obtain a set of target metric data corresponding to each metric data packet;

[0195] Compare the sets of target metric data according to the weights of the respective target metrics input by the user to obtain the measurement results of the respective target simulation tasks.

[0196] Optionally, the refinement function and expansion function of the program can be referred to the above description.

[0197] The embodiment of the present application further provides a storage medium, which can store a program suitable for execution by a processor, and the program is used for:

[0198] Obtain the metric data of at least two target simulation tasks in the same scenario, and the metric data includes the metric data of at least one metric;

[0199] Process the metric data of each target simulation task respectively to obtain a metric data packet corresponding to each target simulation task, and the metric data in the metric data packet has a unified data type;

[0200] Filter the metric data in each metric data packet according to the target metrics input by the user to obtain a set of target metric data corresponding to each metric data packet;

[0201] Compare the sets of target metric data according to the weights of the respective target metrics input by the user to obtain the measurement results of the respective target simulation tasks.

[0202] Optionally, the refinement function and expansion function of the program can be referred to the above description.

[0203] In summary:

[0204] This application first obtains the metric data of at least two target simulation tasks, where the metric data includes data of at least one metric in at least one scenario. Then, the metric data of each target simulation task is processed in a streaming batch processing manner to obtain a metric data packet corresponding to each target simulation task. Among them, the metric data in the metric data packet has a flat data structure and a unified data type. By processing the metric data of each target simulation task in a streaming batch processing manner, the data processing speed can be accelerated; by processing the metric data into a flat data structure with a unified data type, it is convenient for subsequent calculations and simplifies the calculation process. Next, according to the target metrics input by the user, the metric data in each metric data packet is filtered respectively to obtain a set of target metric data corresponding to each metric data packet. Finally, according to the weights of the various target metrics input by the user, the sets of target metric data are compared to obtain the measurement results of each target simulation task. Since the set of target metric data is processed and filtered data, its data structure is simple and the data type is unified, making the comparison process computationally simple. In summary, through the processing of the metric data, this application makes tens of thousands of metric data have a relatively simple data structure and a unified data type. On the one hand, it improves the efficiency of information display and is more convenient for engineering technicians to read; on the other hand, it increases the flexibility of the metric data in screening and comparison, simplifies the calculation process, and can efficiently implement cross-task metric data comparison.

[0205] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0206] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0207] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A simulation task result comparison method, characterized in that: include: Obtaining indicator data for at least two target simulation tasks in the same scenario, the indicator data including indicator data of at least one indicator; wherein the simulation task is a simulation task of an autonomous driving algorithm; the indicator data includes data types including Boolean, integer, and floating-point types, wherein Boolean indicator data is used to indicate whether a test passes, integer indicator data is used to indicate a count in a test, and floating-point indicator data is used to indicate a score in a test; Processing the indicator data of each target simulation task separately to obtain an indicator data packet having a flat data structure corresponding to each target simulation task, wherein the indicator data in the indicator data packet has a uniform data type; filtering the indicator data in each indicator data packet according to the target indicator input by the user to obtain a target indicator data set corresponding to each indicator data packet; According to the weights of each target indicator input by the user, each target indicator data set is compared to obtain the measurement results of each target simulation task, including: Compare the target indicator data sets to obtain the differential results of each target indicator of each target simulation task in each scenario; According to the weight of each target indicator and the differential result, the measurement result of each target simulation task is calculated, including: for each scenario, according to the weight of each target indicator, respectively calculating the score of each target simulation task under each target indicator; according to the score of each target simulation task in each scenario and each target indicator, calculating the first total score of each target simulation task in each scenario; according to the differential result of each target simulation task in each target indicator and each scenario, calculating the second total score of each target simulation task in each target indicator; according to the first total score and the second total score, calculating the measurement result of each target simulation task.

2. The method according to claim 1, characterized in that The process of comparing the target indicator data sets to obtain the differential results of each target indicator of each target simulation task in each scenario includes: According to the scenario and indicator to which the indicator data belongs, the indicator data in each target indicator data set are matched and combined to obtain an indicator data table; wherein the indicator data table is a hash table, with the identifier of the scenario and the identifier of the target indicator as indexes, and the indicator data corresponding to the index under each target simulation task in each target indicator data set as values; For each target simulation task, a difference calculation is performed between the value of the index under the target simulation task and the value of the index under other target simulation tasks to obtain a difference result of the target simulation task under each index.

3. The method according to claim 1, characterized in that The process of processing the indicator data of each target simulation task respectively to obtain an indicator data packet having a flat data structure corresponding to each target simulation task includes: For each target simulation task: Flattening the indicator data of the target simulation task in a streaming batch processing manner to obtain flattened indicator data; The flattened indicator data is converted into floating-point indicator data in a streaming batch processing manner to obtain the indicator data packet.

4. The method according to claim 3, characterized in that The process of flattening the indicator data of the target simulation task includes: The data structure of the indicator data of the target simulation task is converted into a list structure.

5. The method according to claim 3, characterized in that The process of converting the flattened indicator data into floating-point indicator data includes: Obtaining the data type of the flattened indicator data; If the data type is Boolean, convert the indicator data with a value of TRUE into a floating-point data with a value of 1.0, and convert the indicator data with a value of FALSE into a floating-point data with a value of 0.0; If the data type is integer, convert the indicator data into floating-point data of equivalent value.

6. The method according to claim 1, characterized in that The process of obtaining indicator data of at least two target simulation tasks includes: According to the target simulation task input by the user, the indicator data of the target simulation task is obtained from the preset data lake; The indicator data includes structured data and / or unstructured data.

7. A simulation task result comparison device, characterized in that: include: an acquisition unit, configured to acquire indicator data for at least two target simulation tasks in the same scenario, the indicator data including indicator data of at least one indicator; wherein the simulation task is a simulation task of an autonomous driving algorithm; the indicator data includes data types including Boolean, integer, and floating-point types, wherein Boolean indicator data is used to indicate whether a test has passed, integer indicator data is used to indicate a count in a test, and floating-point indicator data is used to indicate a score in a test; A preprocessing unit, configured to process the index data of each target simulation task respectively to obtain an index data packet having a flat data structure corresponding to each target simulation task, wherein the index data in the index data packet has a uniform data type; A filtering unit, configured to filter the indicator data in each indicator data packet according to the target indicator input by the user, and obtain a target indicator data set corresponding to each indicator data packet; The comparison unit is used to compare the target indicator data sets according to the weights of the target indicators input by the user to obtain the measurement results of each target simulation task, including: Compare the target indicator data sets to obtain the differential results of each target indicator of each target simulation task in each scenario; According to the weight of each target indicator and the differential result, the measurement result of each target simulation task is calculated, including: for each scenario, according to the weight of each target indicator, respectively calculating the score of each target simulation task under each target indicator; according to the score of each target simulation task in each scenario and each target indicator, calculating the first total score of each target simulation task in each scenario; according to the differential result of each target simulation task in each target indicator and each scenario, calculating the second total score of each target simulation task in each target indicator; according to the first total score and the second total score, calculating the measurement result of each target simulation task.

8. A simulation task result comparison device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the simulation task result comparison method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the simulation task result comparison method according to any one of claims 1 to 6 is implemented.

Images