An algorithm evaluation method, device, medium and computer device

Abstract

The present disclosure relates to an algorithm evaluation method, device, medium and computer equipment. In at least one embodiment of the present disclosure, by determining the plane range corresponding to the data set of each task, tasks without intersection of the plane range are divided into the same work order; then, taking the work order as the evaluation unit, each evaluation evaluates multiple tasks included in one work order as a whole, compared with the prior art which takes the task as the evaluation unit and evaluates only one task each time, the present disclosure can improve the evaluation efficiency; compared with the prior art which evaluates all tasks as a whole, the present disclosure takes the work order as the evaluation unit, and the same work order includes multiple tasks without intersection of the plane range, so there is no repeated data between the multiple tasks included in the same work order, and thus the interference of repeated data on evaluation can be avoided, and the accuracy of algorithm evaluation is improved.

Description

Technical Field

[0001] This disclosure relates to the field of algorithm evaluation technology, specifically to an algorithm evaluation method, apparatus, medium, and computer equipment. Background Technology

[0002] In the high-definition map production process, different data processing algorithms are used at different stages to process data and ultimately generate a high-definition map. To ensure the reliability of the high-definition map, the data processing algorithms need to be evaluated.

[0003] Currently, a large amount of online task data is input into data processing algorithms. After the algorithms are executed, the output results are evaluated manually or automatically to achieve algorithm evaluation. Here, online tasks can be understood as datasets within a preset range (e.g., a range of 2 kilometers in both length and width) used to evaluate the algorithms.

[0004] It is evident that current algorithm evaluation methods use tasks as the evaluation unit, assessing only the algorithm's output for one task at a time. However, the number of tasks is typically enormous, resulting in low evaluation efficiency. Furthermore, evaluating the algorithm's output across different tasks as a whole can lead to data overlap, interfering with the evaluation and affecting its accuracy. Summary of the Invention

[0005] At least one embodiment of this disclosure provides an algorithm evaluation method, apparatus, medium, and computer device.

[0006] In a first aspect, embodiments of this disclosure propose an algorithm evaluation method, which includes:

[0007] Obtain the algorithm output for each task, where the task is a dataset of spatial data used to evaluate the algorithm.

[0008] Determine the plane extent corresponding to the dataset for each task;

[0009] Based on the plane range corresponding to the dataset of each task, tasks whose plane ranges do not overlap are divided into the same work order, resulting in multiple work orders and a set of algorithm output results corresponding to each work order;

[0010] Using work orders as the evaluation unit, the algorithm output result set corresponding to one of the multiple work orders is evaluated each time to obtain the evaluation result of the algorithm output result set, until all multiple work orders have been evaluated.

[0011] In some embodiments, determining the planar extent corresponding to the dataset for each task includes:

[0012] For any given task, determine the planar geometry of all spatial features included in the dataset for that task, and obtain a set of planar geometry shapes;

[0013] Determine the union of all planar geometric shapes in a set of planar geometric shapes;

[0014] The planar extent corresponding to the dataset for this task is determined based on the first edge of the union.

[0015] In some embodiments, determining the planar extent corresponding to the dataset for the task based on the first edge of the union includes:

[0016] Expand the first edge of the union by a preset distance to obtain the second edge;

[0017] The range formed by the second edge is defined as the planar range corresponding to the dataset of this task.

[0018] In some embodiments, determining the planar extent corresponding to the dataset for the task based on the first edge of the union includes:

[0019] Expand the first edge of the union by a preset distance to obtain the second edge;

[0020] Determine the convex hull of the second edge to obtain the third edge, which is the edge of the convex hull of the second edge;

[0021] The range formed by the third edge is defined as the planar range corresponding to the dataset of this task.

[0022] In some embodiments, based on the planar range corresponding to the dataset of each task, tasks whose planar ranges do not overlap are grouped into the same work order, including:

[0023] Based on the task identifier of each task, determine the order in which work orders are divided for each task;

[0024] For any given task, based on the work order division order, traverse the subsequent tasks of that task and assign the task and subsequent tasks that do not intersect with the plane range to the same work order; for the remaining tasks, repeat this step until all tasks are assigned to work orders.

[0025] In some embodiments, the evaluation result of the algorithm output result set is a correct, incorrect, or missing evaluation result, and the algorithm evaluation method further includes:

[0026] The evaluation results of multiple work orders are summarized to obtain the first number of algorithm outputs rated as correct, the second number of algorithm outputs rated as incorrect, and the third number of tasks rated as missing; where the evaluation result is the correct, incorrect, and missing evaluation result of the algorithm output result set corresponding to the work order.

[0027] Based on the first, second, and third quantities, the values ​​of multiple algorithm evaluation metrics are determined.

[0028] In some embodiments, multiple algorithm evaluation metrics include at least one of the following: accuracy, error rate, false negative rate, and recall rate;

[0029] The accuracy rate is the ratio of the first number to the total number; where the total number is the sum of the first, second, and third numbers.

[0030] The error rate is the ratio of the second-highest number to the total number.

[0031] The underreporting rate is the ratio of the third number to the total number.

[0032] Recall rate is the ratio of the sum of the first and second recalls to the total number of recalls.

[0033] Secondly, this disclosure also proposes an algorithm evaluation device, which includes:

[0034] The acquisition unit is used to acquire the algorithm output results corresponding to each task, where the task is a dataset of spatial data used to evaluate the algorithm.

[0035] The defining unit is used to determine the planar extent corresponding to the dataset for each task;

[0036] The partitioning unit is used to divide tasks whose plane ranges do not overlap into the same work order based on the plane range corresponding to the dataset of each task, so as to obtain multiple work orders and the set of algorithm output results corresponding to each work order;

[0037] The evaluation unit is used to evaluate the algorithm output result set corresponding to one of the multiple work orders at a time, using the work order as the evaluation unit, and to obtain the evaluation result of the algorithm output result set until all multiple work orders have been evaluated.

[0038] Thirdly, embodiments of this disclosure also provide a computer device, comprising at least one computing device and at least one storage device for storing instructions; the instructions, when executed by the at least one computing device, cause the at least one computing device to perform the steps of the algorithm evaluation method as described in any embodiment of the first aspect.

[0039] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the algorithm evaluation method as described in any embodiment of the first aspect.

[0040] Fifthly, embodiments of this disclosure also provide a computer program product, wherein the computer program product includes a computer program stored in a computer-readable storage medium, and at least one processor of the computer reads from the storage medium and executes the computer program, causing the computer to perform the steps of the algorithm evaluation method as described in any embodiment of the first aspect.

[0041] As can be seen, in at least one embodiment of this disclosure, by determining the planar range corresponding to the dataset of each task, tasks whose planar ranges do not overlap are grouped into the same work order; then, using the work order as the evaluation unit, each evaluation assesses multiple tasks included in a work order as a whole. Compared with the prior art, which uses tasks as the evaluation unit and evaluates only one task at a time, this disclosure can improve evaluation efficiency. Compared with the prior art, which evaluates all tasks as a whole, this disclosure uses the work order as the evaluation unit, and the same work order includes multiple tasks whose planar ranges do not overlap. Therefore, there is no duplicate data among the multiple tasks included in the same work order, thus avoiding the interference of duplicate data on the evaluation and improving the accuracy of algorithm evaluation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings.

[0043] Figure 1 This is a diagram illustrating the phenomenon where there is duplicate data between adjacent tasks.

[0044] Figure 2 A flowchart illustrating an algorithm evaluation method provided in this embodiment of the disclosure;

[0045] Figure 3 This invention provides a schematic diagram of the union of planar geometric shapes in a set of planar geometric shapes, according to an embodiment of the present disclosure.

[0046] Figure 4 for Figure 3 A schematic diagram illustrating the planar extent corresponding to the dataset used to determine the task.

[0047] Figure 5 A flowchart illustrating a task partitioning process provided in an embodiment of this disclosure;

[0048] Figure 6 A schematic diagram of the planar extent corresponding to a dataset for different tasks provided in an embodiment of this disclosure;

[0049] Figure 7A schematic diagram of an algorithm evaluation device provided in an embodiment of this disclosure;

[0050] Figure 8 An exemplary block diagram of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0051] To better understand the above-described objectives, features, and advantages of this disclosure, the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It is to be understood that the described embodiments are only some, not all, of the embodiments of this disclosure. The specific embodiments described herein are merely for explaining this disclosure and are not intended to limit it. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure.

[0052] It should be noted that in this article, relational terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0053] In related technologies, the input sources for algorithm evaluation typically include two types: one is data from a master database, which can be understood as a database that summarizes the data after the work (including operations such as drawing, modifying, and adjusting high-precision map production); the other is online tasks, which are datasets within a preset range (e.g., a range with a length and width of 2 kilometers) used to evaluate the algorithm. For example, data within a 10-kilometer range can be divided into 5 online tasks by dividing it into 2-kilometer segments. However, in order to ensure the continuity of the data, there needs to be duplicate data between adjacent tasks. Figure 1 This diagram illustrates the presence of duplicate data between adjacent tasks. Figure 1 In this context, Task 1 and Task 2 are adjacent tasks. For example, Task 1 contains data sets within the first and second kilometers of a 10-kilometer radius, while Task 2 contains data sets within the third and fourth kilometers of the same 10-kilometer radius. To ensure data continuity, there is some overlap between Task 1 and Task 2. Figure 1 The shaded area is shown in the image.

[0054] In related technologies, if data from a master database is used for algorithm evaluation, the input data is determined according to the map sheet range. A map sheet is a grid obtained by dividing the entire Earth or a target area using the map sheet division method; each grid is a map sheet, and the map sheet range is, for example, an area with a length and width of 10 kilometers. It is evident that using data from a master database for algorithm evaluation results in a large data volume and low evaluation efficiency.

[0055] In related technologies, if online task data is used for algorithm evaluation, the algorithm output results for each online task are evaluated manually or automatically. The evaluation method is labeling; for example, correct output results are labeled as correct, and incorrect output results are labeled as incorrect. However, online tasks are usually numerous, resulting in low evaluation efficiency. While evaluating the algorithm output results from different online tasks as a whole can improve evaluation efficiency, the potential for data duplication across different online tasks can interfere with the algorithm evaluation. It becomes impossible to determine whether errors in the algorithm output results are caused by the algorithm itself or by duplicate data, affecting the accuracy of the evaluation.

[0056] To address at least one problem in the existing technology, this disclosure provides an algorithm evaluation method, apparatus, medium, or computer device. By determining the planar range corresponding to the dataset of each task, tasks whose planar ranges do not overlap are grouped into the same work order. Furthermore, using the work order as the evaluation unit, each evaluation treats multiple tasks included in a work order as a whole. Compared to the prior art, which uses tasks as the evaluation unit and evaluates only one task at a time, this disclosure improves evaluation efficiency. Compared to the prior art, which evaluates all tasks as a whole, this disclosure uses the work order as the evaluation unit, and since the same work order includes multiple tasks whose planar ranges do not overlap, there is no duplicate data among the multiple tasks included in the same work order. Therefore, it avoids interference from duplicate data in the evaluation and improves the accuracy of algorithm evaluation.

[0057] Figure 2 This is a flowchart illustrating an algorithm evaluation method provided in an embodiment of the present disclosure. The execution subject of the algorithm evaluation method is an electronic device, including but not limited to in-vehicle devices, smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, servers, etc. The server can be an independent server or a cluster of multiple servers, and can include servers built locally and servers set up in the cloud.

[0058] like Figure 2 As shown, the algorithm evaluation method may include, but is not limited to, steps 201 to 204:

[0059] In step 201, the algorithm output results corresponding to each task are obtained, wherein the task is a dataset of spatial data used to evaluate the algorithm.

[0060] Each task has a task identifier (id), such as a task number, with smaller task numbers indicating earlier task creation. Each task's dataset is a collection of spatial data for map elements within a preset range. Map elements are features used to construct high-precision maps, and they are extracted, edited, and drawn from laser point cloud data collected by radar equipment (e.g., laser scanning equipment) and image data collected by image acquisition equipment (e.g., vehicle-mounted cameras). Map element types include, but are not limited to, ground elements such as directional arrows, lane lines, stop lines, and ground text, as well as above-ground elements such as signs and speedometers. Spatial data for map elements includes, but is not limited to, the location (spatial coordinates), shape, and size of the map elements.

[0061] The algorithm is a data processing algorithm, including but not limited to: data inspection algorithms, data fusion algorithms, data compilation algorithms, etc. Each task is input into the algorithm, and after execution, the algorithm outputs the execution result of each task, i.e., the algorithm output result.

[0062] In step 202, the plane range corresponding to the dataset for each task is determined.

[0063] For any given task, determine the planar geometry of all spatial features included in the dataset for that task, resulting in a set of planar geometry (GeometryCollection). Then, determine the union of all planar geometry in the set of planar geometry. Thus, based on the first edge of the union, determine the planar extent corresponding to the dataset for that task.

[0064] Spatial elements include, but are not limited to: vectorized elements obtained by processing real-world map elements such as points, lines, and polygons, as well as virtual elements from the non-real world. Virtual elements from the non-real world are virtual elements generated by the autonomous driving system that do not exist in the real world and assist in autonomous driving. Virtual elements include, but are not limited to: lane transition points (i.e., separation points) where a lane changes from a single lane to a two-lane lane, lane transition points (i.e., merging points) where a lane changes from a two-lane lane to a single lane, highway entrances, highway exits, and virtual lane lines (for example, if there are no lane lines for left turns at an intersection, the autonomous driving system will generate virtual lane lines to control the vehicle to turn left along the virtual lane lines).

[0065] Planar geometry only considers the planar coordinate information of spatial features, and does not consider the elevation information of spatial features. Elevation is the distance from a point along the vertical direction to a preset reference surface. The preset reference surface can be any of the following surfaces: geodetic reference surface, quasi-geodetic reference surface, reference ellipsoid, etc. It should be noted that the determination of the preset reference surface is a mature technology in the field of mapping, and will not be elaborated further.

[0066] It should be noted that the execution order of steps 202 and 201 can be reversed, that is, step 202 is executed first, and then step 201 is executed.

[0067] In step 203, based on the plane range corresponding to the dataset of each task, tasks whose plane ranges do not overlap are divided into the same work order, resulting in multiple work orders and a set of algorithm output results corresponding to each work order.

[0068] In this embodiment of the disclosure, a work order includes multiple tasks. Since the planar ranges corresponding to the datasets of different tasks do not overlap, there is no duplicate data between different tasks. The set of algorithm output results corresponding to each work order can be saved to the corresponding work order.

[0069] In step 204, the work order is used as the evaluation unit. Each time, the algorithm output result set corresponding to one of the multiple work orders is evaluated to obtain the evaluation result of the algorithm output result set, until the evaluation of multiple work orders is completed.

[0070] In this embodiment of the disclosure, the evaluation result of the algorithm output result set is a right / wrong / missing evaluation result. The right / wrong / missing evaluation of the algorithm output result set can be performed manually or automatically. Automated evaluation follows relevant technologies and will not be elaborated further to avoid repetition. If an algorithm output result is evaluated as right, it is marked as right; if an algorithm output result is evaluated as wrong, it is marked as wrong; if a task has no corresponding algorithm output result, it is evaluated as missing and the task is marked as missing.

[0071] As can be seen, in this embodiment of the present disclosure, by determining the planar range corresponding to the dataset of each task, tasks whose planar ranges do not overlap are grouped into the same work order; then, using the work order as the evaluation unit, each evaluation assesses multiple tasks included in a work order as a whole. Compared with the prior art, which uses tasks as the evaluation unit and evaluates only one task at a time, this embodiment of the present disclosure can improve evaluation efficiency. Compared with the prior art, which evaluates all tasks as a whole, this embodiment of the present disclosure uses the work order as the evaluation unit, and the same work order includes multiple tasks whose planar ranges do not overlap. Therefore, there is no duplicate data among the multiple tasks included in the same work order, thus avoiding the interference of duplicate data on the evaluation and improving the accuracy of algorithm evaluation.

[0072] Based on the above embodiments, step 202, "determining the plane range corresponding to the dataset of each task," specifically involves:

[0073] For any given task, the planar geometry of all spatial features included in the dataset of that task is determined to obtain a set of planar geometry shapes; then, the union of each planar geometry shape in the set of planar geometry shapes can be determined; thus, the planar extent formed by the first edge of the union is determined as the planar extent corresponding to the dataset of that task. Figure 3 The union of planar geometries in a set of planar geometries is shown. The edge of the union is denoted as the first edge. The planar range formed by the first edge is determined as the planar range corresponding to the dataset of this task.

[0074] As can be seen, in this embodiment of the disclosure, the first edge of the union (such as...) Figure 3 The plane range formed by the edges of the plane geometry shown is used as the plane range corresponding to the dataset of the task. However, the following problem may exist: if the first task and the second task are adjacent tasks, the plane range corresponding to the dataset of the first task should have an intersection with the plane range corresponding to the dataset of the second task, that is, there is duplicate data. However, using the plane range determined by the first edge may result in no duplicate data between the first task and the second task.

[0075] To address this issue, step 202, "determining the planar extent corresponding to the dataset for each task," specifically involves:

[0076] For any given task, the planar geometry of all spatial features included in the dataset of that task is determined to obtain a set of planar geometry shapes; then, the union of each planar geometry shape in the set of planar geometry shapes can be determined; thus, the first edge of the union is expanded outward by a preset distance to obtain a second edge; the range formed by the second edge is determined as the planar range corresponding to the dataset of that task. Figure 4 for Figure 3 A schematic diagram illustrating the planar extent corresponding to the dataset used to determine the task. Figure 4 In the figure, reference numeral 41 represents the second edge obtained by expanding the first edge outward.

[0077] As can be seen, in this embodiment, the first edge is extended outward by a preset distance to obtain the second edge, ensuring that there is an intersection between the planar ranges corresponding to the datasets of adjacent tasks, that is, there is duplicate data between adjacent tasks. The preset distance can be set according to the actual application scenario; this embodiment does not limit the specific value of the preset distance. In this embodiment, the preset distance is 10cm.

[0078] However, in this embodiment of the disclosure, using the planar range formed by the second edge of the union as the planar range corresponding to the dataset of the task may have the following problem: the planar range corresponding to the dataset of the task may have a concave surface, such as... Figure 4 As shown by reference numeral 43 in the attached figure, this results in no duplicate data between this task and its adjacent tasks.

[0079] To address this issue, step 202, "determining the planar extent corresponding to the dataset for each task," specifically involves:

[0080] For any given task, the planar geometry of all spatial features included in the dataset of that task is determined to obtain a set of planar geometry shapes. Then, the union of each planar geometry shape in the set of planar geometry shapes can be determined. Thus, the first edge of the union is expanded outward by a preset distance to obtain a second edge. The convex hull of the second edge is determined to obtain a third edge, which is the edge of the convex hull of the second edge. The range formed by the third edge is determined as the planar range corresponding to the dataset of that task. Figure 4 In the attached drawing, reference numeral 42 is the third edge, and the third edge 42 is the edge of the convex hull of the second edge 41.

[0081] Here, the convex hull is a convex polygon that can contain the second edge. The method for determining the convex hull of the second edge can adopt the conventional convex hull analysis method in this field, which will not be elaborated further. Since the third edge is the edge of the convex hull of the second edge, the range formed by the third edge does not have a concave surface, ensuring that there is an intersection between the planar ranges corresponding to the datasets of adjacent tasks, that is, there is duplicate data between adjacent tasks.

[0082] Based on the above embodiments, step 203, which involves dividing tasks whose plane ranges do not overlap into the same work order based on the plane range corresponding to the dataset of each task, is as follows: Figure 5 As shown, including but not limited to the following steps 501 to 503:

[0083] In step 501, the work order division order of each task is determined based on the task identifier of each task.

[0084] In this embodiment of the disclosure, each task has a task identifier (id), such as a task number, where a smaller task number indicates that the task was created earlier. Therefore, based on the task identifier of each task, the division order of each task can be determined. For example, the task identifiers are sorted in ascending order, and then each task is divided into work orders according to the sorting order. For example, if there are 5 tasks with task identifiers of 5, 2, 4, 3, 1, the sorted task identifiers are 1, 2, 3, 4, 5, that is, work order division starts from task 1.

[0085] In step 502, for any task, based on the work order division order, the subsequent tasks of the task are traversed, and the task and subsequent tasks that do not intersect with the plane range are assigned to the same work order; for the remaining tasks, this step is repeated until all tasks are assigned to work orders.

[0086] Figure 6 A schematic diagram of the planar extent corresponding to a dataset for a different task is shown. Figure 6 This shows that there is a relationship of duplicate data between Task 1 and Task 5. Based on the task identifier of each task, the order of work orders for each task is determined as follows: Task 1, Task 2, Task 3, Task 4, and Task 5.

[0087] For Task 1, since Tasks 2, 4, and 5 all contain duplicate data, Task 3 and Task 1 will be assigned to the same work order. For the remaining tasks: Tasks 2, 4, and 5, for Task 2, since Task 4 contains duplicate data, Task 5 and Task 2 will be assigned to the same work order. For the remaining task: Task 4, since it is the only remaining task, it will be assigned to a single work order.

[0088] In this embodiment of the disclosure, after dividing the work orders, the set of algorithm output results corresponding to each work order is saved into the corresponding work order.

[0089] It should be noted that the plane ranges corresponding to the datasets of any two tasks in the same work order do not overlap, that is, there is no duplicate data between any two tasks in the same work order.

[0090] Based on the above embodiments, Figure 2 The algorithm evaluation method shown may also include the following steps:

[0091] The evaluation results of multiple work orders are summarized to obtain the first number of algorithm output results evaluated as correct, the second number of algorithm output results evaluated as incorrect, and the third number of tasks evaluated as missing. The evaluation result is the correct, incorrect, and missing evaluation result of the set of algorithm output results corresponding to the work order. Then, based on the first, second, and third numbers, the values ​​of multiple algorithm evaluation indicators are determined.

[0092] In this embodiment of the disclosure, multiple algorithm evaluation metrics include at least one of the following: accuracy, error rate, false negative rate, and recall. Accuracy is the ratio of the number of accurate algorithm outputs to the total number of tasks; error rate is the ratio of the number of incorrect algorithm outputs to the total number of tasks; false negative rate is the ratio of the number of tasks missed by the algorithm to the total number of tasks; and recall is the ratio of the number of tasks reported by the algorithm (i.e., the number of algorithm outputs) to the total number of tasks.

[0093] The calculation methods for each indicator are as follows:

[0094] The accuracy rate is the ratio of the first number to the total number; where the total number is the sum of the first, second, and third numbers.

[0095] The error rate is the ratio of the second number to the total number.

[0096] The underreporting rate is the ratio of the third number to the total number.

[0097] Recall rate is the ratio of the sum of the first and second quantities to the total quantity, that is, recall rate = (first quantity + second quantity) / total quantity.

[0098] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art will understand that the embodiments of this disclosure are not limited to the described order of actions, because according to the embodiments of this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in the specification are all optional embodiments.

[0099] Figure 7 This diagram illustrates an algorithm evaluation device provided in an embodiment of this disclosure. This device can be applied to electronic devices, including but not limited to in-vehicle devices, smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, smart home devices, and servers. The server can be a standalone server or a cluster of multiple servers, and can include locally located servers and cloud-based servers. The algorithm evaluation device provided in this embodiment can execute the processing flow provided in the algorithm evaluation method embodiment, such as... Figure 7 As shown, the algorithm evaluation device includes: an acquisition unit 71, a determination unit 72, a division unit 73, and an evaluation unit 74.

[0100] The acquisition unit 71 is used to acquire the algorithm output results corresponding to each task, where the task is a dataset of spatial data used to evaluate the algorithm.

[0101] Unit 72 is used to determine the plane range corresponding to the dataset of each task;

[0102] The partitioning unit 73 is used to partition tasks whose plane ranges do not overlap into the same work order based on the plane range corresponding to the dataset of each task, so as to obtain multiple work orders and the set of algorithm output results corresponding to each work order;

[0103] Evaluation unit 74 is used to evaluate the algorithm output result set corresponding to one of the multiple work orders each time, taking the work order as the evaluation unit, and to obtain the evaluation result of the algorithm output result set until all multiple work orders have been evaluated.

[0104] In some embodiments, the determining unit 72 is configured to: for any task, determine the planar geometry of all spatial features included in the dataset of the task to obtain a set of planar geometry; determine the union of each planar geometry in the set of planar geometry; and determine the planar extent corresponding to the dataset of the task based on the first edge of the union.

[0105] In some embodiments, the determining unit 72 determines the planar range corresponding to the dataset of the task based on the first edge of the union, including: expanding the first edge of the union by a preset distance to obtain a second edge; and determining the range formed by the second edge as the planar range corresponding to the dataset of the task.

[0106] In some embodiments, the determining unit 72 determines the planar range corresponding to the dataset of the task based on the first edge of the union, including: expanding the first edge of the union by a preset distance to obtain a second edge; determining the convex hull of the second edge to obtain a third edge, wherein the third edge is the edge of the convex hull of the second edge; and determining the range formed by the third edge as the planar range corresponding to the dataset of the task.

[0107] In some embodiments, the partitioning unit 73 is used to: determine the work order partitioning order of each task based on the task identifier of each task; for any task, based on the work order partitioning order, traverse the subsequent tasks of the task and partition the task and subsequent tasks that do not intersect with the plane range into the same work order; repeat this step for the remaining tasks until all tasks are partitioned into work orders.

[0108] In some embodiments, the evaluation result of the algorithm output result set is the right, wrong, and omission evaluation result. The algorithm evaluation device further includes a summarization unit, which is used to: summarize the evaluation results of multiple work orders to obtain a first number of algorithm output results evaluated as right, a second number of algorithm output results evaluated as wrong, and a third number of tasks evaluated as omission; wherein, the evaluation result is the right, wrong, and omission evaluation result of the algorithm output result set corresponding to the work order; and determine the values ​​of multiple algorithm evaluation indicators based on the first number, the second number, and the third number.

[0109] In some embodiments, multiple algorithm evaluation metrics include at least one of the following: accuracy, error rate, false negative rate, and recall rate;

[0110] The accuracy rate is the ratio of the first number to the total number; where the total number is the sum of the first, second, and third numbers.

[0111] The error rate is the ratio of the second-highest number to the total number.

[0112] The underreporting rate is the ratio of the third number to the total number.

[0113] Recall rate is the ratio of the sum of the first and second recalls to the total number of recalls.

[0114] For details of the embodiments of the algorithm evaluation device disclosed above, please refer to the details of the embodiments of the algorithm evaluation method described above. To avoid repetition, they will not be repeated.

[0115] Figure 8 This is an exemplary block diagram of a computer device provided in an embodiment of this disclosure. Figure 8 As shown, the computer device includes: at least one computing device 81 and at least one storage device 82 for storing instructions. It is understood that the storage device 82 in this embodiment may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0116] In some implementations, storage device 82 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating systems and applications.

[0117] The operating system includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic tasks and handle hardware-based tasks. The application programs include various applications, such as media players and browsers, used to implement various application tasks. The program implementing the algorithm evaluation method provided in this disclosure can be included in the application programs.

[0118] In this embodiment of the disclosure, at least one computing device 81 executes the steps of the various embodiments of the algorithm evaluation method provided in this disclosure by calling a program or instruction stored in at least one storage device 82, specifically, a program or instruction stored in an application program.

[0119] The algorithm evaluation method provided in this disclosure can be applied to or implemented by a computing device 81. The computing device 81 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the hardware or by instructions in software within the computing device 81. The computing device 81 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor.

[0120] The steps of the algorithm evaluation method provided in this disclosure can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in storage device 82, and computing device 81 reads information from storage device 82 and combines it with hardware to complete the steps of the method.

[0121] This disclosure also proposes a computer-readable storage medium that stores a program or instructions that cause a computer to perform the steps of the various embodiments of the algorithm evaluation method; to avoid repetition, these steps will not be repeated here. The computer-readable storage medium can be a non-transitory computer-readable storage medium.

[0122] This disclosure also proposes a computer program product, wherein the computer program product includes a computer program stored in a non-transitory computer-readable storage medium, and at least one processor of the computer reads from the storage medium and executes the computer program, causing the computer to perform the steps of the various embodiments of the algorithm evaluation method, which will not be repeated here to avoid repetition.

[0123] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0124] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this disclosure and form different embodiments.

[0125] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. An algorithm evaluation method, the method comprising: Obtain the algorithm output results corresponding to each task, where the task is a dataset of spatial data used to evaluate the algorithm; Determine the planar extent corresponding to the dataset for each of the tasks; Based on the plane range corresponding to the dataset of each task, tasks whose plane ranges do not overlap are divided into the same work order, resulting in multiple work orders and a set of algorithm output results corresponding to each work order; Using work orders as the evaluation unit, the algorithm output result set corresponding to one of the multiple work orders is evaluated each time to obtain the evaluation result of the algorithm output result set, until all the multiple work orders have been evaluated; Wherein, determining the plane range corresponding to the dataset for each task includes: For any given task, determine the planar geometry of all spatial features included in the dataset for that task, and obtain a set of planar geometry shapes; Determine the union of all planar geometric shapes in the set of planar geometric shapes; The planar extent corresponding to the dataset of this task is determined based on the first edge of the union.

2. The method according to claim 1, wherein, Determining the planar extent corresponding to the dataset for this task based on the first edge of the union includes: The first edge of the union is expanded outward by a predetermined distance to obtain the second edge; The range formed by the second edge is defined as the planar range corresponding to the dataset of this task.

3. The method according to claim 1, wherein, Determining the planar extent corresponding to the dataset for this task based on the first edge of the union includes: The first edge of the union is expanded outward by a predetermined distance to obtain the second edge; The convex hull of the second edge is determined to obtain the third edge, which is the edge of the convex hull of the second edge; The range formed by the third edge is defined as the planar range corresponding to the dataset of this task.

4. The method according to claim 1, wherein, The step of grouping tasks with non-overlapping plane ranges into the same work order based on the plane range corresponding to the dataset of each task includes: Based on the task identifier of each task, determine the work order division order of each task; For any given task, based on the work order division order, traverse the subsequent tasks of that task and assign the task and subsequent tasks that do not intersect with the plane range to the same work order; repeat this step for the remaining tasks until all tasks are assigned to work orders.

5. The method according to any one of claims 1-4, wherein, The algorithm outputs a set of results with evaluations of correctness, error, and omissions. The method also includes: The evaluation results of the multiple work orders are summarized to obtain a first number of algorithm output results evaluated as correct, a second number of algorithm output results evaluated as incorrect, and a third number of tasks evaluated as missing; wherein, the evaluation result is the correct, incorrect, and missing evaluation result of the algorithm output result set corresponding to the work order; Based on the first quantity, the second quantity, and the third quantity, the values ​​of multiple algorithm evaluation metrics are determined.

6. The method according to claim 5, wherein, The algorithm evaluation metrics include at least one of the following: accuracy, error rate, false negative rate, and recall rate; The accuracy rate is the ratio of the first quantity to the total quantity; wherein the total quantity is the sum of the first quantity, the second quantity, and the third quantity; The error rate is the ratio of the second quantity to the total quantity; The underreporting rate is the ratio of the third quantity to the total quantity; The recall rate is the ratio of the sum of the first quantity and the second quantity to the total quantity.

7. An algorithm evaluation device, the device comprising: The acquisition unit is used to acquire the algorithm output results corresponding to each task, wherein the task is a dataset of spatial data used to evaluate the algorithm. A determining unit is configured to determine the planar extent corresponding to the dataset of each task, comprising: for any task, determining the planar geometry of all spatial features included in the dataset of that task, to obtain a set of planar geometry shapes; determining the union of the planar geometry shapes in the set of planar geometry shapes; and determining the planar extent corresponding to the dataset of that task based on the first edge of the union. The partitioning unit is used to partition tasks whose plane ranges do not overlap into the same work order based on the plane range corresponding to the dataset of each task, thereby obtaining multiple work orders and a set of algorithm output results corresponding to each work order; The evaluation unit is used to evaluate the algorithm output result set corresponding to one of the multiple work orders each time, taking the work order as the evaluation unit, and to obtain the evaluation result of the algorithm output result set until all the multiple work orders have been evaluated.

8. A computer-readable storage medium, wherein, The computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the algorithm evaluation method as described in any one of claims 1 to 6.

9. A computer device, wherein, The storage device includes at least one computing device and at least one storage instruction; when the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the steps of the algorithm evaluation method as described in any one of claims 1 to 6.

Images