Evaluation method and system of road surface element prediction model and electronic equipment

Abstract

The invention provides an evaluation method and system for a road surface element prediction model and electronic equipment, and relates to the technical field of model evaluation, and the method comprises the steps: obtaining truth value data and prediction data of a single-frame image, carrying out the classification according to element types, and carrying out the matching, and generating a target data pair; performing first-level evaluation on the target data pair to obtain a first evaluation index of the single-frame image, and further calculating an overall evaluation index, namely a second evaluation index, of the prediction model based on the first evaluation indexes of all the single-frame images, therefore, the prediction performance of the model on continuous elements, discrete elements and cross point elements can be evaluated more accurately and comprehensively, and a better direction is provided for iteration of the prediction model.

Description

Technical Field

[0001] The present application relates to the field of model evaluation technology, and in particular to an evaluation method, system and electronic equipment for a road surface element prediction model. Background Art

[0002] Model evaluation refers to a series of methods and standards for evaluating the performance of deep learning models. In related technologies, models are usually evaluated based on a single type of labeled data. This method of evaluating models based only on a single type of labeled data usually only focuses on the performance indicators of specific dimensions of the model and cannot comprehensively reflect the overall performance of the model. In addition, this evaluation method that relies on a single type of labeled data is difficult to accurately reflect the comprehensive capabilities of the model in real scenarios, which may cause deviations between the evaluation results and actual performance.

[0003] Therefore, a new model evaluation method is urgently needed. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a prediction model evaluation method, system and electronic device to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of the present application, a method for evaluating a road surface element prediction model is provided, the method comprising:

[0006] Obtaining true value data of each of the plurality of single-frame images, and obtaining prediction data of each of the plurality of single-frame images output by the model to be evaluated;

[0007] Classifying the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images;

[0008] Matching the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements, and intersection elements of a road surface;

[0009] Performing a first-level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single-frame image, wherein the first evaluation index at least includes a first distance index;

[0010] Determining a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicator at least includes a second distance indicator;

[0011] The prediction model is evaluated according to the second evaluation indicator.

[0012] Optionally, performing a first level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single frame image includes:

[0013] Determining the element categories of each of the plurality of target data pairs;

[0014] According to the respective element categories of the plurality of target data pairs, respectively calculate the distance similarity value of each target data pair, wherein, when the target element is a continuous element, the distance similarity value is the chamfer distance, when the target element is a discrete element, the distance similarity value is the offset, and when the target element is an intersection element, the distance similarity value is the Euclidean distance;

[0015] The first distance index of the target single-frame image is calculated based on the respective distance similarity values ​​of the plurality of target data pairs.

[0016] Optionally, determining a second evaluation index for the prediction model according to the first evaluation index of each of the target single-frame images includes:

[0017] Obtaining evaluation scenario information for evaluating the model to be evaluated;

[0018] According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information;

[0019] Obtaining a first distance index of each of the target single-frame images that conform to the evaluation scene information;

[0020] Classifying the first distance indicators of the target single-frame images that meet the evaluation scene information according to the element category;

[0021] The weighted average of the first distance index under each of the element categories is calculated respectively to obtain the second distance index of the prediction model under different element categories.

[0022] Optionally, the first evaluation index further includes: a first prediction index; the first-level evaluation of the plurality of target data pairs to obtain the first evaluation index for the target single-frame image includes:

[0023] Obtaining a distance similarity value for each target data pair;

[0024] Compare the distance similarity value of each target data pair with a first distance similarity threshold, and determine a first number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold, and a second number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold.

[0025] The first prediction index is determined according to the first quantity and the second quantity, and the first prediction index at least includes: a first precision index, a first recall index, and a first F-value index.

[0026] Optionally, the second evaluation index further includes: a second prediction index, and the second evaluation index for the prediction model is determined according to the first evaluation index of each of the target single-frame images, including:

[0027] Obtaining evaluation scenario information for evaluating the model to be evaluated;

[0028] According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information;

[0029] Counting a first total number of target data pairs whose distance similarity values ​​of each target single-frame image that meets the evaluation scene information are greater than the first distance similarity threshold, and a second total number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold;

[0030] According to the first total number and the second total number, the second prediction indicator is determined: the second prediction indicator includes at least: a second precision indicator, a second recall rate indicator, a second F value indicator and an AP indicator, and the AP indicator is calculated based on the precision-recall curve drawn based on the second precision indicator and the second recall rate indicator.

[0031] Optionally, the target element category is the continuous element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including:

[0032] Matching each first true value data of the continuous element with each first prediction data of the continuous element to obtain a plurality of first initial data pairs;

[0033] Calculating the chamfer distance of each of the first initial data pairs to obtain a first distance similarity matrix of the continuous elements;

[0034] Determine a bidirectional chamfer distance of each of the first initial data pairs according to the chamfer distance of each of the first initial data pairs in the first distance similarity matrix;

[0035] Comparing the bidirectional chamfer distance of each of the first initial data pairs with a preset chamfer distance threshold;

[0036] The first initial data pair whose bidirectional chamfer distance is less than or equal to the preset chamfer distance threshold is determined as the first target data pair.

[0037] Optionally, the target element category is the discrete element, and the matching of the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category includes:

[0038] Matching each second true value data of the discrete element with each second prediction data of the discrete element to obtain a plurality of second initial data pairs;

[0039] Calculating the offset of each of the second initial data pairs to obtain a second distance similarity matrix of the discrete elements;

[0040] Obtaining a confidence level for each of the second prediction data;

[0041] Sorting all the second prediction data in descending order according to their respective confidence levels;

[0042] From the second distance similarity matrix, a second target true value data is matched in order for each of the sorted second prediction data to obtain multiple second target data pairs of the discrete elements, wherein the offset between the second target true value data and the second prediction data is less than or equal to a preset offset threshold.

[0043] Optionally, the target element category is the intersection element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including:

[0044] Matching each third true value data of the intersection element with each third predicted data of the intersection element to obtain a plurality of third initial data pairs;

[0045] Calculating the Euclidean distance of each of the third initial data pairs to obtain a third distance similarity matrix of the intersection elements;

[0046] Comparing the Euclidean distance of each of the third initial data pairs in the third distance similarity matrix with a preset Euclidean distance threshold;

[0047] The third initial data pair whose Euclidean distance is greater than the preset Euclidean distance threshold is determined as a third target data pair.

[0048] Optionally, the element category is the continuous element, and before classifying the true value data and the predicted data of the target single-frame image, the method further includes:

[0049] Acquiring the resolution of the target single-frame image;

[0050] According to the resolution of the target single-frame image, interpolation processing is performed on the continuous elements in the target single-frame image at preset intervals;

[0051] The target single-frame image after interpolation processing is visualized.

[0052] In a second aspect of the present application, a road surface element prediction model evaluation system is provided, the system comprising:

[0053] An acquisition module, used to acquire true value data of each of a plurality of single-frame images, and to acquire prediction data of each of the plurality of single-frame images output by the model to be evaluated;

[0054] A classification module, used for classifying the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images;

[0055] A matching module, used for matching the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements and intersection elements of the road surface;

[0056] A first evaluation module, configured to perform a first level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, wherein the first evaluation index at least includes a first distance index;

[0057] A second evaluation module, used to determine a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicator at least includes a second distance indicator;

[0058] The third evaluation module is used to evaluate the prediction model according to the second evaluation indicator.

[0059] Optionally, the first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, and the first evaluation module includes:

[0060] A judgment submodule, used for judging the element categories of the plurality of target data pairs;

[0061] A first calculation submodule is used to calculate the distance similarity value of each target data pair according to the respective element categories of the plurality of target data pairs, wherein, when the target element is a continuous element, the distance similarity value is the chamfer distance, when the target element is a discrete element, the distance similarity value is the offset, and when the target element is an intersection element, the distance similarity value is the Euclidean distance;

[0062] The second calculation submodule is used to calculate the first distance index of the target single-frame image according to the respective distance similarity values ​​of the plurality of target data pairs.

[0063] Optionally, according to the first evaluation index of each of the target single-frame images, a second evaluation index for the prediction model is determined, and the second evaluation module includes:

[0064] A first acquisition submodule is used to acquire evaluation scenario information for evaluating the model to be evaluated;

[0065] A first determination submodule is used to perform scene screening on the plurality of target single-frame images according to the evaluation scene information, and determine the target single-frame image that meets the evaluation scene information;

[0066] A second acquisition submodule is used to acquire a first distance index of each of the target single-frame images that conform to the evaluation scene information;

[0067] A first classification submodule, used for classifying the first distance index of each of the plurality of target single-frame images that meet the evaluation scene information according to element categories;

[0068] The second determination submodule is used to calculate the weighted average of the first distance index under each of the element categories respectively, to obtain the second distance index of the prediction model under different element categories.

[0069] Optionally, the first evaluation index further includes: a first prediction index; the first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, and the first evaluation module includes:

[0070] A third acquisition submodule is used to obtain the distance similarity value of each target data pair;

[0071] a first comparison submodule, configured to compare the distance similarity value of each target data pair with a first distance similarity threshold, and determine a first number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold, and a second number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold.

[0072] The third determination submodule is used to determine the first prediction index according to the first quantity and the second quantity, and the first prediction index at least includes: a first precision index, a first recall index and a first F value index.

[0073] Optionally, the second evaluation index further includes: a second prediction index, wherein the second evaluation index for the prediction model is determined according to the first evaluation index of each of the target single-frame images, and the second evaluation module includes:

[0074] The fourth acquisition submodule is used to obtain evaluation scenario information for evaluating the model to be evaluated;

[0075] A first screening submodule, configured to screen the plurality of target single-frame images according to the evaluation scene information, and determine the target single-frame image that meets the evaluation scene information;

[0076] a statistical submodule, configured to count a first total number of target data pairs whose respective distance similarity values ​​of each target single-frame image that meets the evaluation scene information are greater than the first distance similarity threshold, and a second total number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold;

[0077] The fourth determination submodule is used to determine the second prediction indicator based on the first total number and the second total number: the second prediction indicator includes at least: a second precision indicator, a second recall rate indicator, a second F value indicator and an AP indicator, and the AP indicator is calculated based on the precision-recall curve drawn based on the second precision indicator and the second recall rate indicator.

[0078] Optionally, the target element category is the continuous element, the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module includes:

[0079] A first matching submodule, used for matching each first true value data of the continuous element with each first predicted data of the continuous element to obtain a plurality of first initial data pairs;

[0080] A third calculation submodule, used for calculating the chamfer distance of each of the first initial data pairs to obtain a first distance similarity matrix of the continuous elements;

[0081] a fifth determination submodule, configured to determine a bidirectional chamfer distance of each of the first initial data pairs according to the chamfer distance of each of the first initial data pairs in the first distance similarity matrix;

[0082] A second comparison submodule, used for comparing the bidirectional chamfer distance of each of the first initial data pairs with a preset chamfer distance threshold;

[0083] The sixth determination submodule is used to determine the first initial data pair whose bidirectional chamfer distance is less than or equal to the preset chamfer distance threshold as the first target data pair.

[0084] Optionally, the target element category is the discrete element, the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module includes:

[0085] A second matching submodule, used for matching each second true value data of the discrete element with each second predicted data of the discrete element, respectively, to obtain a plurality of second initial data pairs;

[0086] a fourth calculation submodule, configured to calculate an offset of each of the second initial data pairs to obtain a second distance similarity matrix of the discrete elements;

[0087] A fifth acquisition submodule, used to acquire the confidence level of each of the second prediction data;

[0088] A sorting submodule, used to sort all the second prediction data in descending order according to their respective confidence levels;

[0089] The seventh determination submodule is used to match a second target true value data for each sorted second prediction data in order from the second distance similarity matrix, to obtain multiple second target data pairs of the discrete elements, wherein the offset between the second target true value data and the second prediction data is less than or equal to a preset offset threshold.

[0090] Optionally, the target element category is the intersection element, the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module includes:

[0091] A third matching submodule, used for matching each third true value data of the intersection element with each third predicted data of the intersection element, respectively, to obtain a plurality of third initial data pairs;

[0092] A fifth calculation submodule, used for calculating the Euclidean distance of each of the third initial data pairs to obtain a third distance similarity matrix of the intersection elements;

[0093] A third comparison submodule, configured to compare the Euclidean distance of each of the third initial data pairs in the third distance similarity matrix with a preset Euclidean distance threshold;

[0094] The eighth determination submodule is used to determine the third initial data pair whose Euclidean distance is greater than the preset Euclidean distance threshold as a third target data pair.

[0095] Optionally, the system further comprises:

[0096] A sixth acquisition submodule, used to acquire the resolution of the target single-frame image;

[0097] An interpolation processing submodule, used for performing interpolation processing on continuous elements in the target single-frame image at preset intervals according to the resolution of the target single-frame image;

[0098] The visualization processing submodule is used to perform visualization processing on the target single-frame image after the interpolation processing.

[0099] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the evaluation method of the pavement element prediction model as described in the first aspect of the present application.

[0100] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the evaluation method of the pavement element prediction model as described in the first aspect of the present application is implemented.

[0101] Beneficial effects of this application:

[0102] The present application provides an evaluation method for a road feature prediction model, the method comprising: obtaining true value data of each of a plurality of single-frame images, and obtaining predicted data of each of the plurality of single-frame images output by a model to be evaluated; classifying the true value data and predicted data of a target single-frame image according to feature categories, wherein each feature category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images; matching the true value data and predicted data of a target feature category to obtain a plurality of target data pairs under the target feature category, wherein the target feature category is any one of the plurality of feature categories, and the feature categories include: continuous features, discrete features, and intersection features of a road surface; performing a first-level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single-frame image, the first evaluation index at least including a first distance index; determining a plurality of second evaluation indexes for the prediction model according to the first evaluation index of each of the target single-frame images, the second evaluation index at least including a second distance index; and evaluating the prediction model according to the second evaluation index.

[0103] The present application obtains the true value data and predicted data of a single-frame image, classifies them by element category, and matches them to generate a target data pair; performs a first-level evaluation on the target data pair to obtain a first evaluation index for the single-frame image, and further calculates the overall evaluation index of the prediction model, i.e., the second evaluation index, based on the first evaluation index of all single-frame images. This enables a more accurate and comprehensive comprehensive evaluation of the model's prediction performance on continuous elements, discrete elements, and intersection elements, thereby providing a better direction for the iteration of the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0105] Figure 1 It is a schematic diagram of the steps of a method for evaluating a road surface element prediction model provided in an embodiment of the present application;

[0106] Figure 2 It is a schematic diagram of a precision-recall curve provided in an embodiment of the present application;

[0107] Figure 3 It is a schematic diagram of a continuous element interpolation or visualization process provided by an embodiment of the present application;

[0108] Figure 4It is a schematic diagram of a visualization result of discrete element target matching on a single frame image provided by an embodiment of the present application;

[0109] Figure 5 This is a logical diagram of an evaluation system indicator calculation provided by an embodiment of the present application;

[0110] Figure 6 It is a schematic diagram of an evaluation system architecture provided in an embodiment of the present application;

[0111] Figure 7 It is a schematic diagram of an evaluation system of a road surface element prediction model provided in an embodiment of the present application;

[0112] Figure 8 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0113] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0114] Environmental perception tasks play a vital role in complex and ever-changing driving environments, providing a reliable basis for vehicle positioning and state decision-making, thereby ensuring driving safety. Traditional perception methods are usually limited to single-view data and it is difficult to fully capture environmental information. In order to improve the accuracy and efficiency of detection, current perception tasks usually fuse multiple sensor data (such as lidar and multi-channel cameras) to generate a two-dimensional bird's-eye view (BEV) feature map, and perform recognition, classification, and detection tasks on the BEV feature map.

[0115] Among these perception tasks, road feature detection is a key component, responsible for identifying roads and their features, and extracting rich road information and traffic guidance information. Road features can be divided into the following three categories according to their characteristics:

[0116] 1. Continuous elements: including lane lines and curbs, which are used to determine the vehicle's driving trajectory and direction. The detection results are usually output in the form of point sets and combined into instances.

[0117] 2. Discrete elements: including traffic arrows, speed bumps, crosswalks and stop lines, etc., which are used to identify specific traffic rules or driving instructions. The detection results are output in the form of irregular polygonal boxes.

[0118] 3. Intersection: including intersections where the number of lanes changes or points where the lane line changes from solid to dotted in diverging / merging scenarios (e.g., transition points where lane lines change from solid to dotted). These elements provide driving guidance for vehicles, and the detection results are output in the form of points.

[0119] Based on perception tasks such as road surface feature detection, road surface feature prediction models are often used to perform road surface feature detection. The detection performance of road surface feature prediction models directly affects the accuracy of road surface feature prediction. Therefore, the evaluation of road surface feature prediction models is particularly important.

[0120] In a first aspect of the present application, a method for evaluating a road surface element prediction model is provided. Figure 1 As shown, the method includes:

[0121] S101, obtaining true value data of each of a plurality of single-frame images, and obtaining prediction data of each of the plurality of single-frame images output by the model to be evaluated.

[0122] In this step, before evaluating the prediction model, it is first necessary to obtain the basic data for evaluation, including true data and predicted data. True data refers to the annotation results of road surface elements in a single-frame image, which is usually generated by manual or automatic annotation tools. These data come from a high-precision annotation set and can accurately reflect the real road surface elements, including: continuous elements of the road surface: such as lane lines, curbs, etc., marked in the form of point sets. Discrete elements: such as arrows, speed bumps, crosswalks, stop lines, etc., marked as irregular polygonal boxes. Intersection elements: such as lane line intersections or virtual-real change points. The prediction data is generated by the prediction model to be evaluated. By inputting a single-frame image, the model outputs the prediction results for various road surface elements, which also include: continuous elements, discrete elements, and intersection elements.

[0123] In some embodiments, after obtaining the true value data and predicted data of a single frame image, these data need to be preprocessed, including common data preprocessing such as formatting unification, which will not be described in detail in this application.

[0124] S102 classifies the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images.

[0125] In this step, for each target single-frame image, the acquired true value data and predicted data are classified according to the element category for subsequent matching and evaluation. Among them, the target single-frame image refers to any one of the multiple single-frame images, which are described as follows:

[0126] In this application, the element categories are divided into three categories, including continuous elements, discrete elements and intersection elements. Different true value data and different predicted data correspond to their own element categories, so these true value data and predicted data need to be divided into their own category elements.

[0127] In some embodiments, when the true value data and the predicted data are classified into element categories, these data can be integrated again according to the sub-attributes of each element category. By classifying the road surface elements and integrating the sub-attributes, structured data input is provided for subsequent evaluation, ensuring the accuracy and efficiency of the evaluation of different element categories. Among them, each element category has its own sub-attribute, for example, the lane line has the sub-attribute of direction (one-way, two-way), linear attribute (solid line, dashed line, fishbone line), color attribute (white, yellow, blue), functional attribute (lane line to be driven, tidal lane line), multi-line attribute (single line, double line, triple line), etc., and the curb has the sub-attributes of flat curb, road curb, cone barrel, etc.

[0128] S103, matching the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements and intersection elements of the road surface.

[0129] In this step, for the target feature category in the target single-frame image, its true value data and predicted data are matched to generate multiple target data pairs. The matching method needs to match according to the characteristics of the feature category. For example, for the matching of continuous features, the two-way chamfer distance can be used to measure the distance between the true value data and the predicted data. If the two-way chamfer distance meets the preset conditions, the match is considered successful. For the matching of discrete features, the offset can be used to measure the distance between the true value data and the predicted data. Since discrete features are mostly represented in the form of true value boxes and prediction boxes, the offset can also be regarded as the IOU (Intersection over Union, IOU) between the true value box and the prediction box. If the offset meets the preset conditions, the match is considered successful. For the matching of intersection features, the Euclidean distance can be used to measure the distance between the true value data and the predicted data. If the Euclidean distance meets the preset conditions, the match is considered successful.

[0130] S104, performing a first level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, wherein the first evaluation index at least includes a first distance index.

[0131] In this step, a first-level evaluation is performed on multiple target data pairs of the target element category, and a first evaluation index is calculated and output for a single-frame image of the target.

[0132] For example: if the target element category is a continuous element, it is necessary to calculate the bidirectional chamfer distance of each of the multiple target data pairs under the continuous element, so as to obtain the first distance index of the single-frame image of the target, that is, the first evaluation index of the single-frame image of the target. If the target element category is a discrete element, it is necessary to calculate the offset of each of the multiple target data pairs under the discrete element, so as to obtain the first distance index of the single-frame image of the target, that is, the first evaluation index. If the target element category is an intersection element, it is necessary to calculate the Euclidean distance of each of the multiple target data pairs under the intersection element, so as to obtain the first distance index of the single-frame image of the target, that is, the first evaluation index.

[0133] S105, determining a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicators at least include a second distance indicator;

[0134] The second evaluation index is a comprehensive analysis of the evaluation results of all single-frame images, reflecting the performance of the prediction model on the overall test set. In the present application, the second evaluation index includes at least a second distance index. For continuous elements, the second distance index is calculated based on the average value of the two-way chamfer distance of all single-frame images to obtain the second distance index. For discrete elements, the average value of the offset of all single-frame images is calculated to obtain the second distance index. For intersection elements, the average value of the Euclidean distance of all single-frame images of all single-frame images is calculated to obtain the second distance index.

[0135] S106: Evaluate the prediction model according to the second evaluation indicator.

[0136] In this step, the second evaluation index of each element category is summarized, the prediction performance of the prediction model on continuous elements, discrete elements, and intersection elements is analyzed and evaluated, and the evaluation results are output. Finally, combined with the evaluation results, specific suggestions are provided for the optimization and iteration of the prediction model. For example: for continuous elements, optimize the point set interpolation method or strengthen curve feature learning; for discrete elements, improve the positioning accuracy of the detection box or increase the training of complex polygon boxes; for intersection elements, strengthen the training data coverage of complex intersection scenes, etc.

[0137] In a preferred embodiment, the first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, including:

[0138] Determining the element categories of each of the plurality of target data pairs;

[0139] According to the respective element categories of the plurality of target data pairs, respectively calculate the distance similarity value of each target data pair, wherein, when the target element is a continuous element, the distance similarity value is the chamfer distance, when the target element is a discrete element, the distance similarity value is the offset, and when the target element is an intersection element, the distance similarity value is the Euclidean distance;

[0140] The first distance index of the target single-frame image is calculated based on the respective distance similarity values ​​of the plurality of target data pairs.

[0141] In this embodiment, for each target data pair, the element category to which it belongs is first determined. The element categories include continuous elements, discrete elements, and intersection elements. Different distance similarity calculation methods are adopted according to different categories.

[0142] For continuous features, the chamfer distance of the target data pair is calculated. The chamfer distance is used to measure the similarity between two curves or point sets, calculate the minimum distance between each true data and the predicted data, and calculate their average. For discrete features, the offset of the target data pair is calculated. The offset is used to measure the spatial position difference between the predicted data and the true data. For intersection features, the Euclidean distance of the target data pair is calculated. The Euclidean distance measures the straight-line distance between the predicted data and the true data.

[0143] According to the distance similarity values ​​of all target data pairs, the first distance index of the target single frame image is calculated. The first distance index is a quantitative value that measures the overall matching effect of all target data pairs in the image, reflecting the distance error between the predicted data and the true value data. The specific calculation method includes averaging or weighted averaging the distance similarity values ​​of all target data pairs to obtain the first distance index of the single frame image.

[0144] In one embodiment, determining a second evaluation index for the prediction model according to the first evaluation index of each of the target single-frame images includes:

[0145] Obtaining evaluation scenario information for evaluating the model to be evaluated;

[0146] According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information;

[0147] Obtaining a first distance index of each of the target single-frame images that conform to the evaluation scene information;

[0148] Classifying the first distance indicators of the target single-frame images that meet the evaluation scene information according to the element category;

[0149] The weighted average of the first distance index under each of the element categories is calculated respectively to obtain the second distance index of the prediction model under different element categories.

[0150] In this embodiment, first, the evaluation scene information for evaluating the model to be evaluated is obtained. Such scene information may include different test environment conditions, such as weather, light, time, road type (urban road, rural road, etc.) and other factors that may affect the perception results. The evaluation scene information helps to comprehensively evaluate the performance of the model under different environmental conditions.

[0151] According to the obtained evaluation scene information, multiple target single-frame images are screened. The basis for screening is to ensure that the target single-frame images meet the requirements of specific evaluation scenes. For example, if the evaluation scene information is specified as "night driving", all images taken at night are selected; if the scene is "rainy day", all data images under rainy conditions are selected. For each target single-frame image that meets the evaluation scene information, its corresponding first distance index is obtained. The first distance index is the matching effect of each single-frame image calculated in the previous step (such as chamfer distance, offset, Euclidean distance, etc.), which reflects the prediction performance of each target image. The first distance indicators of all target single-frame images that meet the evaluation scene information are classified according to the feature category. Specifically, the first distance indicators of each target single-frame image are grouped according to the feature category (such as continuous feature, discrete feature, intersection feature) in each target single-frame image. This step helps to evaluate the prediction performance of different categories of features separately.

[0152] Furthermore, a weighted average is calculated for the first distance index under each element category. The weighted average can give different weights to different elements according to their importance in the scene. For example, some elements (such as lane lines) may be more important in a specific scene, so they can be given higher weights. According to the weighted average, the second distance index under each element category is obtained. The second distance index measures the overall performance of the model under different element categories, reflecting the model's predictive ability for various elements in a specific scenario. Finally, by calculating the second distance index, the performance of the prediction model under different element categories (such as continuous elements, discrete elements, and intersection elements) can be comprehensively evaluated, and the adaptability and limitations of the model can be further diagnosed in combination with the different conditions of the evaluation scenario.

[0153] In one embodiment, the first evaluation index further includes: a first prediction index; the first level evaluation of the plurality of target data pairs is performed to obtain a first evaluation index for the target single frame image, including:

[0154] Obtaining a distance similarity value for each target data pair;

[0155] Compare the distance similarity value of each target data pair with a first distance similarity threshold, and determine a first number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold, and a second number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold.

[0156] The first prediction index is determined according to the first quantity and the second quantity, and the first prediction index at least includes: a first precision index, a first recall index, and a first F-value index.

[0157] In this embodiment, the first evaluation index includes not only the first distance index, but also the first prediction index. For the determination of the first prediction index, first, the distance similarity value of each target data pair is obtained, such as the chamfer distance, the offset or the Euclidean distance, depending on the element category to which the target data pair belongs, and the distance similarity value of each target data pair is compared with the preset first distance similarity threshold. If the distance similarity value is greater than the first distance similarity threshold, it indicates that the target data pair has a good match. If the distance similarity value is less than or equal to the first distance similarity threshold, it indicates that the target data pair has a poor match.

[0158] Furthermore, the number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold is counted, indicating that the prediction results match well with the data pairs, and the number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold is counted, indicating that the prediction results match poorly with the data pairs. Then, based on the first number and the second number counted above, a first prediction index is calculated. The prediction index reflects the accuracy of the model under the current evaluation conditions, and the first prediction index includes the following parts: a first precision index, a first recall index, and a first F-value index.

[0159] The first precision index is shown in the following formula (1):

[0160]

[0161] Wherein, Precision represents the first precision index; TP represents the first quantity; and FP represents the second quantity.

[0162] The first recall rate indicator is shown in the following formula (2):

[0163]

[0164] Among them, recall represents the first recall rate; Gts represents the number of true value data.

[0165] The first F value index is shown in the following formula (3):

[0166]

[0167] Among them, F-score represents the first F value indicator.

[0168] Finally, the first precision index, the first recall index, and the first F-value index calculated can provide a quantitative evaluation of the model's prediction performance. These indicators help to determine the overall effect of the model on a specific test set and provide guidance for subsequent improvements and optimizations.

[0169] In this application, the first distance index and the second prediction index are collectively referred to as the first evaluation index. By combining the first distance index and the first prediction index, the model evaluation becomes more comprehensive and detailed. The first distance index provides a spatial accuracy assessment of the prediction results, while the precision index, recall index, and F-value index evaluate the performance of the model from the perspective of classification and detection. This evaluation method can help identify and improve the deficiencies of the model and ensure the reliability and accuracy of the model in real applications.

[0170] In one embodiment, the second evaluation index further includes: a second prediction index, and the second evaluation index for the prediction model is determined according to the first evaluation index of each of the target single-frame images, including:

[0171] Obtaining evaluation scenario information for evaluating the model to be evaluated;

[0172] According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information;

[0173] Counting a first total number of target data pairs whose distance similarity values ​​of each target single-frame image that meets the evaluation scene information are greater than the first distance similarity threshold, and a second total number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold;

[0174] According to the first total number and the second total number, determine the second prediction indicator: the second prediction indicator includes at least: a second precision indicator, a second recall rate indicator, a second F value indicator and an AP indicator, and the AP indicator is calculated based on the precision-recall curve drawn based on the second precision indicator and the second recall rate indicator.

[0175] In this embodiment, the second evaluation index includes not only the second distance index but also the second prediction index, specifically:

[0176] First, obtain the evaluation scene information for evaluating the model to be evaluated. Then, according to the evaluation scene information, filter out the target single-frame images that meet the specific scene conditions. For example, if the scene information is specified as "rainy day", only images taken in a rainy environment are selected for subsequent evaluation. And for each target single-frame image that meets the evaluation scene information, count the first total number of target data pairs whose distance similarity value is greater than the first distance similarity threshold, and the number of target data pairs whose distance similarity value is less than or equal to the first distance similarity threshold.

[0177] Then, according to the first total number and the second total number of statistics, a second prediction index is calculated, and the second prediction index includes the following parts: a second precision index, a second recall index, a second F value index, and an AP (Average Precision) index. The calculation formulas of the second precision index, the second recall index, and the second F value index are the same as those of the above formulas (1), (2), and (3), and will not be repeated in this embodiment.

[0178] In this embodiment, the AP index is an index that measures the performance of the model by the approximate area of ​​the precision-recall curve. The specific calculation steps are as follows:

[0179] First, obtain the prediction confidence of each target data for its respective prediction data; then, sort the prediction confidence of each prediction data in descending order according to the target data; then, traverse the precision and recall of each single frame image, and draw the precision-recall curve. The AP indicator refers to the area under the precision-recall curve. Therefore, the interpolation method can be used to calculate the approximate value. For each recall value, find its corresponding maximum precision value, and then calculate the average of these maximum precision values ​​to get the AP indicator.

[0180] like Figure 2 As shown in the precision-recall curve, the horizontal axis is the recall rate and the vertical axis is the precision rate. Figure 2 The precision-recall curves for lane lines and road boundaries are shown in Figure 1, where straight lines represent lane lines and dashed lines represent road boundaries. Figure 2 As can be seen from the figure, the precision-recall curve usually shows a trend in which the precision decreases as the recall increases.

[0181] In some embodiments, in addition to the first evaluation index and the second evaluation index provided in the above embodiment, this embodiment also provides some other indicators as a basis for evaluating the prediction model, specifically including the following Table 1. In addition to showing some commonly used indicators, Table 1 also describes each indicator and annotates the indicator granularity and dimension:

[0182]

[0183]

[0184]

[0185] Table 1

[0186] For each indicator of the prediction box offset in Table 1, such as: dx indicator, dy indicator, dxy indicator, dxyp indicator, dw indicator and dh indicator, this application also provides relevant calculation formulas, as follows:

[0187]

[0188] Where N is the number of TPs; x i is the lateral distance error between the predicted data and the true value data; i is the vertical distance error between the predicted data and the true value data; gt i is the i-th true value instance; w i is the width error between the predicted data and the true value data; h i is the height error between the predicted data and the true data.

[0189] In this application, the final evaluation results will output the evaluation results of three road elements in various driving scenarios according to the element categories. The specific output indicators are shown in Table 2 below:

[0190]

[0191]

[0192] Table 2

[0193] In a preferred embodiment, the target element category is the continuous element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including:

[0194] Matching each first true value data of the continuous element with each first prediction data of the continuous element to obtain a plurality of first initial data pairs;

[0195] Calculating the chamfer distance of each of the first initial data pairs to obtain a first distance similarity matrix of the continuous elements;

[0196] Determine a bidirectional chamfer distance of each of the first initial data pairs according to the chamfer distance of each of the first initial data pairs in the first distance similarity matrix;

[0197] Comparing the bidirectional chamfer distance of each of the first initial data pairs with a preset chamfer distance threshold;

[0198] The first initial data pair whose bidirectional chamfer distance is less than or equal to the preset chamfer distance threshold is determined as the first target data pair.

[0199] In this embodiment, for the case where the target element category is a continuous element, the specific process of matching the true value data and the predicted data of the target element category is as follows:

[0200] Each first true value data of the continuous element is matched with each first predicted data one by one. The matching result generates multiple first initial data pairs, each data pair consists of a true value data and a predicted data. For each first initial data pair, its chamfer distance is calculated. The chamfer distance is used to measure the distance similarity between the true value data and the predicted data. With the true value data and the predicted data as rows and columns, a first distance similarity matrix is ​​constructed, and each element in the matrix represents a chamfer distance between a true value data and the predicted data.

[0201] Furthermore, based on the first distance similarity matrix, the bidirectional chamfer distance is calculated for each first initial data pair, wherein the bidirectional chamfer distance refers to the chamfer distance from the true value data to the predicted data and the chamfer distance from the predicted data to the true value data, that is, the bidirectional chamfer distance is a comprehensive measure of the above two unidirectional distances, which is used to more comprehensively reflect the similarity between the true value data and the predicted data.

[0202] Then, the bidirectional chamfer distance of each first initial data pair is compared with a preset chamfer distance threshold value. For the first initial data pair whose bidirectional chamfer distance is less than or equal to the preset threshold value, it is marked as a first target data pair.

[0203] For example: True value data M i = {P1,P2,P3,…,P m}, to predict data N j = {P1,P2,P3,…,P n}Chamfer distance d xy , the specific calculation method is shown in the following formula (10):

[0204]

[0205] Among them, M is the true value data; N is the predicted data; m is the number of positive data; n is the number of predicted data.

[0206] In one embodiment, the target element category is the discrete element, and the matching of the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category includes:

[0207] Matching each second true value data of the discrete element with each second prediction data of the discrete element to obtain a plurality of second initial data pairs;

[0208] Calculating the offset of each of the second initial data pairs to obtain a second distance similarity matrix of the discrete elements;

[0209] Obtaining a confidence level for each of the second prediction data;

[0210] Sorting all the second prediction data in descending order according to their respective confidence levels;

[0211] From the second distance similarity matrix, a second target true value data is matched in order for each of the sorted second prediction data to obtain multiple second target data pairs of the discrete elements, wherein the offset between the second target true value data and the second prediction data is less than or equal to a preset offset threshold.

[0212] In this embodiment, for the case where the target element category is a discrete element, the specific process of matching the true value data and the predicted data of the target element category is as follows:

[0213] Each second true value data in the discrete elements is matched with each second predicted data one by one. The matching results generate multiple second initial data pairs, each data pair consists of a true value data and a predicted data. For each second initial data pair, its offset (i.e., the center point offset between the true value data and the predicted data) is calculated. According to the offsets of all data pairs, a second distance similarity matrix is ​​constructed, and each element in the matrix represents the offset between a true value data and a predicted data.

[0214] Furthermore, the confidence value of each second prediction data is extracted from the prediction model to reflect the degree of confidence of the model in the prediction result. All second prediction data are sorted in descending order according to the confidence value of the prediction data, so that predictions with high confidence are matched with true value data first. Each sorted second prediction data is traversed, and a second target true value data is matched in order from the second distance similarity matrix: the offset between the prediction data and the true value data is checked. If the offset is less than or equal to the preset offset threshold, the two are determined to match and form a second target data pair. For the matching results of all prediction data and true value data that meet the conditions, multiple second target data pairs of discrete elements are generated.

[0215] In one embodiment, the target element category is the intersection element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including:

[0216] Matching each third true value data of the intersection element with each third predicted data of the intersection element to obtain a plurality of third initial data pairs;

[0217] Calculating the Euclidean distance of each of the third initial data pairs to obtain a third distance similarity matrix of the intersection elements;

[0218] Comparing the Euclidean distance of each of the third initial data pairs in the third distance similarity matrix with a preset Euclidean distance threshold;

[0219] The third initial data pair whose Euclidean distance is greater than the preset Euclidean distance threshold is determined as a third target data pair.

[0220] In this embodiment, for the case where the target element category is an intersection element, the specific process of matching the true value data and the predicted data of the target element category is as follows:

[0221] Each third true value data of the intersection element is matched with each third predicted data. The matching results generate multiple third initial data pairs, each of which consists of a true value data and a predicted data. For each third initial data pair, the Euclidean distance is calculated to measure the spatial position deviation between the true value data and the predicted data. Based on the calculation results, the third distance similarity matrix is ​​constructed, and each element in the matrix represents the Euclidean distance between a true value data and a predicted data.

[0222] The Euclidean distance calculation formula is shown in the following formula (11):

[0223]

[0224] Each Euclidean distance value in the third distance similarity matrix is ​​compared with a preset Euclidean distance threshold: if the Euclidean distance is less than or equal to the preset Euclidean distance threshold, it indicates that the match is successful and the distance between the predicted data and the true value data is within an acceptable range. For all third initial data pairs whose Euclidean distance is greater than the preset Euclidean distance threshold, they are determined as third target data pairs.

[0225] In one embodiment, the element category is the continuous element, and before classifying the true value data and the predicted data of the target single frame image, the method further includes:

[0226] Acquiring the resolution of the target single-frame image;

[0227] According to the resolution of the target single-frame image, interpolation processing is performed on the continuous elements in the target single-frame image at preset intervals;

[0228] The target single-frame image after interpolation processing is visualized.

[0229] In this embodiment, before classifying the true value data and the predicted data of the target single-frame image, the image needs to be interpolated and visualized. Specifically:

[0230] Get the resolution information (such as pixel width and height) of the target single-frame image. The resolution is used to determine the accuracy of the continuous feature interpolation processing. Then, according to the resolution of the target single-frame image, the continuous features are interpolated at a preset interval, where the preset interval can be set to a fixed distance (such as 0.2 meters or 0.8 meters) according to the image resolution, thereby ensuring that the distribution of the continuous feature point set is consistent with the image resolution and improving the accuracy of the subsequent chamfer distance calculation.

[0231] Furthermore, the target single-frame image and its continuous elements after interpolation processing are visualized and rendered to facilitate manual verification of the interpolation quality.

[0232] For example, Figure 3 It is a schematic diagram of the continuous element interpolation, i.e., visualization processing provided by the present application, wherein the interpolation intervals of the left and right single-frame images are different, the interpolation interval of the left image is 0.2 m, and the interpolation interval of the right image is 0.8 m.

[0233] In one embodiment, the present application also provides a Figure 4 The schematic diagram of the visualization result of discrete feature target matching on a single frame image is shown. Figure 4 In the figure, the matching results of the true value box and the predicted box of the road surface element as the road surface arrow are visualized.

[0234] In one embodiment, a multi-process mechanism is also introduced. For example, in the data preprocessing stage, by adopting the multi-process mechanism, the true value data and prediction data of multiple target single-frame images can be read and parsed at one time, thereby improving the efficiency of data preprocessing. In the data pair matching stage, multiple data pairs of continuous elements, discrete elements, and intersection elements can be matched at one time, thereby improving the data matching efficiency. In addition, in the index calculation stage, multiple types of indexes of multiple road surface elements can be calculated and output at one time, thereby improving the efficiency of predictive model evaluation.

[0235] In one embodiment, the present application provides Figure 5 A logical diagram of the evaluation system index calculation is shown in FIG. Figure 5 As shown:

[0236] First, the predicted data and true value data of the acquired target single-frame image need to be classified into elements, which can be divided into continuous elements, discrete elements and intersection elements. For different elements, the distance matrix between the true value data and the predicted data under each element is calculated respectively. The continuous element corresponds to the chamfer distance matrix, the discrete element corresponds to the offset distance matrix, and the intersection element corresponds to the Euclidean distance matrix. For each element, the data pairs are matched based on their respective distance matrices to obtain continuous element data pairs, discrete element data pairs and intersection element data pairs. For continuous elements, the first evaluation index of the continuous elements is calculated based on the continuous element data pairs corresponding to the continuous elements and the preset chamfer distance threshold; for discrete elements, the first evaluation index of the discrete elements is calculated based on the discrete element data pairs corresponding to the discrete elements and the preset offset threshold; for intersection elements, the first evaluation index of the intersection elements is calculated based on the intersection element data pairs corresponding to the intersection elements and the preset Euclidean distance threshold, wherein the first evaluation index of the continuous elements includes: the first distance index of the continuous elements and the first prediction index of the continuous elements; the first evaluation index of the discrete elements includes: the first distance index of the discrete elements and the first prediction index of the discrete elements; the first evaluation index of the intersection elements includes: the first distance index of the intersection elements and the first prediction index of the intersection elements. The first prediction index specifically includes: the first precision index and the first recall index.

[0237] Next, the first evaluation index is aggregated and a second evaluation index is output. When the second evaluation index is determined by the first evaluation index, the screening of the evaluation scene is introduced, so that images of specific scenes that need to be evaluated can be screened out more purposefully.

[0238] Specifically, the first distance index of the continuous elements is averaged to obtain the second distance index of the continuous elements; the first distance index of the discrete elements is averaged to obtain the second distance index of the discrete elements; the first distance index of the intersection elements is averaged to obtain the second distance index of the intersection elements.

[0239] The first prediction indicators of continuous elements in all target single-frame images are integrated to obtain the second prediction indicators of continuous elements; the first prediction indicators of discrete elements in all target single-frame images are integrated to obtain the second prediction indicators of discrete elements; the first prediction indicators of intersection elements in all target single-frame images are integrated to obtain the second prediction indicators of intersection elements. According to the second precision indicator and the second recall indicator in the second prediction indicator, the precision-recall curve is drawn to further obtain the AP indicator.

[0240] In one embodiment, the present application provides Figure 6 A schematic diagram of an evaluation system architecture is shown in FIG. Figure 6 As shown:

[0241] At the data input end, a single-frame image is input, which includes true value data and predicted data. Through the data input port, it first enters the data processing unit. After the data preprocessing by the data processing unit, it is carried out in the evaluation module. First, in the evaluation module, the preprocessed data is classified into elements, and then the classified data is loaded and the metric index is calculated. The evaluation module calculates the single-frame index to obtain the first evaluation index, and then, after scene screening and multi-frame index calculation, the second evaluation index is obtained. Then, the first evaluation index and the second evaluation index are transmitted to the index output module, and reach the visualization unit and the format conversion unit through the output interface. The visualization unit processes and outputs a variety of visualization graphs, and the evaluation index is output after format conversion.

[0242] Based on the same inventive concept, the second aspect of the present application provides an evaluation system for a road surface element prediction model, such as Figure 7 As shown, the system comprises:

[0243] An acquisition module 201 is used to acquire true value data of each of a plurality of single-frame images, and to acquire prediction data of each of the plurality of single-frame images output by the model to be evaluated;

[0244] A classification module 202, for classifying the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images;

[0245] A matching module 203 is used to match the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements and intersection elements of the road surface;

[0246] A first evaluation module 204 is used to perform a first level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, wherein the first evaluation index at least includes a first distance index;

[0247] A second evaluation module 205 is used to determine a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicator at least includes a second distance indicator;

[0248] The third evaluation module 206 is used to evaluate the prediction model according to the second evaluation indicator.

[0249] Optionally, the first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, and the first evaluation module 204 includes:

[0250] A judgment submodule, used for judging the element categories of the plurality of target data pairs;

[0251] A first calculation submodule is used to calculate the distance similarity value of each target data pair according to the respective element categories of the plurality of target data pairs, wherein, when the target element is a continuous element, the distance similarity value is the chamfer distance, when the target element is a discrete element, the distance similarity value is the offset, and when the target element is an intersection element, the distance similarity value is the Euclidean distance;

[0252] The second calculation submodule is used to calculate the first distance index of the target single-frame image according to the respective distance similarity values ​​of the plurality of target data pairs.

[0253] Optionally, according to the first evaluation index of each of the target single-frame images, a second evaluation index for the prediction model is determined, and the second evaluation module 205 includes:

[0254] A first acquisition submodule is used to acquire evaluation scenario information for evaluating the model to be evaluated;

[0255] A first determination submodule is used to perform scene screening on the plurality of target single-frame images according to the evaluation scene information, and determine the target single-frame image that meets the evaluation scene information;

[0256] A second acquisition submodule is used to acquire a first distance index of each of the target single-frame images that conform to the evaluation scene information;

[0257] A first classification submodule, used for classifying the first distance index of each of the plurality of target single-frame images that meet the evaluation scene information according to element categories;

[0258] The second determination submodule is used to calculate the weighted average of the first distance index under each of the element categories respectively, to obtain the second distance index of the prediction model under different element categories.

[0259] Optionally, the first evaluation index further includes: a first prediction index; the first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, and the first evaluation module 204 includes:

[0260] A third acquisition submodule is used to obtain the distance similarity value of each target data pair;

[0261] a first comparison submodule, configured to compare the distance similarity value of each target data pair with a first distance similarity threshold, and determine a first number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold, and a second number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold.

[0262] The third determination submodule is used to determine the first prediction index according to the first quantity and the second quantity, and the first prediction index at least includes: a first precision index, a first recall index and a first F value index.

[0263] Optionally, the second evaluation index further includes: a second prediction index, wherein the second evaluation index for the prediction model is determined according to the first evaluation index of each of the target single-frame images, and the second evaluation module 205 includes:

[0264] The fourth acquisition submodule is used to obtain evaluation scenario information for evaluating the model to be evaluated;

[0265] A first screening submodule, configured to screen the plurality of target single-frame images according to the evaluation scene information, and determine the target single-frame image that meets the evaluation scene information;

[0266] a statistical submodule, configured to count a first total number of target data pairs whose respective distance similarity values ​​of each target single-frame image that meets the evaluation scene information are greater than the first distance similarity threshold, and a second total number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold;

[0267] The fourth determination submodule is used to determine the second prediction indicator based on the first total number and the second total number: the second prediction indicator includes at least: a second precision indicator, a second recall rate indicator, a second F value indicator and an AP indicator, and the AP indicator is calculated based on the precision-recall curve drawn based on the second precision indicator and the second recall rate indicator.

[0268] Optionally, the target element category is the continuous element, the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module 203 includes:

[0269] A first matching submodule, used for matching each first true value data of the continuous element with each first predicted data of the continuous element to obtain a plurality of first initial data pairs;

[0270] A third calculation submodule, used for calculating the chamfer distance of each of the first initial data pairs to obtain a first distance similarity matrix of the continuous elements;

[0271] a fifth determination submodule, configured to determine a bidirectional chamfer distance of each of the first initial data pairs according to the chamfer distance of each of the first initial data pairs in the first distance similarity matrix;

[0272] A second comparison submodule, used for comparing the bidirectional chamfer distance of each of the first initial data pairs with a preset chamfer distance threshold;

[0273] The sixth determination submodule is used to determine the first initial data pair whose bidirectional chamfer distance is less than or equal to the preset chamfer distance threshold as the first target data pair.

[0274] Optionally, the target element category is the discrete element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module 203 includes:

[0275] A second matching submodule, used for matching each second true value data of the discrete element with each second predicted data of the discrete element, respectively, to obtain a plurality of second initial data pairs;

[0276] a fourth calculation submodule, configured to calculate an offset of each of the second initial data pairs to obtain a second distance similarity matrix of the discrete elements;

[0277] A fifth acquisition submodule, used to acquire the confidence level of each of the second prediction data;

[0278] A sorting submodule, used to sort all the second prediction data in descending order according to their respective confidence levels;

[0279] The seventh determination submodule is used to match a second target true value data for each sorted second prediction data in order from the second distance similarity matrix, to obtain multiple second target data pairs of the discrete elements, wherein the offset between the second target true value data and the second prediction data is less than or equal to a preset offset threshold.

[0280] Optionally, the target element category is the intersection element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, and the matching module 203 includes:

[0281] A third matching submodule, used for matching each third true value data of the intersection element with each third predicted data of the intersection element, respectively, to obtain a plurality of third initial data pairs;

[0282] A fifth calculation submodule, used for calculating the Euclidean distance of each of the third initial data pairs to obtain a third distance similarity matrix of the intersection elements;

[0283] A third comparison submodule, configured to compare the Euclidean distance of each of the third initial data pairs in the third distance similarity matrix with a preset Euclidean distance threshold;

[0284] The eighth determination submodule is used to determine the third initial data pair whose Euclidean distance is greater than the preset Euclidean distance threshold as a third target data pair.

[0285] Optionally, the system further comprises:

[0286] A sixth acquisition submodule, used to acquire the resolution of the target single-frame image;

[0287] An interpolation processing submodule, used for performing interpolation processing on continuous elements in the target single-frame image at preset intervals according to the resolution of the target single-frame image;

[0288] The visualization processing submodule is used to perform visualization processing on the target single-frame image after the interpolation processing.

[0289] Based on the same inventive concept, the third aspect of the present application provides a Figure 8 The electronic device 100 shown includes a memory 110, a processor 120 and a computer program stored in the memory 110, and the processor 120 executes the computer program to implement the evaluation method of the road surface element prediction model as described in the first aspect of the present application.

[0290] Based on the same inventive concept, the fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the evaluation method of the road surface element prediction model as described in the first aspect of the present application.

[0291] Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.

[0292] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0293] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0294] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0295] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0296] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0297] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0298] The above provides a detailed introduction to the evaluation method, system and electronic equipment for a road surface element prediction model. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for evaluating a road surface element prediction model, characterized in that: The method comprises: Obtaining true value data of each of the plurality of single-frame images, and obtaining prediction data of each of the plurality of single-frame images output by the model to be evaluated; Classifying the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images; Matching the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements, and intersection elements of a road surface; Performing a first-level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single-frame image, wherein the first evaluation index at least includes a first distance index; Determining a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicator at least includes a second distance indicator; The prediction model is evaluated according to the second evaluation indicator.

2. The evaluation method of the road surface element prediction model according to claim 1, characterized in that: The first level evaluation is performed on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, including: Determine the element category of each of the plurality of target data pairs; According to the respective element categories of the plurality of target data pairs, respectively calculate the distance similarity value of each target data pair, wherein, when the target element is a continuous element, the distance similarity value is the chamfer distance, when the target element is a discrete element, the distance similarity value is the offset, and when the target element is an intersection element, the distance similarity value is the Euclidean distance; The first distance index of the target single-frame image is calculated based on the respective distance similarity values ​​of the plurality of target data pairs.

3. The evaluation method of the road surface element prediction model according to claim 2, characterized in that: Determining a second evaluation index for the prediction model according to the first evaluation index of each of the target single-frame images includes: Obtaining evaluation scenario information for evaluating the model to be evaluated; According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information; Obtaining a first distance indicator for each of the target single-frame images that conform to the evaluation scene information; Classifying the first distance indicators of the target single-frame images that meet the evaluation scene information according to the element category; The weighted average of the first distance index under each of the element categories is calculated respectively to obtain the second distance index of the prediction model under different element categories.

4. The evaluation method of the road surface element prediction model according to claim 2, characterized in that: The first evaluation index also includes: a first prediction index; the first level evaluation of the plurality of target data pairs is performed to obtain a first evaluation index for the target single frame image, including: Obtaining a distance similarity value of each target data pair; Compare the distance similarity value of each target data pair with a first distance similarity threshold, and determine a first number of target data pairs whose distance similarity values ​​are greater than the first distance similarity threshold, and a second number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold. The first prediction index is determined according to the first quantity and the second quantity, and the first prediction index at least includes: a first precision index, a first recall index, and a first F-value index.

5. The evaluation method of the road surface element prediction model according to claim 4, characterized in that: The second evaluation index further includes: a second prediction index, wherein the second evaluation index for the prediction model is determined according to the first evaluation index of each of the target single-frame images, including: Obtaining evaluation scenario information for evaluating the model to be evaluated; According to the evaluation scene information, scene screening is performed on the plurality of target single-frame images to determine the target single-frame image that meets the evaluation scene information; Counting a first total number of target data pairs whose distance similarity values ​​of each target single-frame image that meets the evaluation scene information are greater than the first distance similarity threshold, and a second total number of target data pairs whose distance similarity values ​​are less than or equal to the first distance similarity threshold; According to the first total number and the second total number, the second prediction indicator is determined: the second prediction indicator includes at least: a second precision indicator, a second recall rate indicator, a second F value indicator and an AP indicator, and the AP indicator is calculated based on the precision-recall curve drawn based on the second precision indicator and the second recall rate indicator.

6. The evaluation method of the road surface element prediction model according to claim 1, characterized in that: The target element category is the continuous element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including: Matching each first true value data of the continuous element with each first prediction data of the continuous element to obtain a plurality of first initial data pairs; Calculating the chamfer distance of each of the first initial data pairs to obtain a first distance similarity matrix of the continuous elements; Determine a bidirectional chamfer distance of each of the first initial data pairs according to the chamfer distance of each of the first initial data pairs in the first distance similarity matrix; Comparing the bidirectional chamfer distance of each of the first initial data pairs with a preset chamfer distance threshold; The first initial data pair whose bidirectional chamfer distance is less than or equal to the preset chamfer distance threshold is determined as the first target data pair.

7. The evaluation method of the road surface element prediction model according to claim 1, characterized in that: The target element category is the discrete element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including: Matching each second true value data of the discrete element with each second prediction data of the discrete element to obtain a plurality of second initial data pairs; Calculating the offset of each of the second initial data pairs to obtain a second distance similarity matrix of the discrete elements; Obtaining a confidence level for each of the second prediction data; Sorting all the second prediction data in descending order according to their respective confidence levels; From the second distance similarity matrix, a second target true value data is matched in order for each of the sorted second prediction data to obtain multiple second target data pairs of the discrete elements, wherein the offset between the second target true value data and the second prediction data is less than or equal to a preset offset threshold.

8. The evaluation method of the road surface element prediction model according to claim 1, characterized in that: The target element category is the intersection element, and the true value data and the predicted data of the target element category are matched to obtain a plurality of target data pairs under the target element category, including: Matching each third true value data of the intersection element with each third predicted data of the intersection element to obtain a plurality of third initial data pairs; Calculating the Euclidean distance of each of the third initial data pairs to obtain a third distance similarity matrix of the intersection elements; Comparing the Euclidean distance of each of the third initial data pairs in the third distance similarity matrix with a preset Euclidean distance threshold; The third initial data pair whose Euclidean distance is greater than the preset Euclidean distance threshold is determined as a third target data pair.

9. The evaluation method of the road surface element prediction model according to claim 1, characterized in that: The element category is the continuous element. Before classifying the true value data and the predicted data of the target single-frame image, the method further includes: Acquire the resolution of the target single-frame image; According to the resolution of the target single-frame image, interpolation processing is performed on the continuous elements in the target single-frame image at preset intervals; The target single-frame image after interpolation processing is visualized.

10. A road surface element prediction model evaluation system, characterized in that: The system comprises: An acquisition module, used to acquire true value data of each of a plurality of single-frame images, and to acquire prediction data of each of the plurality of single-frame images output by the model to be evaluated; A classification module, used for classifying the true value data and the predicted data of the target single-frame image according to the element category, wherein each element category includes a plurality of true value data and a plurality of predicted data, and the target single-frame image is any one of the plurality of single-frame images; A matching module, used for matching the true value data and the predicted data of the target element category to obtain a plurality of target data pairs under the target element category, wherein the target element category is any one of the plurality of element categories, and the element categories include: continuous elements, discrete elements and intersection elements of the road surface; A first evaluation module, configured to perform a first level evaluation on the plurality of target data pairs to obtain a first evaluation index for the target single frame image, wherein the first evaluation index at least includes a first distance index; A second evaluation module, used to determine a plurality of second evaluation indicators for the prediction model according to the first evaluation indicator of each of the target single-frame images, wherein the second evaluation indicator at least includes a second distance indicator; The third evaluation module is used to evaluate the prediction model according to the second evaluation indicator.

11. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the evaluation method of the road surface element prediction model according to any one of claims 1-9.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the evaluation method of the road surface element prediction model according to any one of claims 1 to 9 is implemented.

Images