Image dataset processing method, apparatus, device, and storage medium

Abstract

The present application provides an image dataset processing method, apparatus, device, and storage medium, the method including: acquiring an initial autonomous driving image dataset; inputting the initial autonomous driving image dataset into a first autonomous driving detection model and a second autonomous driving detection model, respectively, to obtain a first image detection result and a second image detection result; comparing the first image detection result and the second image detection result to obtain a comparison result; determining a comparison type based on the comparison result; determining an evaluation score for a plurality of initial autonomous driving images based on the comparison type; selecting a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images; and adjusting tags in the target autonomous driving image dataset. The present application can improve the accuracy of image dataset processing.

Description

[Technical Field]

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on 26 July 2023, application number 202310922726.X, and all contents of the above application are incorporated herein by reference.

[0002] The embodiments of this application relate to the technical field of image processing, and more particularly to image dataset processing methods, apparatus, devices, and storage media. [Background technology]

[0003] As is well known, the development of autonomous driving is inextricably linked to advancements in artificial intelligence technology. Data, computing power, and algorithms, as the three elements of artificial intelligence, significantly influence the maturity of autonomous driving technology, and of these three elements, data plays a crucial role. Artificial intelligence can formulate rules after being trained on a large amount of data. In actual applications, when a scenario not included in the training set appears, artificial intelligence is almost entirely guessworking, and predictions may be wrong. Therefore, the role of reliable and high-quality data in the autonomous driving development process is particularly important.

[0004] High-level autonomous driving test vehicles collect terabytes (TB) of data daily and require petabytes (PB) of storage space. However, less than 5% of this data is valuable enough to be used for training autonomous driving systems. Finding the most valuable and challenging samples from this vast amount of data, prioritizing their tagging, and accelerating the iteration of autonomous driving models and the mass production of algorithms are urgent issues that need to be addressed. [Overview of the project] [Problems that the invention aims to solve]

[0005] Embodiments of the present application provide an image dataset processing method, apparatus, device, and storage medium that can improve the accuracy of image dataset processing. [Means for solving the problem]

[0006] In Embodiment 1, the embodiment of the present application is: Obtain an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags, The initial automated driving image dataset is input to the first automated driving detection model and the second automated driving detection model, respectively, and the first image detection result and the second image detection result are obtained, The first image detection result and the second image detection result are compared, and the comparison result is obtained, wherein the first image detection result is used as the reference detection result. Based on the comparison results, the comparison type is determined, and the comparison type includes true positive examples, false positive examples, and false negative examples. Based on the comparison type, the evaluation scores of the multiple initial autonomous driving images are determined, This includes selecting a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images, and adjusting the tags in the target autonomous driving image dataset. This document provides a method for processing image datasets.

[0007] In Embodiment 2, the embodiment of the present application is: An initial autonomous driving image dataset acquisition module for obtaining an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags, The initial automated driving image dataset is input to the first automated driving detection model and the second automated driving detection model, respectively, and an image detection result acquisition module is used to acquire the first image detection result and the second image detection result, respectively. An image detection result comparison module for comparing the first image detection result and the second image detection result and obtaining the comparison result, comprising an image detection result comparison module that uses the first image detection result as the reference detection result, A comparison type determination module for determining the comparison type based on the comparison results, wherein the comparison type includes a true positive example, a false positive example, and a false negative example, An evaluation score determination module for determining the evaluation scores of the plurality of initial autonomous driving images based on the comparison type, The system includes a target autonomous driving image dataset selection module for selecting a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images, and for adjusting the tags in the target autonomous driving image dataset. Further image dataset processing equipment is provided.

[0008] In Embodiment 3, the embodiment of the present application is: One or more processors, A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image dataset processing method according to the embodiment of the present application. We will provide even more electronic devices.

[0009] In Embodiment 4, the embodiment of the present application is: When executed by a computer processor, it includes computer executable instructions for executing the image dataset processing method according to an embodiment of the present application, We will provide more storage media. [Effects of the Invention]

[0010] In the technical solution disclosed in this embodiment, an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags is acquired, the initial autonomous driving image dataset is input to a first autonomous driving detection model and a second autonomous driving detection model, respectively, a first image detection result and a second image detection result are acquired, the first image detection result is used as a reference detection result to compare the first image detection result and the second image detection result to acquire a comparison result, the comparison type is determined based on the comparison result, the comparison type includes true positive examples, false positive examples and false negative examples, the evaluation score of multiple initial autonomous driving images is determined based on the comparison type, a target autonomous driving image dataset is selected from the initial autonomous driving image dataset based on the evaluation score of multiple initial autonomous driving images, and the tags in the target autonomous driving image dataset are adjusted. The embodiment of this application can improve the accuracy of image dataset processing by determining the comparison type based on the comparison result between the first image detection result and the second image detection result, determining the evaluation score of the initial autonomous driving images based on the comparison type, and selecting a target autonomous driving image dataset based on the evaluation score of the initial autonomous driving images. [Brief explanation of the drawing]

[0011] The above and other features, advantages and forms of each embodiment of this application will become clearer when viewed in conjunction with the drawings below. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily manufactured to scale.

[0012] [Figure 1] This is a flowchart of the image dataset processing method according to an embodiment of the present invention. [Figure 2] This is a flowchart of another image dataset processing method according to an embodiment of the present invention. [Figure 3] This is a schematic diagram illustrating the effects of IoU and IoU evaluation scores according to the embodiment of the present application. [Figure 4] This is a schematic diagram illustrating the effect between the reliability and the second evaluation score in the embodiment of the present application. [Figure 5] It is an effect schematic diagram between the reliability and the first evaluation score according to an embodiment of the present application. [Figure 6] It is an effect schematic diagram between the area and the second evaluation score according to an embodiment of the present application. [Figure 7] It is an effect schematic diagram between the area and the first evaluation score according to an embodiment of the present application. [Figure 8] It is an effect schematic diagram of the first image detection result according to an embodiment of the present application. [Figure 9] It is an effect schematic diagram of the second image detection result according to an embodiment of the present application. [Figure 10] It is a structural schematic diagram of an image dataset processing device according to an embodiment of the present application. [Figure 11] It is a structural schematic diagram of an electronic device according to an embodiment of the present application.

Embodiments for Implementing the Invention

[0013] Hereinafter, embodiments of the present application will be described in more detail with reference to the drawings. Although several embodiments of the present application are shown in the drawings, the present application can be realized in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, it should be understood that these embodiments are provided to make the present application more clearly and completely understood. It should be understood that the drawings and embodiments of the present application are merely illustrative and not intended to limit the protection scope of the present application.

[0014] It should be understood that each step described in the embodiments of the method of the present application may be executed in a different order and / or may be executed in parallel. Also, the embodiments of the method may include additional steps and / or steps whose execution is omitted. The scope of the present application is not limited in this regard.

[0015] As used in this paper, the term "includes" and its variations are open inclusions, meaning "includes, but not limited to." The term "based on" means "based at least partially." The term "one embodiment" means "at least one embodiment." The term "another embodiment" means "at least one other embodiment." The term "several embodiments" means "at least several embodiments." Relevant definitions of other terms are given below.

[0016] Furthermore, the concepts of "first," "second," etc., as used in this application are merely for distinguishing between devices, modules, or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules, or units.

[0017] Furthermore, the modifiers “one” and “multiple” as used in this application are schematic but not restrictive, and a person skilled in the art should understand that they should be understood as “one or multiple” unless the context clearly indicates an exception.

[0018] It is understood that the data relating to this proposed technology (including, but not limited to, the data itself, or the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations, and relevant provisions.

[0019] Figure 1 is a flowchart of an image dataset processing method according to an embodiment of the present invention. This embodiment can be applied when selecting a target autonomous driving image dataset from an initial autonomous driving image dataset. This embodiment is not limited to the target detection task of an autonomous driving front camera scene, but may also be used for other scene tasks. When switching scene tasks, the relevant configuration information is modified according to the model capabilities and project needs. In this embodiment, selecting a target autonomous driving image dataset from an initial autonomous driving image dataset may be understood as data mining, where the most valuable and difficult samples are found from a vast amount of data, tagged preferentially, and model iteration is accelerated. This method can be performed by an image dataset processing device and specifically includes the following steps.

[0020] In S110, the initial automated driving image dataset is acquired.

[0021] Here, the initial autonomous driving image dataset includes multiple initial autonomous driving images and corresponding tags. Here, the tags include information such as detection frame information, category, confidence level, and area corresponding to the initial autonomous driving image. When this embodiment is used for a 3D scene task, the area may be the volume or projected area of ​​the 3D target.

[0022] Here, initial autonomous driving images can be collected by a high-level autonomous driving test vehicle. The autonomous driving image dataset may or may not include tags, and this is not limited to the dataset in this embodiment.

[0023] In S120, the initial automated driving image dataset is input to the first automated driving detection model and the second automated driving detection model, respectively, and the first image detection result and the second image detection result are obtained.

[0024] Here, the first autonomous driving detection model may be a server-side model, which has sufficient computing power and can perform offline processing, and can be called a large-scale model. The second autonomous driving detection model may be an edge-side model, which has limited computing power and a high real-time requirement, and can be called a small-scale model.

[0025] Preferably, here, each initial autonomous driving image includes at least one target object, the first image detection result includes first detection frame information for at least one target object, a first category corresponding to each of the at least one target object, a first confidence level for the first category, and a first area corresponding to the first detection frame information, and the second image detection result includes second detection frame information for at least one target object, a second category corresponding to each of the at least one target object, a second confidence level corresponding to the second category, and a second area corresponding to the second detection frame information.

[0026] Here, the target object may be a vehicle, railing, pedestrian, house, etc., and this embodiment is not limited to these. The detection frame information may include the length, width, and height of the detection frame.

[0027] In S130, the first image detection result is used as the reference detection result, and the first image detection result and the second image detection result are compared to obtain the comparison result.

[0028] In this embodiment, the first image detection result and the second image detection result are compared in correspondence, and the corresponding comparison result is obtained, which includes comparison matches and comparison mismatches.

[0029] Preferably, the comparison results include category comparison results, and comparing the first image detection result and the second image detection result to obtain the comparison results includes comparing the first category and the second category to obtain category comparison results including category comparison matches and category comparison mismatches.

[0030] In this embodiment, a category comparison result can be obtained by comparing the first category and the second category, and the comparison type is determined based on the category comparison result.

[0031] In S140, the comparison type is determined based on the comparison results.

[0032] Here, comparison types include true positive examples (TP), false positive examples (FP), and false negative examples (FN). Exemplary examples, if there is a categorical comparison match, the comparison type is a true positive example; if there is a categorical comparison mismatch, the comparison type is either a false positive example or a false negative example.

[0033] Preferably, determining the comparison type based on the comparison results includes determining the IoU based on first detection frame information and second detection frame information, determining a first IoU setting value based on the first area and / or second area, determining the comparison type as a true positive example if the IoU is greater than or equal to the first IoU setting value and the category comparison result is a category comparison match, determining the comparison type as a false positive example and a false negative example if the IoU is less than the first IoU setting value and / or the category comparison result is a category comparison mismatch, or determining the comparison type as a false negative example if the first image detection result includes first detection frame information of the target object and the second image detection result does not include second detection frame information of the target object, and determining the comparison type as a false positive example if the first image detection result does not include first detection frame information of the target object and the second image detection result includes second detection frame information of the target object.

[0034] In this embodiment, the IoU can be determined based on the first detection frame information and the second detection frame information, the size of the target object can be determined based on the first area and / or the second area, if the first area and / or the second area is smaller than 32*32, the target object is a small target and the corresponding first IoU setting value may be 0.3, and if the first area and / or the second area is 32*32 or larger, the target object is a normal target and the corresponding first IoU setting value may be 0.5.

[0035] If IoU is greater than or equal to the first IoU setting and the category comparison result is a category comparison match, the comparison type is determined to be a true positive example. If IoU is less than the first IoU setting and / or the category comparison result is a category comparison mismatch, the comparison type is determined to be a false positive example and a false negative example. Alternatively, if the first image detection result includes the first detection frame information of the target object, and the second image detection result does not include the second detection frame information of the target object, the comparison type is determined to be a false negative example; that is, for detection frames that are greater in the first image detection result than in the second image detection result, the corresponding comparison type is a false negative example. If the first image detection result does not include the first detection frame information of the target object, and the second image detection result includes the second detection frame information of the target object, the comparison type is determined to be a false positive example; that is, for detection frames that are greater in the second image detection result than in the first image detection result, the corresponding comparison type is a false positive example.

[0036] This embodiment can accurately determine the comparison type by either determining the comparison type based on the comparison results and IoU, or by determining whether the first image detection result and the second image detection result contain detection frame information for the same target object.

[0037] In S150, evaluation scores for multiple initial autonomous driving images are determined based on the comparison type.

[0038] In this embodiment, the method for determining the evaluation score of each initial autonomous driving image is as follows: Based on the comparison type, the evaluation score of all target objects in the initial autonomous driving image is determined. The evaluation score of a target object includes the IoU score, confidence score, area score, and category score. Based on the evaluation scores of all target objects and the scene score of the initial autonomous driving image, the evaluation score of the initial autonomous driving image is obtained, thereby obtaining the evaluation score of all initial autonomous driving images.

[0039] In S160, a target autonomous driving image dataset is selected from the initial autonomous driving image dataset based on the evaluation scores of multiple initial autonomous driving images, and the tags in the target autonomous driving image dataset are adjusted.

[0040] In this embodiment, a target autonomous driving image dataset can be selected from the initial autonomous driving image dataset based on the evaluation scores of all initial autonomous driving images. Here, the target autonomous driving image dataset may be understood as a dataset that significantly affects the detection accuracy of the second autonomous driving detection model (which may be any other autonomous driving detection model), or as a dataset with a high target value. By adjusting the tags in the target autonomous driving image dataset, the detection accuracy of the autonomous driving detection model can be improved.

[0041] In this embodiment, the comparison is not limited to two models, but may also be between two models at a time. Multiple value scores can be obtained from the evaluation score of a single initial autonomous driving image, and the highest score is taken, or the final value score is obtained by using another score fusion algorithm.

[0042] In this embodiment, only one model may be used, but it is necessary to perform forward calculations for two different modes, such as TTA (Test Time Augmentation) mode and normal mode, and obtain two different detection results for two groups.

[0043] In the proposed technology disclosed in this embodiment, an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags is acquired, the initial autonomous driving image dataset is input to a first autonomous driving detection model and a second autonomous driving detection model, respectively, a first image detection result and a second image detection result are acquired, the first image detection result is used as a reference detection result to compare the first image detection result and the second image detection result, a comparison type is determined based on the comparison result, the comparison type includes true positive examples, false positive examples and false negative examples, an evaluation score for multiple initial autonomous driving images is determined based on the comparison type, a target autonomous driving image dataset is selected from the initial autonomous driving image dataset based on the evaluation scores for multiple initial autonomous driving images, and tags in the target autonomous driving image dataset are adjusted. The embodiment of this application can improve the accuracy of image dataset processing by determining the comparison type based on the comparison result between the first image detection result and the second image detection result, determining the evaluation score for the initial autonomous driving images based on the comparison type, and selecting a target autonomous driving image dataset based on the evaluation score for the initial autonomous driving images.

[0044] Figure 2 is a flowchart of another image dataset processing method according to an embodiment of the present application. The embodiment of the present application is a concrete manifestation based on the embodiment of the present invention, and referring to Figure 2, the method according to the embodiment of the present application specifically includes the following steps.

[0045] In S201, the initial automated driving image dataset is acquired.

[0046] In S202, the initial automated driving image dataset is input to the first automated driving detection model and the second automated driving detection model, respectively, and the first image detection result and the second image detection result are obtained.

[0047] In S203, the first image detection result and the second image detection result are compared, and the comparison result is obtained.

[0048] In S204, the comparison type is determined based on the comparison results.

[0049] In S205, the evaluation score of the target object in the initial autonomous driving image is determined based on the comparison type for the evaluation score of any of the initial autonomous driving images.

[0050] Here, the evaluation score of the target object includes the IoU score, confidence score, area score, and category score of the target object. In this embodiment, the evaluation score of the target object differs depending on the comparison type. Based on the comparison type, the evaluation scores of all target objects or some target objects in the initial autonomous driving image can be determined.

[0051] Preferably, determining the evaluation score of the target object in the initial autonomous driving image based on the comparison type includes determining at least one of the IoU score, confidence score, and area score of the target object based on the comparison type, obtaining a pre-set category score corresponding to the target object, and determining the evaluation score of the target object based on at least one of the IoU score, confidence score, area score, and category score.

[0052] In this embodiment, the evaluation score for each target object is obtained by the following method: Based on the comparison type, the IoU score, confidence score, and area score of each target object are obtained, and a pre-set category score corresponding to the target object is obtained. The IoU score, confidence score, area score, and category score are multiplied or weighted averaged, and the result after multiplication or weighted average is taken as the evaluation score for the target object. In this embodiment, the elements that determine the evaluation score of the target object are not limited to the IoU score, confidence score, area score, and category score, and corresponding elements may be added or removed.

[0053] Each target category has a corresponding category weight pre-set, meaning a category score is pre-set, with the category score range being [0,1]. After predicting the category to which the target belongs, the category score corresponding to the predicted category can be obtained. Regarding category weights, the distribution information of the initial autonomous driving image dataset can be merged, and higher category weights can be set for the long tail portion. The evaluation results of the model on the test set can also be merged, and the weights can be increased for categories with poor accuracy.

[0054] This embodiment allows for the accurate determination of the evaluation score of a target object by determining the evaluation score of the target object based on at least one of the IoU score, confidence score, area score, and category score.

[0055] Preferably, determining the target IoU score based on the comparison type includes determining a second IoU setting value based on a first area and / or a second area, and determining an IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU.

[0056] Here, the second IoU setting value is greater than the first IoU setting value, and the second IoU setting value may be understood as the largest IoU threshold, while the first IoU setting value may be understood as the smallest IoU threshold. In this embodiment, the size of the target object is determined based on the first area and / or the second area. If the first area and / or the second area is less than 32*32, the target object is a small target, and the corresponding second IoU setting value may be 0.7. If the first area and / or the second area is 32*32 or larger, the target object is a normal target, and the corresponding first IoU setting value may be 0.9.

[0057] For example, the formula for the IoU threshold is as follows:

number

[0058] Specifically, if the comparison type is a false positive example or a false negative example, and the IoU is smaller than the first IoU setting, the IoU evaluation score can be obtained directly. If the comparison type is a true positive example, and the IoU is greater than or equal to the second IoU setting, the IoU evaluation score can be obtained directly. If the comparison type is a true positive example, and the IoU is within the interval formed by the first and second IoU setting, the IoU evaluation score can be determined based on the first IoU setting, the second IoU setting, and the IoU.

[0059] This embodiment allows for the accurate determination of the IoU evaluation score by determining the IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU.

[0060] Preferably, determining the IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU means that if the comparison type is a false positive example or a false negative example and the IoU is smaller than the first IoU setting value, the IoU evaluation score is the first setting IoU evaluation score, and if the comparison type is a true positive example and the IoU is greater than or equal to the second IoU setting value, the IoU evaluation score is the second setting IoU evaluation score. This includes determining the first IoU rating score based on the second IoU setting and the IoU if the comparison type is a true positive example and the IoU falls within the interval defined by the first IoU setting and the second IoU setting, determining the second IoU rating score based on the second IoU setting and the first IoU setting, and determining the IoU rating score based on the first IoU rating score and the second IoU rating score.

[0061] For example, the formula for the IoU evaluation score is as follows:

number

[0062] Here, if the comparison type is a false positive example or a false negative example, and the IoU is smaller than the first IoU setting, the value is highest and the IoU evaluation score is 1. If the comparison type is a true positive example, and the IoU is greater than or equal to the second IoU setting, the model does not need to pay attention to small edge differences, there is no value, and the IoU evaluation score is considered to be 0. If the comparison type is a true positive example, and the IoU is within the interval formed by the first and second IoU settings, the lower the IoU, the higher the value, the first IoU evaluation score is max_iou-iou, the second IoU evaluation score is max_iou-min_iou, and the IoU evaluation score is (max_iou-iou) / (max_iou-min_iou).

[0063] As an example, Figure 3 is a schematic diagram illustrating the effects of IoU and IoU evaluation score according to an embodiment of the present application. As shown in Figure 3, the horizontal axis represents IoU (iou), and the vertical axis represents the IoU evaluation score (iou_score). It can be seen that when the target is a small target or a normal target, there is an inverse relationship between IoU and the IoU evaluation score.

[0064] The reason for distinguishing and treating targets according to their size is that absolute coordinate shifts (using pixels as the unit of measurement) for targets with the same position have a greater impact on the IoU value of smaller targets during alignment. Therefore, generally, the IoU value of smaller targets is not high, and a higher evaluation score is given to more small targets, which does not align with the principle that "larger targets are of greater value."

[0065] In this embodiment, the IoU evaluation score can be accurately determined by a method in which, if the comparison type is a false positive example or a false negative example and the IoU is smaller than the first IoU setting value, the first set IoU evaluation score is used as the IoU evaluation score; if the comparison type is a true positive example and the IoU is greater than or equal to the second IoU setting value, the second set IoU evaluation score is used as the IoU evaluation score; and if the comparison type is a true positive example and the IoU is within the interval formed by the first IoU setting value and the second IoU setting value, the IoU evaluation score is determined based on the first IoU evaluation score and the second IoU evaluation score.

[0066] Preferably, determining the confidence score of the target object based on the comparison type includes: determining the first evaluation score of the second confidence level and using the first evaluation score of the second confidence level as the confidence score of the target object if the comparison type is a false positive example; determining the first evaluation score of the first confidence level and using the first evaluation score of the first confidence level as the confidence score of the target object if the comparison type is a false negative example; and determining the second evaluation score of the first confidence level and the second evaluation score of the second confidence level, respectively, and determining the confidence score of the target object based on the second evaluation score of the first confidence level and the second evaluation score of the second confidence level if the comparison type is a true positive example.

[0067] For example, the formula for determining the confidence score of a target is as follows:

number

[0068] In this embodiment, the confidence score of the target can be accurately determined by the following methods: when the comparison type is a false positive example, the first evaluation score of the second confidence level output from the second autonomous driving detection model is determined, and the first evaluation score of the second confidence level is used as the confidence score of the target when the comparison type is a false positive example; when the comparison type is a false negative example, the first evaluation score of the first confidence level output from the first autonomous driving detection model is determined, and the first evaluation score of the first confidence level is used as the confidence score of the target when the comparison type is a false negative example; and when the comparison type is a true positive example, the second evaluation score of the first confidence level and the second evaluation score of the second confidence level are determined, respectively, and the confidence score of the target when the comparison type is a true positive example is determined based on the second evaluation score of the first confidence level and the second evaluation score of the second confidence level.

[0069] Preferably, determining the first evaluation score for the first confidence level, or determining the first evaluation score for the second confidence level, involves obtaining the first confidence threshold and the second confidence threshold, and if the second confidence threshold is greater than the first confidence threshold, and the first or second confidence level is less than the first confidence threshold, the first set confidence score is set as the first evaluation score for the first confidence level or the first evaluation score for the second confidence level, and if the first confidence level is greater than or equal to the second confidence threshold, or if the second confidence level is greater than or equal to the second confidence threshold, the second set confidence score is set as the first evaluation score for the first confidence level or the first evaluation score for the second confidence level, and if the first confidence level is within the interval formed by the first confidence threshold and the second confidence threshold, This includes determining a first confidence score based on a first confidence level, a second confidence threshold, and the first confidence threshold; determining a second confidence score based on a second confidence threshold and the first confidence threshold; determining a first evaluation score for the first confidence level based on the first confidence score and the second confidence score; or, if the second confidence level falls within the interval defined by the first confidence threshold and the second confidence threshold, determining a third confidence score based on a second confidence level, a second confidence threshold, and the first confidence threshold; determining a fourth confidence score based on a second confidence threshold and the first confidence threshold; and determining a first evaluation score for the second confidence level based on the third confidence score and the fourth confidence score.

[0070] For example, if the comparison type is FP, the formula for determining the first evaluation score of the second confidence level is as follows:

number

[0071] Here, if the comparison type is a false positive example, conf is the second confidence level; if the second confidence level is less than the first confidence level threshold, the first evaluation score for the second confidence level is 0, meaning it has no value; if the second confidence level is greater than or equal to the second confidence level threshold, the first evaluation score for the second confidence level is 1, meaning it has the greatest value; and if the second confidence level falls within the interval formed by the first and second confidence level thresholds, then {0.5*(conf+max_conf)-min_conf} / (max_conf-min_conf) is the first evaluation score for the second confidence level.

[0072] In this embodiment, if the second confidence level is less than the first confidence threshold, the first set confidence score is used as the first evaluation score for the second confidence level; if the second confidence level is equal to or greater than the second confidence threshold, the second set confidence score is used as the first evaluation score for the second confidence level; and if the second confidence level falls within the interval formed by the first and second confidence thresholds, the first evaluation score for the second confidence level is determined based on the third and fourth confidence scores. This method allows for the accurate determination of the first evaluation score for the second confidence level.

[0073] For example, when the comparison type is FN, the formula for determining the first evaluation score of the first confidence level is as follows:

number

[0074] In this embodiment, if the first confidence level is less than the first confidence threshold, the first set confidence score is used as the first evaluation score for the first confidence level; if the first confidence level is equal to or greater than the second confidence threshold, the second set confidence score is used as the first evaluation score for the first confidence level; and if the first confidence level falls within the interval formed by the first and second confidence thresholds, the first evaluation score for the first confidence level is determined based on the first and second confidence scores. This method allows for the accurate determination of the first evaluation score for the first confidence level.

[0075] Preferably, determining the second evaluation score for the first confidence level, or determining the second evaluation score for the second confidence level, means setting the first set confidence score as the second evaluation score for the first confidence level or the second evaluation score for the second confidence level if the first confidence level or the second confidence level is less than the first confidence level threshold; setting the first set confidence score as the second evaluation score for the first confidence level or the second evaluation score for the second confidence level if the first confidence level is greater than or equal to the first confidence level threshold, or if the second confidence level is greater than or equal to the second confidence level threshold; and if the first confidence level is within the interval formed by the first confidence level threshold and the second confidence level threshold. This includes determining the fifth confidence score based on the following: determining the sixth confidence score based on the second confidence threshold and the first confidence threshold; determining the second evaluation score of the first confidence score based on the fifth confidence score and the sixth confidence score; or, if the second confidence score is within the interval formed by the first confidence threshold and the second confidence threshold, determining the seventh confidence score based on the second confidence score and the second confidence threshold; determining the eighth confidence score based on the second confidence threshold and the first confidence threshold; and determining the second evaluation score of the second confidence score based on the seventh confidence score and the eighth confidence score.

[0076] For example, if the comparison type is a true positive example, the formula for determining the second evaluation score of the second confidence level is as follows:

number

[0077] In this embodiment, if the second confidence level is less than the first confidence threshold, the first set confidence score is used as the second evaluation score for the second confidence level; if the second confidence level is equal to or greater than the second confidence threshold, the first set confidence score is used as the second evaluation score for the second confidence level; and if the second confidence level falls within the interval formed by the first and second confidence thresholds, the second evaluation score for the second confidence level is determined based on the seventh and eighth confidence scores. This method allows for the accurate determination of the second evaluation score for the second confidence level.

[0078] For example, if the comparison type is a true positive example, the formula for determining the second evaluation score of the first confidence level is as follows:

number

[0079] In this embodiment, if the first confidence level is less than the first confidence threshold, the first set confidence score is used as the second evaluation score for the first confidence level; if the first confidence level is equal to or greater than the first confidence threshold, the first set confidence score is used as the second evaluation score for the first confidence level; and if the first confidence level falls within the interval formed by the first and second confidence thresholds, the second evaluation score for the first confidence level is determined based on the fifth and sixth confidence scores. This method allows for the accurate determination of the second evaluation score for the first confidence level.

[0080] Exemplary, Figure 4 is a schematic diagram illustrating the effect between confidence and the second evaluation score in an embodiment of the present application. As shown in Figure 4, when the comparison type is a true positive example, the horizontal axis is conf, where conf is the first or second confidence level, min_conf is the first confidence threshold, and max_conf is the second confidence threshold. The vertical axis is conf_score, where conf_score is the second evaluation score for the first or second confidence level. If the first or second confidence level is less than the first confidence threshold, or greater than or equal to the second confidence threshold, the second evaluation score is 0 and has no value. If conf is between the first and second confidence thresholds, the relationship between the second evaluation score (which may be understood as value) and the confidence level is inversely proportional.

[0081] Exemplary, Figure 5 is a schematic diagram illustrating the effect between confidence and first evaluation score in an embodiment of the present application. As shown in Figure 5, when the comparison type is a false negative example or a false positive example, the horizontal axis is conf, where conf is the first or second confidence level, min_conf is the first confidence threshold, and max_conf is the second confidence threshold; the vertical axis is conf_score, where conf_score is the first evaluation score for the first or second confidence level. If the first or second confidence level is less than the first confidence threshold, the first evaluation score is all 0 and has no value. If the first or second confidence level is greater than or equal to the second confidence threshold, the value is greatest, and the first evaluation score is all 1. When conf is between the first and second confidence thresholds, there is a proportional relationship between the first evaluation score (which may be understood as value) and the confidence level.

[0082] Preferably, determining the area score of the target object based on the comparison type includes: when the comparison type is a false positive example, determining the first evaluation score of the second area and using the first evaluation score of the second area as the area score of the target object; when the comparison type is a false negative example, determining the first evaluation score of the first area and using the first evaluation score of the first area as the area score of the target object; and when the comparison type is a true positive example, determining the second evaluation score of the first area and the second evaluation score of the second area respectively, and determining the area score of the target object based on the second evaluation score of the first area and the second evaluation score of the second area.

[0083] Exemplarily, the formula for determining the area score of the target object is as follows.

Number

[0084] In this embodiment, if the comparison type is a false positive example, the first evaluation score of the second area is determined and the first evaluation score of the second area is used as the area score of the target object when the comparison type is a false positive example; if the comparison type is a false negative example, the first evaluation score of the first area is determined and the first evaluation score of the first area is used as the area score of the target object when the comparison type is a false negative example; and if the comparison type is a true positive example, the area score of the target object when the comparison type is a true positive example is determined based on the second evaluation score of the first area and the second evaluation score of the second area. This method allows for the accurate determination of the area score of the target object.

[0085] Preferably, determining the first evaluation score for the first area or determining the first evaluation score for the second area involves obtaining the first area threshold and the second area threshold, and if the second area threshold is greater than the first area threshold and the first or second area is less than the first area threshold, setting the first set area score as the first evaluation score for the first area or the first evaluation score for the second area; if the first area is greater than or equal to the second area threshold, or if the second area is greater than or equal to the second area threshold, setting the second set area score as the first evaluation score for the first area or the first evaluation score for the second area; and if the first area is within the interval formed by the first area threshold and the second area threshold, This includes determining a first area score based on a first area, a second area threshold, and the first area threshold; determining a second area score based on a second area threshold and the first area threshold; determining a first evaluation score for the first area based on the first area score and the second area score; or, if the second area falls within the interval defined by the first area threshold and the second area threshold, determining a third area score based on the second area, the second area threshold, and the first area threshold; determining a fourth area score based on the second area threshold and the first area threshold; and determining a first evaluation score for the second area based on the third area score and the fourth area score.

[0086] Here, the first area threshold may be 30*30 and the second area threshold may be 500*500, the first set area score may be 0, and the second set area score may be 1.

[0087] For example, if the comparison type is FN, the formula for determining the first evaluation score of the first area is as follows:

number

[0088] In this embodiment, if the first area is smaller than the first area threshold, the first set area score is used as the first evaluation score for the first area; if the first area is equal to or greater than the second area threshold, the second set area score is used as the first evaluation score for the first area; and if the first area falls within the interval formed by the first area threshold and the second area threshold, the first evaluation score for the first area is determined based on the first area score and the second area score. This method allows for the accurate determination of the first evaluation score for the first area.

[0089] For example, if the comparison type is FP, the formula for determining the first evaluation score of the second area is as follows:

number

[0090] In this embodiment, if the second area is smaller than the first area threshold, the first set area score is used as the first evaluation score for the second area; if the second area is equal to or greater than the second area threshold, the second set area score is used as the first evaluation score for the second area; and if the second area falls within the interval formed by the first and second area thresholds, the first evaluation score for the second area is determined based on the third and fourth area scores. This method allows for the accurate determination of the first evaluation score for the second area.

[0091] Preferably, determining the second evaluation score for the first area or the second evaluation score for the second area means that if the first or second area is smaller than the first area threshold, the first set area score is set as the second evaluation score for the first area or the second evaluation score for the second area; if the first area is greater than or equal to the first area threshold, or if the second area is greater than or equal to the first area threshold, the second set area score is set as the second evaluation score for the first area or the second evaluation score for the second area; and if the first area is within the interval formed by the first area threshold and the second area threshold, the first area and the first area threshold This includes determining the fifth area score based on the second area threshold and the first area threshold, determining the sixth area score based on the fifth area score and the sixth area score, or, if the second area is within the interval defined by the first area threshold and the second area threshold, determining the seventh area score based on the second area and the first area threshold, determining the eighth area score based on the second area threshold and the first area threshold, and determining the second evaluation score of the second area based on the seventh area score and the eighth area score.

[0092] For example, if the comparison type is a true positive example, the formula for determining the second evaluation score of the first area is as follows:

number

[0093] In this embodiment, if the first area is smaller than the first area threshold, the first set area score is used as the second evaluation score for the first area; if the first area is equal to or greater than the first area threshold, the second set area score is used as the second evaluation score for the first area; and if the first area falls within the interval formed by the first area threshold and the second area threshold, the second evaluation score for the first area is determined based on the fifth area score and the sixth area score. This method allows for the accurate determination of the second evaluation score for the first area.

[0094] For example, if the comparison type is a true positive example, the formula for determining the second evaluation score for the second area is as follows:

number

[0095] In this embodiment, if the second area is smaller than the first area threshold, the first set area score is used as the second evaluation score for the second area; if the second area is equal to or greater than the first area threshold, the second set area score is used as the second evaluation score for the second area; and if the second area falls within the interval formed by the first and second area thresholds, the second evaluation score for the second area is determined based on the seventh area score and the eighth area score. This method allows for the accurate determination of the second evaluation score for the second area.

[0096] Exemplary, Figure 6 is a schematic diagram illustrating the effect between area and second evaluation score in an embodiment of the present application. As shown in Figure 6, when the comparison type is a true positive example, the horizontal axis is area, where area is the first or second area, min_area is the first area threshold, and max_conf is the second area threshold. The vertical axis is area_score, where area_score is the second evaluation score. If the first or second area is smaller than the first area threshold, the second evaluation score is 0 and has no value. If the first or second area is greater than or equal to the second area threshold, the value is the greatest, and the second evaluation score is 1 for all. If area is between the first and second area thresholds, the relationship between the second evaluation score (which may be understood as value) and area is proportional.

[0097] Exemplary, Figure 7 is a schematic diagram illustrating the effect between area and first evaluation score in an embodiment of the present application. As shown in Figure 7, when the comparison type is a false negative example or a false positive example, the horizontal axis is area, where area is the first or second area, min_area is the first area threshold, max_area is the second area threshold, and the vertical axis is area_score, where area_score is the first evaluation score of the area. If the first or second area is smaller than the first area threshold, the first evaluation score is all 0 and has no value. If the first or second area is greater than or equal to the second area threshold, the value is the greatest and the first evaluation score is all 1. If the area is between the first and second area thresholds, the relationship between the first evaluation score (which may be understood as value) and the area is proportional.

[0098] In S206, the scene score of the initial autonomous driving image is determined.

[0099] In this embodiment, the initial autonomous driving image scene is not restricted and can include, for example, rain scenes, snow scenes, traffic jam scenes, off-road scenes, etc.

[0100] In this embodiment, actual scene information can be acquired based on the initial autonomous driving image, the initial autonomous driving image can be input into the autonomous driving scene model, predicted scene information can be output, and a scene score can be obtained based on the similarity between the predicted scene information and the actual scene information.

[0101] Preferably, determining the scene score of the initial autonomous driving image includes inputting the initial autonomous driving image into an autonomous driving scene model and outputting predicted scene information, obtaining setting scene information corresponding to the initial autonomous driving image, determining scene similarity based on the predicted scene information and setting scene information, and determining the scene score of the initial autonomous driving image based on the scene similarity.

[0102] Here, the autonomous driving scene model may be a scene model based on any deep learning algorithm. The set scene information may be real scene information corresponding to the initial autonomous driving image. In this embodiment, the initial autonomous driving image is input to the autonomous driving scene model, the corresponding predicted scene information is output, the set scene information corresponding to the initial autonomous driving image is obtained, the scene similarity between the predicted scene information and the set scene information is calculated based on an arbitrary similarity algorithm, and the scene score of the initial autonomous driving image is determined based on the scene similarity. Here, the range of the scene similarity is [0,1], that is, the range of the scene score is [0,1].

[0103] This embodiment accurately determines the scene score of the initial autonomous driving image by determining the scene score of the initial autonomous driving image based on the scene similarity between the predicted scene information and the set scene information.

[0104] In S207, the evaluation score of the initial autonomous driving image is determined based on the evaluation score of the target object and the scene score.

[0105] In this embodiment, the calculation method for the evaluation score of any initial autonomous driving image is as follows: The evaluation scores of all target objects in the current initial autonomous driving image are cumulatively added together, the cumulative sum result is obtained, and then the cumulative sum result is multiplied by the scene score to obtain the evaluation score of the current initial autonomous driving image.

[0106] For example, the formula for determining the evaluation score of the initial autonomous driving image is as follows:

number

[0107] In S208, a target autonomous driving image dataset is selected from the initial autonomous driving image dataset based on evaluation scores of multiple initial autonomous driving images, and tags in the target autonomous driving image dataset are adjusted.

[0108] In this embodiment, the initial autonomous driving images in the initial autonomous driving image dataset are ranked in descending order according to the evaluation score of each initial autonomous driving image. The top number of initial autonomous driving images are extracted and designated as the target number of autonomous driving images, and all tags corresponding to the target number of autonomous driving images are adjusted.

[0109] Preferably, selecting a target autonomous driving image dataset from an initial autonomous driving image dataset based on evaluation scores of multiple initial autonomous driving images includes ranking multiple initial autonomous driving images based on evaluation scores of multiple initial autonomous driving images, obtaining an initial autonomous driving image dataset after ranking, and selecting a set number of target autonomous driving image datasets from the initial autonomous driving image dataset after ranking.

[0110] Specifically, based on the evaluation scores of multiple initial autonomous driving images, multiple initial autonomous driving images are ranked in descending order, a dataset of the ranked initial autonomous driving images is obtained, and a dataset of target autonomous driving images with a set number of images at the top is selected from the ranked initial autonomous driving image dataset, that is, the initial autonomous driving images with a set number of images from the beginning are extracted and used as the target autonomous driving images with a set number of images. Alternatively, based on the evaluation scores of multiple initial autonomous driving images, multiple initial autonomous driving images are ranked in descending order, a dataset of the ranked initial autonomous driving images is obtained, and a dataset of target autonomous driving images with a set number of images at the bottom is selected from the ranked initial autonomous driving image dataset, that is, the initial autonomous driving images with a set number of images from the end are extracted and used as the target autonomous driving images with a set number of images.

[0111] In this embodiment, by selecting a set number of target autonomous driving image datasets from the initial autonomous driving image dataset after ranking, the target autonomous driving images can be accurately selected.

[0112] For example, Figure 8 is a schematic diagram illustrating the effect of the first image detection result according to an embodiment of the present application. Figure 9 is a schematic diagram illustrating the effect of the second image detection result according to an embodiment of the present application. Figure 8 shows the first image detection result detected by the first autonomous driving detection model, and Figure 9 shows the second image detection result detected by the second autonomous driving detection model. As can be seen from Figures 8 and 9, In the first autonomous driving detection model, one group of bicycles is outside of a single frame, but in the second autonomous driving detection model, each bicycle is outside of its own frame individually. As a result, the frames of all the bicycles do not match, and the comparison type is FN or FP. In Figure 8 or Figure 9, the rearmost pillar is detected by the first autonomous driving detection model but not by the second autonomous driving detection model, and its comparison type is FN, with a high value score. Therefore, the evaluation score of the target object is mostly FN and FP, and the score is high.

[0113] Figure 10 is a schematic diagram of the structure of an image data set processing device according to an embodiment of the present invention. As shown in Figure 10, the device comprises an initial automatic driving image data set acquisition module 1001, an image detection result acquisition module 1002, an image detection result comparison module 1003, a comparison type determination module 1004, an evaluation score determination module 1005, and a target automatic driving image data set selection module 1006.

[0114] The initial autonomous driving image dataset acquisition module 1001 is used to acquire an initial autonomous driving image dataset containing multiple initial autonomous driving images and their corresponding tags.

[0115] The image detection result acquisition module 1002 is used to input the initial automated driving image dataset into the first automated driving detection model and the second automated driving detection model, respectively, and to acquire the first image detection result and the second image detection result, respectively.

[0116] The image detection result comparison module 1003 is used to compare the first image detection result and the second image detection result and obtain the comparison result, with the first image detection result being used as the reference detection result.

[0117] The comparison type determination module 1004 is used to determine the comparison type based on the comparison result, and the comparison type includes true positive examples, false positive examples, and false negative examples.

[0118] The evaluation score determination module 1005 is used to determine the evaluation scores of the plurality of initial automated driving images based on the comparison type.

[0119] The target autonomous driving image dataset selection module 1006 is used to select a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images, and to adjust the tags in the target autonomous driving image dataset.

[0120] In the proposed technology disclosed in this embodiment, an initial autonomous driving image dataset acquisition module acquires an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags; an image detection result acquisition module inputs the initial autonomous driving image dataset into a first autonomous driving detection model and a second autonomous driving detection model, respectively, and acquires a first image detection result and a second image detection result; an image detection result comparison module compares the first image detection result and the second image detection result using the first image detection result as a reference detection result and acquires a comparison result; a comparison type determination module determines the comparison type based on the comparison result, and the comparison type includes true positive examples, false positive examples and false negative examples; an evaluation score determination module determines the evaluation score of multiple initial autonomous driving images based on the comparison type; and a target autonomous driving image dataset selection module selects a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation score of multiple initial autonomous driving images and adjusts the tags in the target autonomous driving image dataset. The embodiment of the present invention can improve the accuracy of image dataset processing by determining the comparison type based on the comparison result between the first image detection result and the second image detection result, determining the evaluation score of the initial autonomous driving image based on the comparison type, and selecting a target autonomous driving image dataset based on the evaluation score of the initial autonomous driving image.

[0121] Preferably, each initial autonomous driving image includes at least one target object, the first image detection result includes first detection frame information for at least one target object, a first category corresponding to each of the at least one target object, a first confidence level for the first category, and a first area corresponding to the first detection frame information, the second image detection result includes second detection frame information for at least one target object, a second category corresponding to each of the at least one target object, a second confidence level for the second category, and a second area corresponding to the second detection frame information, the comparison result includes a category comparison result, and preferably, the image detection result comparison module is used specifically to compare the first category and the second category and to obtain a category comparison result including category comparison matches and category comparison mismatches.

[0122] Preferably, the comparison type determination module is used to determine the IoU based on the first detection frame information and the second detection frame information; to determine a first IoU setting value based on the first area and / or the second area; to determine the comparison type as a true positive example if the IoU is greater than or equal to the first IoU setting value and the category comparison result is a category comparison match; to determine the comparison type as a false positive example and a false negative example if the IoU is less than the first IoU setting value and / or the category comparison result is a category comparison mismatch; or to determine the comparison type as a false negative example if the first image detection result includes the first detection frame information of the target object and the second image detection result does not include the second detection frame information of the target object; and to determine the comparison type as a false positive example if the first image detection result does not include the first detection frame information of the target object and the second image detection result includes the second detection frame information of the target object.

[0123] Preferably, the evaluation score determination module is used to determine the evaluation score of the target object in an initial autonomous driving image based on the comparison type, to determine the scene score of the initial autonomous driving image, and to determine the evaluation score of the initial autonomous driving image based on the evaluation score of the target object and the scene score.

[0124] Preferably, the evaluation score determination module is used to determine at least one of the IoU score, confidence score, and area score of the target object based on the comparison type, to obtain a pre-set category score corresponding to the target object, and to determine the evaluation score of the target object based on at least one of the IoU score, confidence score, area score, and category score.

[0125] Preferably, the evaluation score determination module is used to further determine a second IoU setting value based on the first area and / or second area, where the second IoU setting value is greater than the first IoU setting value, and to determine an IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU.

[0126] Preferably, the evaluation score determination module is used to determine the IoU evaluation score to be the first set IoU evaluation score when the comparison type is a false positive example or a false negative example and the IoU is smaller than the first IoU set value; to determine the IoU evaluation score to be the second set IoU evaluation score when the comparison type is a true positive example and the IoU is greater than or equal to the second IoU set value; to determine the first IoU evaluation score based on the second IoU set value and the IoU when the comparison type is a true positive example and the IoU is within the interval formed by the first IoU set value and the second IoU set value; to determine the second IoU evaluation score based on the second IoU set value and the first IoU set value; and to determine the IoU evaluation score based on the first IoU evaluation score and the second IoU evaluation score.

[0127] Preferably, the evaluation score determination module is used to determine the first evaluation score of the second confidence level and set the first evaluation score of the second confidence level as the confidence score of the target if the comparison type is a false positive example; to determine the first evaluation score of the first confidence level and set the first evaluation score of the first confidence level as the confidence score of the target if the comparison type is a false negative example; and to determine the second evaluation score of the first confidence level and the second evaluation score of the second confidence level, respectively, and determine the confidence score of the target based on the second evaluation score of the first confidence level and the second evaluation score of the second confidence level if the comparison type is a true positive example.

[0128] Preferably, the evaluation score determination module further obtains a first confidence threshold and a second confidence threshold, and if the second confidence threshold is greater than the first confidence threshold, and the first confidence or the second confidence is less than the first confidence threshold, the first set confidence score is set as the first evaluation score for the first confidence or the first evaluation score for the second confidence; if the first confidence is greater than or equal to the second confidence threshold, or if the second confidence is greater than or equal to the second confidence threshold, the second set confidence score is set as the first evaluation score for the first confidence or the first evaluation score for the second confidence; and if the first confidence is within the interval formed by the first confidence threshold and the second confidence threshold, the first confidence, the second confidence threshold and the first It is used to determine a first confidence score based on a confidence threshold, to determine a second confidence score based on the second confidence threshold and the first confidence threshold, to determine a first evaluation score for the first confidence based on the first confidence score and the second confidence score, or, if the second confidence is within the interval formed by the first confidence threshold and the second confidence threshold, to determine a third confidence score based on the second confidence, the second confidence threshold and the first confidence threshold, to determine a fourth confidence score based on the second confidence threshold and the first confidence threshold, and to determine a first evaluation score for the second confidence based on the third confidence score and the fourth confidence score.

[0129] Preferably, the evaluation score determination module further determines the first set confidence score as the second evaluation score of the first confidence level or the second evaluation score of the second confidence level if the first confidence level or the second confidence level is less than the first confidence level threshold; determines the first set confidence score as the second evaluation score of the first confidence level or the second evaluation score of the second confidence level if the first confidence level is greater than or equal to the first confidence level threshold, or if the second confidence level is greater than or equal to the second confidence level threshold; and determines the fifth confidence score based on the first confidence level and the second confidence level threshold if the first confidence level is within the interval formed by the first confidence level threshold and the second confidence level threshold. It is used to determine a sixth confidence score based on a second confidence threshold and a first confidence threshold, to determine a second evaluation score for the first confidence score based on the fifth confidence score and the sixth confidence score, or, if the second confidence score is within the interval formed by the first confidence threshold and the second confidence threshold, to determine a seventh confidence score based on the second confidence score and the second confidence threshold, to determine an eighth confidence score based on the second confidence threshold and the first confidence threshold, and to determine a second evaluation score for the second confidence score based on the seventh confidence score and the eighth confidence score.

[0130] Preferably, the evaluation score determination module is used to determine the first evaluation score of the second area and set the first evaluation score of the second area as the target area score when the comparison type is a false positive example; to determine the first evaluation score of the first area and set the first evaluation score of the first area as the target area score when the comparison type is a false negative example; and to determine the second evaluation score of the first area and the second evaluation score of the second area, respectively, when the comparison type is a true positive example, and to determine the target area score based on the second evaluation score of the first area and the second evaluation score of the second area.

[0131] Preferably, the evaluation score determination module further obtains a first area threshold and a second area threshold, and if the second area threshold is greater than the first area threshold, and the first area or the second area is less than the first area threshold, the first set area score is set to the first evaluation score of the first area or the first evaluation score of the second area; if the first area is greater than or equal to the second area threshold, or if the second area is greater than or equal to the second area threshold, the second set area score is set to the first evaluation score of the first area or the first evaluation score of the second area; and if the first area is within the interval formed by the first area threshold and the second area threshold, the first area, the second area threshold and It is used to determine a first area score based on a first area threshold, to determine a second area score based on a second area threshold and a first area threshold, to determine a first evaluation score for the first area based on the first area score and the second area score, or, if the second area is within the interval formed by the first area threshold and the second area threshold, to determine a third area score based on the second area, the second area threshold and the first area threshold, to determine a fourth area score based on the second area threshold and the first area threshold, and to determine a first evaluation score for the second area based on the third area score and the fourth area score.

[0132] Preferably, the evaluation score determination module further determines the first set area score as the second evaluation score for the first area or the second evaluation score for the second area if the first area or the second area is smaller than the first area threshold; determines the second set area score as the second evaluation score for the first area or the second evaluation score for the second area if the first area is greater than or equal to the first area threshold, or if the second area is greater than or equal to the first area threshold; and determines the fifth area score based on the first area and the first area threshold if the first area is within the interval formed by the first area threshold and the second area threshold. It is used to determine the sixth area score based on the second area threshold and the first area threshold, to determine the second evaluation score of the first area based on the fifth area score and the sixth area score, or, if the second area is within the interval formed by the first area threshold and the second area threshold, to determine the seventh area score based on the second area and the first area threshold, to determine the eighth area score based on the second area threshold and the first area threshold, and to determine the second evaluation score of the second area based on the seventh area score and the eighth area score.

[0133] Preferably, the evaluation score determination module is used to input the initial autonomous driving image into the autonomous driving scene model and output predicted scene information, to acquire setting scene information corresponding to the initial autonomous driving image, to determine the scene similarity based on the predicted scene information and the setting scene information, and to determine the scene score of the initial autonomous driving image based on the scene similarity.

[0134] Preferably, the target autonomous driving image dataset selection module is used to rank the multiple initial autonomous driving images based on their evaluation scores, obtain a dataset of the ranked initial autonomous driving images, and select a set number of target autonomous driving image datasets from the ranked initial autonomous driving image datasets.

[0135] The image data set processing device according to the embodiment of the present application can perform an image data set processing method according to any embodiment of the present application and is equipped with a functional module and beneficial effects corresponding to the performance of the method.

[0136] Furthermore, the units and modules of the above-mentioned device are merely divided according to functional logic and are not limited to the above division; they only need to be able to realize the corresponding function. Also, the specific names of each functional unit are merely for ease of distinction and do not limit the scope of protection of the embodiments of this application.

[0137] Figure 11 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Hereinafter, with reference to Figure 11, schematic diagrams of electronic devices (e.g., terminal devices or servers in Figure 11) 1100 suitable for realizing the embodiment of the present application are shown. The terminal devices in the embodiment of the present application may include, but are not limited to, mobile devices such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. The electronic devices shown in Figure 11 are merely examples and do not impose any limitations on the functions and scope of use of the embodiment of the present application.

[0138] As shown in Figure 11, the electronic device 1100 may include a processing unit (e.g., a central processor, a graphics processor, etc.) 1101, which can perform various appropriate operations and processes based on a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 into a random access memory (RAM) 1103. The RAM 1103 further stores various programs and data necessary for the operation of the electronic device 1100. The processing unit 1101, ROM 1102, and RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 115 is also connected to the bus 1104.

[0139] Typically, input devices 1106, including, for example, touch panels, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc., output devices 1107, including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc., storage devices 1108, including, for example, magnetic tape, hard disks, etc., and communication devices 1109 can be connected to the I / O interface 1105. The communication devices 1109 enable the electronic device 1100 to exchange data with other devices via wireless or wired communication. Figure 11 shows an electronic device 1100 with various devices, but it should be understood that it is not necessary to implement or have all of the devices shown. Instead, more or fewer devices may be implemented or have all of them.

[0140] In particular, according to the embodiments of the present application, the process described with reference to the flowchart above can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product which includes a computer program carried on a non-temporary computer-readable medium, the computer program which includes program code for performing the method shown in the flowchart. In such embodiments, the computer program may be downloaded and installed from a network by a communication device 1109, installed from a storage device 1108, or installed from a ROM 1102. When the computer program is executed by the processing device 1101, it performs the above-mentioned functions limited by the method of the embodiments of the present application.

[0141] The names of messages or information exchanged between multiple devices in the embodiments of this application are for descriptive purposes only and are not intended to limit the scope of such messages or information.

[0142] The electronic device according to the embodiment of the present application belongs to the same inventive concept as the image data set processing method according to the above embodiment, and technical details not described in detail in this embodiment can be referenced to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0143] The embodiment of the present invention provides a computer storage medium that stores a computer program that, when executed by a processor, implements the image dataset processing method according to the above embodiment.

[0144] The computer-readable medium described in this application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the above. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Further specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more leads, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used in or in conjunction with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium may include data signals propagated in the baseband or as part of a carrier wave, in which computer-readable program code is carried. Such propagated data signals can take various forms and include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, and such computer-readable signal medium can transmit, propagate, or transmit programs used in or in conjunction with instruction execution systems, apparatus, or devices. The program code contained in the computer-readable medium can be transmitted by any suitable medium and may include, but are not limited to, wires, optical cables, RF (radio frequency), or any suitable combination thereof.

[0145] In some embodiments, clients and servers can communicate using any currently known or future-to-be-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communications (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), network-off-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), and any currently known or future-to-be-developed networks.

[0146] The computer-readable medium described above may be included in the electronic device described above, or it may exist independently and not be attached to the electronic device.

[0147] When one or more programs are contained on the computer-readable medium described above, and when the one or more programs are executed by the electronic device, the electronic device will An initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags is acquired; the initial autonomous driving image dataset is input to a first autonomous driving detection model and a second autonomous driving detection model, respectively; a first image detection result and a second image detection result are acquired; the first image detection result is used as a reference detection result and compared with the second image detection result; a comparison result is acquired; a comparison type is determined based on the comparison result, and the comparison type includes true positive examples, false positive examples, and false negative examples; an evaluation score is determined based on the comparison type; a target autonomous driving image dataset is selected from the initial autonomous driving image dataset based on the evaluation scores of the multiple initial autonomous driving images; and tags in the target autonomous driving image dataset are adjusted.

[0148] Computer program code for performing the operations of the present invention can be written in one or more programming languages ​​or a combination thereof, and such programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, and further include conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, run as a single standalone software package, run partially on the user's computer and partially on a remote computer, or run entirely on a remote computer or business server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (for example, connected via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the drawings illustrate the implementable architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code, which includes one or more executable instructions for implementing a given logic function. In some alternative implementations, the functions described in a block may occur in an order different from that shown in the drawings. For example, two blocks shown as connected may actually be executed almost in parallel or in reverse order, depending on the functions they relate to. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, may be implemented in a system based on dedicated hardware for performing a given function or operation, or in a combination of dedicated hardware and computer instructions.

[0150] The unit according to the embodiment of the present application may be implemented in software or in hardware. Here, the name of the unit is not limited to the unit itself in some cases; for example, the first acquisition unit may be described as "a unit that acquires at least two Internet Protocol addresses."

[0151] The functions described in this paper may be performed at least partially by one or more hardware logic components. For example, non-limiting exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), system-on-a-chip systems (SOCs), and composite programmable logic devices (CPLDs).

[0152] In the specification of this application, an instrument-readable medium may be a tangible medium that contains or stores a program used in or in conjunction with an instruction execution system, apparatus or device. An instrument-readable medium may be an instrument-readable signal medium or an instrument-readable storage medium. An instrument-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. Further specific examples of instrument-readable storage media include one or more wired electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only disks (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0153] The above description is merely an explanation of preferred embodiments and the technical principles used in the present application. Those skilled in the art will understand that the scope of the disclosure in this application is not limited to technical solutions consisting of specific combinations of the above technical features, but also includes other technical solutions formed by arbitrary combinations of the above technical features or their equivalents, provided that such combinations do not deviate from the spirit of the disclosure. For example, technical solutions formed by substituting the above features with similar functional technical features disclosed in this application (but not limited to these).

[0154] Furthermore, although the operations are described in a specific order, it should not be understood that these operations must be performed in the specific order or in the forward direction indicated. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above considerations, these should not be understood as limiting the scope of the present application. Some features described in the description of a single embodiment may be implemented in combination in a single embodiment. Conversely, various features described in the description of a single embodiment may be implemented individually or in any appropriate sub-combination in multiple embodiments.

[0155] Although this subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter limited to the attached claims is not necessarily limited to the specific features or behaviors described above. Conversely, the specific features and behaviors described above are merely illustrative forms of realizing the claims.

Claims

1. An image dataset processing method performed by an image dataset processing device comprising a storage device and a processor, The processor acquires from the storage device an initial autonomous driving image dataset, which includes multiple initial autonomous driving images and corresponding tags collected by a high-level autonomous driving test vehicle. The processor inputs the initial automated driving image dataset to the first automated driving detection model, which is a server-side model, and the second automated driving detection model, which is an edge-side model, both stored in a memory device, and obtains the first image detection result and the second image detection result, respectively. The processor compares the first image detection result and the second image detection result and obtains the comparison result, wherein the first image detection result is used as the reference detection result. The processor determines the comparison type based on the comparison result, and that the comparison type includes true positive examples, false positive examples, and false negative examples. The processor determines the evaluation score of the plurality of initial autonomous driving images based on the comparison type, The processor includes selecting a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images, and adjusting the tags in the target autonomous driving image dataset. Image dataset processing methods.

2. Each initial autonomous driving image includes at least one target object. The first image detection result includes first detection frame information for at least one target object, a first category corresponding to each of the at least one target object, a first confidence level for the first category, and a first area corresponding to the first detection frame information. The second image detection result includes second detection frame information for at least one target object, a second category corresponding to each of the at least one target object, a second confidence level corresponding to the second category, and a second area corresponding to the second detection frame information. The aforementioned comparison results include category comparison results, The processor compares the first image detection result and the second image detection result and obtains the comparison result. This includes comparing the first category with the second category and obtaining category comparison results, including category matches and category mismatches. The method according to claim 1.

3. The processor determines the comparison type based on the comparison result, The IoU is determined based on the first detection frame information and the second detection frame information, The first IoU setting value is determined based on the first area and / or the second area, If the IoU is greater than or equal to the first IoU setting value, and the category comparison result is a category comparison match, the comparison type is determined to be a true positive example. If the IoU is smaller than the first IoU setting value and / or the category comparison result is a category comparison mismatch, the comparison type is determined to be a false positive example and a false negative example, or If the first image detection result includes first detection frame information of the target object, and the second image detection result does not include second detection frame information of the target object, the comparison type is determined to be a false negative example. If the first image detection result does not include first detection frame information of the target object, and the second image detection result includes second detection frame information of the target object, the comparison type is determined to be a false positive example, The method according to claim 2.

4. The processor determines the evaluation score of the plurality of initial autonomous driving images based on the comparison type, For each initial autonomous driving image, the evaluation score of the target object in the initial autonomous driving image is determined based on the comparison type, To determine the scene score of the initial autonomous driving image, This includes determining the evaluation score of the initial autonomous driving image based on the evaluation score of the target object and the scene score, The method according to claim 3.

5. Determining the evaluation score of the target in the initial autonomous driving image based on the comparison type is Based on the comparison type, determine at least one of the target IoU score, confidence score, and area score, Obtain a pre-set category score corresponding to the target object, This includes determining the evaluation score of the target based on at least one of the IoU score, the confidence score, the area score, and the category score. The method according to claim 4.

6. Determining the target IOU score based on the comparison type is A second IoU setting value is determined based on the first area and / or second area, and the second IoU setting value is greater than the first IoU setting value. This includes determining an IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU, The method according to claim 5.

7. Determining the IoU evaluation score based on the comparison type, the first IoU setting value, the second IoU setting value, and the IoU is: If the comparison type is a false positive example or a false negative example, and the IoU is smaller than the first IoU setting value, the IoU evaluation score is the first setting IoU evaluation score. If the comparison type is a true positive example and the IoU is greater than or equal to the second IoU setting value, the IoU evaluation score is the second setting IoU evaluation score. If the comparison type is a true positive example and the IoU falls within the interval defined by the first IoU setting value and the second IoU setting value, the first IoU evaluation score is determined based on the second IoU setting value and the IoU. The second IoU evaluation score is determined based on the second IoU setting value and the first IoU setting value, This includes determining the IoU evaluation score based on the first IoU evaluation score and the second IoU evaluation score, The method according to claim 6.

8. Determining the confidence score of the target based on the comparison type is If the comparison type is a false positive example, the first evaluation score of the second confidence level is determined, and the first evaluation score of the second confidence level is used as the confidence score of the target object. If the comparison type is a false negative example, the first evaluation score of the first confidence level is determined, and the first evaluation score of the first confidence level is used as the confidence score of the target object. If the comparison type is a true positive example, the following steps are taken: determine the second evaluation score for the first confidence level and the second evaluation score for the second confidence level, and determine the confidence score of the target based on the second evaluation score for the first confidence level and the second evaluation score for the second confidence level. The method according to claim 5.

9. Determining the first evaluation score for the first confidence level, or determining the first evaluation score for the second confidence level, Obtain a first confidence threshold and a second confidence threshold, and confirm that the second confidence threshold is greater than the first confidence threshold. If the first confidence level or the second confidence level is less than the first confidence level threshold, the first set confidence score is set to the first evaluation score of the first confidence level or the first evaluation score of the second confidence level. If the first confidence level is equal to or greater than the second confidence threshold, or if the second confidence level is equal to or greater than the second confidence threshold, the second set confidence score shall be the first evaluation score of the first confidence level or the first evaluation score of the second confidence level. If the first confidence level falls within the interval defined by the first confidence threshold and the second confidence threshold, the first confidence score is determined based on the first confidence level, the second confidence threshold, and the first confidence threshold. The second confidence score is determined based on the second confidence threshold and the first confidence threshold, The first evaluation score of the first confidence level is determined based on the first confidence score and the second confidence score, or If the second confidence level falls within the interval defined by the first confidence threshold and the second confidence threshold, the third confidence score is determined based on the second confidence level, the second confidence threshold, and the first confidence threshold. The fourth confidence score is determined based on the second confidence threshold and the first confidence threshold, This includes determining the first evaluation score of the second confidence level based on the third confidence score and the fourth confidence score, The method according to claim 8.

10. Determining the second evaluation score for the first confidence level, or determining the second evaluation score for the second confidence level, If the first confidence level or the second confidence level is less than the first confidence level threshold, the first set confidence score is set to the second evaluation score of the first confidence level or the second evaluation score of the second confidence level. If the first confidence level is equal to or greater than the first confidence threshold, or if the second confidence level is equal to or greater than the second confidence threshold, the first set confidence score shall be the second evaluation score of the first confidence level or the second evaluation score of the second confidence level. If the first confidence level falls within the interval defined by the first confidence threshold and the second confidence threshold, a fifth confidence score is determined based on the first confidence level and the second confidence threshold. The sixth confidence score is determined based on the second confidence threshold and the first confidence threshold, The second evaluation score of the first confidence level is determined based on the fifth confidence score and the sixth confidence score, or If the second confidence level falls within the interval defined by the first confidence threshold and the second confidence threshold, the seventh confidence score is determined based on the second confidence level and the second confidence threshold. The eighth confidence score is determined based on the second confidence threshold and the first confidence threshold, This includes determining the second evaluation score of the second confidence level based on the seventh confidence score and the eighth confidence score, The method according to claim 9.

11. Determining the target area score based on the comparison type is If the comparison type is a false positive example, the first evaluation score of the second area is determined, and the first evaluation score of the second area is set as the area score of the target. If the comparison type is a false negative example, the first evaluation score of the first area is determined, and the first evaluation score of the first area is set as the area score of the target. If the comparison type is a true positive example, the method includes determining the second evaluation score for the first area and the second evaluation score for the second area, and determining the area score of the target based on the second evaluation score for the first area and the second evaluation score for the second area. The method according to claim 5.

12. Determining the first evaluation score for the first area, or determining the first evaluation score for the second area, Obtain a first area threshold and a second area threshold, and if the second area threshold is greater than the first area threshold, If the first area or the second area is smaller than the first area threshold, the first set area score is set to the first evaluation score of the first area or the first evaluation score of the second area. If the first area is equal to or greater than the second area threshold, or if the second area is equal to or greater than the second area threshold, the second set area score shall be the first evaluation score of the first area or the first evaluation score of the second area. If the first area falls within the interval defined by the first area threshold and the second area threshold, the first area score is determined based on the first area, the second area threshold, and the first area threshold. The second area score is determined based on the second area threshold and the first area threshold, The first evaluation score of the first area is determined based on the first area score and the second area score, or If the second area falls within the interval defined by the first area threshold and the second area threshold, the third area score is determined based on the second area, the second area threshold, and the first area threshold. The fourth area score is determined based on the second area threshold and the first area threshold, This includes determining a first evaluation score for the second area based on the third area score and the fourth area score, The method according to claim 11.

13. Determining the second evaluation score for the first area, or determining the second evaluation score for the second area, If the first area or the second area is smaller than the first area threshold, the first set area score is set to the second evaluation score of the first area or the second evaluation score of the second area. If the first area is equal to or greater than the first area threshold, or if the second area is equal to or greater than the first area threshold, the second set area score shall be the second evaluation score of the first area or the second evaluation score of the second area. If the first area falls within the interval defined by the first area threshold and the second area threshold, the fifth area score is determined based on the first area and the first area threshold. The sixth area score is determined based on the second area threshold and the first area threshold, The second evaluation score of the first area is determined based on the fifth area score and the sixth area score, or If the second area falls within the interval defined by the first area threshold and the second area threshold, the seventh area score is determined based on the second area and the first area threshold. The eighth area score is determined based on the second area threshold and the first area threshold, This includes determining a second evaluation score for the second area based on the seventh area score and the eighth area score, The method according to claim 12.

14. Determining the scene score of the initial autonomous driving image is The initial autonomous driving image is input into the autonomous driving scene model, and predicted scene information is output. To acquire setting scene information corresponding to the initial autonomous driving image, The scene similarity is determined based on the aforementioned predicted scene information and the aforementioned set scene information, This includes determining the scene score of the initial autonomous driving image based on the aforementioned scene similarity, The method according to claim 4.

15. The processor selects a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images. The process involves ranking the multiple initial autonomous driving images based on their evaluation scores, and obtaining a dataset of the ranked initial autonomous driving images. This includes selecting a set number of target autonomous driving image datasets from the initial autonomous driving image datasets after ranking, The method according to claim 1.

16. An initial autonomous driving image dataset acquisition module is configured to acquire an initial autonomous driving image dataset containing multiple initial autonomous driving images and corresponding tags, An image detection result acquisition module is configured to input the initial automated driving image dataset into a first automated driving detection model and a second automated driving detection model, respectively, and to acquire a first image detection result and a second image detection result, respectively. An image detection result comparison module configured to compare the first image detection result with the second image detection result and obtain a comparison result, comprising an image detection result comparison module that uses the first image detection result as the reference detection result, A comparison type determination module configured to determine the comparison type based on the comparison results, wherein the comparison type includes true positive examples, false positive examples and false negative examples, An evaluation score determination module configured to determine the evaluation score of the plurality of initial autonomous driving images based on the comparison type, The system includes a target autonomous driving image dataset selection module configured to select a target autonomous driving image dataset from the initial autonomous driving image dataset based on the evaluation scores of the plurality of initial autonomous driving images, and to adjust the tags in the target autonomous driving image dataset. Image dataset processing device.

17. At least one processor, A memory device configured to store at least one program, When the at least one program is executed by the at least one processor, the at least one processor implements the image dataset processing method according to any one of claims 1 to 15. electronic equipment.

18. When executed by a computer processor, the computer executable instructions include instructions for performing the image dataset processing method described in any one of claims 1 to 15. storage medium.

Images