Algorithm evaluation method, device, electronic device and medium

Abstract

The embodiments of the present invention provide an algorithm evaluation method, device, electronic device and medium, which relate to the field of computer technology. The method includes: using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set; using the first large model to process the specified data set to obtain a second processing result set; determining a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set; determining a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set; comparing the first missed detection rate with the second missed detection rate, and / or, comparing the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested. The method can accurately evaluate the performance of the algorithm and eliminate evaluation errors caused by data quality, data labeling and other reasons.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an algorithm evaluation method, device, electronic equipment and medium. Background Art

[0002] An algorithm is a description of the steps to solve a specific problem, which is represented as an ordered sequence of instruction sets in a computer. Before applying an algorithm, it is necessary to further evaluate whether the performance of the algorithm meets the requirements or to evaluate multiple algorithms to select the search algorithm with the best performance. Traditional algorithm evaluation methods usually analyze the design principles of the algorithm and evaluate the performance of the algorithm based on the performance of software and hardware. For example, based on the selected test strategy, the evaluation of hardware device indicators (such as memory and other resource utilization, power consumption) and algorithm indicators (such as accuracy) is completed. However, this method cannot accurately locate the evaluation dimension. For example, the poor effect of algorithm evaluation indicators may be a problem with the quality of the data set, or a problem with the scene, an annotation, or an algorithm. These cannot be effectively confirmed, which leads to the possibility that the indicator may cause evaluation bias of the algorithm (thinking that the algorithm is poorly effective). Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, an embodiment of the present invention provides an algorithm evaluation method, device, electronic device and medium.

[0004] In a first aspect, an embodiment of the present invention provides an algorithm evaluation method, comprising:

[0005] Using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set;

[0006] Using a preset first large model to process each data sample in the specified data set to obtain a second processing result set;

[0007] Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set;

[0008] Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set;

[0009] The first missed detection rate is compared with the second missed detection rate, and / or the first false detection rate is compared with the second false detection rate to determine the performance of the algorithm to be tested.

[0010] Optionally, comparing the first missed detection rate with the second missed detection rate, and / or comparing the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested includes:

[0011] Inputting the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate as input data into a preset second large model, so that the second large model analyzes the input data based on a preset prompt word;

[0012] Obtain a performance analysis report output by the second largest model, wherein the performance analysis report includes performance information, existing problems, and adjustment suggestions of the algorithm to be tested.

[0013] Optionally, when the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the first processing result set includes a first target recognition frame, the second processing result set includes a second target recognition frame, and the label information of each data sample in the specified data set includes a target calibration frame;

[0014] Determining a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set includes:

[0015] For an image corresponding to each processing result of the second processing result set, calculating a first intersection-over-union ratio between a second target recognition frame and a target calibration frame corresponding to the image;

[0016] When the first intersection-to-combination ratio is less than or equal to a third preset threshold, the first missed detection number is increased by 1;

[0017] After traversing the images corresponding to each processing result of the second processing result set, calculating a first missed detection rate according to the first missed detection number and the total number of the second target recognition frames;

[0018] and / or,

[0019] For each image in the specified data set, calculating a second intersection-over-union ratio between a target calibration frame corresponding to the image and the second target recognition frame;

[0020] When the second intersection-over-combination ratio is less than or equal to a third preset threshold, the first false detection number is increased by 1;

[0021] When all images in the designated data set are traversed, a first false detection rate is calculated according to the first false detection number and the total number of target calibration frames.

[0022] Optionally, determining a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set includes:

[0023] For an image corresponding to each processing result of the second processing result set, calculating a third intersection-over-union ratio between a second target recognition frame and a first target recognition frame corresponding to the image;

[0024] When the third intersection-to-combination ratio is less than or equal to the third preset threshold, the second missed detection number is increased by 1;

[0025] After traversing the images corresponding to each processing result of the second processing result set, calculating a second missed detection rate according to the second missed detection number and the total number of the second target recognition frames;

[0026] and / or,

[0027] For each image in the specified data set, calculating a fourth intersection-over-union ratio between a first target recognition frame and a second target recognition frame corresponding to the image;

[0028] When the fourth intersection-over-combination ratio is less than or equal to the third preset threshold, the second false detection number is increased by 1;

[0029] When all images in the designated data set are traversed, a second false detection rate is calculated according to the second false detection number and the total number of the first target recognition frames.

[0030] Optionally, the method further comprises: determining an evaluation index value according to the first processing result set and the label information of each data sample in the specified data set, wherein the evaluation index comprises one or more of the following: accuracy, recall, false alarm rate and third false negative rate;

[0031] The comparing of the first missed detection rate with the second missed detection rate, and / or the comparing of the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested includes: comparing the first missed detection rate with the second missed detection rate, and / or the comparing of the first false detection rate with the second false detection rate, to obtain a comparison result; and determining the performance of the algorithm to be tested based on the evaluation index value and the comparison result.

[0032] Optionally, the method further includes: when there are multiple designated data sets, screening the multiple designated data sets according to the first preset threshold and the evaluation index value corresponding to each designated data set;

[0033] The method of using the preset first large model to process each data sample in the specified data set to obtain a second processing result set includes: using the preset first large model to process each data sample in the screened specified data set to obtain a second processing result set.

[0034] Optionally, the using a preset first large model to process each data sample in the specified data set to obtain a second processing result set includes:

[0035] In the case where the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the algorithm category information corresponding to the algorithm to be tested and each data sample in the specified data set are input as input data into a preset first large model to perform image-text similarity calculation, and obtain data samples whose image-text similarity is greater than a second preset threshold value;

[0036] The preset first large model is used to process data samples whose image-text similarity is greater than a second preset threshold to obtain a second processing result set.

[0037] In a second aspect, an embodiment of the present invention provides an algorithm evaluation device, including:

[0038] A testing module, used to process each data sample in a specified data set using the algorithm to be tested to obtain a first processing result set;

[0039] A reference module, used to process each data sample in the specified data set using a preset first large model to obtain a second processing result set;

[0040] A calculation module, used to determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set; and determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set;

[0041] A comparison module is used to compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to determine the performance of the algorithm to be tested.

[0042] Optionally, the comparison module is used to: input the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate as input data into a preset second large model, so that the second large model analyzes the input data based on preset prompt words; obtain a performance analysis report output by the second large model, the performance analysis report including performance information of the algorithm to be tested, existing problems and adjustment suggestions.

[0043] Optionally, when the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the first processing result set includes a first target recognition frame, the second processing result set includes a second target recognition frame, and the label information of each data sample in the specified data set includes a target calibration frame;

[0044] The calculation module is used to: for each image corresponding to the processing result of the second processing result set, calculate the first intersection-and-union ratio between the second target recognition box corresponding to the image and the target calibration box; when the first intersection-and-union ratio is less than or equal to a third preset threshold, increase the first missed detection number by 1; after traversing the images corresponding to each processing result of the second processing result set, calculate the first missed detection rate according to the first missed detection number and the total number of the second target recognition boxes; and / or, for each image in the specified data set, calculate the second intersection-and-union ratio between the target calibration box corresponding to the image and the second target recognition box; when the second intersection-and-union ratio is less than or equal to the third preset threshold, increase the first false detection number by 1; after traversing the images in the specified data set, calculate the first false detection rate according to the first false detection number and the total number of the target calibration boxes.

[0045] Optionally, the calculation module is used to: for each image corresponding to the processing result of the second processing result set, calculate the third intersection-and-union ratio between the second target recognition box corresponding to the image and the first target recognition box; when the third intersection-and-union ratio is less than or equal to the third preset threshold, increase the second missed detection number by 1; after traversing the images corresponding to each processing result of the second processing result set, calculate the second missed detection rate based on the second missed detection number and the total number of the second target recognition boxes; and / or, for each image in the specified data set, calculate the fourth intersection-and-union ratio between the first target recognition box corresponding to the image and the second target recognition box; when the fourth intersection-and-union ratio is less than or equal to the third preset threshold, increase the second false detection number by 1; after traversing the images in the specified data set, calculate the second false detection rate based on the second false detection number and the total number of the first target recognition boxes.

[0046] Optionally, the device further includes an indicator determination module, which is used to: determine an evaluation indicator value according to the first processing result set and the label information of each data sample in the specified data set, wherein the evaluation indicator includes one or more of the following: accuracy, recall rate, false alarm rate and third false negative rate;

[0047] The comparison module is also used to: compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to obtain a comparison result; and determine the performance of the algorithm to be tested based on the evaluation index value and the comparison result.

[0048] Optionally, the device further comprises a screening module, which is used to: when there are multiple designated data sets, screen the multiple designated data sets according to the first preset threshold and the evaluation index value corresponding to each designated data set;

[0049] The reference module is used to: use a preset first large model to process each data sample in the screened designated data set to obtain a second processing result set.

[0050] Optionally, the reference module is used to: when the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, input the algorithm category information corresponding to the algorithm to be tested and each data sample in the specified data set as input data into a preset first large model to perform image-text similarity calculation to obtain data samples whose image-text similarity is greater than a second preset threshold; use the preset first large model to process the data samples whose image-text similarity is greater than the second preset threshold to obtain a second processing result set.

[0051] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the algorithm evaluation method provided by any embodiment of the present invention when executing the program stored in the memory.

[0052] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the algorithm evaluation method provided by any embodiment of the present invention.

[0053] The technical solution provided by the embodiment of the present invention brings at least the following beneficial effects:

[0054] The algorithm evaluation method provided by the embodiment of the present invention uses the algorithm to be tested and a preset first large model to process each data sample in a specified data set, respectively, to obtain a first processing result set and a second processing result set, and determines a first missed detection rate and / or a first false detection rate based on the second processing result set and the label information of each data sample in the specified data set; determines a second missed detection rate and / or a second false detection rate based on the first processing result set and the second processing result set, compares the first missed detection rate with the second missed detection rate, and / or compares the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested. The method uses a first large model to process a specified data set to obtain a first missed detection rate and / or a first false detection rate. The first missed detection rate and / or the first false detection rate can be used to indicate the quality of the specified data set, and can eliminate evaluation errors caused by problems such as data quality and data labeling, and can accurately locate the root cause of the problem of good or bad evaluation indicators (data quality, data scenario, labeling or algorithm problem); the performance of the algorithm to be tested is determined according to the comparison result of the first missed detection rate and the second missed detection rate, and / or the performance of the algorithm to be tested is determined according to the comparison result of the first false detection rate and the second false detection rate, so as to evaluate the performance of the algorithm to be tested with the first large model as a reference, and can accurately evaluate the difference between the algorithm to be tested and the first large model, and then accurately evaluate the performance of the algorithm to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0056] Figure 1 A schematic diagram showing a flow chart of an algorithm evaluation method according to an embodiment of the present invention;

[0057] Figure 2 A schematic diagram showing a flow chart of an algorithm evaluation method according to another embodiment of the present invention;

[0058] Figure 3 A schematic diagram showing a flow chart of an algorithm evaluation method according to another embodiment of the present invention;

[0059] Figure 4 A schematic flow chart showing an algorithm evaluation method according to another embodiment of the present invention is shown;

[0060] Figure 5 A schematic diagram showing a flow chart of an algorithm evaluation device according to an embodiment of the present invention is shown;

[0061] Figure 6 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0062] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0063] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0064] Figure 1 FIG. 1 is a flow chart showing the main steps of the algorithm evaluation method provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] Step S101: using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set.

[0066] Among them, the algorithm to be tested can be any algorithm whose performance needs to be verified. The specified data set corresponds to the algorithm to be tested. For example, if the algorithm to be tested is an image processing algorithm (such as an image detection algorithm, an image recognition algorithm, an image classification algorithm, etc.), each data sample in the specified data set is an image including a detection target, and the processing results in the first processing result set are, for example, targets detected by the algorithm to be tested (such as target boxes predicted by the algorithm to be tested), target categories, etc. If the algorithm to be tested is a text processing algorithm (such as a summary generation algorithm, a text sentiment analysis algorithm, a text translation algorithm, a text classification algorithm, etc.), each data sample in the specified data set is text data, and the processing results in the first processing result set are, for example, generated summaries, sentiment tendencies, translation data, text categories, etc.

[0067] Step S102: using a preset first large model to process each data sample in a specified data set to obtain a second processing result set.

[0068] Among them, the large model refers to a machine learning model with large-scale parameters and complex computing structure. The preset first large model of this embodiment has the same function as the algorithm to be tested, that is, if the algorithm to be tested is an image processing algorithm, the first large model is also an image processing model. In this step, the first large model is a trained model with good performance and stable performance. The purpose of using the first large model to process each data sample in the specified data set is to use the first large model as a reference for evaluating the algorithm to be tested. For example, if the processing result of the algorithm to be tested on the specified data set is similar to the processing result of the first large model on the specified data set, it means that the performance of the algorithm to be tested is similar to the performance of the first large model. If the processing result of the algorithm to be tested on the specified data set is significantly different from the processing result of the first large model on the specified data set, it means that the performance of the algorithm to be tested is significantly different from the performance of the first large model.

[0069] Step S103: Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set.

[0070] Among them, the label information of each data sample in the specified data set is a manually annotated label. For example, if the data sample is an image, the label information of the data sample can be the category of the image or the category of the target detected on the image, etc. If the data sample is a face image, the label can include a face ID and a manually annotated face frame. The missed detection rate refers to the percentage of the number of unqualified products not found in the detection process to the actual number of unqualified products. The false detection rate refers to the percentage of the number of qualified products mistakenly detected as unqualified products to the actual number of qualified products. Taking the first large model as an image processing model as an example, the first missed detection rate is the percentage of the number of non-target images not found by the first large model during the processing process to the actual number of non-target images, and the first false detection rate is the percentage of the number of target images mistakenly identified as non-target images by the first large model during the processing process to the actual number of target images.

[0071] In the embodiment of the present invention, since the label information of each data sample in the specified data set is manually annotated, there may be data samples with incorrect or unlabeled labels. The first missed detection rate and / or the first false positive rate can be used to indicate the quality of the specified data set. If the first missed detection rate is large, it indicates that there are many unlabeled images in the specified data set. If the first false positive rate is large, it indicates that there are many images with incorrect labels in the specified data set.

[0072] Step S104: Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set.

[0073] In this step, the second processing result set of the first large model on the specified data set is used as the basis to calculate the missed detection rate and the false detection rate corresponding to the algorithm to be tested.

[0074] Step S105: Compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to determine the performance of the algorithm to be tested.

[0075] For example, if the difference between the first missed detection rate and the second missed detection rate is within the first range, it means that the performance of the algorithm to be tested is similar to that of the first large model. If the first missed detection rate is large (if the first missed detection rate is large, it means that there are many unlabeled images in the specified data set), the second missed detection rate is also large, and the reason why the result of the algorithm to be tested on the specified data set is poor is that there are many unlabeled images in the specified data set. If the difference between the first missed detection rate and the second missed detection rate exceeds the first range (for example, the first missed detection rate is large, the second missed detection rate is small, or the first missed detection rate is small, the second missed detection rate is large), it means that the performance of the algorithm to be tested is greatly different from that of the first large model, and the performance of the algorithm to be tested is poor. The conclusion of comparing the first false positive rate with the second false positive rate is similar to the conclusion of comparing the first missed detection rate with the second missed detection rate, and the present invention will not be repeated here.

[0076] The algorithm evaluation method provided by the embodiment of the present invention uses the first large model to process the specified data set to obtain a first missed detection rate and / or a first false detection rate. The first missed detection rate and / or the first false detection rate can be used to indicate the quality of the specified data set, and can eliminate evaluation errors caused by problems such as data quality and data labeling, and can accurately locate the root cause of the problem of good or bad evaluation indicators (data quality, data scenario, labeling or algorithm problem); determine the performance of the algorithm to be tested based on the comparison result of the first missed detection rate and the second missed detection rate, and / or determine the performance of the algorithm to be tested based on the comparison result of the first false detection rate and the second false detection rate, so as to evaluate the performance of the algorithm to be tested with the first large model as a reference, and can accurately evaluate the difference between the algorithm to be tested and the first large model, and then accurately evaluate the performance of the algorithm to be tested.

[0077] Figure 2 FIG. 2 is a flow chart of an algorithm evaluation method provided by another embodiment of the present invention. Figure 2 As shown, the method includes:

[0078] Step S201: using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set.

[0079] Step S202: Use the preset first large model to process each data sample in the specified data set to obtain a second processing result set.

[0080] Step S203: Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set.

[0081] Step S204: Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set.

[0082] Step S205: Input the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate as input data into the preset second largest model, so that the second largest model analyzes the input data based on preset prompt words, and obtains a performance analysis report output by the second largest model, wherein the performance analysis report includes performance information of the algorithm to be tested, existing problems, and adjustment suggestions.

[0083] Among them, steps S201-S204 can refer to Figure 1 To avoid repetition, the present invention will not be described in detail in the embodiments shown.

[0084] For step S205, the large model prompt is a text or instruction used when interacting with the large model, which is intended to guide the model to generate a specific type of output. The prompt helps the model better understand the user's intentions and needs by providing information such as background, goals, style, etc., so as to generate content that meets expectations. The prompt can be designed according to the structure of roles, tasks, requirements, and prompts. As an optional example, the prompt can include the following content:

[0085] You are an intelligent assistant. Please refer to the conclusions given in the example, analyze the various input parameters, and then identify the quality of the specified data set, evaluate the performance of the algorithm to be tested, and give adjustment suggestions.

[0086] Example:

[0087] Input parameters: xxxxx;

[0088] Output conclusion: xxxxxx.

[0089] In this step, after the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate are input as input data into the preset second largest model, the second largest model analyzes the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate according to the prompt word, and outputs a performance analysis report. Optionally, the performance analysis report can be generated in a word format, and the generated name is in the format of timestamp-title.

[0090] The algorithm evaluation algorithm provided in the embodiment of the present invention uses the first large model to process the specified data set to obtain the first missed detection rate and / or the first false detection rate. The first missed detection rate and / or the first false detection rate can be used to indicate the quality of the specified data set, and can eliminate evaluation errors caused by problems such as data quality and data labeling; the performance of the algorithm to be tested is determined according to the comparison result of the first missed detection rate and the second missed detection rate, and / or the performance of the algorithm to be tested is determined according to the comparison result of the first false detection rate and the second false detection rate, so as to evaluate the performance of the algorithm to be tested with the first large model as a reference, accurately evaluate the difference between the algorithm to be tested and the first large model, and then accurately evaluate the performance of the algorithm to be tested; the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate are automatically analyzed by the second large model to generate a performance analysis report, thereby improving the evaluation efficiency, and the performance analysis results and rationalization suggestions of the algorithm to be tested are given, which provide a certain reference for the improvement of the subsequent algorithm and save manpower.

[0091] To facilitate understanding of the algorithm evaluation method provided by an embodiment of the present invention, the following is an example in which the algorithm to be tested is an image processing algorithm (such as a face detection algorithm, a vehicle recognition algorithm) and the first large model is an image processing model. The label information of each data sample in the specified data set includes a target calibration frame (i.e., a GT frame (ground truth frame), a manually calibrated label used to indicate the true position and shape of the target object in the image). The result set obtained by the algorithm to be tested processing each data sample in the specified data set includes a first target recognition frame. The result obtained by the first large model processing each data sample in the specified data set includes a second target recognition frame.

[0092] After obtaining the first target recognition frame and the second target recognition frame, the first missed detection rate and the first false detection rate are calculated. Figure 3 As shown, the process of calculating the first missed detection rate and the first false detection rate includes:

[0093] Step S301: for an image corresponding to each processing result of the second processing result set, a first intersection-over-union ratio between a second target recognition frame and a target calibration frame corresponding to the image is calculated.

[0094] Step S302: when the first intersection-over-combination ratio is less than or equal to a third preset threshold, the first missed detection number is increased by 1.

[0095] Step S303: After traversing the images corresponding to each processing result of the second processing result set, the first missed detection rate is calculated according to the first missed detection number and the total number of second target recognition frames. For example, as shown in the following formula (1):

[0096]

[0097] Step S304: For each image in the specified data set, a second intersection-over-union ratio between the target calibration frame corresponding to the image and the second target recognition frame is calculated.

[0098] Step S305: when the second intersection-over-combination ratio is less than or equal to the third preset threshold, the first false detection number is increased by 1.

[0099] Step S306: After traversing all images in the specified data set, calculate the first false detection rate according to the first false detection number and the total number of target calibration frames. For example, as shown in the following formula (2):

[0100]

[0101] Among them, the intersection over union (IoU) is an important metric in object detection and segmentation tasks. It represents the ratio of the intersection area of ​​two bounding boxes (or segmentation masks) to their union area. The value range of the intersection over union is between 0 and 1. The larger the value, the higher the overlap between the two bounding boxes. When IoU is 1, it means that the two bounding boxes overlap completely; when IoU is 0, it means that the two bounding boxes do not overlap at all.

[0102] The process of calculating the second missed detection rate and the second false detection rate is similar to the process of calculating the first missed detection rate and the first false detection rate, for example including: for each image corresponding to the processing result of the second processing result set, calculating the third intersection-and-union ratio between the second target recognition frame corresponding to the image and the first target recognition frame; when the third intersection-and-union ratio is less than or equal to the third preset threshold, the second missed detection number is increased by 1; after traversing the images corresponding to each processing result of the second processing result set, the second missed detection rate is calculated based on the second missed detection number and the total number of second target recognition frames; and / or, for each image in the specified data set, calculating the fourth intersection-and-union ratio between the first target recognition frame corresponding to the image and the second target recognition frame; when the fourth intersection-and-union ratio is less than or equal to the third preset threshold, the second false detection number is increased by 1; after traversing the images in the specified data set, the second false detection rate is calculated based on the second false detection number and the total number of first target recognition frames.

[0103] After obtaining the first missed detection rate, the first false detection rate, the second missed detection rate, and the second false detection rate, the first missed detection rate, the first false detection rate, the second missed detection rate, and the second false detection rate are input into the second largest model, a performance analysis report output by the second largest model is obtained, and the performance analysis results and adjustment suggestions of the algorithm to be tested are determined.

[0104] In an optional embodiment, the algorithm testing algorithm further includes: determining an evaluation index value according to the label information of each data sample in the first processing result set and the specified data set, the evaluation index including one or more of the following: accuracy, recall, false alarm rate and third false negative rate; comparing the first false negative rate with the second false negative rate, and / or comparing the first false positive rate with the second false positive rate to obtain a comparison result; determining the performance of the algorithm to be tested according to the evaluation index value and the comparison result. Among them, the third false negative rate is determined according to the target calibration frame and the first target recognition frame.

[0105] The algorithm evaluation method of the embodiment of the present invention combines the comparison results of the first missed detection rate and the second missed detection rate (and / or the comparison results of the first false detection rate and the second false detection rate) with the evaluation index value to evaluate the performance of the algorithm to be tested, and can provide more accurate evaluation results.

[0106] In an optional embodiment, the number of designated data sets may be one or more. When the number of designated data sets is more than one, in order to improve the evaluation accuracy and evaluation efficiency, data sets with poor data quality may be screened out during the testing of the algorithm to be tested.

[0107] Figure 4 FIG. 2 is a flow chart of an algorithm evaluation method provided by another embodiment of the present invention. Figure 4 As shown, the method includes:

[0108] Step S401: determine the algorithm to be tested and the evaluation index, and select multiple specified data sets from multiple optional data sets.

[0109] Optionally, the user can select the algorithm to be tested, multiple specified data sets, and evaluation indicators through the visual interface. The algorithm to be tested can be, for example, a face recognition algorithm, a vehicle recognition algorithm, etc. The user can also select a self-uploaded custom algorithm as the algorithm to be tested. The uploaded custom algorithm needs to include the development kit of the entire inference code of the algorithm (including the model and the calling interface). The optional data set is a business scenario data set, and each data set includes multiple images and label information of each image. For example, each data set folder is divided into two folders, namely GT and images. The GT folder stores the true value label, which is json by default and supports txt format. Each image corresponds to 1 label. The images stored in the images folder are in formats such as jpg, png, jpeg, etc. When the algorithm to be tested and the first large model complete the processing of the specified data set, a pred folder will be generated. The pred folder includes an obj folder and a result folder. The result folder stores the result of drawing the prediction box (i.e., the first target recognition box or the second target recognition box) and the target calibration box (GT box (groundtruth box), a manually calibrated label used to indicate the true position and shape of the target object in the image) in a picture. The obj folder stores the prediction result information of each image in json format. Each file name in the obj folder is consistent with the predicted image name.

[0110] Evaluation indicators include, for example, precision-based evaluation indicators and performance-based evaluation indicators, such as accuracy, recall, false alarm rate, false negative rate, F1-score (F1 score, defined as the harmonic mean of precision and recall), etc. Users can select one or more evaluation indicators according to business scenarios. Optionally, mandatory evaluation indicators can also be set, such as setting accuracy, recall, false alarm rate, and false negative rate as mandatory evaluation indicators.

[0111] Step S402: Use the algorithm to be tested to process each data sample in the multiple specified data sets to obtain multiple first processing result sets. The first processing result set includes the evaluation index value corresponding to each specified data set, the first target recognition box corresponding to each data sample, and the total index result, wherein the total index result refers to the weighted average of each index.

[0112] Step S403: screening multiple designated data sets according to the first preset threshold and the evaluation index value corresponding to each designated data set.

[0113] In order to improve the accuracy and efficiency of algorithm evaluation, multiple specified data sets can be screened according to the evaluation index values ​​corresponding to each specified data set. For example, the specified data sets corresponding to the accuracy, recall or F1-score less than the first preset threshold can be screened out, and the specified data sets with the accuracy, recall or F1-score greater than or equal to the first preset threshold are retained.

[0114] Step S404: input the algorithm category information corresponding to the algorithm to be tested and each data sample in the specified data set as input data into a preset first large model to calculate the image-text similarity, and obtain data samples whose image-text similarity is greater than or equal to a second preset threshold.

[0115] Use the first large model to take the algorithm category information as input (such as face capture, vehicle recognition, etc.), and calculate the image and text similarity for each data set respectively (the image target information will be output in json format when extracting image features, and each target information will be saved separately). A folder will be automatically generated for different data sets. The folder is named after the data set name. The folder only saves images whose image and text similarity is greater than or equal to the second preset threshold, and does not save images whose text-image similarity is lower than the second preset threshold.

[0116] Since there may be unlabeled images or incorrectly labeled images in the specified data set, in order to ensure the accuracy of the subsequent calculation of the missed detection rate and the false detection rate, the first large model is first used to screen the images in the specified data set, and the images whose image-text similarity is less than the second preset threshold are screened out, and only the images whose image-text similarity is greater than or equal to the second preset threshold are retained. As an optional example, the second preset threshold is 90%.

[0117] Step S405: Use the preset first large model to process data samples whose image-text similarity is greater than or equal to the second preset threshold to obtain a second processing result set. The second processing result set includes the second target recognition box corresponding to each image in each screened designated data set.

[0118] Step S406: Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set.

[0119] This step performs false positive and false negative analysis. For each data set’s target calibration frame and the second target recognition frame identified by the first large model, IOU index statistics are performed. The index statistics are based on the result of the first large model (theoretically, the false positive rate and false negative rate are extremely low, almost 0, when the index is calculated between the second target recognition frame identified by the large model and the target calibration frame). The specific process is as follows:

[0120] (1) Number of missed detections: Traverse each image of the output result of the first large model, and obtain the second target recognition box and target calibration box corresponding to the image according to the image name. For each second target recognition box recognized by the first large model, perform IOU calculation between it and the corresponding target calibration box. If the IOU value is greater than or equal to the third preset threshold, the number of missed detections remains unchanged. If the IOU value is less than the third preset threshold, the number of missed detections is increased by 1.

[0121] (2) False positives: Traverse each image in the specified data set and calculate the IOU value between the target calibration frame and the second target recognition frame of the image. If the IOU value is greater than or equal to the third preset threshold, the false positives remain unchanged. If the IOU value is less than the third preset threshold, the false positives are increased by 1.

[0122] (3) Calculate the first missed detection rate and the first false detection rate. The first missed detection rate = the number of missed detections / the total number of second target recognition frames recognized by the first large model; the first false detection rate = the number of false detections / the total number of target calibration frames.

[0123] If the first missed detection rate is large, it indicates that there are many unlabeled images in the specified data set; if the first missed detection rate is small or 0, it indicates that there are some (or no) unlabeled images in the specified data set.

[0124] If the first false positive rate is large, it indicates that there are many image mislabelings in the specified data set; if the first false positive rate is small (or 0), it indicates that there are few (or no) image mislabelings in the specified data set.

[0125] In an embodiment of the present invention, the quality of each specified data set can be determined based on the first missed detection rate and the first false detection rate. For a specified data set, if the first missed detection rate and / or the first false detection rate corresponding to the specified data set is large, and the evaluation index value after the algorithm to be tested processes the specified data set, such as the accuracy rate, is low, it can be explained that the reason for the low evaluation index value of the algorithm to be tested may be poor data quality, rather than poor performance of the algorithm to be tested. If the first missed detection rate and / or the first false detection rate corresponding to the specified data set is small, and the evaluation index value after the algorithm to be tested processes the specified data set, such as the accuracy rate, is low, it can be explained that the reason for the low evaluation index value of the algorithm to be tested may be poor performance of the algorithm to be tested, rather than poor data quality.

[0126] Step S407: Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set.

[0127] In this step, the second target recognition frame recognized by the first large model is used as the basis to calculate the second missed detection rate and / or the second false detection rate corresponding to the algorithm to be tested. The specific process is as follows:

[0128] (1) Number of missed detections: Traverse each image of the output result of the first large model, and obtain the second target recognition box and the first target recognition box corresponding to the image according to the image name. For each second target recognition box recognized by the first large model, perform IOU calculation with the corresponding first target recognition box. If the IOU value is greater than or equal to the third preset threshold, the number of missed detections remains unchanged. If the IOU value is less than the third preset threshold, the number of missed detections is increased by 1.

[0129] (2) False positives: traverse each image in the specified data set and calculate the IOU value between the first target recognition box and the second target recognition box of the image. If the IOU value is greater than or equal to the third preset threshold, the false positives remain unchanged. If the IOU value is less than the third preset threshold, the false positives are increased by 1.

[0130] (3) Calculate the missed detection rate and false detection rate. The missed detection rate = the number of missed detections / the total number of second target recognition frames recognized by the first large model; the false detection rate = the number of false detections / the total number of first target recognition frames.

[0131] If the second missed detection rate is large, it indicates that there are many unlabeled images in the prediction results of the algorithm to be tested. If the second missed detection rate is small or 0, it indicates that there are some (or no) unlabeled images in the prediction results of the algorithm to be tested.

[0132] If the second false positive rate is larger, it indicates that there are more image error labels in the prediction results of the algorithm to be tested. If the second false positive rate is smaller (or 0), it indicates that there are fewer (or no) image error labels in the prediction results of the algorithm to be tested.

[0133] Step S408: Input the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate, and the evaluation index value as input data into the preset second largest model, so that the second largest model analyzes the input data based on the preset prompt words, and obtains a performance analysis report output by the second largest model, wherein the performance analysis report includes performance information of the algorithm to be tested, existing problems, and adjustment suggestions.

[0134] In this step, the first missed detection rate, the first false detection rate, the second missed detection rate, the second false detection rate, the evaluation index value, and the total evaluation index value are input as input data into the second largest model, and the second largest model is used to perform result analysis.

[0135] Optionally, the analysis process of the second largest model is as follows:

[0136] (1) The input content includes the title (optional) and the algorithm analysis results. The title is optional and will be automatically filled in if the second largest model is not provided. The algorithm analysis results include: the first processing result set corresponding to the algorithm to be tested and the second processing result set output by the first largest model. The input format can be online input or document txt format.

[0137] (2) After the content is entered, the second largest model algorithm will automatically add prompt words and respond to the input information. The response result can be generated in the form of word, with the generated name in the format of timestamp-title, and the generated result will be displayed on the interface.

[0138] The possible results of the algorithm result analysis are as follows: If the first missed detection rate or the first false detection rate corresponding to each screened data set is large, it means that there is a problem with the data set. If the first false detection rate is large, it indicates that there are more image errors in the specified data set; if the first false detection rate is small (or 0), it indicates that there are fewer (or no) image errors in the specified data set. If the first missed detection rate and / or the first false detection rate corresponding to the specified data set is large, and the evaluation index value after the algorithm to be tested processes the specified data set, such as the accuracy rate, is low, it can be explained that the reason for the low evaluation index value of the algorithm to be tested may be poor data quality, rather than poor performance of the algorithm to be tested. If the first missed detection rate and / or the first false detection rate corresponding to the specified data set is small, and the evaluation index value after the algorithm to be tested processes the specified data set, such as the accuracy rate, is low, it can be explained that the reason for the low evaluation index value of the algorithm to be tested may be poor performance of the algorithm to be tested, rather than poor data quality. If the second false detection rate is similar to the first false detection rate, it means that the performance of the algorithm to be tested is similar to that of the first large model.

[0139] The algorithm evaluation method provided by the embodiment of the present invention uses the first large model to process the specified data set to obtain a first missed detection rate and / or a first false detection rate. The first missed detection rate and / or the first false detection rate can be used to indicate the quality of the specified data set, and can eliminate evaluation errors caused by problems such as data quality and data labeling, and can accurately locate the root cause of the problem of good or bad evaluation indicators (data quality, data scenario, labeling or algorithm problem); determine the performance of the algorithm to be tested based on the comparison result of the first missed detection rate and the second missed detection rate, and / or determine the performance of the algorithm to be tested based on the comparison result of the first false detection rate and the second false detection rate, so as to evaluate the performance of the algorithm to be tested with the first large model as a reference, and can accurately evaluate the difference between the algorithm to be tested and the first large model, and then accurately evaluate the performance of the algorithm to be tested.

[0140] Figure 5 A schematic diagram of the structure of an algorithm evaluation device provided by an embodiment of the present invention is provided. Figure 5 As shown, the algorithm evaluation device 500 includes:

[0141] The testing module 501 is used to process each data sample in the specified data set using the algorithm to be tested to obtain a first processing result set;

[0142] A reference module 502 is used to process each data sample in the specified data set using a preset first large model to obtain a second processing result set;

[0143] A calculation module 503 is used to determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set; and a second missed detection rate and / or a second false detection rate are determined according to the second processing result set and the first processing result set;

[0144] The comparison module 504 is used to compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to determine the performance of the algorithm to be tested.

[0145] Optionally, the comparison module is used to: input the first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate as input data into a preset second large model, so that the second large model analyzes the input data based on preset prompt words; obtain a performance analysis report output by the second large model, the performance analysis report including performance information of the algorithm to be tested, existing problems and adjustment suggestions.

[0146] Optionally, when the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the first processing result set includes a first target recognition frame, the second processing result set includes a second target recognition frame, and the label information of each data sample in the specified data set includes a target calibration frame;

[0147] The calculation module is used to: for each image corresponding to the processing result of the second processing result set, calculate the first intersection-and-union ratio between the second target recognition box corresponding to the image and the target calibration box; when the first intersection-and-union ratio is less than or equal to a third preset threshold, increase the first missed detection number by 1; after traversing the images corresponding to each processing result of the second processing result set, calculate the first missed detection rate according to the first missed detection number and the total number of the second target recognition boxes; and / or, for each image in the specified data set, calculate the second intersection-and-union ratio between the target calibration box corresponding to the image and the second target recognition box; when the second intersection-and-union ratio is less than or equal to the third preset threshold, increase the first false detection number by 1; after traversing the images in the specified data set, calculate the first false detection rate according to the first false detection number and the total number of the target calibration boxes.

[0148] Optionally, the calculation module is used to: for each image corresponding to the processing result of the second processing result set, calculate the third intersection-and-union ratio between the second target recognition box corresponding to the image and the first target recognition box; when the third intersection-and-union ratio is less than or equal to the third preset threshold, increase the second missed detection number by 1; after traversing the images corresponding to each processing result of the second processing result set, calculate the second missed detection rate based on the second missed detection number and the total number of the second target recognition boxes; and / or, for each image in the specified data set, calculate the fourth intersection-and-union ratio between the first target recognition box corresponding to the image and the second target recognition box; when the fourth intersection-and-union ratio is less than or equal to the third preset threshold, increase the second false detection number by 1; after traversing the images in the specified data set, calculate the second false detection rate based on the second false detection number and the total number of the first target recognition boxes.

[0149] Optionally, the device further includes an indicator determination module, which is used to: determine an evaluation indicator value according to the first processing result set and the label information of each data sample in the specified data set, wherein the evaluation indicator includes one or more of the following: accuracy, recall rate, false alarm rate and third false negative rate;

[0150] The comparison module is also used to: compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to obtain a comparison result; and determine the performance of the algorithm to be tested based on the evaluation index value and the comparison result.

[0151] Optionally, the device further comprises a screening module, which is used to: when there are multiple designated data sets, screen the multiple designated data sets according to the first preset threshold and the evaluation index value corresponding to each designated data set;

[0152] The reference module is used to: use a preset first large model to process each data sample in the screened designated data set to obtain a second processing result set.

[0153] Optionally, the reference module is used to: when the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, input the algorithm category information corresponding to the algorithm to be tested and each data sample in the specified data set as input data into a preset first large model to perform image-text similarity calculation to obtain data samples whose image-text similarity is greater than a second preset threshold; use the preset first large model to process the data samples whose image-text similarity is greater than the second preset threshold to obtain a second processing result set.

[0154] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the algorithm evaluation method provided by the embodiment of the present invention.

[0155] Figure 6 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present invention. Figure 6 As shown, the electronic device includes:

[0156] Processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602, memory 603 communicate with each other via communication bus 604.

[0157] Memory 603, used for storing computer programs;

[0158] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0159] Using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set;

[0160] Using a preset first large model to process each data sample in the specified data set to obtain a second processing result set;

[0161] Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; the first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set;

[0162] Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set;

[0163] The first missed detection rate is compared with the second missed detection rate, and / or the first false detection rate is compared with the second false detection rate to determine the performance of the algorithm to be tested.

[0164] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0165] The communication interface is used for communication between the above terminal and other devices.

[0166] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0167] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0168] In another embodiment provided by the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the algorithm evaluation method described in any one of the above embodiments.

[0169] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the algorithm evaluation method described in any one of the above embodiments.

[0170] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0171] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0172] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An algorithm evaluation method, characterized in that: include: Using the algorithm to be tested to process each data sample in the specified data set to obtain a first processing result set; Using a preset first large model to process each data sample in the specified data set to obtain a second processing result set; Determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; The first missed detection rate and / or the first false detection rate are used to indicate the quality of the specified data set; Determine a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set; The first missed detection rate is compared with the second missed detection rate, and / or the first false detection rate is compared with the second false detection rate to determine the performance of the algorithm to be tested.

2. The method according to claim 1, characterized in that The comparing the first missed detection rate with the second missed detection rate, and / or comparing the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested includes: The first missed detection rate and / or the first false detection rate, the second missed detection rate and / or the second false detection rate are input as input data into a preset second large model, so that the second large model analyzes the input data based on preset prompt words, and obtains a performance analysis report output by the second large model, wherein the performance analysis report includes performance information of the algorithm to be tested, existing problems and adjustment suggestions.

3. The method according to claim 1 or 2, characterized in that: In the case where the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the first processing result set includes a first target recognition frame, the second processing result set includes a second target recognition frame, and the label information of each data sample in the specified data set includes a target calibration frame; Determining a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set includes: For an image corresponding to each processing result of the second processing result set, calculating a first intersection-over-union ratio between a second target recognition frame and a target calibration frame corresponding to the image; When the first intersection-to-combination ratio is less than or equal to a third preset threshold, the first missed detection number is increased by 1; After traversing the images corresponding to each processing result of the second processing result set, calculating a first missed detection rate according to the first missed detection number and the total number of the second target recognition frames; and / or, For each image in the specified data set, calculating a second intersection-over-union ratio between a target calibration frame corresponding to the image and the second target recognition frame; When the second intersection-over-combination ratio is less than or equal to a third preset threshold, the first false detection number is increased by 1; When all images in the designated data set are traversed, a first false detection rate is calculated according to the first false detection number and the total number of target calibration frames.

4. The method according to claim 3, characterized in that Determining a second missed detection rate and / or a second false detection rate according to the second processing result set and the first processing result set includes: For an image corresponding to each processing result of the second processing result set, calculating a third intersection-over-union ratio between a second target recognition frame and a first target recognition frame corresponding to the image; When the third intersection-and-combination ratio is less than or equal to the third preset threshold, the second missed detection number is increased by 1; After traversing the images corresponding to each processing result of the second processing result set, calculating a second missed detection rate according to the second missed detection number and the total number of the second target recognition frames; and / or, For each image in the specified data set, calculating a fourth intersection-over-union ratio between a first target recognition frame and a second target recognition frame corresponding to the image; When the fourth intersection-over-combination ratio is less than or equal to the third preset threshold, the second false detection number is increased by 1; When all images in the designated data set are traversed, a second false detection rate is calculated according to the second false detection number and the total number of the first target recognition frames.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Determine an evaluation index value according to the first processing result set and the label information of each data sample in the specified data set, wherein the evaluation index includes one or more of the following: accuracy, recall, false alarm rate, and third false negative rate; The comparing the first missed detection rate with the second missed detection rate, and / or comparing the first false detection rate with the second false detection rate, to determine the performance of the algorithm to be tested includes: Comparing the first missed detection rate with the second missed detection rate, and / or comparing the first false detection rate with the second false detection rate to obtain a comparison result; The performance of the algorithm to be tested is determined according to the evaluation index value and the comparison result.

6. The method according to claim 5, characterized in that The method further comprises: In the case where there are multiple designated data sets, screening the multiple designated data sets according to the first preset threshold and the evaluation index value corresponding to each designated data set; The method of using the preset first large model to process each data sample in the specified data set to obtain a second processing result set includes: The preset first large model is used to process each data sample in the screened designated data set to obtain a second processing result set.

7. The method according to claim 1, characterized in that The method of using the preset first large model to process each data sample in the specified data set to obtain a second processing result set includes: In the case where the algorithm to be tested is an image recognition algorithm and each data sample in the specified data set is an image, the algorithm category information corresponding to the algorithm to be tested and each data sample in the specified data set are input as input data into a preset first large model to perform image-text similarity calculation, and obtain data samples whose image-text similarity is greater than a second preset threshold value; The preset first large model is used to process data samples whose image-text similarity is greater than a second preset threshold to obtain a second processing result set.

8. An algorithm evaluation device, characterized in that: include: A testing module, used to process each data sample in a specified data set using the algorithm to be tested to obtain a first processing result set; A reference module, used to process each data sample in the specified data set using a preset first large model to obtain a second processing result set; A calculation module, used to determine a first missed detection rate and / or a first false detection rate according to the second processing result set and the label information of each data sample in the specified data set; The first missed detection rate and / or the first false positive rate are used to indicate the quality of the specified data set; a second missed detection rate and / or a second false positive rate are determined according to the second processing result set and the first processing result set; A comparison module is used to compare the first missed detection rate with the second missed detection rate, and / or compare the first false detection rate with the second false detection rate to determine the performance of the algorithm to be tested.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images