Automated model quality assessment method, device, equipment and storage medium

Abstract

The present application discloses an automated model quality assessment method, apparatus, device and storage medium, wherein the automated model quality assessment method comprises: obtaining a model file of a model to be assessed, wherein the model file comprises model prediction results and sample labels; recommending an optimal detection threshold based on the model prediction results and the sample labels; calculating the scoring index of the model to be assessed under the optimal detection threshold, and generating a quality assessment report corresponding to the scoring index. The present application belongs to the field of computer technology, and the optimal detection threshold is calculated through the model prediction results and sample labels under the model file of the model to be assessed, rather than selecting the detection threshold by manual setting, and the various evaluation indicators of the model to be assessed are calculated with the optimal detection threshold, so as to realize the quality assessment of the inference model, and improve the quality assessment accuracy and evaluation efficiency of the inference model.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an automated model quality assessment method, device, equipment and storage medium. Background Art

[0002] At present, with the development of deep learning technology, image detection models are widely used in the field of intelligent detection, such as call gesture recognition, outdoor human and vehicle detection, etc. In this field, image detection models are often required to have high accuracy, so how to accurately evaluate the quality of image detection inference models is a technical problem that needs to be solved urgently.

[0003] In related technologies, in different model scenarios, various evaluation indicators of the inference model to be evaluated are calculated through manually set detection thresholds to achieve quality evaluation of the inference model. However, the manually set detection threshold in this method is usually not the optimal detection threshold in the model scenario, so there are errors in the corresponding calculated evaluation indicators, resulting in low accuracy of quality evaluation of the inference model. Summary of the invention

[0004] The main purpose of this application is to provide an automated model quality assessment method, device, equipment and storage medium, aiming to solve the technical problem of low quality assessment accuracy of reasoning models in the prior art.

[0005] To achieve the above objectives, the present application provides an automated model quality assessment method, the automated model quality assessment method comprising:

[0006] Obtain a model file of the model to be evaluated, wherein the model file includes model prediction results and sample labels;

[0007] Based on the model prediction results and the sample labels, recommend an optimal detection threshold;

[0008] Calculate the scoring index of the model to be evaluated under the optimal detection threshold, and generate a quality assessment report corresponding to the scoring index.

[0009] The step of recommending an optimal detection threshold based on the model prediction result and the sample label includes:

[0010] Get the detection threshold set to be traversed;

[0011] Based on the model prediction results and the sample labels, a confusion matrix of each detection threshold under the detection threshold set to be traversed is calculated;

[0012] Based on the confusion matrix of each detection threshold, an optimal detection threshold is recommended.

[0013] Optionally, the confusion matrix includes a first sample classification category, the first sample classification category refers to a negative sample detection that is correctly predicted as a negative sample, and a negative sample refers to a target that does not need to be detected in an image detection task; the negative sample includes an image background, and when the image background is correctly predicted as a negative sample, the value under the first sample classification category in the confusion matrix remains unchanged.

[0014] Optionally, the step of calculating the confusion matrix of each detection threshold under the set of detection thresholds to be traversed based on the model prediction result and the sample label includes:

[0015] Determine a first target labeling result corresponding to the model prediction result, and determine a second target labeling result corresponding to the sample label;

[0016] Based on the detection threshold set to be traversed, the first target labeling result and the second target labeling result are compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed;

[0017] Based on the comparison result, a confusion matrix of each detection threshold under the detection threshold set to be traversed is formed.

[0018] Optionally, the step of comparing the first target labeling result with the second target labeling result based on the detection threshold set to be traversed to obtain the comparison results of each detection threshold under the detection threshold set to be traversed includes:

[0019] Based on the detection threshold set to be traversed, determine a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein the positive sample refers to a target that needs to be detected in the image detection task, and the negative sample refers to a target that does not need to be detected in the image detection task;

[0020] The first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed are matched and compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed.

[0021] Optionally, the step of recommending an optimal detection threshold based on the confusion matrix of each detection threshold comprises:

[0022] Calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set;

[0023] The highest comprehensive score is selected from the comprehensive score set, and the detection threshold corresponding to the highest comprehensive score is used as the optimal detection threshold.

[0024] Optionally, the step of calculating the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set includes:

[0025] The confusion matrix of each detection threshold is calculated by the comprehensive score calculation formula to obtain a comprehensive score set, wherein the comprehensive score calculation formula includes the following formula:

[0026]

[0027] in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

[0028] Optionally, the step of obtaining a model file of the model to be evaluated includes:

[0029] Obtaining a test set image and a sample label corresponding to the test set image;

[0030] The test set images are input into the model to be evaluated to obtain model prediction results, and a model file of the model to be evaluated is formed based on the model prediction results and the sample labels corresponding to the test set images.

[0031] Optionally, the step of generating a quality assessment report corresponding to the scoring indicator includes:

[0032] Parsing the model file to obtain hardware information of the model to be evaluated;

[0033] Based on the scoring index of the model to be evaluated, a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information is generated.

[0034] Optionally, the step of generating a quality assessment report corresponding to the scoring indicator includes:

[0035] Based on the model file, determine the MD5 information and version information of the model to be evaluated;

[0036] Based on the MD5 information and the version information, a model ID of the model to be evaluated is formed;

[0037] Based on the scoring index of the model to be evaluated and the model ID, a scoring trend result of the model to be evaluated and a quality assessment report corresponding to the scoring trend result are formed.

[0038] The present application also provides an automated model quality assessment device, the automated model quality assessment device comprising:

[0039] An acquisition module is used to acquire a model file of a model to be evaluated, wherein the model file includes a model prediction result and a sample label;

[0040] A recommendation module, used for recommending an optimal detection threshold based on the model prediction result and the sample label;

[0041] The calculation module is used to calculate the scoring index of the model to be evaluated under the optimal detection threshold and generate a quality evaluation report corresponding to the scoring index.

[0042] Optionally, the recommendation module includes:

[0043] A set acquisition module, used to acquire a set of detection thresholds to be traversed;

[0044] A matrix calculation module, used to calculate the confusion matrix of each detection threshold under the detection threshold set to be traversed based on the model prediction result and the sample label;

[0045] The comprehensive score calculation module is used to recommend an optimal detection threshold based on the confusion matrix of each detection threshold.

[0046] Optionally, the matrix calculation module includes:

[0047] A result determination module, used to determine a first target labeling result corresponding to the model prediction result, and to determine a second target labeling result corresponding to the sample label;

[0048] A comparison module, configured to compare the first target labeling result with the second target labeling result based on the detection threshold set to be traversed, and obtain a comparison result of each detection threshold under the detection threshold set to be traversed;

[0049] The confusion matrix forming module is used to form a confusion matrix of each detection threshold under the detection threshold set to be traversed based on the comparison result.

[0050] Optionally, the comparison module includes:

[0051] A sample determination module, configured to determine, based on the detection threshold set to be traversed, a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and to determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein a positive sample refers to a target that needs to be detected in an image detection task, and a negative sample refers to a target that does not need to be detected in an image detection task;

[0052] A matching module is used to match and compare the first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed, so as to obtain a comparison result of each detection threshold under the detection threshold set to be traversed.

[0053] Optionally, the comprehensive score calculation module includes:

[0054] A calculation module is used to calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set;

[0055] The selection module is used to select the highest comprehensive score from the comprehensive score set, and use the detection threshold corresponding to the highest comprehensive score as the optimal detection threshold.

[0056] Optionally, the calculation module includes:

[0057] The scoring formula calculation module is used to calculate the confusion matrix of each detection threshold through a comprehensive scoring calculation formula to obtain a comprehensive scoring set, wherein the comprehensive scoring calculation formula includes the following formula:

[0058]

[0059] in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

[0060] Optionally, the information determination module includes:

[0061] A test set acquisition module, used to acquire test set images and sample labels corresponding to the test set images;

[0062] The prediction module is used to input the test set images into the model to be evaluated, obtain the model prediction results, and form a model file of the model to be evaluated based on the model prediction results and the sample labels corresponding to the test set images.

[0063] Optionally, the calculation module includes:

[0064] A parsing module, used for parsing the model file to obtain the hardware information of the model to be evaluated;

[0065] The report generation module is used to generate a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information based on the scoring index of the model to be evaluated.

[0066] The present application also provides an automated model quality assessment device, the automated model quality assessment device comprising: a memory, a processor, and a program stored in the memory for implementing the automated model quality assessment method,

[0067] The memory is used to store a program for implementing an automated model quality assessment method;

[0068] The processor is used to execute a program for implementing the automatic model quality assessment method to implement the steps of the automatic model quality assessment method.

[0069] The present application also provides a storage medium, on which is stored a program for implementing the automated model quality assessment method, and the program for implementing the automated model quality assessment method is executed by a processor to implement the steps of the automated model quality assessment method.

[0070] This application calculates the optimal detection threshold through the model prediction results and sample labels under the model file of the model to be evaluated, rather than selecting the detection threshold by manual setting. The optimal detection threshold is used to calculate various evaluation indicators of the model to be evaluated, thereby realizing the quality evaluation of the inference model and improving the accuracy of the quality evaluation of the inference model. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0072] Figure 1 This is a flow chart of the first embodiment of the automatic model quality assessment method of the present application;

[0073] Figure 2 A schematic diagram showing the summary of scores of the model to be evaluated under different recommended thresholds for the automated model quality evaluation method of this application;

[0074] Figure 3 A step-by-step scoring diagram of the automated model quality assessment method for this application;

[0075] Figure 4 This is a schematic diagram of the modules of the automatic model quality assessment device of this application;

[0076] Figure 5 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present application.

[0077] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0078] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0079] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the automatic model quality assessment method of the present application.

[0080] In a first embodiment, the automated model quality assessment method comprises the following steps:

[0081] Step S100, obtaining a model file of a model to be evaluated, wherein the model file includes a model prediction result and a sample label;

[0082] It should be noted that the executor of the automated model quality assessment method is an automated model quality assessment device. Preferably, the automated model quality assessment device is a software client, or other terminals with data transmission and data processing functions, which are not specifically limited here.

[0083] It is understandable that the model to be evaluated refers to an inference model that is iteratively trained through a deep neural network learning method. In this application, an image detection inference model is preferred, specifically including but not limited to call gesture recognition, outdoor human and vehicle detection, etc.

[0084] Furthermore, the model file includes model prediction results and sample labels, wherein the model prediction results refer to the prediction results of the output corresponding to the input samples of the model to be evaluated. Since the model to be evaluated may have precision errors, the prediction results may be accurate or inaccurate; the sample labels refer to the labels corresponding to the manually annotated input samples. The results of the labels are correct results, that is, non-predicted, manually annotated results, which are accurate results from the machine's perspective.

[0085] Preferably, the model file of the model to be evaluated is a txt file, which includes but is not limited to hardware information, manually annotated test set image label file, test set image information and model prediction results, etc., wherein the hardware information refers to the information of the hardware used by the model to be evaluated.

[0086] In a specific implementation, the method for the device to obtain the model file of the model to be evaluated also includes the following steps:

[0087] The device obtains a test set image and a sample label corresponding to the test set image; inputs the test set image into the model to be evaluated to obtain a model prediction result, and forms a model file of the model to be evaluated based on the model prediction result and the sample label corresponding to the test set image.

[0088] It should be noted that the test set images refer to a set of samples used to perform accuracy testing on the model to be evaluated, including a certain number of image samples. The device obtains the model prediction results of the model to be evaluated by inputting the test set images into the model to be evaluated, and forms a model file of the model to be evaluated based on the model prediction results and the sample labels corresponding to the test set images.

[0089] Step S200, recommending an optimal detection threshold based on the model prediction result and the sample label;

[0090] It should be noted that the optimal detection threshold refers to the detection threshold that the device automatically recommends according to different model scenarios, rather than manually setting the prediction threshold of the model. Specifically, due to different evaluation requirements of different scenario models, this application proposes to calculate the optimal detection threshold of each model to be evaluated to achieve more accurate model quality evaluation.

[0091] In a specific implementation, the method for recommending an optimal detection threshold based on the model prediction result and the sample label also includes the following steps:

[0092] The device obtains a set of detection thresholds to be traversed; based on the model prediction results and the sample labels, calculates the confusion matrix of each detection threshold redefined under the detection threshold set to be traversed; based on the confusion matrix of each detection threshold, recommends the optimal detection threshold.

[0093] It should be noted that the detection threshold set to be traversed refers to a set of detection thresholds within a certain detection threshold range, and the device traverses the detection threshold set to select the optimal detection threshold that best suits the model scenario from the detection threshold set. Specifically, the device starts from the lowest detection threshold, increments by 0.01, and ends at the highest detection threshold, to obtain the detection threshold set to be traversed, wherein the lowest detection threshold and the highest detection threshold can be set by oneself.

[0094] It is understandable that, generally speaking, the confusion matrix, also known as the error matrix, is a standard format for expressing accuracy evaluation, expressed in the form of a matrix with n rows and n columns. Specific evaluation indicators include overall accuracy, mapping accuracy, user accuracy, etc. These accuracy indicators reflect the accuracy of image classification from different aspects. In artificial intelligence, the confusion matrix is ​​a visualization tool, especially for supervised learning, and is generally called a matching matrix in unsupervised learning. In image accuracy evaluation, it is mainly used to compare classification results with actual measured values, and the accuracy of classification results can be displayed in a confusion matrix. The confusion matrix is ​​calculated by comparing the position and classification of each measured pixel with the corresponding position and classification in the classified image.

[0095] For the model quality assessment method of the detection class, under the conditions of different detection thresholds of the detection threshold set to be traversed, according to the model prediction results and the sample labels, each detection threshold can calculate the corresponding confusion matrix. Specifically, in the detection class algorithm, the target that needs to be detected in the image detection task in the information in the drawing frame is used as a positive sample, and the target that does not need to be detected in the image detection task is used as a negative sample, and the confusion matrix is ​​statistically analyzed. For example, the target that needs to be detected of the model A to be evaluated is a pedestrian, and the remaining targets are negative samples. The picture X includes pedestrian Y1, vehicle Y2 and street lamp Y3, among which the detection frame of pedestrian Y1 is a positive sample, and the detection frames of vehicle Y2 and street lamp Y3 are negative samples.

[0096] It should be noted that the specific meaning of the confusion matrix is ​​as follows. For the convenience of description, a simple description is given with pedestrians as positive samples: TP: correctly predicted positive samples (i.e., people in the image are detected as people);

[0097] FP: non-positive samples are predicted as positive samples (that is, other objects in the image are predicted as people); FN: positive samples are detected as negative samples (people are detected as other things); TN: negative samples are correctly predicted as negative samples (non-human parts are not detected as people).

[0098] It should be noted that this application proposes a confusion matrix with a different definition from that in the related art, in which the statistics of the entire confusion matrix are not calculated in units of pictures, but in units of frames, and the picture background is taken as a special negative sample. Specifically, a person is taken as a positive sample. When a person is not detected in the background part, the TN value does not change, and when a person is not detected in the car part, the TN value increases. This application redefines the confusion matrix by taking the background as a negative sample, making it more suitable for detection model scenarios and improving the accuracy of model quality assessment.

[0099] In a specific implementation, the method for calculating the confusion matrix of each detection threshold under the detection threshold set to be traversed based on the model prediction result and the sample label also includes the following steps:

[0100] The device determines a first target labeling result corresponding to the model prediction result, and determines a second target labeling result corresponding to the sample label; based on the detection threshold set to be traversed, the first target labeling result is compared with the second target labeling result to obtain a comparison result of each detection threshold under the detection threshold set to be traversed; based on the comparison result, a confusion matrix of each detection threshold under the detection threshold set to be traversed is formed.

[0101] It should be noted that the first target labeling result corresponding to the model prediction result refers to the target detection box of each image in the test set image by the model to be evaluated, and the second target labeling result corresponding to the sample label refers to the target box of each manually labeled image.

[0102] In a specific implementation, the device compares the first target labeling result and the second target labeling result based on the detection threshold set to be traversed, and obtains the comparison results of each detection threshold under the detection threshold set to be traversed, wherein a confusion matrix of each detection threshold under the detection threshold set to be traversed is formed according to different comparison results. Specifically, the confusion matrix is ​​a comparison between the model prediction results and the manual labeling results. By comparing each box, all the box detections are counted to see whether the two are consistent, and the comparison result is the value in the confusion matrix. For example, compare the box information (exceeding the detection threshold) of all the labels output by the model in a picture and the manually annotated box information to see if they match. If they match, the corresponding TP will be +1, and if they do not match, the other values ​​of the confusion matrix will be increased by 1 according to the specific mismatch results.

[0103] In a specific implementation, the device compares the first target labeling result with the second target labeling result based on the detection threshold set to be traversed, and obtains the comparison result of each detection threshold under the detection threshold set to be traversed, and the method also includes the following steps:

[0104] The device determines, based on the detection threshold set to be traversed, a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and determines a second positive sample and a second negative sample for image detection in the second target annotation result, wherein a positive sample refers to a target that needs to be detected in an image detection task, and a negative sample refers to a target that does not need to be detected in an image detection task; matches and compares the first positive sample, the first negative sample, the second positive sample, and the second negative sample of each detection threshold under the detection threshold set to be traversed, and obtains a comparison result of each detection threshold under the detection threshold set to be traversed.

[0105] It should be noted that the present application proposes that the first negative sample and the second negative sample include the image background. Specifically, the calculation of the confusion matrix performed by this algorithm is slightly different from the official definition. The initial calculation of the confusion matrix is ​​for the classification algorithm, that is, the positive sample and the negative sample are not in the same picture, and then the classification statistics are performed. In the detection algorithm, the information in the box drawn by the category itself is the positive sample, and the different categories are negative samples for the confusion matrix statistics. In order to more accurately evaluate the false alarm situation of the model, this application also classifies the background as a negative sample, that is, the information in the non-prediction category box of a picture is all negative samples.

[0106] In a specific implementation, the device compares and matches the target detection frames including the first positive sample and the first negative sample about image detection in the first target labeling result, and the second positive sample and the second negative sample about image detection in the second target labeling result, that is, the positive sample and the negative sample, and performs incremental statistics on the initial confusion matrix under each corresponding detection threshold until the comparison results of each detection threshold under the detection threshold set to be traversed are obtained.

[0107] In a specific implementation, the method for recommending an optimal detection threshold based on the confusion matrix of each detection threshold also includes the following steps:

[0108] The device calculates the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set; selects the highest comprehensive score from the comprehensive score set, and uses the detection threshold corresponding to the highest comprehensive score as the optimal detection threshold.

[0109] It should be noted that the present application obtains the comprehensive score corresponding to the confusion matrix of each detection threshold through comprehensive score calculation, and selects the detection threshold corresponding to the highest comprehensive score as the optimal detection threshold, which means that the quality assessment accuracy of the inference model under this detection threshold is the highest.

[0110] In a specific implementation, the device calculates the comprehensive score corresponding to the confusion matrix of each detection threshold, and the method for obtaining the comprehensive score set also includes the following steps:

[0111] The device calculates the confusion matrix of each detection threshold through a comprehensive score calculation formula to obtain a comprehensive score set, wherein the comprehensive score calculation formula includes the following formula:

[0112]

[0113] in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

[0114] It should be noted that in order to meet the evaluation requirements of different scenario models, the model threshold automatically recommended by the system has a certain adaptability to the scenario information. Therefore, when calculating the comprehensive score, the evaluation model introduces a hyperparameter recommendation coefficient that is relevant to the scenario, taking into account the scenario requirements. ,in It is a weight information, and the value range of this coefficient is a closed interval from 0 to 1.

[0115] For example, when When the value is 0.5, the comprehensive scoring formula is the calculation formula of ACC, that is, the threshold value is recommended according to the ACC value of the model. If the value is 0 to 0.5, the recommended threshold value pays more attention to reducing the false alarm rate, and the lower the recommended coefficient, the more inclined to reduce the false alarm rate. On the contrary, if the recommended coefficient is 0.5 to 1, the recommended model prediction threshold value is more inclined to reduce the false alarm rate, and the closer the recommended coefficient is to 1, the lower the false alarm rate of the recommended threshold value.

[0116] In the specific implementation, refer to Figure 2 ,The scoring indicators and comprehensive scores under different recommended detection thresholds are different, and the recommended threshold with the highest comprehensive score is selected as the optimal threshold, that is, 0.65 with a comprehensive score of 86.1 is taken as the optimal threshold.

[0117] Step S300, calculating the scoring index of the model to be evaluated under the optimal detection threshold, and generating a quality evaluation report corresponding to the scoring index.

[0118] It should be noted that the scoring indicators include but are not limited to the model's accuracy (ACC), precision (P), recall (R), F1 value, false positive rate (FPR), mAP, etc. The optimal detection threshold is the evaluation result of the model that the device considers to be the best under the threshold.

[0119] Further, refer to Figure 3 According to the calculated scoring indicators, the device collects the md5 information of the model to form a model evaluation summary table (including model version, md5, comprehensive information, false alarm rate (and the increase or decrease compared with the online version), missed alarm rate (and the increase or decrease compared with the online version), label category), model scoring table (including model version, precision rate / false alarm rate, recall rate / missing alarm rate, accuracy, F1 value, false certificate rate, AP50, comprehensive score and threshold recommendation), scoring trend line chart (including comprehensive score, missed alarm rate, false alarm rate), model scoring table under specified threshold (including model version, false alarm rate, missed alarm rate, comprehensive score), horizontal comparison table with same missed alarm rate strategy, and horizontal comparison table with same false alarm rate strategy.

[0120] It is understandable that in order to facilitate reading and understanding of the model, the report will automatically include information such as the hardware output by the model and the evaluation label, and automatically generate a PDF evaluation report with the fourth information + false positive and false negative samples. The evaluation report forms a report directory based on hardware information, predicted category label situation, false positive samples, and false negative samples. In the false positive and false negative samples, the positive samples in the test data, as well as the specific examples of false positives and false negatives are drawn onto the image sample information through boxes of different colors and label information for easy manual reading and viewing.

[0121] It should be noted that the present application also proposes an evaluation switch function for small targets. The device can set information such as whether the small target participates in the evaluation. This item will affect the results of the confusion matrix. For example, small targets below 16*16 pixels will not participate in the evaluation.

[0122] It should be noted that the present application also proposes to draw PR curves and ROC curves before threshold recommendations as reference information for model upgrades and display them to users.

[0123] In a specific implementation, the method for the device to generate a quality assessment report corresponding to the scoring index further includes the following steps:

[0124] The device parses the model file to obtain the hardware information of the model to be evaluated; and based on the scoring index of the model to be evaluated, generates a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information.

[0125] It should be noted that the device supports the automatic expansion of different hardware model evaluations. You only need to generate a txt file of a specified format for the test set image of the corresponding hardware model, and then put the txt file in a specified location named by the hardware. The device will determine the hardware information based on the file information and automatically generate the corresponding hardware evaluation module in the report. That is, the device supports the free expansion of different hardware model evaluations and automatically generates the corresponding model in the hardware evaluation result information in the report.

[0126] In a specific implementation, the method for the device to generate a quality assessment report corresponding to the scoring index further includes the following steps:

[0127] The device determines the MD5 information and version information of the model to be evaluated based on the model file; forms a model ID of the model to be evaluated based on the MD5 information and the version information; forms a scoring trend result of the model to be evaluated and a quality assessment report corresponding to the scoring trend result based on the scoring index of the model to be evaluated and the model ID.

[0128] In the specific implementation, the device automatically generates hardware information, prediction category information, etc. into the model evaluation report based on the scene information and the model hardware output results. The automatically generated report also includes model quality evaluation information in five dimensions, and automatically inserts false positive and missed negative samples, and draws relevant information in the samples to facilitate a comprehensive assessment of the model situation.

[0129] This application calculates the optimal detection threshold through the model prediction results and sample labels under the model file of the model to be evaluated, rather than selecting the detection threshold by manual setting. The optimal detection threshold is used to calculate various evaluation indicators of the model to be evaluated, thereby realizing the quality evaluation of the inference model and improving the accuracy of the quality evaluation of the inference model.

[0130] Based on the above first embodiment, the present application further proposes another embodiment, wherein the automatic model quality assessment method comprises:

[0131] 1. First, obtain the txt information of the model to be evaluated on the corresponding hardware platform for the test set image. The txt needs to be in a specific format. The txt content contains the image information and the corresponding output box information, category information, and box confidence information. The txt is named version number_hardware name.txt;

[0132] 2. The device reads all txt files that need to be evaluated, the corresponding test set images, and the corresponding label files;

[0133] 3. The device will calculate the confusion matrix of the corresponding target category (such as people, cars, etc.) under the corresponding threshold based on the txt file and the label file;

[0134] 4. The device calculates the comprehensive score under the threshold according to the confusion matrix and the comprehensive score calculation formula;

[0135] 5. The device repeats the above 3-4 steps for all thresholds to obtain a comprehensive score list. The threshold corresponding to the highest comprehensive score is the optimal threshold recommended by the system;

[0136] 6. The device calculates various scoring indicators of the model to be evaluated under the optimal threshold, and automatically generates a report of the corresponding scoring indicators and the corresponding hardware, category and other information to the corresponding hardware module;

[0137] 7. Draw the false positive and false negative information of the test set image B into the image and generate it into the module corresponding to the report, finally forming an automated evaluation report.

[0138] In a specific implementation, the beneficial effects of this application include:

[0139] 1. This application can automatically recommend a detection threshold suitable for the scenario based on different model scenarios, rather than manually setting the prediction threshold of the model;

[0140] 2. This application redefines the confusion matrix of the evaluation model, that is, all information except the information in the positive sample frame is negative sample, which is more in line with the evaluation of the detection model than the open source code;

[0141] 3. This application automatically calculates the model's md5 information based on the model file, and together with the version information forms the model's unique ID, and plots the model's score trend based on the model generation time;

[0142] 4. This application automatically generates hardware information, prediction category information, etc. into the model evaluation report based on the scenario information and the model hardware output results. The automatically generated report also includes model quality evaluation information in five dimensions, and automatically inserts false positive and false negative samples and draws relevant information in the samples to facilitate a comprehensive inspection of the model situation;

[0143] 5. This application supports the free expansion of different hardware model evaluations, and automatically generates the corresponding model evaluation result information on the hardware in the report.

[0144] This application also provides an automated model quality assessment device, referring to Figure 4 , the automatic model quality assessment device comprises:

[0145] An acquisition module is used to acquire a model file of a model to be evaluated, wherein the model file includes a model prediction result and a sample label;

[0146] A recommendation module, used for recommending an optimal detection threshold based on the model prediction result and the sample label;

[0147] The calculation module is used to calculate the scoring index of the model to be evaluated under the optimal detection threshold and generate a quality evaluation report corresponding to the scoring index.

[0148] Optionally, the recommendation module includes:

[0149] A set acquisition module, used to acquire a set of detection thresholds to be traversed;

[0150] A matrix calculation module, used to calculate the confusion matrix of each detection threshold under the detection threshold set to be traversed based on the model prediction result and the sample label;

[0151] The comprehensive score calculation module is used to recommend an optimal detection threshold based on the confusion matrix of each detection threshold.

[0152] Optionally, the matrix calculation module includes:

[0153] A result determination module, used to determine a first target labeling result corresponding to the model prediction result, and to determine a second target labeling result corresponding to the sample label;

[0154] A comparison module, configured to compare the first target labeling result with the second target labeling result based on the detection threshold set to be traversed, and obtain a comparison result of each detection threshold under the detection threshold set to be traversed;

[0155] The confusion matrix forming module is used to form a confusion matrix of each detection threshold under the detection threshold set to be traversed based on the comparison result.

[0156] Optionally, the comparison module includes:

[0157] A sample determination module, configured to determine, based on the detection threshold set to be traversed, a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and to determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein a positive sample refers to a target that needs to be detected in an image detection task, and a negative sample refers to a target that does not need to be detected in an image detection task;

[0158] A matching module is used to match and compare the first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed, so as to obtain a comparison result of each detection threshold under the detection threshold set to be traversed.

[0159] Optionally, the comprehensive score calculation module includes:

[0160] A calculation module is used to calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set;

[0161] The selection module is used to select the highest comprehensive score from the comprehensive score set, and use the detection threshold corresponding to the highest comprehensive score as the optimal detection threshold.

[0162] Optionally, the calculation module includes:

[0163] The scoring formula calculation module is used to calculate the confusion matrix of each detection threshold through a comprehensive scoring calculation formula to obtain a comprehensive scoring set, wherein the comprehensive scoring calculation formula includes the following formula:

[0164]

[0165] in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

[0166] Optionally, the information determination module includes:

[0167] A test set acquisition module, used to acquire test set images and sample labels corresponding to the test set images;

[0168] The prediction module is used to input the test set images into the model to be evaluated, obtain the model prediction results, and form a model file of the model to be evaluated based on the model prediction results and the sample labels corresponding to the test set images.

[0169] Optionally, the calculation module includes:

[0170] A parsing module, used for parsing the model file to obtain the hardware information of the model to be evaluated;

[0171] The report generation module is used to generate a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information based on the scoring index of the model to be evaluated.

[0172] Optionally, the calculation module further includes:

[0173] A version information determination module, used to determine the MD5 information and version information of the model to be evaluated based on the model file;

[0174] A model ID forming module, used to form a model ID of the model to be evaluated based on the MD5 information and the version information;

[0175] The quality assessment report forming module is used to form a scoring trend result of the model to be evaluated and a quality assessment report corresponding to the scoring trend result based on the scoring index of the model to be evaluated and the model ID.

[0176] The specific implementation of the automatic model quality assessment device of the present application is basically the same as the embodiments of the above-mentioned automatic model quality assessment method, and will not be repeated here.

[0177] Reference Figure 5 , Figure 5 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present application.

[0178] like Figure 5 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0179] Optionally, the automated model quality assessment device may also include a rectangular user interface, a network interface, a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, etc. The rectangular user interface may include a display screen (Display), an input submodule such as a keyboard (Keyboard), and the optional rectangular user interface may also include a standard wired interface and a wireless interface. The network interface may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0180] Those skilled in the art will understand that Figure 4 The structure of the automated model quality assessment device shown in the figure does not constitute a limitation on the automated model quality assessment device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] like Figure 5 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, and an automated model quality assessment program. The operating system is a program that manages and controls the hardware and software resources of the automated model quality assessment device, and supports the operation of the automated model quality assessment program and other software and / or programs. The network communication module is used to realize the communication between the components inside the memory 1005, and the communication with other hardware and software in the automated model quality assessment system.

[0182] exist Figure 5 In the automated model quality assessment device shown, the processor 1001 is used to execute the automated model quality assessment program stored in the memory 1005 to implement the steps of any of the automated model quality assessment methods described above.

[0183] The specific implementation of the automatic model quality assessment device of the present application is basically the same as the embodiments of the above-mentioned automatic model quality assessment method, and will not be repeated here.

[0184] The present application also provides a storage medium, on which is stored a program for implementing the automated model quality assessment method, and the program for implementing the automated model quality assessment method is executed by a processor to implement the automated model quality assessment method as described below:

[0185] Obtain a model file of the model to be evaluated, wherein the model file includes model prediction results and sample labels;

[0186] Based on the model prediction results and the sample labels, recommend an optimal detection threshold;

[0187] Calculate the scoring index of the model to be evaluated under the optimal detection threshold, and generate a quality assessment report corresponding to the scoring index.

[0188] The step of recommending an optimal detection threshold based on the model prediction result and the sample label includes:

[0189] Get the detection threshold set to be traversed;

[0190] Based on the model prediction results and the sample labels, a confusion matrix of each detection threshold under the detection threshold set to be traversed is calculated;

[0191] Based on the confusion matrix of each detection threshold, an optimal detection threshold is recommended.

[0192] Optionally, the confusion matrix includes a first sample classification category, the first sample classification category refers to a negative sample detection that is correctly predicted as a negative sample, and a negative sample refers to a target that does not need to be detected in an image detection task; the negative sample includes an image background, and when the image background is correctly predicted as a negative sample, the value under the first sample classification category in the confusion matrix remains unchanged.

[0193] Optionally, the step of calculating the confusion matrix of each detection threshold under the set of detection thresholds to be traversed based on the model prediction result and the sample label includes:

[0194] Determine a first target labeling result corresponding to the model prediction result, and determine a second target labeling result corresponding to the sample label;

[0195] Based on the detection threshold set to be traversed, the first target labeling result and the second target labeling result are compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed;

[0196] Based on the comparison result, a confusion matrix of each detection threshold under the detection threshold set to be traversed is formed.

[0197] Optionally, the step of comparing the first target labeling result with the second target labeling result based on the detection threshold set to be traversed to obtain the comparison results of each detection threshold under the detection threshold set to be traversed includes:

[0198] Based on the detection threshold set to be traversed, determine a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein the positive sample refers to a target that needs to be detected in the image detection task, and the negative sample refers to a target that does not need to be detected in the image detection task;

[0199] The first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed are matched and compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed.

[0200] Optionally, the step of recommending an optimal detection threshold based on the confusion matrix of each detection threshold comprises:

[0201] Calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set;

[0202] The highest comprehensive score is selected from the comprehensive score set, and the detection threshold corresponding to the highest comprehensive score is used as the optimal detection threshold.

[0203] Optionally, the step of calculating the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set includes:

[0204] The confusion matrix of each detection threshold is calculated by the comprehensive score calculation formula to obtain a comprehensive score set, wherein the comprehensive score calculation formula includes the following formula:

[0205]

[0206] in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

[0207] Optionally, the step of obtaining a model file of the model to be evaluated includes:

[0208] Obtaining a test set image and a sample label corresponding to the test set image;

[0209] The test set images are input into the model to be evaluated to obtain model prediction results, and a model file of the model to be evaluated is formed based on the model prediction results and the sample labels corresponding to the test set images.

[0210] Optionally, the step of generating a quality assessment report corresponding to the scoring indicator includes:

[0211] Parsing the model file to obtain hardware information of the model to be evaluated;

[0212] Based on the scoring index of the model to be evaluated, a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information is generated.

[0213] Optionally, the step of generating a quality assessment report corresponding to the scoring indicator includes:

[0214] Based on the model file, determine the MD5 information and version information of the model to be evaluated;

[0215] Based on the MD5 information and the version information, a model ID of the model to be evaluated is formed;

[0216] Based on the scoring index of the model to be evaluated and the model ID, a scoring trend result of the model to be evaluated and a quality assessment report corresponding to the scoring trend result are formed.

[0217] The specific implementation of the storage medium of the present application is basically the same as the embodiments of the above-mentioned automatic model quality assessment method, and will not be repeated here.

[0218] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned automatic model quality assessment method when executed by a processor.

[0219] The specific implementation of the computer program product of the present application is basically the same as the various embodiments of the above-mentioned automatic model quality assessment method, and will not be repeated here.

[0220] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0221] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0222] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0223] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An automated model quality assessment method, characterized in that: The automated model quality assessment method comprises: Obtain a model file of the model to be evaluated, wherein the model file includes model prediction results and sample labels; Based on the model prediction results and the sample labels, recommend an optimal detection threshold; The step of recommending an optimal detection threshold based on the model prediction result and the sample label includes: Get the detection threshold set to be traversed; Based on the model prediction result and the sample label, a confusion matrix of each detection threshold under the detection threshold set to be traversed is calculated, wherein the confusion matrix includes a first sample classification category, the first sample classification category refers to a negative sample detection that is correctly predicted as a negative sample, and a negative sample refers to a target that does not need to be detected in an image detection task; the negative sample includes an image background, and when the image background is correctly predicted as a negative sample, the value under the first sample classification category in the confusion matrix remains unchanged; Based on the confusion matrix of each detection threshold, recommend an optimal detection threshold; The step of calculating the confusion matrix of each detection threshold under the detection threshold set to be traversed based on the model prediction result and the sample label includes: Determine a first target labeling result corresponding to the model prediction result, and determine a second target labeling result corresponding to the sample label; Based on the detection threshold set to be traversed, the first target labeling result and the second target labeling result are compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed; Based on the comparison result, a confusion matrix of each detection threshold under the detection threshold set to be traversed is formed; Calculate the scoring index of the model to be evaluated under the optimal detection threshold, and generate a quality assessment report corresponding to the scoring index.

2. The automated model quality assessment method according to claim 1, wherein: The step of comparing the first target labeling result with the second target labeling result based on the detection threshold set to be traversed to obtain the comparison results of each detection threshold under the detection threshold set to be traversed includes: Based on the detection threshold set to be traversed, determine a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein the positive sample refers to a target that needs to be detected in the image detection task, and the negative sample refers to a target that does not need to be detected in the image detection task; The first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed are matched and compared to obtain comparison results of each detection threshold under the detection threshold set to be traversed.

3. The automated model quality assessment method according to claim 1, wherein: The step of recommending an optimal detection threshold based on the confusion matrix of each detection threshold comprises: Calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set; The highest comprehensive score is selected from the comprehensive score set, and the detection threshold corresponding to the highest comprehensive score is used as the optimal detection threshold.

4. The automated model quality assessment method according to claim 3, characterized in that: The step of calculating the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set includes: The confusion matrix of each detection threshold is calculated by the comprehensive score calculation formula to obtain a comprehensive score set, wherein the comprehensive score calculation formula includes the following formula: in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

5. The automated model quality assessment method according to claim 1, wherein: The step of obtaining the model file of the model to be evaluated includes: Obtaining a test set image and a sample label corresponding to the test set image; The test set images are input into the model to be evaluated to obtain model prediction results, and a model file of the model to be evaluated is formed based on the model prediction results and the sample labels corresponding to the test set images.

6. The automated model quality assessment method according to claim 1, wherein: The step of generating a quality assessment report corresponding to the scoring indicator comprises: Parsing the model file to obtain hardware information of the model to be evaluated; Based on the scoring index of the model to be evaluated, a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information is generated.

7. The automated model quality assessment method according to claim 1, wherein: The step of generating a quality assessment report corresponding to the scoring indicator comprises: Based on the model file, determine the MD5 information and version information of the model to be evaluated; Based on the MD5 information and the version information, a model ID of the model to be evaluated is formed; Based on the scoring index of the model to be evaluated and the model ID, a scoring trend result of the model to be evaluated and a quality assessment report corresponding to the scoring trend result are formed.

8. An automated model quality assessment device, characterized in that: The automatic model quality assessment device comprises: An acquisition module is used to acquire a model file of a model to be evaluated, wherein the model file includes a model prediction result and a sample label; A recommendation module, used for recommending an optimal detection threshold based on the model prediction result and the sample label; The recommendation module includes: A set acquisition module, used to acquire a set of detection thresholds to be traversed; A matrix calculation module, for calculating the confusion matrix of each detection threshold under the detection threshold set to be traversed based on the model prediction result and the sample label, wherein the confusion matrix includes a first sample classification category, the first sample classification category refers to a negative sample detection that is correctly predicted as a negative sample, and a negative sample refers to a target that does not need to be detected in an image detection task; the negative sample includes an image background, and when the image background is correctly predicted as a negative sample, the value under the first sample classification category in the confusion matrix remains unchanged; A comprehensive score calculation module, used for recommending an optimal detection threshold based on the confusion matrix of each detection threshold; The matrix calculation module comprises: A result determination module, used to determine a first target labeling result corresponding to the model prediction result, and to determine a second target labeling result corresponding to the sample label; A comparison module, configured to compare the first target labeling result with the second target labeling result based on the detection threshold set to be traversed, and obtain a comparison result of each detection threshold under the detection threshold set to be traversed; A confusion matrix forming module, used for forming a confusion matrix of each detection threshold under the detection threshold set to be traversed based on the comparison result; The calculation module is used to calculate the scoring index of the model to be evaluated under the optimal detection threshold and generate a quality evaluation report corresponding to the scoring index.

9. The automatic model quality assessment device according to claim 8, characterized in that: The comparison module comprises: A sample determination module, configured to determine, based on the detection threshold set to be traversed, a first positive sample and a first negative sample for image detection in the first target annotation result under the detection threshold set to be traversed, and to determine a second positive sample and a second negative sample for image detection in the second target annotation result, wherein a positive sample refers to a target that needs to be detected in an image detection task, and a negative sample refers to a target that does not need to be detected in an image detection task; A matching module is used to match and compare the first positive sample, the first negative sample, the second positive sample and the second negative sample of each detection threshold under the detection threshold set to be traversed, so as to obtain a comparison result of each detection threshold under the detection threshold set to be traversed.

10. The automatic model quality assessment device according to claim 8, characterized in that: The comprehensive score calculation module includes: A calculation module is used to calculate the comprehensive score corresponding to the confusion matrix of each detection threshold to obtain a comprehensive score set; The selection module is used to select the highest comprehensive score from the comprehensive score set, and use the detection threshold corresponding to the highest comprehensive score as the optimal detection threshold.

11. The automatic model quality assessment device according to claim 10, characterized in that: The computing module comprises: The scoring formula calculation module is used to calculate the confusion matrix of each detection threshold through a comprehensive scoring calculation formula to obtain a comprehensive scoring set, wherein the comprehensive scoring calculation formula includes the following formula: in, Represents the comprehensive rating value. represents the number of correct predictions of positive samples in the confusion matrix, Indicates the number of times a negative sample is predicted as a positive sample in the confusion matrix. Indicates the number of times a positive sample in the confusion matrix is ​​predicted as a negative sample, represents the number of correct predictions of negative samples in the confusion matrix, represents the weight coefficient, The value range is a closed interval from 0 to 1.

12. The automatic model quality assessment device according to claim 8, characterized in that: The acquisition module comprises: A test set acquisition module, used to acquire test set images and sample labels corresponding to the test set images; The prediction module is used to input the test set images into the model to be evaluated, obtain the model prediction results, and form a model file of the model to be evaluated based on the model prediction results and the sample labels corresponding to the test set images.

13. The automatic model quality assessment device according to claim 8, characterized in that: The computing module comprises: A parsing module, used for parsing the model file to obtain the hardware information of the model to be evaluated; The report generation module is used to generate a quality evaluation report of the model to be evaluated under the hardware corresponding to the hardware information based on the scoring index of the model to be evaluated.

14. An automated model quality assessment device, characterized in that: The automatic model quality assessment device comprises: a memory, a processor, and a program stored in the memory for implementing the automatic model quality assessment method. The memory is used to store a program for implementing an automated model quality assessment method; The processor is used to execute a program for implementing the automatic model quality assessment method to implement the steps of the automatic model quality assessment method as claimed in any one of claims 1 to 7.

15. A storage medium, characterized in that: The storage medium stores a program for implementing the automated model quality assessment method, and the program for implementing the automated model quality assessment method is executed by a processor to implement the steps of the automated model quality assessment method as claimed in any one of claims 1 to 7.

Images