Model evaluation method, system and computer storage medium

Abstract

The present application provides a model evaluation method, device and computer storage medium, including using the model to be evaluated to predict sample images based on each preset category, and obtaining the category detection result corresponding to each preset category for each pixel in the sample image; according to the sample label of the sample image, analyzing the category detection result corresponding to each preset category for each pixel in the sample image based on the pixel dimension and image dimension, and determining the pixel dimension evaluation value and image dimension evaluation value of the model to be evaluated; according to the pixel dimension evaluation value and image dimension evaluation value of the model to be evaluated, obtaining the performance evaluation result of the model to be evaluated. Accordingly, the present application can provide a model performance evaluation result with higher accuracy to improve the model training effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of model training technology, and in particular to a model evaluation method, device and computer storage medium. Background Art

[0002] In the actual application scenarios of fire and smoke detection, how to select a suitable effect evaluation method to determine the effect of the fire and smoke detection model is a very important issue. A good evaluation method can assist in selecting a better model, allowing a more accurate model to be put into practical application, which is of great significance for early warning of fire and smoke.

[0003] Currently, for the evaluation of the training effect of the fireworks detection model, if the model is based on different technical routes, there are often different evaluation methods. For example, the model based on image classification usually uses the image classification indicator evaluation; the model based on target detection usually uses the mAP indicator evaluation; the model based on semantic segmentation will use the mIoU indicator for pixel-level evaluation.

[0004] At present, there is no unified, objective and targeted model evaluation method for the task of fireworks detection. Summary of the invention

[0005] In view of the above problems, the present application provides a model evaluation method, device and computer storage medium, which can at least partially solve the problems in the prior art.

[0006] The first aspect of the present application provides a model performance evaluation method, including: using the model to be evaluated to predict sample images based on each preset category, and obtaining a category detection result corresponding to each preset category for each pixel in the sample image; according to the sample label of the sample image, analyzing the category detection result corresponding to each preset category for each pixel in the sample image based on the pixel dimension and the image dimension, and determining the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated; according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated, obtaining a performance evaluation result of the model to be evaluated.

[0007] The second aspect of the present application provides a model performance evaluation device, including: a category detection module, used to use the model to be evaluated to predict a sample image based on each preset category, and obtain a category detection result corresponding to each preset category for each pixel in the sample image; an evaluation module, used to analyze the category detection result corresponding to each preset category for each pixel in the sample image based on the pixel dimension and the image dimension according to the sample label of the sample image, determine the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated, and obtain the performance evaluation result of the model to be evaluated according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated.

[0008] A third aspect of the present application provides a computer storage medium, wherein the computer storage medium stores instructions for executing the steps in the method described in the first aspect.

[0009] To sum up, the model evaluation scheme provided by various aspects of the present application, by combining the pixel dimension and the image dimension, performs analysis on the sample image prediction results output by the model to be evaluated, and can obtain more accurate model performance evaluation results, which is particularly suitable for the performance evaluation of semantic segmentation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0011] Figure 1 This is a processing flow chart of a model performance evaluation method according to an exemplary embodiment of the present application.

[0012] Figure 2 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0013] Figure 3 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0014] Figure 4 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0015] Figure 5 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0016] Figure 6 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0017] Figure 7 A processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application.

[0018] Figure 8 A processing flow chart of a model performance evaluation device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] At present, there is no unified, objective and targeted technical solution for the evaluation of fireworks detection models, and the current evaluation method cannot meet the evaluation of the effectiveness of fireworks detection models for practical applications.

[0021] In addition, when the fire and smoke detection model is used in actual industrial applications, it is usually necessary to set a threshold to balance the precision and recall of the model. However, the current evaluation method also lacks support for threshold selection.

[0022] Compared with the existing performance evaluation methods of fireworks detection models in the industry, the mainstream ideas are: 1) Models based on image classification use classification indicators for evaluation, such as Accuracy; 2) Models based on object detection use the mAP indicator for evaluation; 3) Models based on semantic segmentation use the mIoU indicator and other indicators such as pixel-level Accuracy for evaluation.

[0023] This application is mainly aimed at semantic segmentation models, and designs a multi-dimensional combination model performance evaluation method. On the basis of evaluating the pixel dimension effect, it introduces the image dimension effect evaluation, which can more intuitively reflect the effect of the model in industrial applications. At the same time, it supports threshold selection and searches for the appropriate threshold for the model to be deployed.

[0024] The specific embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0025] Figure 1 This is a processing flow chart of the model performance evaluation method of the exemplary embodiment of the present application. As shown in the figure, this embodiment mainly includes the following processing steps:

[0026] Step S102: using the model to be evaluated to predict the sample image based on each preset category, and obtaining a category detection result corresponding to each preset category for each pixel in the sample image.

[0027] Optionally, the model to be evaluated includes a semantic segmentation model.

[0028] Optionally, the sample image may include flames and / or smoke, so that the model to be evaluated can make predictions on the flames and / or smoke in the sample image.

[0029] Optionally, the evaluated model may be used to predict each pixel in the sample image based on each preset category, so as to obtain a category prediction value of each pixel in the sample image corresponding to each preset category.

[0030] In one embodiment, if the training loss function of the model to be evaluated is ordinary cross entropy, a softmax calculation is performed for each pixel point based on each preset category to obtain a category prediction value of each pixel point corresponding to each preset category.

[0031] In another embodiment, if the training loss function of the model to be evaluated is binary cross entropy, a sigmoid calculation is performed for each pixel based on each preset category to obtain a category prediction value of each pixel corresponding to each preset category.

[0032] Step S104, according to the sample label of the sample image, analyze the category detection result of each pixel point in the sample image corresponding to each preset category based on the pixel dimension and the image dimension, and determine the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated.

[0033] Optionally, the pixel dimension evaluation value of the model to be evaluated may be determined based on the sample label of the sample image, the category prediction value of each pixel point in the sample image corresponding to each preset category, and the position information of each pixel point.

[0034] Optionally, the image dimension evaluation value of the model to be evaluated corresponding to each preset category can be determined based on the sample label of the sample image and the category prediction value of each pixel point in the sample image corresponding to each preset category.

[0035] Step S106, obtaining the performance evaluation result of the model to be evaluated according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated.

[0036] Optionally, based on the evaluation values ​​of each image dimension of the sample image corresponding to each preset category, the accuracy value and recall value of the sample image corresponding to each evaluation threshold can be obtained, and a traversal search can be performed to obtain the accuracy value closest to the expected threshold, and based on the evaluation threshold and recall value corresponding to the accuracy, the performance evaluation result of the model to be evaluated can be obtained.

[0037] To sum up, the model performance evaluation method of the embodiment of the present application evaluates the category detection results output by the model to be evaluated based on both the pixel dimension and the image dimension, which can improve the accuracy of the model performance evaluation results, and is particularly suitable for performing performance evaluation on semantic segmentation models, which can help improve the training effect of semantic segmentation models.

[0038] Figure 2 This is a processing flow chart of a model performance evaluation method of another exemplary embodiment of the present application. This embodiment is a specific implementation scheme for determining the pixel dimension evaluation value of the model to be evaluated in the above step S104. As shown in the figure, this embodiment mainly includes the following steps:

[0039] Step S202 , determining the predicted category of each pixel point according to the highest of the predicted values ​​of each category corresponding to each preset category of each pixel point.

[0040] For example, the model to be evaluated can perform prediction for pixel point a in the sample image according to preset categories (such as category A, category B, category C), and output the probability values ​​of pixel point a corresponding to each preset category. If the probability value of category A is the highest, category A is determined as the predicted category of pixel point a.

[0041] Step S204, obtaining an intersection-over-union ratio of each preset category according to the predicted category and the actual category of each pixel point and the position information of each pixel point.

[0042] Optionally, the sample label of the sample image includes the true category of each pixel in the sample image.

[0043] Optionally, the intersection-and-union ratio of each preset category may be calculated according to the predicted category and the true category of each pixel point.

[0044] Step S206, performing mean calculation on the intersection-and-union ratio of each preset category to obtain a pixel dimension evaluation value of the model to be evaluated.

[0045] Specifically, after obtaining the IoU value of each preset category, the mean IoU of all preset categories, that is, mIoU, is calculated to determine the pixel dimension evaluation value of the model to be evaluated.

[0046] It should be noted that the present application does not need to pay attention to the background category in the sample image, that is, when determining the pixel dimension evaluation value of the model to be evaluated, the background part in the sample image can be eliminated, and only the IoU mean of the flame category and the smoke category, for example, is calculated, so as to improve the efficiency of the model evaluation processing.

[0047] Figure 3 The following is a processing flow chart of a model performance evaluation method according to another exemplary embodiment of the present application. This embodiment is a specific implementation scheme of the above step S204. As shown in the figure, this embodiment mainly includes the following steps:

[0048] Step S302: determine a preset category as the current category.

[0049] Step S304: determine each pixel point whose predicted category matches the current category as a first candidate pixel point, and determine each pixel point whose actual category matches the current category as a second candidate pixel point.

[0050] Specifically, according to the current category (for example, category A), if the predicted category of pixel a is consistent with the current category, pixel a is determined as the first candidate pixel, and if the true category of pixel a is consistent with the current category, pixel a is determined as the second candidate pixel. In other words, when the predicted category of pixel a is consistent with the true category, pixel a is both the first candidate pixel and the second candidate pixel.

[0051] Step S306: determining a first region of the current category according to the position information of each first candidate pixel point, and determining a second region of the current category according to the position information of each second candidate pixel point.

[0052] Specifically, the coordinate information of all first candidate pixel points in the sample image is integrated to obtain the first region of the current category, and the coordinate information of all second candidate pixel points in the sample image is integrated to obtain the second region of the current category.

[0053] Step S308: determining the intersection-and-union ratio of the current category according to the intersection and union of the first region and the second region.

[0054] Specifically, the intersection between the first region and the second region may be divided by the union between the first region and the second region to obtain an intersection-and-union ratio of the current category.

[0055] Step S310, determining whether the intersection-and-union ratios of all preset categories have been obtained, if so, exiting this step flow, if not, returning to step S302 to obtain the next preset category and perform the intersection-and-union ratio calculation.

[0056] In summary, the present application evaluates the prediction results of the sample image from the pixel dimension based on the predicted category and the true category of each pixel point to determine the pixel dimension evaluation result of the model to be evaluated.

[0057] Figure 4 The processing flow chart of the model evaluation method of another exemplary embodiment of the present application is shown. This embodiment is a specific implementation scheme for determining the image dimension evaluation value of the model to be evaluated in the above step S104. As shown in the figure, this embodiment mainly includes the following steps:

[0058] Step S402: determine a preset category as the current category.

[0059] Step S404, obtaining a category prediction value corresponding to the current category for each pixel point.

[0060] For example, if the current category is category A, the probability values ​​of all pixels in the sample image corresponding to category A are obtained.

[0061] Step S406, determining whether the sample image contains the current category, if so, executing step S408, if not, executing step S410.

[0062] Optionally, whether the sample image contains the current label may be determined based on the real label of the sample image.

[0063] For example, if the true label of the sample image contains flame and the current category is flame, then it is determined that the sample image contains the current category.

[0064] For example, if the true label of the sample image contains flames and the current category is smoke, it is determined that the sample image does not contain the current category.

[0065] Step S408, determine the category area of ​​the current category in the sample image, and based on the category prediction value of each pixel point in the category area corresponding to the current category, determine the maximum category prediction value as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

[0066] Specifically, if the sample image contains the current category (for example, flame), the real flame area marked in the sample image is determined, and based on the probability value of each pixel point in the real flame area corresponding to the flame, the maximum probability value is determined as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

[0067] Step S410, based on the category prediction value of each pixel point in the sample image corresponding to the current category, the maximum category prediction value is determined as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

[0068] Specifically, if the sample image does not contain the current category (eg, smoke), obtain the probability value of each pixel in the sample image corresponding to smoke, and determine the maximum probability value as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

[0069] Step S412, determine whether the image dimension evaluation values ​​of all preset categories have been obtained, if so, exit this step process, if not, return to step S402 to obtain the next preset category and perform the intersection-and-union ratio operation.

[0070] In summary, the present application determines whether the sample image contains the current category, and based on the judgment result, uses different methods to determine the image dimension evaluation value of the model to be evaluated corresponding to the current category, so as to improve the accuracy of subsequent model evaluation results.

[0071] Figure 5 This is a processing flow chart of a model performance evaluation method of another exemplary embodiment of the present application. This embodiment is a specific implementation of the above step S106. As shown in the figure, this embodiment mainly includes the following steps:

[0072] Step S502, based on each evaluation threshold and the sample annotation of the sample image, analyze the evaluation values ​​of each image dimension of the sample image corresponding to each preset category, and obtain the accuracy value and recall value of the sample image corresponding to each evaluation threshold.

[0073] Optionally, each evaluation threshold may be determined based on the evaluation granularity of the model to be evaluated.

[0074] Optionally, the evaluation granularity of the model to be evaluated may include at least one of 1 / 10, 1 / 100, and 1 / 1000.

[0075] For example, if the evaluation granularity is 1 / 10, the evaluation threshold can be set to 0.3, 0.5, 0.8, etc.; if the evaluation granularity is 1 / 100, the evaluation threshold can be set to 0.03, 0.05, 0.08, etc., and so on.

[0076] In this embodiment, each evaluation threshold is between 0 and 1.

[0077] Preferably, each evaluation threshold may be between 0.5 and 0.8.

[0078] Optionally, based on each evaluation threshold, the sample annotation of the sample image, and the image dimension evaluation value of the sample image corresponding to each preset category, the positive example identification result of the sample image corresponding to each evaluation threshold can be obtained, and based on the positive example identification result of the sample image corresponding to each evaluation threshold, the accuracy value and recall rate value of the sample image corresponding to each evaluation threshold can be obtained.

[0079] Step S504, obtaining a performance evaluation result of the model to be evaluated according to the accuracy value and recall value of the sample image corresponding to each evaluation threshold, the expected threshold, and the pixel dimension evaluation value of the model to be evaluated.

[0080] Optionally, the expected threshold may be adjusted arbitrarily according to the actual application scenario of the model to be evaluated, and this application does not impose any restrictions on this.

[0081] Figure 6 The following is a processing flow chart of a model performance evaluation method according to another embodiment of the present application. This embodiment is a specific implementation scheme of the above step S502. As shown in the figure, this embodiment mainly includes the following steps:

[0082] Step S602: determine an evaluation threshold as the current evaluation threshold.

[0083] Step S604: determine a preset category as the current category.

[0084] Step S606, based on the sample annotation of the sample image, the current evaluation threshold, and the image dimension evaluation value of the sample image corresponding to the current category, determine the positive example identification result of the sample image corresponding to the current category.

[0085] Optionally, when the sample image contains the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a sample recognition result that the sample image is a true positive example is obtained; if the image dimension evaluation value of the sample image corresponding to the current category is not greater than the current evaluation threshold, a positive example identification result that the sample image is a false positive example is obtained.

[0086] Optionally, when the sample image does not contain the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a positive example identification result indicating that the sample image is a false positive example is obtained.

[0087] In step S608, it is determined whether all preset categories have been analyzed. If so, step S610 is performed. If not, step S604 is performed to obtain the next preset category and perform analysis.

[0088] Step S610, determining whether all evaluation thresholds have been analyzed, if so, proceeding to step S612, if not, executing step S602 to obtain the next evaluation threshold to perform analysis.

[0089] Step S612: Obtain the precision value and recall value of the sample image corresponding to each evaluation threshold according to the positive example identification result of the sample image corresponding to each evaluation threshold.

[0090] Specifically, the identification results of true positive examples and false positive examples of the sample image corresponding to each evaluation threshold are counted, and the precision value and recall value of the sample image corresponding to each evaluation threshold are determined accordingly.

[0091] In summary, this embodiment sets multiple evaluation thresholds, and based on the evaluation values ​​of each image dimension of the sample image corresponding to each preset category, and based on the accuracy value and recall value of the sample image corresponding to each evaluation threshold, provides a reference for determining the target evaluation threshold, which can be suitable for targeted evaluation of the models to be evaluated in different application scenarios.

[0092] Figure 7 The processing flow chart of the model performance evaluation method of another exemplary embodiment of the present application is shown. This embodiment is a specific implementation of the above step S504. As shown in the figure, this embodiment mainly includes the following steps:

[0093] Step S702, according to the expected threshold and the accuracy value of the sample image corresponding to each evaluation threshold, determine the accuracy value with the smallest difference from the expected threshold, and determine the evaluation threshold corresponding to the accuracy value as the target evaluation threshold.

[0094] Optionally, an expected threshold may be set according to the actual application scenario of the model to be evaluated.

[0095] Specifically, the accuracy values ​​of the sample images corresponding to each evaluation threshold may be traversed to query the accuracy value closest to the expected threshold, and the evaluation threshold corresponding to the accuracy value may be determined as the target evaluation threshold.

[0096] Step S704, determining the recall value of the target evaluation threshold as the target recall value, and obtaining the performance evaluation result of the model to be evaluated according to the target recall value, the pixel dimension evaluation value of the model to be evaluated, and the preset evaluation conversion rule.

[0097] In this embodiment, the preset evaluation conversion rule can be expressed as:

[0098]

[0099] Among them, Score total Indicates the performance evaluation result of the model to be evaluated; Score pix Represents the pixel dimension evaluation value of the model to be evaluated; Represents the target recall value.

[0100] Among them, ε is an adjustment factor, and its default value can be set to 0.5, but it is not limited to this and can be adjusted arbitrarily according to actual needs.

[0101] In summary, this embodiment combines pixel dimension, image dimension, and expected threshold determined based on different application scenarios, which can effectively improve the accuracy of the performance evaluation results of the model to be evaluated, and is particularly suitable for evaluating the performance of semantic detection models with irregular detection areas.

[0102] Figure 8 This is a structural block diagram of a model performance evaluation device according to an exemplary embodiment of the present application. As shown in the figure, the model performance evaluation device 800 of this embodiment mainly includes: a category detection module 802 and an evaluation module 804.

[0103] The category detection module 802 is used to predict the sample image based on each preset category using the model to be evaluated, and obtain the category detection result of each pixel point in the sample image corresponding to each preset category.

[0104] The evaluation module 804 is used to analyze the category detection results of each pixel point in the sample image corresponding to each preset category based on the pixel dimension and the image dimension according to the sample label of the sample image, determine the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated, and obtain the performance evaluation result of the model to be evaluated based on the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated.

[0105] Optionally, the model to be evaluated includes a semantic segmentation model.

[0106] Optionally, the category detection module 802 is also used to: use the model to be evaluated to predict each pixel in the sample image based on each preset category, and obtain the category prediction value of each pixel in the sample image corresponding to each preset category; wherein, if the training loss function of the model to be evaluated is ordinary cross entropy, softmax calculation is performed for each pixel based on each preset category to obtain the category prediction value of each pixel corresponding to each preset category; if the training loss function of the model to be evaluated is binary cross entropy, sigmoid calculation is performed for each pixel based on each preset category to obtain the category prediction value of each pixel corresponding to each preset category.

[0107] Optionally, the evaluation module 804 is also used to: determine the pixel dimension evaluation value of the model to be evaluated based on the sample label of the sample image, the category prediction value of each pixel point in the sample image corresponding to each preset category, and the position information of each pixel point; determine the image dimension evaluation value of the model to be evaluated corresponding to each preset category based on the sample label of the sample image and the category prediction value of each pixel point in the sample image corresponding to each preset category.

[0108] Optionally, the sample label includes the true category of each pixel in the sample image.

[0109] Optionally, the evaluation module 804 is also used to: determine the predicted category of each pixel point based on the highest of the predicted values ​​of each category corresponding to each preset category; obtain the intersection-and-union ratio of each preset category based on the predicted category and the true category of each pixel point and the position information of each pixel point; perform mean calculation on the intersection-and-union ratio of each preset category to obtain the pixel dimension evaluation value of the model to be evaluated.

[0110] Optionally, the evaluation module 804 is also used to: determine a preset category as the current category; determine each pixel point whose predicted category is consistent with the current category as a first candidate pixel point, and determine each pixel point whose real category is consistent with the current category as a second candidate pixel point; determine the first area of ​​the current category based on the position information of each first candidate pixel point, and determine the second area of ​​the current category based on the position information of each second candidate pixel point; determine the intersection-union ratio of the current category based on the intersection and union of the first area and the second area.

[0111] Optionally, the sample label includes a real category contained in the sample image.

[0112] Optionally, the evaluation module 804 is also used to: determine a preset category as the current category; obtain a category prediction value corresponding to the current category for each pixel; obtain a judgment result of whether the sample image contains or does not contain the current category based on the sample label of the sample image; if the sample image contains the current category, determine the category area of ​​the current category in the sample image, and determine the maximum category prediction value as the image dimension evaluation value of the model to be evaluated corresponding to the current category based on the category prediction value corresponding to the current category for each pixel in the category area; if the sample image does not contain the current category, determine the maximum category prediction value as the image dimension evaluation value of the model to be evaluated corresponding to the current category based on the category prediction value corresponding to the current category for each pixel in the sample image.

[0113] Optionally, the evaluation module 804 is also used to: based on each evaluation threshold and the sample annotation of the sample image, analyze the evaluation values ​​of each image dimension of the sample image corresponding to each preset category, and obtain the accuracy value and recall value of the sample image corresponding to each evaluation threshold; obtain the performance evaluation result of the model to be evaluated according to the accuracy value and recall value of the sample image corresponding to each evaluation threshold, the expected threshold, and the pixel dimension evaluation value of the model to be evaluated.

[0114] Optionally, each evaluation threshold can be determined based on the evaluation granularity of the model to be evaluated; the evaluation granularity includes at least one of 1 / 10, 1 / 100, and 1 / 1000; each evaluation threshold is between 0 and 1; preferably, each evaluation threshold is between 0.5 and 0.8.

[0115] Optionally, the evaluation module 804 is also used to: execute the positive example identification step, determine an evaluation threshold as the current evaluation threshold, and compare the current evaluation threshold with the image dimension evaluation value of the sample image corresponding to each preset category according to the sample annotation of the sample image, to obtain the positive example identification result of the sample image corresponding to the current evaluation threshold; repeat the positive example identification step to obtain the positive example identification result of the sample image corresponding to each evaluation threshold; and obtain the accuracy value and recall value of the sample image corresponding to each evaluation threshold according to the positive example identification result of the sample image corresponding to each evaluation threshold.

[0116] Optionally, the evaluation module 804 is also used to: determine a preset category as the current category; determine a positive example identification result of the sample image corresponding to the current category based on the sample annotation of the sample image, the current evaluation threshold, and the image dimension evaluation value of the sample image corresponding to the current category; wherein, in the case where the sample image contains the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a sample recognition result that the sample image is a true positive example is obtained; if the image dimension evaluation value of the sample image corresponding to the current category is not greater than the current evaluation threshold, a positive example identification result that the sample image is a false positive example is obtained; in the case where the sample image does not contain the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a positive example identification result that the sample image is a false positive example is obtained.

[0117] Optionally, the evaluation module 804 is further used to: determine the accuracy value with the smallest difference from the expected threshold value according to the expected threshold value and the accuracy value of the sample image corresponding to each evaluation threshold value, and determine the evaluation threshold corresponding to the accuracy value as the target evaluation threshold value; determine the recall value of the target evaluation threshold value as the target recall value, and obtain the performance evaluation result of the model to be evaluated according to the target recall value, the pixel dimension evaluation value of the model to be evaluated, and a preset evaluation conversion rule; the preset evaluation conversion rule can be expressed as:

[0118]

[0119] Among them, the Score total represents the performance evaluation result of the model to be evaluated; the Scorepix represents the pixel dimension evaluation value of the model to be evaluated; represents the target recall value, and ε is the adjustment factor.

[0120] Another embodiment of the present application further provides a computer storage medium, in which instructions for executing the steps described in the embodiments of the model performance evaluation method are stored.

[0121] To sum up, the model performance evaluation scheme provided in each embodiment of the present application no longer focuses solely on pixel dimension evaluation. By introducing image dimension effect evaluation, it can more intuitively reflect the detection effect of the model, which is conducive to improving the accuracy of the model detection results.

[0122] In addition, the effect evaluation based on the image dimension is oriented to multiple threshold selections, which can solve the problem of different threshold selections for detection models in different application scenarios. Through the evaluation scheme of traversing thresholds one by one, it can be suitable for evaluating the models to be evaluated in different application scenarios.

[0123] Finally, by making a multi-dimensional combined judgment and providing a set adjustment factor, different weights are assigned to the pixel dimension and the image dimension, so that the performance of the model to be evaluated can be evaluated more comprehensively.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model performance evaluation method, comprising: Using the model to be evaluated to predict the sample image based on each preset category, obtaining a category detection result corresponding to each preset category for each pixel in the sample image; According to the sample label of the sample image, analyzing the category detection result of each pixel point in the sample image corresponding to each preset category based on the pixel dimension and the image dimension, and determining the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated; Obtaining a performance evaluation result of the model to be evaluated according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated; The step of obtaining a performance evaluation result of the model to be evaluated according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated comprises: Based on each evaluation threshold and the sample annotation of the sample image, analyzing the evaluation values ​​of each image dimension of the sample image corresponding to each preset category, and obtaining the precision value and recall value of the sample image corresponding to each evaluation threshold; Obtaining a performance evaluation result of the model to be evaluated according to the accuracy value and recall value of the sample image corresponding to each evaluation threshold, the expected threshold, and the pixel dimension evaluation value of the model to be evaluated; Each preset category is a flame category and / or a smoke category; The sample label includes the real category contained in the sample image; The step of determining the image dimension evaluation value of the model to be evaluated corresponding to each preset category according to the sample label of the sample image and the category prediction value of each pixel point in the sample image corresponding to each preset category includes: Determine a preset category as the current category; Obtaining a category prediction value corresponding to the current category for each pixel; Obtaining, according to the sample label of the sample image, a judgment result of whether the sample image contains or does not contain the current category; If the sample image contains the current category, determine the category area of ​​the current category in the sample image, and according to the category prediction value of each pixel in the category area corresponding to the current category, determine the maximum category prediction value as the image dimension evaluation value of the model to be evaluated corresponding to the current category; If the sample image does not contain the current category, based on the category prediction value of each pixel in the sample image corresponding to the current category, the maximum category prediction value is determined as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

2. The method according to claim 1, wherein: The model to be evaluated includes a semantic segmentation model.

3. The method according to claim 1, wherein: The method of using the model to be evaluated to predict the sample image based on each preset category to obtain a category detection result corresponding to each preset category for each pixel in the sample image includes: The model to be evaluated is used to predict each pixel in the sample image based on each preset category, and obtains a category prediction value of each pixel in the sample image corresponding to each preset category; wherein, If the training loss function of the model to be evaluated is ordinary cross entropy, a softmax calculation is performed for each pixel point based on each preset category to obtain a category prediction value of each pixel point corresponding to each preset category; If the training loss function of the model to be evaluated is binary cross entropy, a sigmoid calculation is performed for each pixel point based on each preset category to obtain a category prediction value of each pixel point corresponding to each preset category.

4. The method according to claim 1 or 3, wherein: The step of analyzing the category detection result of each pixel point in the sample image corresponding to each preset category based on the pixel dimension and the image dimension according to the sample label of the sample image, and determining the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated includes: Determine the pixel dimension evaluation value of the model to be evaluated according to the sample label of the sample image, the category prediction value of each pixel point in the sample image corresponding to each preset category, and the position information of each pixel point; According to the sample label of the sample image and the category prediction value of each pixel point in the sample image corresponding to each preset category, the image dimension evaluation value of the model to be evaluated corresponding to each preset category is determined.

5. The method according to claim 4, wherein: The sample label includes the true category of each pixel in the sample image; and wherein, The step of determining the pixel dimension evaluation value of the model to be evaluated according to the sample label of the sample image, the category prediction value of each pixel point in the sample image corresponding to each preset category, and the position information of each pixel point includes: Determine the predicted category of each pixel point according to the highest of the predicted values ​​of each category corresponding to each preset category of each pixel point; According to the predicted category and the actual category of each pixel point and the position information of each pixel point, the intersection-and-union ratio of each preset category is obtained; The mean value calculation is performed on the intersection-and-union ratio of each preset category to obtain the pixel dimension evaluation value of the model to be evaluated.

6. The method according to claim 5, wherein: The method of obtaining the intersection-and-union ratio of each preset category according to the predicted category and the actual category of each pixel point and the position information of each pixel point includes: Determine a preset category as the current category; Determine each pixel point whose predicted category matches the current category as a first candidate pixel point, and determine each pixel point whose actual category matches the current category as a second candidate pixel point; Determine a first region of the current category according to the position information of each first candidate pixel point, and determine a second region of the current category according to the position information of each second candidate pixel point; An intersection-and-union ratio of the current category is determined according to the intersection and union of the first area and the second area.

7. The method according to claim 1, wherein: Each evaluation threshold is determined based on the evaluation granularity of the model to be evaluated; The evaluation granularity fineness includes at least one of 1 / 10, 1 / 100, and 1 / 1000; Each evaluation threshold is between 0 and 1.

8. The method according to claim 1, wherein: The step of analyzing the image dimension evaluation values ​​of the sample image corresponding to each preset category based on each evaluation threshold and the sample annotation of the sample image, and obtaining the accuracy value and recall value of the sample image corresponding to each evaluation threshold includes: a positive example identification step, determining an evaluation threshold as a current evaluation threshold, and comparing the current evaluation threshold with the image dimension evaluation value of the sample image corresponding to each preset category according to the sample annotation of the sample image, to obtain a positive example identification result of the sample image corresponding to the current evaluation threshold; Repeating the positive example identification step to obtain a positive example identification result of the sample image corresponding to each evaluation threshold; According to the positive example identification result of the sample image corresponding to each evaluation threshold, the accuracy value and the recall value of the sample image corresponding to each evaluation threshold are obtained.

9. The method according to claim 8, wherein: The positive example identification step comprises: Determine a preset category as the current category; Based on the sample annotation of the sample image, the current evaluation threshold, and the image dimension evaluation value of the sample image corresponding to the current category, a positive example identification result of the sample image corresponding to the current category is determined; wherein, In the case where the sample image contains the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a sample recognition result indicating that the sample image is a true positive example is obtained; if the image dimension evaluation value of the sample image corresponding to the current category is not greater than the current evaluation threshold, a positive example identification result indicating that the sample image is a false positive example is obtained; In the case that the sample image does not contain the current category, if the image dimension evaluation value of the sample image corresponding to the current category is greater than the current evaluation threshold, a positive example identification result indicating that the sample image is a false positive example is obtained.

10. The method according to claim 1, wherein: The step of obtaining the performance evaluation result of the model to be evaluated according to the accuracy value and the recall value of the sample image corresponding to each evaluation threshold, the expected threshold, and the pixel dimension evaluation value of the model to be evaluated comprises: According to the expected threshold and the accuracy value of the sample image corresponding to each evaluation threshold, determine the accuracy value with the smallest difference from the expected threshold, and determine the evaluation threshold corresponding to the accuracy value as the target evaluation threshold; Determining the recall value of the target evaluation threshold as the target recall value, and obtaining the performance evaluation result of the model to be evaluated according to the target recall value, the pixel dimension evaluation value of the model to be evaluated, and a preset evaluation conversion rule; The preset evaluation conversion rule can be expressed as: Among them, the Score total represents the performance evaluation result of the model to be evaluated; the Score pix represents the pixel dimension evaluation value of the model to be evaluated; represents the target recall value, and ε is the adjustment factor.

11. A model performance evaluation device, comprising: A category detection module is used to predict the sample image based on each preset category using the model to be evaluated, and obtain a category detection result corresponding to each preset category for each pixel in the sample image; An evaluation module, configured to analyze, according to the sample label of the sample image, the category detection result of each pixel point in the sample image corresponding to each preset category based on the pixel dimension and the image dimension, determine the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated, and obtain the performance evaluation result of the model to be evaluated according to the pixel dimension evaluation value and the image dimension evaluation value of the model to be evaluated; The evaluation module is also used to: analyze the evaluation values ​​of each image dimension of the sample image corresponding to each preset category based on each evaluation threshold and the sample annotation of the sample image, and obtain the accuracy value and recall value of the sample image corresponding to each evaluation threshold; obtain the performance evaluation result of the model to be evaluated according to the accuracy value and recall value of the sample image corresponding to each evaluation threshold, the expected threshold, and the pixel dimension evaluation value of the model to be evaluated; Each preset category is a flame category and / or a smoke category; The sample label includes the real category contained in the sample image; The step of determining the image dimension evaluation value of the model to be evaluated corresponding to each preset category according to the sample label of the sample image and the category prediction value of each pixel point in the sample image corresponding to each preset category includes: Determine a preset category as the current category; Obtaining a category prediction value corresponding to the current category for each pixel; Obtaining, according to the sample label of the sample image, a judgment result of whether the sample image contains or does not contain the current category; If the sample image contains the current category, determine the category area of ​​the current category in the sample image, and according to the category prediction value of each pixel in the category area corresponding to the current category, determine the maximum category prediction value as the image dimension evaluation value of the model to be evaluated corresponding to the current category; If the sample image does not contain the current category, based on the category prediction value of each pixel in the sample image corresponding to the current category, the maximum category prediction value is determined as the image dimension evaluation value of the model to be evaluated corresponding to the current category.

12. A computer storage medium, characterized in that: The computer storage medium stores instructions for executing the steps in the method according to any one of claims 1 to 10.

Images