Performance testing method, device, equipment and storage medium of target detection model

Abstract

The present invention discloses a performance test method, device, equipment and storage medium of a target detection model, and relates to the technical field of model testing. The method comprises: obtaining an image data set, the image data set comprising: an image with a target, a label of the image and a semantic map of the image; for each image in the image data set, based on the label of the image and the semantic map of the image, extracting all targets in the image and the number of pixels of each target, and determining the pixel level of the target based on the number of pixels of the target; processing the image in the image data set by the target detection model to obtain a prediction result, and based on the prediction result and the label, counting the number of true positive examples, the number of false positive examples and the number of false negative examples of each target category at each pixel level; based on the number of true positive examples, the number of false positive examples and the number of false negative examples, calculating the average precision mean of the target detection model at each pixel level. The present invention better measures the detection performance of the target detection model for small targets.

Description

Technical Field

[0001] The present invention relates to the field of model testing technology, and in particular to a performance testing method, device, equipment and storage medium for a target detection model. Background Art

[0002] Object detection is a long-standing and challenging fundamental problem in the field of computer vision, which aims to determine whether there are object instances of interest in a visual image, and if so, return the category and location information of each object instance. As the basis for solving more advanced visual tasks (such as segmentation, scene understanding, and object tracking), object detection is widely used in many fields, including consumer electronics, autonomous driving, and intelligent video surveillance.

[0003] In recent years, with the rapid development of deep learning technology, object detection based on deep learning has greatly surpassed traditional object detection based on manually extracted features in terms of accuracy and test speed. Object detection based on deep learning has become an important research hotspot in the field of computer vision.

[0004] However, most current target detection models are targeted at medium-to-large targets that are sparsely distributed in natural scenes, and have certain limitations in detecting small targets that occupy fewer pixels in the original image and carry limited information. After multiple downsampling, the resolution of small targets is further reduced, the feature information is gradually weakened, and the detection difficulty increases. Therefore, the general target detection model has the disadvantage of low detection accuracy for small targets. Small targets are widely present in aerospace images and remote sensing images taken at long distances. On the COCO dataset, which is commonly used to test the performance of methods, the detection accuracy of small targets is usually only half of that of large targets. The small target problem affects the overall performance of the detector on the COCO dataset. Summary of the invention

[0005] In order to overcome the limitations of existing small target detection methods, the present invention discloses a performance testing method, device, equipment and storage medium for a target detection model.

[0006] The technical solution of the present invention includes:

[0007] A performance testing method for a target detection model, the method comprising:

[0008] Acquire an image dataset, the image dataset comprising: an image with a target, a label of the image, and a semantic map of the image;

[0009] For each image in the image dataset, based on the label of the image and the semantic map of the image, extract all objects in the image and the number of pixels of each object, and determine the pixel level of the object based on the number of pixels of the object;

[0010] Processing the images in the image data set by the target detection model to obtain prediction results, and based on the prediction results and the labels, counting the number of true positive examples, false positive examples, and false negative examples of each target category at each pixel level;

[0011] Based on the number of true positive examples, the number of false positive examples, and the number of false negative examples, the average precision mean of the target detection model at each pixel level is calculated.

[0012] Optionally, extracting all objects in the image and the number of pixels of each object based on the label of the image and the semantic graph of the image includes:

[0013] Constructing a semantic graph matrix based on the semantic graph of the image;

[0014] Initializing the number of targets based on the number of targets indicated by the label of the image;

[0015] When traversing to the w-th target, the semantic information of the w-th target is obtained, and the number of pixels of the w-th target is assigned to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the w-th target.

[0016] Optionally, determining the pixel level of the target based on the number of pixels of the target includes:

[0017] Acquire a correspondence table, wherein the correspondence table includes at least one set of correspondences between pixel quantities and pixel levels;

[0018] Based on the correspondence table, a pixel level corresponding to the number of pixels of the target is obtained and determined as the pixel level of the target.

[0019] Optionally, the process of processing the images in the image dataset by the target detection model to obtain a prediction result, and counting the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level based on the prediction result and the label, includes:

[0020] When traversing to the sth image in the image data set, inputting the sth image into the target detection model to obtain all prediction results in the sth image whose prediction label is the target category C;

[0021] Based on the label of the s-th image, extract all objects whose true labels are the object category C in the s-th image;

[0022] Based on the intersection-and-union ratio between all targets of the target category C with their true labels and each prediction result, the number of true positive examples, the number of false positive examples, and the number of false negative examples of the target category C at each pixel level are obtained;

[0023] Until all target categories and all images in the image data set are traversed, the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level are obtained.

[0024] Optionally, obtaining the number of true positive examples, the number of false positive examples, and the number of false negative examples of the target category C at each pixel level based on the intersection-and-union ratio between all targets whose true labels are the target category C and each prediction result, comprises:

[0025] When traversing to the i-th prediction result, determining the pixel level k of the i-th prediction result based on the number of pixels of the i-th prediction result;

[0026] Calculate the intersection-over-union ratio between all targets whose true labels are the target category C and the i-th prediction result;

[0027] If there is an IoU ratio between a target and the i-th prediction result, which is greater than the IoU ratio threshold and greater than the IoU ratio between any other target and the i-th prediction result, then the i-th prediction result is determined to be a true positive example of the target category C at the pixel level k;

[0028] If the intersection-over-union ratios between all targets and the i-th prediction result are less than the intersection-over-union ratio threshold, then the i-th prediction result is determined to be a false positive example of the target category C at the pixel level k;

[0029] If there is a t-th target that does not match any of the prediction results and the t-th target belongs to the pixel level k, then the t-th target is a false negative example of the target category C at the pixel level k;

[0030] Until all prediction results and all targets are traversed, the number of true positive examples, false positive examples, and false negative examples of the target category C at each pixel level are obtained.

[0031] Optionally, the calculating the average precision mean of the target detection model at each pixel level based on the number of true positive examples, the number of false positive examples, and the number of false negative examples includes:

[0032] When traversing to the target category C and the pixel level k, based on the number of true positive examples and the number of false positive examples of the target category C at the pixel level k, the accuracy of the target category C at the pixel level k is calculated;

[0033] Calculate the recall rate of the target category C at the pixel level k based on the number of true positive examples and the number of false negative examples of the target category C at the pixel level k;

[0034] Until all pixel levels in the target category C are traversed to obtain the precision and recall rate of the target category C at all pixel levels;

[0035] Based on the precision and recall of the target category C at all pixel levels, calculate the average precision of the target category C at each pixel level;

[0036] Until all target categories are traversed, the average accuracy of each target category at each pixel level is obtained;

[0037] For the pixel level k, the average precision of all target categories at the pixel level k is averaged to obtain the mean of the average precision of the target detection model at the pixel level k.

[0038] Optionally, the semantic map is in the form of a grayscale map.

[0039] A performance testing device for a target detection model, the device comprising:

[0040] An image data set acquisition module, used to acquire an image data set, wherein the image data set includes: an image with a target, a label of the image, and a semantic map of the image;

[0041] A pixel level determination module, for extracting, for each image in the image data set, all objects in the image and the number of pixels of each object based on the label of the image and the semantic map of the image, and determining the pixel level of the object based on the number of pixels of the object;

[0042] A prediction result analysis module, used to process the images in the image data set through the target detection model to obtain prediction results, and based on the prediction results and the labels, count the number of true positive examples, false positive examples and false negative examples of each target category at each pixel level;

[0043] The model performance calculation module is used to calculate the average precision mean of the target detection model at each pixel level based on the number of true positive examples, the number of false positive examples and the number of false negative examples.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned performance testing method for a target detection model.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the above-mentioned performance testing method for the target detection model.

[0046] A computer program product, when the computer program product is run on a computer device, enables the computer device to execute the performance testing method for the target detection model as described above.

[0047] The present invention divides small targets into multiple pixel levels according to the number of pixels occupied by the small targets, and then evaluates the detection performance of the target detection model at each pixel level, giving the influence of target size on the detection performance of the target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flowchart of the performance testing methodology for object detection models.

[0049] Figure 2 Flowchart of the method for obtaining the number of pixels and the pixel level.

[0050] Figure 3 Flowchart of the forecast results analysis method.

[0051] Figure 4 Block diagram of the performance testing setup for object detection models. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0053] The present invention proposes a performance testing method for a target detection model, such as Figure 1 As shown, the following steps are included (steps 110 to 140).

[0054] Step 110: Obtain an image dataset, where the image dataset includes: images with targets, labels of the images, and semantic maps of the images.

[0055] In the present invention, the computer device can obtain an image data set from other devices, can also obtain an image data set from local storage, and can also generate and obtain an image data set in real time. Among them, the image data set includes an image with a target, an image label, and an image semantic map. In the present invention, an image includes at least one target, and the image label carries the category, detailed coordinate information, semantic information, etc. of each target in the image, and the image semantic map details the pixel position of each target in the image. Optionally, the image semantic map is in the form of a grayscale image.

[0056] Step 120: For each image in the image dataset, based on the label of the image and the semantic map of the image, all objects in the image and the number of pixels of each object are extracted, and the pixel level of the object is determined based on the number of pixels of the object.

[0057] In order to better measure the detection performance of the target detection model for small targets, the present invention divides several pixel levels based on the number of pixels occupied by the target (i.e., "the number of pixels of the target"). For each image in the image data set, the computer device obtains the number of targets in the image based on the label of the image, and takes out each target in turn. When taking out the target, the semantic information belonging to the corresponding target can be obtained based on the semantic map of the image, and then the number of semantic information belonging to the corresponding target can be counted to obtain the number of pixels of the corresponding target, and then according to the corresponding relationship between the number of pixels and the pixel level, the pixel level of the corresponding target can be obtained.

[0058] In one example, if Figure 2 As shown, the method for obtaining the number of pixels and the pixel level of the present invention includes the following steps (steps 121 to 125).

[0059] Step 121: construct a semantic graph matrix based on the semantic graph of the image.

[0060] Optionally, the dimensions of the semantic map matrix segmanticMatrix are the height and width of the image, and the number of channels is 1.

[0061] Step 122: Initialize the number of targets based on the number of targets indicated by the label of the image.

[0062] The label of the image includes the category, detailed coordinate information, semantic information, etc. of each target in the image. The computer device initializes the target number objectNum to the target number indicated by the label, and then traverses and takes out each target.

[0063] Step 123: When traversing to the w-th target, the semantic information of the w-th target is obtained, and the number of pixels of the w-th target is assigned to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the w-th target.

[0064] If the traversed and retrieved target is the wth target (w is a positive integer), the semantic information segmenticID of the wth target is obtained, and the number of pixels ithPObjectPixel of the wth target is assigned to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the wth target. For other targets in the image, the same method is used to obtain the number of pixels of the target.

[0065] Step 124: Obtain a corresponding relationship table.

[0066] The correspondence table includes at least one set of correspondence between the number of pixels and the pixel level. Optionally, as shown in Table 1, the present invention divides the pixel levels into 11 levels.

[0067] Table 1 Correspondence between pixel number and pixel level

[0068]

[0069]

[0070] Step 125: Based on the correspondence table, the pixel level corresponding to the number of pixels of the target is obtained, and determined as the pixel level of the target.

[0071] Based on the corresponding relationship table shown in Table 1 above, the computer device can obtain the pixel level corresponding to the number of pixels of the target, and determine and save it as the pixel level of the target.

[0072] In one example, after obtaining the pixel level of each target in each image, the computer device may further add a scale attribute to represent the pixel level of the target based on the annotation of the image dataset.

[0073] Step 130: Process the images in the image dataset through the target detection model to obtain prediction results, and based on the prediction results and labels, count the number of true positive examples, false positive examples, and false negative examples of each target category at each pixel level.

[0074] In order to evaluate the detection performance of the target detection model at different pixel levels, the computer equipment needs to obtain the prediction results of the target detection model for all images in the image dataset, and calculate the prediction accuracy based on the prediction results, that is, the number of true positive examples, false positive examples, false negative examples, etc.

[0075] In one example, if Figure 3 As shown, the prediction result analysis method of the present invention includes the following steps (steps 131 to 134).

[0076] Step 131: When traversing to the sth image in the image data set, the sth image is input into the target detection model to obtain all prediction results whose predicted label is the target category C in the sth image.

[0077] The computer device takes a single image image (the sth image), an image label label, an object detection model OD, an IOU threshold iouthreshold, and a specified object category C as input. First, the computer device inputs the image image into the object detection model OD, and processes the image through the object detection model OD to obtain all prediction results predictResult whose predicted label is the object category C in the image image.

[0078] Step 132: Based on the label of the s-th image, extract all objects whose true labels are the object category C in the s-th image.

[0079] like Figure 3 As shown, for a specified target category C, the computer device obtains all prediction results predictResult whose predicted labels are target category C in the s-th image through step 131, and obtains all target groundtruth gtBBox whose real labels are target category C in the s-th image through step 132.

[0080] It should be understood that step 131 may be performed before step 132, may be performed after step 132, or may be performed simultaneously with step 132, and the present invention is not limited to this.

[0081] Step 133: Based on the intersection-and-union ratios between all targets whose true labels are target category C and each prediction result, the number of true positive examples, false positive examples, and false negative examples of target category C at each pixel level is obtained.

[0082] Optionally, the process of obtaining the number of true positive examples, the number of false positive examples, and the number of false negative examples includes the following steps (steps 1331 to 1336).

[0083] Step 1331: when traversing to the i-th prediction result, determine the pixel level k of the i-th prediction result based on the number of pixels of the i-th prediction result.

[0084] The computer device sequentially traverses all prediction results obtained in step 131. Optionally, when obtaining all prediction results whose prediction labels are target category C in the s-th image in step 131, the confidence of each prediction result is also obtained at the same time, so that the computer device sequentially traverses all prediction results in order of confidence from high to low or from low to high. Figure 3As shown, when traversing to the i-th (i is a positive integer) prediction result ithDet, the i-th prediction result ithDet is taken out, and the pixel level k of the i-th prediction result is determined based on the number of pixels of the i-th prediction result.

[0085] Step 1332: Calculate the intersection-over-union ratio between all targets whose true labels are target category C and the i-th prediction result.

[0086] like Figure 3 As shown, when traversing to the i-th prediction result ithDet, the computer device also calculates the intersection-over-union (IOU) ratio between the i-th prediction result and each target gtBBox obtained in step 132.

[0087] It should be understood that step 1331 may be executed before step 1332, may be executed after step 1332, or may be executed simultaneously with step 1332, and the present invention is not limited to this.

[0088] Step 1333: If there is an IoU between a target and the i-th prediction result that is greater than the IoU threshold and greater than the IoU between any other target and the i-th prediction result, then the i-th prediction result is determined to be a true example of target category C at pixel level k.

[0089] In this step, if there is an intersection-and-union ratio between the jth target and the i-th prediction result among the targets obtained in step 132, which is greater than the intersection-and-union ratios between other targets and the i-th prediction result (that is, the maximum value of the intersection-and-union ratios between the i-th prediction result and each target obtained in step 132 is the intersection-and-union ratio between the j-th target and the i-th prediction result), and is greater than a pre-set intersection-and-union ratio threshold iouThreshold, then it means that the j-th target is detected (the j-th target is detected as the i-th prediction result) and the detection is correct, that is, the i-th prediction result is a true positive (TP) in pixel level k of category C.

[0090] To ensure statistical accuracy, optionally, if the ith prediction result is a true positive example in pixel level k of category C, the computer device removes the jth target corresponding to the ith prediction result from the targets acquired in step 132 .

[0091] Step 1334: If the IoU ratios between all targets and the i-th prediction result are less than the IoU threshold, the i-th prediction result is determined to be a false positive example of target category C at pixel level k.

[0092] In this step, if the intersection-and-union ratios between each target in the targets obtained in step 132 and the i-th prediction result are less than or equal to the intersection-and-union ratio threshold, it means that there is no target matching the i-th prediction result in the targets obtained in step 132, and the i-th prediction result is detected incorrectly, that is, the i-th prediction result is a false positive (FP) in pixel level k of category C.

[0093] Step 1335: If there is a t-th target that does not match any prediction result and the t-th target belongs to pixel level k, then the t-th target is a false negative example of target category C at pixel level k.

[0094] In this step, after the computer device traverses all the prediction results obtained in step 131, if there is a target t that does not match all the prediction results among the targets obtained in step 132, it means that the target t has not been detected. Further, if the target t belongs to pixel level k, the target t is a false negative (FN) in pixel level k of category C.

[0095] Step 1336: until all prediction results and all targets are traversed, the number of true positive examples, false positive examples, and false negative examples of target category C at each pixel level are obtained.

[0096] Step 134: until all target categories and all images in the image data set are traversed, the number of true positive examples, the number of false positive examples, and the number of false negative examples at each pixel level for each target category are obtained.

[0097] For each image in the image data set and each specified target category, the computer device repeatedly executes the above steps 131 to 134 and counts the number of true positive examples, false positive examples, and false negative examples, so as to obtain the number of true positive examples, false positive examples, and false negative examples at each pixel level of each category.

[0098] For example, Figure 3 As shown, before traversing all the prediction results obtained in step 131, the computer device initializes the array of true positive examples tp of all pixel categories of array category C, false positive examples fp of all pixel categories of array category C, and false negative examples fn of all pixel categories of array category C, with a length equal to the number of pixel categories, and each element of the array is an empty list. If the situation of step 1333 occurs, set tp[k]=tp[k]+1; if the situation of step 1334 occurs, set fp[k]=fp[k]+1; if the situation of step 1335 occurs, set fn[k]=fn[k]+1. Finally, the computer device outputs the tp, fp, and fn arrays.

[0099] Step 140: Based on the number of true positive examples, the number of false positive examples, and the number of false negative examples, calculate the average precision of the target detection model at each pixel level.

[0100] In one example, the method for calculating the average precision mean at each pixel level of the present invention includes the following steps (steps 141 to 146).

[0101] Step 141: When traversing to the target category C and the pixel level k, based on the number of true positive examples and the number of false positive examples of the target category C at the pixel level k, the accuracy of the target category C at the pixel level k is calculated.

[0102] The accuracy of target category C at pixel level k is the proportion of correct predictions at pixel level k among all prediction results whose prediction labels are target category C. Ck represents the number of true positive examples of target category C at pixel level k, FP Ck represents the number of false positive examples of target category C at pixel level k, then the accuracy P of target category C at pixel level k Ck The calculation formula is as follows.

[0103]

[0104] Step 142: Calculate the recall rate of the target category C at the pixel level k based on the number of true positive examples and the number of false negative examples of the target category C at the pixel level k.

[0105] The recall rate of target category C at pixel level k is the proportion of all targets whose true labels are target category C that are correctly predicted at pixel level k. Ck represents the number of true positive examples of target category C at pixel level k, FN Ck represents the number of false negative examples of target category C at pixel level k, then the recall rate R of target category C at pixel level k Ck The calculation formula is as follows.

[0106]

[0107] Step 143: until all pixel levels in the target category C are traversed, the precision and recall rate of the target category C at all pixel levels are obtained.

[0108] Step 144: Based on the precision and recall of the target category C at all pixel levels, calculate the average precision of the target category C at each pixel level.

[0109] The AP (Average Precision) of a single target category can better evaluate the accuracy of a model. In the present invention, based on the accuracy and recall rate of the target category C at all pixel levels, the computer device can draw a PR (Precision Recall) curve. Usually, the precision P Ck Set as the vertical axis and set the recall rate R Ck Set as the horizontal axis. After completing the PR curve drawing for the target category C at different pixel levels, for the pixel level k of the target category C, the computer device can obtain the area of ​​the pixel level k of the target category C under the PR curve (for example, the computer device uniformly samples, that is, respectively takes the vertical axis (precision) p value corresponding to the horizontal axis (recall rate) r = 0.00, 0.01, 0.02, ..., 1.00, and then calculates the average of these 101 p values), and the area is the AP of the target category C at the pixel level k.

[0110] Step 145: until all target categories are traversed, the average accuracy of each target category at each pixel level is obtained.

[0111] For each specified target category, the computer device repeatedly executes steps 141 to 144 to obtain the AP of each target category at each pixel level.

[0112] Step 146: For pixel level k, average the average precision of all target categories at pixel level k to obtain the mean average precision of the target detection model at pixel level k.

[0113] In the present invention, the detection performance of the image dataset for objects of different sizes is evaluated by the mAP (mean Average Precision) of the image dataset at different pixel levels. Taking pixel level k as an example, the computer device averages the average precision of pixel level k in all specified target categories to obtain the mAP of the image dataset at pixel level k.

[0114] To sum up, due to the variable shape of the target, the target annotation box often occupies the target's real pixels and background pixels or other target pixels at the same time, which leads to errors when using the target annotation box to classify the target size. In order to better measure the detection performance of the target detection model for small targets, the present invention divides small targets into multiple pixel levels according to the number of pixels occupied, and then evaluates the detection performance of the target detection model at each pixel level, and gives the impact of the target size on the detection performance of the target detection model.

[0115] Please refer to Figure 4, which shows a block diagram of a performance testing device for a target detection model provided by an embodiment of the present invention. The device may be a computer device, or may be arranged in a computer device. Figure 4 As shown, the device includes the following modules: an image data set acquisition module 410, a pixel level determination module 420, a prediction result analysis module 430, and a model performance calculation module 440.

[0116] The image data set acquisition module 410 is used to acquire an image data set, where the image data set includes: an image with a target, a label of the image, and a semantic map of the image;

[0117] A pixel level determination module 420 is used to extract all objects in the image and the number of pixels of each object based on the label of the image and the semantic map of the image for each image in the image data set, and determine the pixel level of the object based on the number of pixels of the object;

[0118] The prediction result analysis module 430 is used to process the images in the image data set through the target detection model to obtain the prediction results, and based on the prediction results and labels, count the number of true positive examples, false positive examples and false negative examples of each target category at each pixel level;

[0119] The model performance calculation module 440 is used to calculate the average precision of the target detection model at each pixel level based on the number of true positive examples, the number of false positive examples, and the number of false negative examples.

[0120] Optionally, the level determination module 420 is used to:

[0121] Constructing a semantic graph matrix based on the semantic graph of the image;

[0122] Initializing the number of targets based on the number of targets indicated by the label of the image;

[0123] When traversing to the w-th target, the semantic information of the w-th target is obtained, and the number of pixels of the w-th target is assigned to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the w-th target.

[0124] Optionally, the level determination module 420 is used to:

[0125] Acquire a correspondence table, wherein the correspondence table includes at least one set of correspondences between pixel quantities and pixel levels;

[0126] Based on the correspondence table, a pixel level corresponding to the number of pixels of the target is obtained and determined as the pixel level of the target.

[0127] Optionally, the image analysis module 430 is used to:

[0128] When traversing to the sth image in the image data set, inputting the sth image into the target detection model to obtain all prediction results in the sth image whose prediction label is the target category C;

[0129] Based on the label of the s-th image, extract all objects whose true labels are the object category C in the s-th image;

[0130] Based on the intersection-and-union ratio between all targets of the target category C with their true labels and each prediction result, the number of true positive examples, the number of false positive examples, and the number of false negative examples of the target category C at each pixel level are obtained;

[0131] Until all target categories and all images in the image data set are traversed, the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level are obtained.

[0132] Optionally, the image analysis module 430 is further used to:

[0133] When traversing to the i-th prediction result, determining the pixel level k of the i-th prediction result based on the number of pixels of the i-th prediction result;

[0134] Calculate the intersection-over-union ratio between all targets whose true labels are the target category C and the i-th prediction result;

[0135] If there is an IoU ratio between a target and the i-th prediction result, which is greater than the IoU ratio threshold and greater than the IoU ratio between any other target and the i-th prediction result, then the i-th prediction result is determined to be a true positive example of the target category C at the pixel level k;

[0136] If the intersection-over-union ratios between all targets and the i-th prediction result are less than the intersection-over-union ratio threshold, then the i-th prediction result is determined to be a false positive example of the target category C at the pixel level k;

[0137] If there is a t-th target that does not match any of the prediction results and the t-th target belongs to the pixel level k, then the t-th target is a false negative example of the target category C at the pixel level k;

[0138] Until all prediction results and all targets are traversed, the number of true positive examples, false positive examples, and false negative examples of the target category C at each pixel level are obtained.

[0139] Optionally, the performance calculation module 440 is used to:

[0140] When traversing to the target category C and the pixel level k, based on the number of true positive examples and the number of false positive examples of the target category C at the pixel level k, the accuracy of the target category C at the pixel level k is calculated;

[0141] Calculate the recall rate of the target category C at the pixel level k based on the number of true positive examples and the number of false negative examples of the target category C at the pixel level k;

[0142] Until all pixel levels in the target category C are traversed to obtain the precision and recall rate of the target category C at all pixel levels;

[0143] Based on the precision and recall of the target category C at all pixel levels, calculate the average precision of the target category C at each pixel level;

[0144] Until all target categories are traversed, the average accuracy of each target category at each pixel level is obtained;

[0145] For the pixel level k, the average precision of all target categories at the pixel level k is averaged to obtain the mean of the average precision of the target detection model at the pixel level k.

[0146] Optionally, the semantic map is in the form of a grayscale map.

[0147] For the detailed description of the implementation process, beneficial effects, etc. of the device module, please refer to the introduction of the above method embodiment, which will not be elaborated here.

[0148] In an exemplary embodiment, a computer device is also provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the performance testing method of the target detection model mentioned above.

[0149] In an exemplary embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the performance testing method of the target detection model as described above is implemented.

[0150] In an exemplary embodiment, a computer program product is also provided. When the computer program product is run on a computer device, the computer device executes the performance testing method of the target detection model as described above.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A performance testing method for a target detection model, characterized in that: The method comprises: Acquire an image dataset, the image dataset comprising: an image with a target, a label of the image, and a semantic map of the image; For each image in the image data set, based on the label of the image and the semantic map of the image, all objects in the image and the number of pixels of each object are extracted, and the pixel level of the object is determined based on the number of pixels of the object; wherein, based on the label of the image and the semantic map of the image, all objects in the image and the number of pixels of each object are extracted, including: Constructing a semantic graph matrix based on the semantic graph of the image; Initializing the number of targets based on the number of targets indicated by the label of the image; When traversing to the w-th target, obtaining the semantic information of the w-th target, and assigning the number of pixels of the w-th target to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the w-th target; The images in the image data set are processed by the target detection model to obtain prediction results, and based on the prediction results and the labels, the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level are counted; wherein the images in the image data set are processed by the target detection model to obtain prediction results, and based on the prediction results and the labels, the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level are counted, including: When traversing to the sth image in the image data set, inputting the sth image into the target detection model to obtain all prediction results in the sth image whose prediction label is the target category C; Based on the label of the s-th image, extract all objects whose true labels are the object category C in the s-th image; Based on the intersection-and-union ratio between all targets of the target category C with their true labels and each prediction result, the number of true positive examples, the number of false positive examples, and the number of false negative examples of the target category C at each pixel level are obtained; Until all target categories and all images in the image data set are traversed to obtain the number of true positive examples, false positive examples, and false negative examples of each target category at each pixel level; Based on the number of true positive examples, the number of false positive examples, and the number of false negative examples, the average precision mean of the target detection model at each pixel level is calculated.

2. The method according to claim 1, characterized in that Determining the pixel level of the target based on the number of pixels of the target includes: Acquire a correspondence table, wherein the correspondence table includes at least one set of correspondences between pixel quantities and pixel levels; Based on the correspondence table, a pixel level corresponding to the number of pixels of the target is obtained and determined as the pixel level of the target.

3. The method according to claim 1, characterized in that The method of obtaining the number of true positive examples, false positive examples, and false negative examples of the target category C at each pixel level based on the intersection-and-union ratio between all targets of the target category C with the true label and each prediction result includes: When traversing to the i-th prediction result, determining the pixel level k of the i-th prediction result based on the number of pixels of the i-th prediction result; Calculate the intersection-over-union ratio between all targets whose true labels are the target category C and the i-th prediction result; If there is an IoU ratio between a target and the i-th prediction result, which is greater than the IoU ratio threshold and greater than the IoU ratio between any other target and the i-th prediction result, then the i-th prediction result is determined to be a true positive example of the target category C at the pixel level k; If the intersection-over-union ratios between all targets and the i-th prediction result are less than the intersection-over-union ratio threshold, then the i-th prediction result is determined to be a false positive example of the target category C at the pixel level k; If there is a t-th target that does not match any of the prediction results and the t-th target belongs to the pixel level k, then the t-th target is a false negative example of the target category C at the pixel level k; Until all prediction results and all targets are traversed, the number of true positive examples, false positive examples, and false negative examples of the target category C at each pixel level are obtained.

4. The method according to claim 1, characterized in that The calculating the average precision mean of the target detection model at each pixel level based on the number of true positive examples, the number of false positive examples, and the number of false negative examples includes: When traversing to the target category C and the pixel level k, based on the number of true positive examples and the number of false positive examples of the target category C at the pixel level k, the accuracy of the target category C at the pixel level k is calculated; Calculate the recall rate of the target category C at the pixel level k based on the number of true positive examples and the number of false negative examples of the target category C at the pixel level k; Until all pixel levels in the target category C are traversed to obtain the precision and recall rate of the target category C at all pixel levels; Based on the precision and recall of the target category C at all pixel levels, calculate the average precision of the target category C at each pixel level; Until all target categories are traversed, the average accuracy of each target category at each pixel level is obtained; For the pixel level k, the average precision of all target categories at the pixel level k is averaged to obtain the mean of the average precision of the target detection model at the pixel level k.

5. The method according to claim 1, characterized in that The semantic map is in the form of a grayscale map.

6. A performance testing device for a target detection model, characterized in that: The device comprises: An image data set acquisition module, used to acquire an image data set, wherein the image data set includes: an image with a target, a label of the image, and a semantic map of the image; A pixel level determination module is used to extract, for each image in the image data set, all objects in the image and the number of pixels of each object based on the label of the image and the semantic map of the image, and determine the pixel level of the object based on the number of pixels of the object; wherein, extracting all objects in the image and the number of pixels of each object based on the label of the image and the semantic map of the image includes: Constructing a semantic graph matrix based on the semantic graph of the image; Initializing the number of targets based on the number of targets indicated by the label of the image; When traversing to the w-th target, obtaining the semantic information of the w-th target, and assigning the number of pixels of the w-th target to the number of pixels whose values ​​in the semantic map matrix are equal to the semantic information of the w-th target; A prediction result analysis module, used to process the images in the image data set through the target detection model to obtain prediction results, and based on the prediction results and the labels, count the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level; wherein, the processing of the images in the image data set through the target detection model to obtain prediction results, and based on the prediction results and the labels, counting the number of true positive examples, the number of false positive examples, and the number of false negative examples of each target category at each pixel level, includes: When traversing to the sth image in the image data set, inputting the sth image into the target detection model to obtain all prediction results in the sth image whose prediction label is the target category C; Based on the label of the s-th image, extract all objects whose true labels are the object category C in the s-th image; Based on the intersection-and-union ratio between all targets of the target category C with their true labels and each prediction result, the number of true positive examples, the number of false positive examples, and the number of false negative examples of the target category C at each pixel level are obtained; Until all target categories and all images in the image data set are traversed to obtain the number of true positive examples, false positive examples, and false negative examples of each target category at each pixel level; The model performance calculation module is used to calculate the average precision mean of the target detection model at each pixel level based on the number of true positive examples, the number of false positive examples and the number of false negative examples.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute any one of the methods of claims 1-5 when run.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method according to any one of claims 1 to 5.

Images