Target detection performance evaluation method and system based on average inter-class detection precision difference

Abstract

The application relates to a target detection performance evaluation method and system based on average inter-class detection precision difference, which comprises the following steps: constructing a target detection data set; the target detection data set comprises multiple labeled target images; a target detection model is used to detect the target detection data set to obtain detection precision values of different labeled categories; each labeled category is numbered in ascending order, and the detection precision value of each number is sequentially subtracted from the detection precision value of the subsequent number and the absolute value is taken to obtain multiple subtraction result values; the average value of the subtraction result values is obtained to obtain the average inter-class detection precision difference; and the average inter-class detection precision difference is used to evaluate the balance of the detection performance of the target detection model on different categories. The average value of the absolute values of the detection precision difference values between all different categories is calculated, so that the balance of the detection performance of the model on different categories on the data set can be accurately evaluated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a target detection performance evaluation method and system based on average inter-class detection precision difference. BACKGROUND

[0002] With the development of computer technology and the wide application of computer vision, using computer image processing technology to detect targets has become a research hotspot and has been widely applied in many fields. Visual target detection is a classic task of computer vision, aiming to locate the position of existing objects in an image and identify the specific categories of the objects. Among them, target detection based on deep learning has become the mainstream method.

[0003] When using a detection model based on deep learning to detect targets, since there are multiple target categories in the data set, the detection accuracy of the model for different target categories is not the same, and there is a large difference in detection accuracy between some target categories. The model detection result shows a kind of imbalance, and the existing evaluation index cannot fully explain the detection ability of the model for targets. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a target detection performance evaluation method and system based on average inter-class detection precision difference.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A target detection performance evaluation method based on average inter-class detection precision difference, comprising:

[0007] constructing a target detection data set; the target detection data set includes multiple labeled target images;

[0008] detecting the target detection data set according to a target detection model to obtain detection precision values of different labeled categories;

[0009] numbering each labeled category in order from small to large, and sequentially subtracting the detection precision value of each numbered category from the detection precision value of the subsequent numbered category and taking the absolute value to obtain multiple subtraction result values;

[0010] averaging the subtraction result values to obtain an average inter-class detection precision difference; the average inter-class detection precision difference is used to evaluate the balance of the detection performance of the target detection model for different categories.

[0011] Preferably, the target detection data set is constructed, comprising:

[0012] labeling the collected target images using image labeling software to obtain labeled target images;

[0013] determining the target detection data set according to the target image.

[0014] Preferably, the target detection data set is detected according to the target detection model to obtain detection precision values of different annotation categories, including:

[0015] The target detection data set is divided into a training set and a test set according to a predetermined proportion;

[0016] The target detection model is constructed.

[0017] The target detection model is trained on the training set to obtain a trained model;

[0018] The trained model is tested on the test set to obtain the detection precision values of the different annotation categories.

[0019] Preferably, the calculation formula of the average inter-class detection precision difference is:

[0020]

[0021] wherein, ADPD is the average inter-class detection precision difference, N is the number of numbers, N≥2, P i is the detection precision of the annotation category numbered i, P j is the detection precision of the annotation category numbered j, is the number of difference result values.

[0022] Preferably, the image annotation software is IabelImg.

[0023] A target detection performance evaluation system based on an average inter-class detection precision difference, comprising:

[0024] A data set construction module for constructing a target detection data set; the target detection data set includes multiple annotated target images;

[0025] A detection module for detecting the target detection data set according to a target detection model to obtain detection precision values of different annotation categories;

[0026] A numbering difference module for numbering each of the annotation categories in ascending order, and sequentially differencing and taking absolute values of each numbered detection precision value and the detection precision value of the subsequent number to obtain multiple difference result values;

[0027] An average precision difference determination module is configured to average the difference result values to obtain an average inter-class detection precision difference, and the average inter-class detection precision difference is used to evaluate the balance of the detection performance of the target detection model on different classes.

[0028] Preferably, the data set construction module specifically comprises:

[0029] A labeling unit is configured to label the collected target images by using image labeling software to obtain labeled target images.

[0030] A data set determination unit is configured to determine the target detection data set according to the target images.

[0031] Preferably, the detection module specifically comprises:

[0032] A division unit is configured to divide the target detection data set according to a predetermined proportion to obtain a training set and a test set.

[0033] A model construction unit is configured to construct a target detection model.

[0034] A training unit is configured to train the target detection model on the training set to obtain a trained model.

[0035] A test unit is configured to test the trained model on the test set to obtain the detection precision values of different labeled classes.

[0036] According to the embodiments of the present application, the following technical effects are provided:

[0037] The present application provides a target detection performance evaluation method and system based on average inter-class detection precision difference, which comprises the following steps: constructing a target detection data set; the target detection data set comprises a plurality of labeled target images; detecting the target detection data set by using a target detection model to obtain detection precision values of different labeled classes; numbering each labeled class in ascending order, and sequentially subtracting the detection precision value of each number from the detection precision value of the subsequent number and taking the absolute value to obtain a plurality of difference result values; averaging the difference result values to obtain an average inter-class detection precision difference; and the average inter-class detection precision difference is used to evaluate the balance of the detection performance of the target detection model on different classes. By calculating the average value of the absolute values of the detection precision difference values between all different classes, the balance of the detection performance of the model on different classes can be accurately evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative labor.

[0039] Figure 1 The method flowchart in the embodiments provided by the present application;

[0040] Figure 2 The technical solution flowchart in the embodiments provided by the present application;

[0041] Figure 3 The calculation ADPD difference process schematic diagram in the embodiments provided by the present application

[0042] Figure 4 The inter-class detection precision difference result schematic diagram in the embodiments provided by the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0044] In this document, reference to“an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0045] The terms“first”,“second”,“third”, and“fourth” and the like in the description and in the claims of the present application and the drawings referred to herein are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. Moreover, the terms“include”,“have”, and“comprise” and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, product, or apparatus that comprises a list of steps or components does not necessarily comprise only those steps or components but can include additional steps or components not expressly listed or inherent to such process, method, product, or apparatus.

[0046] The application aims to provide a target detection performance evaluation method and system based on average inter-class detection accuracy difference, which can accurately evaluate the balance of model detection performance on different classes in a dataset.

[0047] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below in combination with the drawings and specific embodiments.

[0048] Figure 1 The method flowchart in the embodiment provided by the present application is shown in Figure 1 The present application provides a target detection performance evaluation method based on average inter-class detection accuracy difference, which comprises the following steps:

[0049] Step 100: constructing a target detection dataset; the target detection dataset comprises multiple labeled target images;

[0050] Step 200: detecting the target detection dataset by using a target detection model to obtain detection accuracy values of different labeled classes;

[0051] Step 300: numbering each of the labeled classes in ascending order, and sequentially subtracting the detection accuracy values of each number from the detection accuracy values of subsequent numbers and taking absolute values to obtain multiple subtraction result values;

[0052] Step 400: averaging the subtraction result values to obtain an average inter-class detection accuracy difference; the average inter-class detection accuracy difference is used to evaluate the balance of detection performance of the target detection model on different classes.

[0053] Figure 2 The technical solution flowchart in the embodiment provided by the present application is shown in Figure 2 The technical solution flowchart of the present application comprises the following steps:

[0054] Step 1: using image labeling software to label the collected images to construct a target detection dataset;

[0055] Step 2: using a target detection model to detect the dataset to obtain detection accuracy values of different classes;

[0056] Step 3: numbering different classes as 0, 1, …, N-1 respectively;

[0057] Step 4: subtracting detection accuracy between different classes, starting from number 0 to sequentially subtract from numbers greater than 0 to take absolute values, after completing the operation of number 0, number 1 is subtracted in the same way to take absolute values, and the same operation is performed on all numbers to obtain values;

[0058] Step 5: averaging the values to obtain an average inter-class detection accuracy difference; The average inter-class detection precision difference is calculated by averaging the values, and the smaller the value, the more similar the detection capabilities of the model on different classes, and the larger the value, the greater the gap in detection capabilities of the model on different classes.

[0059] The calculation process of the average inter-class detection precision difference is shown in formula (1):

[0060]

[0061] In the formula, N represents the number of classes, N≥2, P represents the number of times of difference, P i P represents the detection precision of the class numbered i, P j P represents the detection precision of the class numbered j.

[0062] Preferably, the step 100 specifically comprises:

[0063] The target image collected is labeled by using an image labeling software to obtain a labeled target image.

[0064] The target detection data set is determined according to the target image.

[0065] Specifically, the image labeling software is IabelImg.

[0066] In this embodiment, the bolt defect detection of a power transmission line is taken as an example for further description.

[0067] In this embodiment, the bolt image of a power transmission line collected is labeled by using an IabelImg image labeling software, such as the pin defect of a bolt labeled as “pin losing”, and a power transmission line bolt defect detection data set is constructed.

[0068] Preferably, the step 200 specifically comprises:

[0069] The target detection data set is divided according to a predetermined proportion to obtain a training set and a test set.

[0070] A target detection model is constructed.

[0071] The target detection model is trained on the training set to obtain a trained model.

[0072] The trained model is used to test on the test set to obtain the detection precision values of different labeled classes.

[0073] The bolt defect detection data set is divided into a training set and a test set according to a certain proportion, the target detection model is trained on the training set to obtain weight parameters, and then the trained model is tested on the test set to obtain detection accuracy values of different categories.

[0074] Preferably, the calculation formula of the average inter-class detection accuracy difference is:

[0075]

[0076] wherein ADPD is the average inter-class detection accuracy difference, N is the number of numbers, N>=2, P i is the detection accuracy of the labeled category numbered i, P j is the detection accuracy of the labeled category numbered j, and is the number of difference results.

[0077] In this embodiment, different categories are numbered 0, 1, …, N-1, and the average inter-class detection accuracy difference is calculated; then the detection accuracies between different categories are subtracted, as shown in formula (1), to obtain |P0-P1|, |P0-P2|……|P0-P Figure 3 N-1 |, and N-1 difference values are obtained. After the number 0 is completed, the number 1 is subtracted in the same way to obtain |P1-P2|, |P1-P3|……|P1-P N-1 |, and N-2 difference values are obtained. In this way, the absolute values of the detection accuracies between different categories are obtained, as shown in formula (2), and a total of Figure 4 values are obtained.

[0078] In this embodiment, the average value of the values is calculated, and the calculated value is the average inter-class detection accuracy difference. The smaller the value, the more similar the detection abilities of the model on different categories are, and the larger the value, the greater the gap between the detection abilities of the model on different categories.

[0079] Corresponding to the above method, this embodiment further provides a target detection performance evaluation system based on an average inter-class detection accuracy difference, comprising:

[0080] A data set construction module is configured to construct a target detection data set, wherein the target detection data set comprises a plurality of labeled target images.

[0081] A detection module is configured to detect the target detection data set according to a target detection model to obtain detection accuracy values of different labeled categories.

[0082] ​​A number difference module is configured to number each of the annotation categories in ascending order, and sequentially subtract the detection accuracy value of each number from the detection accuracy value of the next number and take the absolute value to obtain a plurality of difference result values;

[0083] An average precision difference determination module is configured to average the difference result values to obtain an average inter-class detection precision difference, which is used to evaluate the balance of the detection performance of the target detection model on different categories.

[0084] Preferably, the data set construction module specifically comprises:

[0085] An annotation unit is configured to use image annotation software to annotate the collected power line bolt images to obtain the annotated target images.

[0086] A data set determination unit is configured to determine the target detection data set according to the target images.

[0087] Preferably, the detection module specifically comprises:

[0088] A division unit is configured to divide the target detection data set according to a predetermined proportion to obtain a training set and a test set.

[0089] A model construction unit is configured to construct a target detection model.

[0090] A training unit is configured to train the target detection model on the training set to obtain a trained model.

[0091] A test unit is configured to test the trained model on the test set to obtain the detection accuracy values of the different annotation categories.

[0092] The present application has the following advantages:

[0093] The target detection evaluation method based on the average inter-class detection precision difference can accurately evaluate the balance of the detection performance of the model on different categories by calculating the average value of the absolute values of the detection precision difference values between all different categories.

[0094] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0095] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A target detection performance evaluation method based on the average inter-class detection accuracy difference, characterized in that, include: Construct an object detection dataset; The target detection dataset includes multiple labeled target images; The target detection dataset is then analyzed using the target detection model to obtain detection accuracy values ​​for different labeled categories. The label categories are numbered in ascending order, and the detection accuracy value of each number is subtracted from the detection accuracy value of the subsequent numbers, and the absolute value is taken to obtain multiple difference result values. The average value of the difference results is calculated to obtain the average inter-class detection accuracy difference; the average inter-class detection accuracy difference is used to evaluate the balance of the target detection model's performance in detecting different categories.

2. The target detection performance evaluation method based on the average inter-class detection accuracy difference according to claim 1, characterized in that, The construction of the target detection dataset includes: The collected target images are annotated using image annotation software to obtain an annotated target image; The target detection dataset is determined based on the target image.

3. The target detection performance evaluation method based on the average inter-class detection accuracy difference according to claim 1, characterized in that, The step of detecting the target dataset according to the target detection model to obtain detection accuracy values ​​for different labeled categories includes: The target detection dataset is divided into a training set and a test set according to a predetermined ratio; Construct an object detection model; The target detection model is trained on the training set to obtain a trained model; The trained model is tested on the test set to obtain the detection accuracy values ​​for different labeled categories.

4. The target detection performance evaluation method based on the average inter-class detection accuracy difference according to claim 1, characterized in that, The formula for calculating the average inter-class detection accuracy difference is: ； in, ADPD The average inter-class detection accuracy difference. The number of numbers, N ≥2, For the number The detection accuracy of the labeled categories, For the number The detection accuracy of the labeled categories, This represents the number of difference results.

5. The target detection performance evaluation method based on the average inter-class detection accuracy difference according to claim 2, characterized in that, The image annotation software is IabelImg.

6. A target detection performance evaluation system based on the average inter-class detection accuracy difference, characterized in that, include: The dataset building module is used to build object detection datasets; The target detection dataset includes multiple labeled target images; The detection module is used to detect the target detection dataset according to the target detection model and obtain the detection accuracy values ​​for different labeled categories; The numbering and difference module is used to number each of the labeled categories in ascending order, and to subtract the detection accuracy value of each number from the detection accuracy value of the subsequent numbers in turn and take the absolute value to obtain multiple difference result values. The average accuracy difference determination module is used to calculate the average value of the difference results to obtain the average inter-class detection accuracy difference; the average inter-class detection accuracy difference is used to evaluate the balance of the target detection model's performance for different categories of detection.

7. The target detection performance evaluation system based on the average inter-class detection accuracy difference according to claim 6, characterized in that, The dataset construction module specifically includes: The annotation unit is used to annotate the acquired target image using image annotation software to obtain the annotated target image; A dataset determination unit is used to determine the target detection dataset based on the target image.

8. The target detection performance evaluation system based on the average inter-class detection accuracy difference according to claim 6, characterized in that, The detection module specifically includes: A partitioning unit is used to divide the target detection dataset according to a predetermined ratio to obtain a training set and a test set; Model building unit, used to build object detection models; A training unit is used to train the target detection model on the training set to obtain a trained model; The testing unit is used to test the trained model on the test set to obtain the detection accuracy values ​​for different labeled categories.

Images