A confusion matrix generation method based on PaddleDetection

Abstract

The present invention provides a confusion matrix generation method based on PaddleDetection, comprising: using the PaddleDetection target detection development kit to train a target detection network under the PaddlePaddle platform; obtaining a labeled test data set, dividing it into two folders, one folder for storing json files, and the other for storing corresponding image files, and obtaining a real target object circumscribed rectangular frame; using the infer.py image file for prediction to obtain a predicted label; deriving a rectangular detection frame obtained by prediction correction; performing cyclic matching between the real rectangular frame and the rectangular detection frame obtained by prediction correction, and recording the statistically obtained values ​​in a matrix; creating an excel table, filling the recorded matrix into the excel table, and drawing a confusion matrix for evaluating the actual application effect of the model. The method of the present invention can optimize the training method of the network model.

Description

Technical Field

[0001] The present invention relates to the field of deep learning target detection, and in particular to a confusion matrix generation method based on PaddleDetection. Background Art

[0002] PaddlePaddle is an open source deep learning platform independently developed by Baidu that integrates deep learning core framework, tool components and service platform. PaddleDetection is an excellent target detection development kit under the PaddlePaddle deep learning platform, which provides a variety of mainstream target detection, instance segmentation, key point detection algorithms, modularizes each network component, provides data enhancement strategy, loss function strategy, etc. The compression of the model and the high-performance deployment across platforms can help industrial projects to be better implemented.

[0003] For target detection, the accuracy of detection and recognition depends largely on the recognition accuracy of the model itself. Therefore, when training the neural network recognition model in the early stage, it is necessary to find a suitable evaluation index suitable for actual production. In the PaddleDetection suite, the evaluation index of the neural network model is evaluated by mAP, which cannot reflect the recognition accuracy of each category in actual sorting.

[0004] The confusion matrix is ​​also called the error matrix, which is represented by a matrix of n rows and n columns. In the evaluation of the accuracy of artificial intelligence image classification, it is mainly used to compare the classification results with the actual measured values. Each column of the confusion matrix represents the predicted category, and the total number of each column represents the number of data predicted to be in that category; each row represents the true category of the data, and the total number of data in each row represents the number of data instances in that category. The value in each column represents the number of real data predicted to be in that category. Figure 1 As shown, the 96 in the first row and the first column means that 96 instances that actually belong to the first category are predicted to be the first category. Similarly, the 2 in the first row and the second column means that 2 instances that actually belong to the first category are wrongly predicted to be the second category, and so on. Summary of the invention

[0005] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and propose a confusion matrix generation method based on PaddleDetection, which can match the detection box predicted by the target detection network with the actual target detection box, record the actual matching results, and draw them on the confusion matrix, so as to conduct statistical research on the recognition results of various types of models and optimize the training method of the network model.

[0006] The present invention adopts the following technical solution:

[0007] Step 1: Use the PaddleDetection target detection development kit on the PaddlePaddle platform to train an object detection network.

[0008] Step 2: Prepare a new batch of labeled test data sets, divided into two folders, one for the json file and the other for the corresponding image files. Use the openCV rect function to process the contour labels under the json file to get the actual target object's circumscribed rectangular frame;

[0009] Step 3: Use infer.py in the PaddleDetection development kit to predict the image file in step 2 and get the predicted label;

[0010] Step 4: loop through the number of image files i, use the model to predict each image, get the detection frame of each object in the image, perform iou judgment on the detection frame of the object, delete other detection frames of the same type with iou greater than the threshold, and only retain the detection frame with the highest confidence score; for different types with iou greater than the threshold, only retain one detection frame, delete the others, and change the corresponding label to others, so that the predicted and corrected rectangular detection frame can be obtained;

[0011] Step 5: cyclically match the actual target object circumscribed rectangular frame obtained in step 2 with the rectangular detection frame obtained by prediction and correction in step 4, and record the statistical values ​​in the matrix;

[0012] Step 6: Create an Excel table, fill the matrix obtained in step 5 into the table, and draw a confusion matrix for evaluating the actual application effect of the model.

[0013] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0014] (1) The present invention proposes a confusion matrix generation method based on PaddleDetection, which can match the detection box predicted by the target detection network with the actual target detection box, record the actual matching results, and plot them on the confusion matrix, so as to conduct statistical research on the recognition results of various types of models and optimize the training method of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is an example diagram of the confusion matrix;

[0016] Figure 2 is a method for calculating iou in an embodiment of the present invention;

[0017] Figure 3A schematic diagram of deleting overlapping boxes predicted by target detection in an embodiment of the present invention;

[0018] Figure 4 Schematic diagram of a program flow for matching a predicted frame with a real frame in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of a confusion matrix obtained according to an embodiment of the present invention.

[0020] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0021] The present invention proposes a confusion matrix generation method based on PaddleDetection, which is used to evaluate the recognition effect of a neural network model on actual target detection. The neural network is a target detection network including Mask-RCNN. The method provided by the present invention can match the detection frame predicted by the target detection network with the actual target detection frame, record the actual matching results, and plot them on the confusion matrix, so that we can conduct statistical research on the recognition results of various types of models and optimize the training method of the network model.

[0022] The specific embodiment of the present invention takes the actual working conditions of the back-end sorting of domestic waste as an example. The collected image data is the image of low-value recyclables of domestic waste on the conveyor belt. Labelme is used to annotate the image data, and the target objects in the image are divided into ten categories, including Trans_HDPE, Color_HDPE, White_HDPE, Trans_PP, White_PP, Black_PP, Color_PP, Trans_PET, Bottle_PET, and Others.

[0023] Reference Figure 2 , Figure 3 , Figure 4 , Figure 5 The specific steps are as follows:

[0024] Step 1: Collect and annotate 12,000 low-value recyclables datasets of domestic waste, 80% of which are divided into training sets and 20% into validation sets. Use GPU for training, the computer configuration is Windows 10, Intel (R) i5-10400F CPU, NVIDIA GeForce GTX3090 graphics card, 24G video memory, and the platform is PaddlePaddle. Use the Mask_RCNN neural network model in the static version of the PaddleDetection 2.0 development kit, the initial learning rate of training is 0.00125, the maximum number of iterations max_iters is 48,000 steps, and the target detection network model is trained;

[0025] Step 2: Prepare a new batch of labeled test data sets, divided into two folders, one for json files and the other for corresponding image files; the json file is generated by labelme annotation, which records the type and location information of each target object. Use the rect function of openCV to process the contour labels under the json file to obtain the actual external rectangular frame of the target object.

[0026] Step 3: Loop through the number of image files i in step 2, export the object detection model trained in step 1, and execute infer.py under the PaddleDetection2.0 development kit to predict each image in step 2. You will get a list of np_boxes, each of which contains the rectangular detection box, category, and confidence of each object in the image.

[0027] Step 4: Due to the complex working conditions, the neural network model predicts the existence of check boxes and overlapping boxes, which will affect the statistics of the final detection accuracy, so the predicted box needs to be corrected. The position of the predicted box may be slightly different from the actual box. After experimental statistics, if the IOU value of the predicted box and the actual box is greater than 0.7, it is considered that the two boxes overlap, so the IOU threshold is set to 0.7.

[0028] The prediction result of each target object in the image is obtained by step 3. Each target object will get one or more np_boxes after the prediction in step 3. Each np_boxes contains the rectangular detection box, category, and confidence of each object in the image. The actual result of each target object in the image is obtained by step 2. The real label of each target object in the image is stored in the corresponding json file. The minimum enclosing rectangular box of the target object can be obtained by using the point coordinates of the target object in the label.

[0029] If there are multiple np_boxes predicted for the same object (each np_boxes contains the rectangular detection box, category, and confidence of each object in the image), compare each predicted np_boxes. If each np_boxes predicted for the same object is of the same type, keep the iou (the calculation method of iou is as follows Figure 2 As shown in the figure, the np_boxes with iou greater than the threshold and the highest confidence score are selected, and the other np_boxes are deleted; for the case where the np_boxes predicted for the same object belong to different categories, the np_boxes with iou greater than the threshold and the highest confidence score are retained and the corresponding category labels are changed to others, and the other np_boxes are deleted. The prediction label of a single object is modified as follows Figure 3As shown, the predicted and corrected rectangular detection frame can be obtained.

[0030] Step 5: Compare the real label obtained in step 2 with the predicted label corrected in step 4. The matching flow chart is as follows: Figure 4 As shown. The outermost loop of the program traverses the number of pictures that need to be predicted, and then the second loop loops the number of predicted labels in each picture. The third loop loops the number of actual labels in each picture. The actual label of each object is judged by the predicted label. When the iou is greater than the threshold of 0.7, it is considered to be matched and counted at the corresponding position of the matrix. If the actual label does not match the predicted label, it is considered to be missed and counted at the corresponding position of the matrix.

[0031] Step 6: Create an excel table and fill the recorded matrix into the table to draw a confusion matrix, with the vertical direction being the actual category and the horizontal direction being the predicted category, such as Figure 5 shown. Figure 5 The neural network model shown is for 11 categories (including background). Taking the first row as an example, in practice, 64 Color_PP are correctly identified as Color_PP, 10 are incorrectly identified as Black_PP, and so on.

[0032] The present invention proposes a confusion matrix generation method based on PaddleDetection, which can match the detection box predicted by the target detection network with the actual target detection box, record the actual matching results, and plot them on the confusion matrix, so as to conduct statistical research on the recognition results of various types of models and optimize the training method of the network model.

[0033] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A confusion matrix generation method based on PaddleDetection, characterized in that: The following steps are involved: Step 1: Use the PaddleDetection object detection development kit to train the object detection network on the PaddlePaddle platform; Step 2: Get the labeled test data set and divide it into two folders, one for storing json files and the other for storing corresponding image files; use openCV's rect function to process the contour labels under the json file to get the actual target object's circumscribed rectangular frame; Step 3: Use infer.py in the PaddleDetection development kit to predict the image file in step 2 and get the predicted label; Step 4: loop through the number of image files i, use the model to predict each image, obtain the rectangular detection frame of each object in the image, perform iou judgment on the detection frame of the object, and retain the detection frame with the highest confidence score for the detection frames of the same type and with iou greater than the threshold; for detection frames of different types and with iou greater than the threshold, only retain one detection frame and change the corresponding label to others to obtain the predicted and corrected rectangular detection frame; Step 5: perform cyclic matching between the actual target object circumscribed rectangular frame obtained in step 2 and the predicted and corrected rectangular detection frame in step 4, and record the statistically obtained values ​​in a matrix; Step 6: Create an Excel table, fill the matrix obtained in step 5 into the table, and draw a confusion matrix for evaluating the actual application effect of the model.

Images