Segmentation model evaluation method based on industrial defect detection

Abstract

The present invention relates to a segmentation model evaluation method based on industrial defect detection. Using the inference result as a reference, the labeled area of each defect in the annotation is compared with the defect area segmented in the inference result, and the areas are compared to see if there is any intersection between the areas and whether the categories of the two areas with intersection are consistent. The number of correctly detected defects, the number of misclassified defects, and the number of missed defects are counted to obtain an evaluation value for the number of missed defects. Using the annotation as a reference, each defect area segmented in the inference result is compared with the labeled area of each defect in the annotation to count the number of areas in the inference result that do not have an intersection with the annotation, i.e., the number of overkill areas, and obtain an evaluation value for the number of overkill areas. The evaluation value of the final segmentation model is obtained by using the evaluation value of the number of missed defects and the evaluation value of the number of overkill areas. The present invention reduces the workload of manually checking the segmentation test results; the model selected during training is more in line with the effects required in actual application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image visual detection, and in particular to a segmentation model evaluation method based on industrial defect detection. Background Art

[0002] Segmentation models classify every pixel in an image. Currently, the most popular semantic segmentation evaluation methods are based on pixel labeling. In actual industrial applications, the detection capability of a model is typically evaluated by comparing the number of defects detected, which differs from mainstream pixel-level evaluation metrics.

[0003] PA (Pixel Accuracy) is the simplest metric calculation, which is the ratio of the total number of pixels to the predicted correct pixels. This metric is not very effective for comparison when the target is small and the black background is large. MPA (Mean Pixel Accuracy) is the mean value obtained by summing the ratio of the total number of correct pixels in each category to the total number of each category. MIoU (mean Intersection over Union) is measured by calculating the average value of the accumulated IoU values of each class of image pixels. FWIoU (Frequency Weighted Intersection over Union) is an improved version of MIoU, which gives different weights to each category based on the frequency of occurrence of each category.

[0004] However, in existing industrial inspections, defect areas typically occupy only a small percentage of the entire image, perhaps only a few pixels in size. Traditional evaluation methods can still achieve high scores for small, severe defects even when they are missed. Therefore, existing evaluation methods cannot accurately reflect the model's detection performance. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: to provide a segmentation model evaluation method based on industrial defect detection to solve the problem of tedious manual inspection and statistical detection results; to solve the problem that the model selected by the traditional model evaluation method is difficult to accurately reflect whether the number of over-killing and missed detection in industrial detection is optimal.

[0006] The technical solution adopted by the present invention to solve the technical problem is: a segmentation model evaluation method based on industrial defect detection, comprising the following steps:

[0007] S1. Select the connected domains of the annotation and reasoning results, and check whether the connected domains have intersections;

[0008] S2, detect whether the center of the maximum inscribed circle of any connected domain is within another connected domain;

[0009] S3, determine whether the connected domain categories are the same;

[0010] S4. Find the connected domain of the corresponding category, determine that the current category inference result is correct, and end the current connected domain search;

[0011] S5. Compare the inference results with the annotations to find the nearest annotation area for each inference result area;

[0012] S6. Determine whether the annotation corresponding to the inference result area is correctly detected;

[0013] S7, whether the reasoning and annotation categories are the same;

[0014] S8, comparing the area and distance between the inference region and the marked region with a threshold value to detect whether the current inference region is an overkill region;

[0015] S9, hierarchically cluster the inference regions of the same category and calculate the number of cluster centers;

[0016] S10. Count the number of cluster centers of each class. The number of cluster centers is the number of overkill that needs to be counted.

[0017] The present invention uses the inference result as a reference, compares the annotated area of each defect in the label with the segmented defect area in the inference result, compares whether there is any intersection between the areas, and whether the categories of the two areas with intersection are consistent, and counts the number of correctly detected and misclassified areas. Areas that do not intersect with the inference result in the label are missed detection areas. Using the inference result as a reference, after comparing the areas in all labels, the number of correctly detected areas in the current inference result can be counted. Number of misclassifications Number of defects missed And get the evaluation value of the number of missed defects Using the label as a reference, compare each defect area segmented in the inference result with each defect marked area in the label, and count the number of areas in the inference result that do not intersect with the label, which is the number of overkill areas. And get the evaluation value of the number of overkill Then, the evaluation value of the number of missed defects is and the number of overkill areas Evaluation value The evaluation value of the final segmentation model is obtained.

[0018] The beneficial effects of the present invention are that it solves the defects existing in the background technology, compares the inference results with the annotated mask images through an algorithm, analyzes the relationship between the annotated connected domain and the connected domain of the inference result, and counts the number of correct defect detections, over-kills, and missed detections; uses the number of over-kills and missed detections combined with the area of over-kills and missed detections to calculate the evaluation value, and evaluates the model training status from the level of the number of over-kills and missed detections during the training process; reduces the workload of manual inspection of segmentation test results; and the model selected in training is more in line with the effects required in actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is the main statistical flow chart of the present invention;

[0020] Figure 2 This is the statistical auxiliary flow chart of the present invention. DETAILED DESCRIPTION

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0022] like Figure 1-Figure 2 The segmentation model evaluation method shown here is based on industrial defect detection. It performs confusion matrix statistics based on the annotation results and inference results, calculates the number of missed detections and over-detections for all test samples, and then calculates the evaluation value of the segmentation model based on the number of missed detections and over-detections. The specific steps are as follows:

[0023] S1. Select the connected domains of the annotation and reasoning results, and check whether the connected domains have intersections;

[0024] If yes, proceed to step S2; if not, determine whether the current connected domain search is complete; if it is determined that the search is complete, mark the connected domain as not finding the connected domain corresponding to the inference result, and the current category connected domain has missed detection;

[0025] S2, detect whether the center of the maximum inscribed circle of any connected domain is within another connected domain;

[0026] If yes, proceed to step S3; if no, compare the IOU of the connected domain of the annotation and the inference result with the threshold; if it is greater than the set threshold, proceed to step S3; if it is less than the set threshold, it is determined that there is a missed detection in the connected domain of the current category;

[0027] S3, determine whether the connected domain categories are the same;

[0028] If yes, proceed to step S4; if no, it is determined that the current connected domain is misclassified;

[0029] S4. Find the connected domain of the corresponding category, determine that the current category inference result is correct, and end the current connected domain search;

[0030] Figure 1 The main purpose is to count the number of correct detections, misclassifications, and missed detections for each category in the confusion matrix. Using the inference result as a reference, the labeled area of each defect in the label is compared with the defect area segmented in the inference result. Compare whether there is any intersection between the areas, whether the categories of the two areas with intersection are consistent, and count the number of correctly detected and misclassified areas. Areas that do not intersect with the inference result in the label are missed detection areas. Using the inference result as a reference, after comparing the areas in all labels, the number of correct detections in the current inference result can be counted. Number of misclassifications Number of defects missed Evaluation value on the number of missed defects The calculation method is shown in formula (1):

[0031]

[0032] S5. Compare the inference results with the annotations to find the nearest annotation area for each inference result area;

[0033] S6. Determine whether the annotation corresponding to the inference result area is correctly detected;

[0034] If yes, proceed to step S7; if no, determine that the current inference area is an overkill area;

[0035] S7, whether the reasoning and annotation categories are the same;

[0036] If yes, proceed to step S8; if no, compare the area and distance between the inference area and the marked area with the set threshold. If the set threshold is exceeded, compare whether the defect level of the inference area is higher than that of the marked area. If so, the current inference area is an overkill area; if not, proceed to step S9;

[0037] S8, comparing the area and distance between the inference region and the marked region with a threshold value to detect whether the current inference region is an overkill region;

[0038] S9, hierarchically cluster the inference regions of the same category and calculate the number of cluster centers;

[0039] S10. Count the number of cluster centers of each class. The number of cluster centers is the number of overkill that needs to be counted.

[0040] Figure 2The main purpose is to count the number of overkills. Using the label as a reference, each defect area segmented in the inference result is compared with the labeled area of each defect in the label. The number of areas in the inference result that do not intersect with the label is counted, which is the number of overkill areas. Evaluation value of the number of overkill The calculation method is shown in formula (2):

[0041]

[0042] The evaluation method is calculated based on the number of defects over-detected and missed detected in the confusion matrix of the first two steps. The evaluation method of over-detected and missed detection area refers to the calculation method of MIoU:

[0043]

[0044] Where k represents the number of categories, p ii Indicates the number of pixels whose category is i and the inference result is i, p ij and p ji They represent the number of pixels whose category is i and the inference result is j, and the number of pixels whose category is j and the inference result is i.

[0045] The calculation method of the evaluation value of the final segmentation model is modified as follows:

[0046]

[0047] where p i0 Indicates that category i is inferred to be a normal category, which means missed detection. Similarly, p 0i Indicates excessive killing.

[0048] Since it is difficult to accurately evaluate the segmentation effect of the model on industrial defect detection at the pixel level, this paper compares the inference results with the annotated mask images, analyzes the relationship between the annotated connected domains and the inference result connected domains, and counts the number of defects correctly detected, over-detected, and missed.

[0049] However, there are still problems in evaluating the model only by the number of defects detected by the model. When the model converges to a state where the entire image is segmented during training, the missed detection rate is almost zero, and the entire image is counted as just another over-detection. Therefore, when calculating the evaluation value of the number of over-detections and the evaluation value of the number of missed detections, it is also necessary to calculate the evaluation value of the missed detection area and the over-detection area.

[0050] This method reduces the workload of manually inspecting and segmenting test results by automatically calculating the number of over-detected and missed defects on the test set. By calculating evaluation values based on the number of over-detected and missed defects and their area, the model selected during training can better meet the needs of actual application scenarios. It can more accurately reflect the defect detection status during model training. It can also quickly and automatically analyze the defect detection status of the model on the test set, assisting developers in model tuning.

[0051] The above description only describes specific embodiments of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can make modifications or variations to the specific embodiments described above without departing from the essence and scope of the invention.

Claims

1. A segmentation model evaluation method based on industrial defect detection, characterized by: The following steps are included: S1, select the connected domain of the marked defect area and the inference result output by the segmentation model, and check whether the connected domains have an intersection; if there is an intersection, execute S2; if there is no intersection, determine that the current marked connected domain is a missed detection area, and end the current connected domain search; S2: Check whether the center of the largest inscribed circle of any connected domain is within another connected domain; if so, execute S3; if not, compare the IOU of the two connected domains with the threshold. If it is greater than the threshold, execute S3; if it is less than the threshold, determine that the currently marked connected domain is a missed detection area; S3, determine whether the categories of the connected domains are the same; if so, execute S4; if not, determine that the current connected domain is misclassified; S4. Find the connected domain of the corresponding category, determine that the current category inference result is correct, and end the current connected domain search; S5. Compare the inference results with the annotations to find the nearest annotation area to each inference result area (i.e., the inference result area); S6, determining whether the annotation corresponding to the inference result area is correctly detected; if so, executing S7; if not, determining that the current inference result area is an overkill area; S7: Determine whether the inferenced and annotated categories are the same; if so, execute S8; if not, compare the area and distance between the inferenced result area and the annotated area with the threshold. If the threshold is exceeded, further determine whether the defect level of the inferenced result area is higher than that of the annotated area. If so, it is an overkill area; if not, execute S9; S8, comparing the area and distance between the inference result region and the marked region with a threshold value to determine whether the current inference result region is an overkill region; S9, hierarchically clustering the inference result regions of the same category and calculating the number of cluster centers; S10. Count the number of cluster centers of each class, which is the overkill number.

2. The segmentation model evaluation method based on industrial defect detection according to claim 1, characterized in that: The areas that do not intersect with the inference results in the annotations are called missed areas.

3. The segmentation model evaluation method based on industrial defect detection according to claim 2, characterized in that: After S1-S10 are all executed, the number of correct detections in the current inference result can be counted by comparing all the marked areas. Number of misclassifications Number of defects missed 4. The segmentation model evaluation method based on industrial defect detection according to claim 3, characterized in that: The evaluation value of the number of missed defects The calculation formula is:

5. The segmentation model evaluation method based on industrial defect detection according to claim 4, characterized in that: The number of overkill areas Evaluation value The calculation formula is:

6. The segmentation model evaluation method based on industrial defect detection according to claim 5, characterized in that: Evaluation value based on the number of missed defects and the number of overkill areas Evaluation value The calculation formula for the evaluation value of the final segmentation model is: Among them, k represents the number of categories, p ii Indicates the number of pixels whose category is i and the inference result is i, p ij and p ji They represent the number of pixels whose category is i and the inference result is j, and the number of pixels whose category is j and the inference result is i, respectively. i0 Indicates that category i is inferred to be a normal category, which means missed detection. Similarly, p 0i Indicates excessive killing.

Images