A sewer defect detection method and system based on a confusion matrix

Abstract

The application discloses a sewer defect detection method and system based on a confusion matrix, which comprises the following steps: building an artificial neural network three-stage model; collecting a training data set and a verification data set to train the artificial neural network three-stage model; constructing a multi-classification percentage confusion matrix based on the training result and a true label to determine an easily-confused class; using a third-order model to finely classify the easily-confused class, and correcting the classification prediction of the easily-confused class based on the fine classification result to obtain a final detection result. The system comprises a data acquisition module, a model building module, a model training module, a prediction module, an easily-confused coarse classification module, a fine classification module, a correction module and a post-processing module. By using the application, the problem of the increase of false positive samples caused by similar features among sewer defect classes can be solved, and the detection accuracy of sewer defects is improved. The application can be widely applied to the technical field of sewer defect detection.

Description

Technical Field

[0001] This invention relates to the field of sewer defect detection technology, and in particular to a sewer pipe defect detection method and system based on a confusion matrix. Background Technology

[0002] Urban sewer pipes have a long service life and are prone to various defects, leading to sewage pipe failures. If these defects are not promptly removed or repaired, the deterioration of the pipe system will be more severe, and repair or replacement costs will be higher. Currently, closed-circuit television (CCTV) technology is widely used for inspecting underground sewage pipes. Closed-circuit robots automatically crawl inside the pipes, detecting and photographing defects such as cracks, deposits, tree roots, and misalignments, while simultaneously recording the pipe's health status, thus transforming hidden underground pipelines into internal video recordings visible on a computer. However, current defect identification in urban drainage networks mainly relies on manual review and judgment of the inspection image data, resulting in low efficiency and susceptibility to subjective bias.

[0003] Currently, some research has applied deep learning to sewer pipe defect detection. Deep learning object detection is mainly divided into one-stage object detection and two-stage object detection. One-stage object detection models, such as the YOLO series, SSD, and RetinaNet, are characterized by small model size, high speed, and low accuracy. Two-stage detection models, such as R-CNN, Fast R-CNN, and Faster R-CNN, are characterized by large model size, slow speed, and high accuracy. However, both one-stage and two-stage object detection models need to balance the classification and localization performance of objects, making it impossible to achieve optimal performance for each. In addition, some sewer defects have similar characteristics, such as misaligned joints and disconnections, sediment and obstacles. These models do not perform well in distinguishing these defects and have a serious false positive problem. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a method and system for detecting defects in sewer pipes based on a confusion matrix. This method can solve the problem of increased false positive samples caused by similar features among different categories of sewer defects, thereby improving the detection accuracy of sewer pipe defects.

[0005] The first technical solution adopted in this invention is: a method for detecting defects in sewer pipes based on a confusion matrix, comprising the following steps:

[0006] Images of sewer defects of multiple categories were collected and labeled with their categories and locations to obtain training and validation datasets;

[0007] A three-stage model of artificial neural networks is built based on the first-order model, the second-order model, and the third-order model;

[0008] The three-stage artificial neural network model was trained and validated based on the training dataset and the validation dataset, and the three-stage artificial neural network model with the highest mean accuracy was obtained.

[0009] Obtain the prediction dataset and input it into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset;

[0010] The easily confused category groups are determined based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset;

[0011] The predicted bounding boxes of easily confused category groups in the prediction dataset are cropped and input into the third-order model with the highest average accuracy for fine classification, thus obtaining the fine classification results of easily confused category groups.

[0012] The prediction results of easily confused category groups in the prediction dataset are corrected based on the fine classification results of easily confused category groups to obtain the detection results.

[0013] Furthermore, the first-order model is a region proposal model, used to perform binary classification of the foreground and background parts of the features and to initially locate the foreground target; the second-order model is a detection model, used to classify the foreground target of the first-order model, fine-tune the position of the foreground target prediction box, and obtain the classification and localization results of each category; the third-order model is a fine-classification fine-tuning model, used to perform fine-classification of the easily confused category group samples output by the second-order model, and obtain the fine-classification results of the easily confused category group.

[0014] Furthermore, the step of training and validating the three-stage artificial neural network model based on the training and validation datasets to obtain the three-stage artificial neural network model with the highest mean accuracy specifically includes:

[0015] The first and second order models of the three-stage artificial neural network model are trained based on the training dataset to obtain the training results.

[0016] The model weights of the first-order and second-order models are adjusted based on the loss function of the training results and the true labels of the training dataset.

[0017] The validation dataset is input into the first-order and second-order models after weight adjustment to obtain the validation results.

[0018] Calculate the mean and average precision of the validation results and the true labels of the validation dataset, and retain the first-order model and the second-order model with the highest mean and average precision.

[0019] A percentage confusion matrix analysis was performed on the training and validation results to construct the percentage confusion matrix and identify the easily confused category groups.

[0020] The labeled regions of easily confused category groups in the training dataset and the labeled regions of easily confused category groups in the validation dataset are cropped to obtain the cropped training set and the cropped validation set.

[0021] The third-order model of the three-stage artificial neural network model is trained based on the pruned training set to obtain the training result of the third-order model.

[0022] The model weights of the third-order model are adjusted based on the loss function of the training results of the third-order model and the true labels of the cropped training set.

[0023] The clipped validation set is input into the weighted third-order model to obtain the validation results of the third-order model;

[0024] Calculate the mean and average precision of the validation results of the third-order model and the true labels on the pruned validation set, and retain the third-order model with the highest mean and average precision.

[0025] This optimization process trains and validates the three-stage artificial neural network model twice, retaining the model with the highest average accuracy and improving the model's detection accuracy.

[0026] Furthermore, the step of performing percentage confusion matrix analysis on the training and validation results, constructing a percentage confusion matrix, and identifying easily confused category groups specifically includes:

[0027] Calculate the percentage of true positive, false positive, and false negative samples for each category in the training and validation results;

[0028] Construct a percentage confusion matrix based on the sum of the percentages of false positive samples across all categories;

[0029] Based on the percentage threshold, category combinations that exceed the threshold are selected and identified as easily confused category groups.

[0030] This optimization step constructs a percentage confusion matrix for coarse classification, filters out easily confused category groups in the prediction dataset, and enables subsequent fine classification of these easily confused category groups.

[0031] Furthermore, the step of obtaining the prediction dataset and inputting it into the first-order model and the second-order model with the highest average accuracy to obtain the prediction result of the prediction dataset specifically includes:

[0032] Feature extraction and feature integration are performed on the prediction dataset to obtain feature maps at different scales;

[0033] Based on the first-order model, feature maps of different scales are extracted to obtain predicted anchor boxes;

[0034] The predicted anchor boxes are classified into foreground and background, and the optimal candidate regions are obtained by filtering based on the foreground score and overlap of the anchor boxes.

[0035] The feature map within the optimal candidate region is mapped based on the second-order model to obtain a low-dimensional feature vector.

[0036] The predicted class is obtained by calculating the class probability of the feature map within each candidate region based on the low-dimensional feature vector;

[0037] Based on the low-dimensional feature vector, the bounding box offset of the candidate region is predicted, and the position and size of the corrected candidate region are obtained.

[0038] This optimization step enables the selection of optimal candidate regions, accurately extracting which targets exist in the prediction dataset, as well as their locations and categories.

[0039] Furthermore, the step of correcting the prediction results of easily confused category groups in the prediction dataset based on the fine classification results of easily confused category groups to obtain the detection results specifically includes:

[0040] When the number of samples in easily confused categories differs greatly and the features between categories are similar, the sub-classification results of the easily confused category group are used as the detection results;

[0041] When the prediction results of the easily confused category group in the prediction dataset can distinguish the easily confused categories, the prediction boxes and confidence scores of the sub-classification results of the easily confused category group and the prediction results of the easily confused category group in the prediction dataset are fused to obtain the detection results.

[0042] This optimization step refines the classification of easily confused categories in the prediction results obtained from the second-order model, thereby enhancing the accuracy of the prediction results.

[0043] Furthermore, it also includes a step of removing duplicate and low-quality prediction boxes from the detection results to obtain the final detection result, which specifically includes:

[0044] The predicted bounding boxes in the detection results are sorted according to their confidence scores and weights;

[0045] Starting with the prediction box with the highest confidence, traverse the prediction boxes to obtain all prediction boxes with an overlap higher than a certain threshold.

[0046] All the predicted bounding boxes found are assigned to the same cluster, and the weighted average coordinates, confidence score, and category are calculated to obtain the final detection result.

[0047] This optimization step removes duplicate and low-quality prediction boxes from the detection results, resulting in a higher confidence level for the final output detection results.

[0048] Furthermore, the loss function based on the training results and the true labels of the training dataset is calculated as follows:

[0049]

[0050] Among them, L sum L represents the sum of the loss functions of the first-order model and the second-order model. rpn L represents the loss function of the first-order model. det L represents the loss function of the second-order model. cls L represents the classification loss function. reg Let λ1 and λ2 represent the regression loss function, and let λ1 and λ2 represent the balancing parameters of the first-order and second-order models, respectively, which are used to control the relative importance of classification and regression.

[0051] Furthermore, the loss function based on the training results of the third-order model and the true labels of the cropped training set is calculated as follows:

[0052] L ft =-α t (1-p t ) γ log(p t )

[0053] Among them, L ft p represents the multi-class focus loss function. t α represents the predicted probability. t γ represents the weighting factor, and γ represents the adjustment factor.

[0054] The second technical solution adopted in this invention is: a sewer pipe defect detection system based on a confusion matrix, comprising:

[0055] The data acquisition module is used to collect images of various types of sewer defects and label them with their categories and locations to obtain training and validation datasets.

[0056] The model building module constructs a three-stage artificial neural network model based on the first-order model, the second-order model, and the third-order model.

[0057] The model training module trains and validates the three-stage artificial neural network model based on the training dataset and the validation dataset, and obtains the three-stage artificial neural network model with the highest mean accuracy.

[0058] The prediction module is used to acquire the prediction dataset and input the prediction dataset into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset;

[0059] The easily confused coarse classification module determines the easily confused category groups based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset;

[0060] The fine classification module clips the prediction boxes of easily confused category groups in the prediction dataset and inputs them into the third-order model with the highest average accuracy for fine classification, thus obtaining the fine classification results of easily confused category groups.

[0061] The correction module corrects the prediction results of easily confused category groups in the prediction dataset based on the fine classification results of easily confused category groups, and obtains the detection results;

[0062] The post-processing module is used to remove duplicate and low-quality prediction boxes from the detection results to obtain the final detection results.

[0063] The beneficial effects of the method and system of this invention are as follows: This invention constructs a three-stage artificial neural network model, obtains training and validation datasets to train and test it, and determines the network weights and model performance of the three-stage artificial neural network model; it constructs a multi-class percentage confusion matrix to determine the easily confused class groups of sewer defects, and merges the category labels within the easily confused class groups as coarse classifications; it uses the third-stage model to perform fine classification training and testing on the easily confused classes, corrects the classification predictions in the easily confused classes, and obtains the final detection results; ultimately, it overcomes the problem of increased false positive samples caused by similar features between sewer defect categories, and improves the detection accuracy of sewer pipe defects. Attached Figure Description

[0064] Figure 1 This is a flowchart of the steps of a sewage pipe defect detection method based on a confusion matrix according to the present invention;

[0065] Figure 2 This is a structural diagram of a sewer pipe defect detection system based on a confusion matrix according to the present invention.

[0066] Figure 3 This is a structural diagram of a sewer pipe defect detection method based on a confusion matrix according to the present invention;

[0067] Figure 4 This is a sample diagram of a multi-class percentage confusion matrix for a sewer pipe defect detection method based on a confusion matrix according to the present invention;

[0068] Figure 5 This is a sample diagram of the multi-class percentage confusion matrix after fusion correction of the present invention's method for detecting defects in sewer pipes based on a confusion matrix;

[0069] Figure 6 This is an example diagram illustrating the model detection effect of a sewer pipe defect detection method based on a confusion matrix according to the present invention. Detailed Implementation

[0070] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0071] Reference Figure 1 This invention provides a method for detecting defects in sewer pipes based on a confusion matrix. The method includes the following steps:

[0072] S1. Collect images of various types of sewer defects and label them with their categories and locations to obtain training and validation datasets;

[0073] Specifically, a closed-loop pipeline robot is used to detect and photograph defects within the pipeline, collecting images of various types of sewer defects. These images are then manually labeled using image annotation software, generating XML-formatted annotation files. This process is repeated to annotate all images. For model training, selection, and testing, the annotation files need to be divided into training, validation, and prediction datasets according to a specific ratio, with the number of images in the datasets being training > validation >= prediction. The ratios include, but are not limited to, 8:1:1, 8:0.5:1, and 6:2:2. Since the datasets contain both normal and multiple defect categories, non-random and random data augmentation are applied to the datasets before training the model. Non-random data augmentation is used across all datasets, such as applying deterministic histogram equalization to all images, increasing image sharpness, and dehazing, to improve image quality. Random data augmentation is used on the training set, such as randomly adjusting brightness, contrast, and image size, artificially increasing the size of the training set.

[0074] S2. Construct a three-stage artificial neural network model based on the first-order model, the second-order model, and the third-order model;

[0075] Specifically, the first-order model is a region proposal model, used to perform binary classification of the foreground and background parts of the features and to initially locate the foreground target; the second-order model is a detection model, used to classify the foreground target from the first-order model, fine-tune the position of the foreground target prediction box, and obtain the classification and localization results of each category; the third-order model is a fine-classification fine-tuning model, used to perform fine-classification of the easily confused category group samples output by the second-order model, and obtain the fine-classification results of the easily confused category group.

[0076] Reference Figure 3The dataset images are processed by a feature extraction network to generate multi-layer feature maps. Preferably, in this specific embodiment of the invention, a Swing Transformer structure is used as the feature extraction network, and the weights of the feature extraction network are initialized using weights learned from the ImageNet dataset. The generated feature maps have strong semantic information in higher-level feature maps and strong detail information in lower-level feature maps. The multi-layer feature maps are then fused using a feature fusion network. Preferably, in this specific embodiment of the invention, an FPN structure is used to integrate higher-level features into lower-level features, increasing the semantic information of the lower-level features. The fused feature map possesses both strong semantic information and strong detail information.

[0077] Preferably, the first-order model in a specific embodiment of the present invention consists of a 3×3 convolutional layer and two 1×1 convolutional layers, which are used to extract shared feature maps and make binary classification judgments on whether the anchor box at each position contains a foreground target and to correct the coordinates of the anchor box. Using a non-maximum suppression algorithm, the optimal candidate region is selected based on the anchor box score and overlap, thus completing the first stage of prediction.

[0078] Preferably, the second-order model in this specific embodiment of the invention consists of two fully connected layers and two output layers. The fully connected layers map the candidate region feature maps extracted by the Roi Pooling layer into a low-dimensional feature vector for subsequent classification and regression. The classification output layer is a softmax layer that calculates the class probability of each candidate region based on the feature vector, including the background class. The regression output layer is a linear regression layer that predicts the bounding box offset of each candidate region based on the feature vector, used to correct the position and size of the candidate regions. The role of the second-stage detection model is to achieve the basic objective of object detection, namely, determining which objects exist in the image, as well as their location and class.

[0079] Preferably, in this specific embodiment of the invention, the third-order model is a combination of multiple classification networks, with the number of combinations equal to the number of easily confused classes. The feature extraction network adopts a Swing Transformer structure, and the weights of the feature extraction network are initialized using weights learned from the ImageNet dataset. The classifier uses a multi-class focus loss function. The fine-grained classification performance of the third stage depends on the capabilities of the feature extraction network model and the classification loss function. Generally, the larger the parameters of the feature extraction network model, the more complex the model, and the more the classification loss function focuses on difficult samples and imbalanced samples, resulting in better fine-grained classification results. However, if the model is too large or too complex, its performance will decrease due to overfitting.

[0080] S3. Train and validate the three-stage artificial neural network model based on the training dataset and validation dataset to obtain the three-stage artificial neural network model with the highest mean accuracy.

[0081] S3.1. Train the first and second order models of the three-stage artificial neural network model based on the training dataset to obtain the training results;

[0082] Specifically, the training dataset is input into the three-stage artificial neural network model in S2 to initialize the feature extraction network. The feature extraction network, feature integration network, first-order model and second-order model are trained using the training dataset and AdamW optimizer. Multiple forward computations and backpropagation processes are performed, and the results of model training output are collected.

[0083] S3.2 Adjust the model weights of the first-order and second-order models based on the loss function of the training results and the true labels of the training dataset;

[0084] Specifically, the loss function based on the training results and the true labels of the training dataset is calculated as follows:

[0085]

[0086] Among them, L sum L represents the sum of the loss functions of the first-order model and the second-order model. rpn L represents the loss function of the first-order model. det L represents the loss function of the second-order model. cls L represents the classification loss function. reg Let λ1 and λ2 represent the regression loss function, and let λ1 and λ2 represent the balancing parameters of the first-order and second-order models, respectively, which are used to control the relative importance of classification and regression.

[0087] S3.3 Input the validation dataset into the first-order and second-order models after adjusting the weights to obtain the validation results;

[0088] Specifically, the validation dataset is input into the first-order and second-order models after weight adjustment. The performance of the first-order and second-order models after weight adjustment is tested using the validation dataset. Multiple forward calculations and backpropagation processes are performed, and the results of model validation are collected.

[0089] S3.4 Calculate the mean and average precision of the validation results and the true labels of the validation dataset, and retain the first-order model and the second-order model with the highest mean and average precision.

[0090] Specifically, the average longitude is calculated using the following expression:

[0091]

[0092]

[0093]

[0094] r′>r

[0095]

[0096]

[0097] Where mAP represents the average longitude, C represents the total number of categories, and AP C Let r' represent the average precision of the Cth category, r' represent the recall rate, p(r') represent the precision rate, TP represent the number of true positives, FP represent the number of false positives, and FN represent the number of false negatives.

[0098] S3.5 Perform percentage confusion matrix analysis on the training and validation results, construct the percentage confusion matrix, and identify the easily confused category groups;

[0099] Specifically, some sewer defect classes are very similar in features, making model classification difficult. Therefore, a percentage confusion matrix is ​​needed to analyze the training and validation results generated by the second-stage detection model to identify easily confused class groups that are prone to classification errors. For each class in the training and validation results, the IoU matrix between the bounding boxes and ground truth bounding boxes of that class is calculated, with the matrix shape (num_bboxes, num_gt). Here, num_bboxes is the number of detected bounding boxes, and num_gt is the number of ground truth bounding boxes. For each bounding box of that class, if its score is greater than or equal to the confidence threshold, all ground truth labels are iterated. If its IoU with a ground truth bounding box is greater than or equal to the IoU threshold, and the ground truth label and the detected label are the same, then a successful match is considered, the number of true positive samples is incremented by 1, and the corresponding position in the confusion matrix is ​​updated. For each detected label, if no match is found, it is considered a false positive, the number of false positive samples is incremented by 1, and the corresponding position in the confusion matrix is ​​updated. For each ground truth label, if no match is found, it is considered a missed detection, the number of false negatives is incremented by 1, and the corresponding position in the confusion matrix is ​​updated. The multi-class percentage confusion matrix has the shape (num_classes+1, num_classes+1), where num_classes is the number of classes, and the last row and last column represent the percentage of samples with an IoU of 0 between the predicted bounding box and the labeled box, and the percentage of false negative samples, respectively. The percentages of true positives, false positives, and false negatives among all training and validation results are calculated for each class, and the sum of the percentages of false positives among each class is added to the matrix to obtain the percentage confusion matrix. Based on a threshold percentage, class combinations exceeding the threshold are identified as easily confused class groups. (Refer to...) Figure 4Excluding the last row and last column, if the sum of the percentages of false positive samples between categories exceeds a threshold, the corresponding categories are grouped into a group of easily confused categories, and these easily confused categories are then grouped into a single major category. Specific embodiments of this invention use the following categories as examples: Normal (zc), Misaligned (CK), Deposited (CJ), Deformed (BX), Tree Root (SG), Disjointed (TJ), Dark Connection (AJ), Introduced (CR), Fractured (PL), and Obstacle (ZW). (Refer to...) Figure 4 The number of object detection classes is 10. The percentages of (CK,TJ)+(TJ,CK) and (CJ,ZW)+(ZW,CJ) are greater than our set threshold of 20%. Therefore, we define the CK and TJ classes, and the CJ and ZW classes as easily confused class groups, and combine the categories of each easily confused class group into a coarse classification.

[0100] S3.6. Crop the labeled regions of easily confused category groups in the training dataset and the labeled regions of easily confused category groups in the validation dataset to obtain the cropped training set and the cropped validation set.

[0101] Specifically, based on the percentage confusion matrix constructed in step 3.5 and the true labels in the training and validation datasets, easily confused samples are selected from the training and validation datasets. The labeled regions in the easily confused samples are then cropped to obtain the cropped training set and the cropped validation set.

[0102] S3.7. Train the third-order model of the three-stage artificial neural network model based on the pruned training set to obtain the training result of the third-order model;

[0103] Specifically, the pruned training set is input into the third stage of the three-stage artificial neural network model. The pruned training set and the AdamW optimizer are used to train the third stage model. Multiple forward computations and backpropagation processes are performed, and the training results of the third stage model are collected.

[0104] S3.8 Adjust the model weights of the third-order model based on the loss function of the training results of the third-order model and the true labels of the cropped training set;

[0105] Specifically, the loss function based on the training results of the third-order model and the true labels of the cropped training set is calculated as follows:

[0106] L ft =-α t (1-p t ) γ log(p t )

[0107] Among them, L ft p represents the multi-class focus loss function.t α represents the predicted probability. t γ represents the weighting factor, and γ represents the adjustment factor.

[0108] S3.9 Input the clipped validation set into the third-order model after weight adjustment to obtain the validation results of the third-order model;

[0109] Specifically, the pruned validation set is input into the weighted third-order model, and the performance of the weighted third-order model is tested using the pruned validation set. Multiple forward calculations and backpropagation processes are performed to collect the validation results of the third-order model.

[0110] S3.10 Calculate the mean and average precision of the third-order model validation results and the true labels on the pruned validation set, and retain the third-order model with the highest mean and average precision.

[0111] Specifically, the mean accuracy calculation refers to the mean longitude calculation formula in S3.4.

[0112] S4. Obtain the prediction dataset and input the prediction dataset into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset;

[0113] Specifically, the prediction dataset is input into a pre-trained feature extraction network and a pre-trained feature fusion network to perform feature extraction and feature fusion on the prediction dataset, resulting in feature maps at different scales. Based on the first-order model with the highest average accuracy, feature maps at different scales are extracted to obtain predicted anchor boxes. In the first-order model, the predicted anchor boxes are classified into foreground and background, and the optimal candidate regions are obtained by filtering based on the foreground score and overlap of the anchor boxes. Based on the second-order model with the highest average accuracy, the feature maps within the optimal candidate regions are mapped to obtain low-dimensional feature vectors. In the second-order model, the class probability of the feature map within each candidate region is calculated based on the low-dimensional feature vectors to obtain the predicted class. The bounding box offsets of the candidate regions are predicted using the low-dimensional feature vectors to obtain the corrected position and size of the candidate regions.

[0114] S5. Determine the easily confused category groups based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset;

[0115] Based on the categories shown in the prediction results of the prediction dataset, compare them with the percentage confusion matrix constructed in step S3.5. If the predicted category is a category within the easily confused category group, then the sample is identified as an easily confused category group.

[0116] S6. The prediction boxes of easily confused category groups in the prediction dataset are cropped and input into the third-order model with the highest average accuracy for fine classification, so as to obtain the fine classification results of easily confused category groups.

[0117] S7. Based on the fine classification results of the easily confused category groups, the prediction results of the easily confused category groups in the prediction dataset are corrected to obtain the detection results;

[0118] Specifically, when the number of easily confused category samples differs greatly and the features between categories are similar, the performance of the second-order model is insufficient to distinguish easily confused categories. The third-order model can solve the problem of insufficient performance of the second-order model. The detection result depends entirely on the sub-classification result of the easily confused category group. For example, if the features of the CK and TJ classes in the sewer are similar and the number of labels in the dataset differs greatly, this merging method is adopted.

[0119] When the prediction results of easily confused class groups in the prediction dataset can distinguish easily confused classes, the detection result is jointly determined by the sub-classification results of the easily confused class groups and the prediction results of the easily confused class groups in the prediction dataset. For example, in the case of CJ and ZW classes in a sewer, during synthesis, the predicted bounding boxes and confidence scores of the sub-classification results of the easily confused class groups and the prediction results of the easily confused class groups in the prediction dataset are fused. The region class confidence scores of the easily confused class groups in the prediction dataset are multiplied element-wise with the class confidence scores in the sub-classification results of the easily confused class groups, and then summed to obtain the detection result. This approach can relatively improve the final prediction confidence score, which is high for both models, achieving the effect of model fusion. (Refer to...) Figure 5 As can be seen, through the fusion correction of this scheme, the number of true positive samples in CJ and ZW, CK and TJ categories has increased significantly, while the number of false positive samples has decreased significantly.

[0120] S8. Perform duplicate and low-quality prediction box removal operations on the detection results to obtain the final detection results.

[0121] Specifically, for the predicted bounding boxes in the detection results, they are sorted in descending order of confidence score and placed into a set B. Two empty lists are initialized: a cluster list L and a result list F. The cluster list L stores the set of predicted bounding boxes for each target, and the result list F stores the fused bounding boxes for each target. The predicted bounding boxes in set B are iterated through. For each box, the IoU (Intersection over Union) is calculated with the fused bounding boxes in result list F. If the IoU is greater than the fusion threshold, the box is added to the corresponding cluster list L. The coordinates and confidence score of the fused bounding boxes in result list F are updated using the fusion bounding box formula, which is expressed as follows:

[0122]

[0123]

[0124]

[0125] Where (x1, y1) represents the coordinates of the upper left corner of the predicted box, (x2, y2) represents the coordinates of the lower right corner of the predicted box, s represents the confidence level of the predicted box, A represents predicted box A, B represents predicted box B, and C represents fused box C.

[0126] Predicted box C is obtained by fusing predicted box A and predicted box B. If the value is less than the fusion threshold, this box is added as a new target to the cluster list L and the result list F. Preferably, the fusion threshold is set to 0.65. After processing all boxes in set B, the fused boxes in result list F are taken as the final result. This method can be applied after merging classification results to remove some low-quality overlapping boxes.

[0127] This invention tests real sewer images captured by a closed-loop pipeline robot camera on a platform equipped with an NVIDIA GPU. The results are as follows... Figure 6 As shown, the sewer pipe defect detection method of the present invention can solve the problem of increased false positive samples caused by similar features among sewer defect categories, and improve the detection accuracy of sewer pipe defects.

[0128] Reference Figure 2 This invention provides a sewer pipe defect detection system based on a confusion matrix, comprising:

[0129] The data acquisition module is used to collect images of various types of sewer defects and label them with their categories and locations to obtain training and validation datasets.

[0130] The model building module constructs a three-stage artificial neural network model based on the first-order model, the second-order model, and the third-order model.

[0131] The model training module trains and validates the three-stage artificial neural network model based on the training dataset and the validation dataset, and obtains the three-stage artificial neural network model with the highest mean accuracy.

[0132] The prediction module is used to acquire the prediction dataset and input the prediction dataset into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset;

[0133] The easily confused coarse classification module determines the easily confused category groups based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset;

[0134] The fine classification module clips the prediction boxes of easily confused category groups in the prediction dataset and inputs them into the third-order model with the highest average accuracy for fine classification, thus obtaining the fine classification results of easily confused category groups.

[0135] The correction module corrects the prediction results of easily confused category groups in the prediction dataset based on the fine classification results of easily confused category groups, and obtains the detection results;

[0136] The post-processing module is used to remove duplicate and low-quality prediction boxes from the detection results to obtain the final detection results.

[0137] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting defects in sewer pipes based on a confusion matrix, characterized in that, Includes the following steps: Images of sewer defects of multiple categories were collected and labeled with their categories and locations to obtain training and validation datasets; A three-stage model of artificial neural networks is built based on the first-order model, the second-order model, and the third-order model; The three-stage artificial neural network model was trained and validated based on the training dataset and the validation dataset, and the three-stage artificial neural network model with the highest mean accuracy was obtained. Obtain the prediction dataset and input it into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset; The easily confused category groups are determined based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset; The predicted bounding boxes of easily confused category groups in the prediction dataset are cropped and input into the third-order model with the highest average accuracy for fine classification, thus obtaining the fine classification results of easily confused category groups. The prediction results of easily confused category groups in the prediction dataset are corrected based on the fine classification results of easily confused category groups to obtain the detection results; The first-order model is a region proposal model, used to perform binary classification of the foreground and background parts of the features and to initially locate the foreground target; the second-order model is a detection model, used to classify the foreground target from the first-order model, fine-tune the position of the foreground target prediction box, and obtain the classification and localization results of each category; the third-order model is a fine-tuning model, used to perform fine-tuning of the easily confused category group samples output by the second-order model, and obtain the fine-tuning results of the easily confused category group. The step of training and validating the three-stage artificial neural network model based on the training and validation datasets to obtain the three-stage artificial neural network model with the highest mean accuracy specifically includes: The first and second order models of the three-stage artificial neural network model are trained based on the training dataset to obtain the training results. The model weights of the first-order and second-order models are adjusted based on the loss function of the training results and the true labels of the training dataset. The validation dataset is input into the first-order and second-order models after weight adjustment to obtain the validation results. Calculate the mean and average precision of the validation results and the true labels of the validation dataset, and retain the first-order model and the second-order model with the highest mean and average precision. A percentage confusion matrix analysis was performed on the training and validation results to construct the percentage confusion matrix and identify the easily confused category groups; The labeled regions of easily confused category groups in the training dataset and the labeled regions of easily confused category groups in the validation dataset are cropped to obtain the cropped training set and the cropped validation set. The third-order model of the three-stage artificial neural network model is trained based on the pruned training set to obtain the training result of the third-order model. The model weights of the third-order model are adjusted based on the loss function of the training results of the third-order model and the true labels of the cropped training set. The clipped validation set is input into the weighted third-order model to obtain the validation results of the third-order model; Calculate the mean and average precision of the third-order model validation results and the true labels on the pruned validation set, and retain the third-order model with the highest mean and average precision. The step of performing percentage confusion matrix analysis on the training and validation results, constructing the percentage confusion matrix, and identifying easily confused category groups specifically includes: Calculate the percentage of true positive, false positive, and false negative samples for each category in the training and validation results; Construct a percentage confusion matrix based on the sum of the percentages of false positive samples across all categories; Based on the percentage threshold, category combinations that exceed the threshold are selected and identified as easily confused category groups.

2. The method for detecting defects in sewer pipes based on a confusion matrix according to claim 1, characterized in that, The step of obtaining the prediction dataset and inputting it into the first-order and second-order models with the highest average accuracy to obtain the prediction results of the prediction dataset specifically includes: Feature extraction and feature integration are performed on the prediction dataset to obtain feature maps at different scales; Based on the first-order model, feature maps of different scales are extracted to obtain predicted anchor boxes; The predicted anchor boxes are classified into foreground and background, and the optimal candidate regions are obtained by filtering based on the foreground score and overlap of the anchor boxes. The feature map within the optimal candidate region is mapped based on the second-order model to obtain a low-dimensional feature vector. The predicted class is obtained by calculating the class probability of the feature map within each candidate region based on the low-dimensional feature vector; Based on the low-dimensional feature vector, the bounding box offset of the candidate region is predicted, and the position and size of the corrected candidate region are obtained.

3. The method for detecting defects in sewer pipes based on a confusion matrix according to claim 1, characterized in that, The step of correcting the prediction results of easily confused category groups in the prediction dataset based on the fine classification results of easily confused category groups to obtain the detection results specifically includes: When the number of samples in easily confused categories differs greatly and the features between categories are similar, the sub-classification results of the easily confused category group are used as the detection results. When the prediction results of the easily confused category group in the prediction dataset can distinguish the easily confused categories, the prediction boxes and confidence scores of the sub-classification results of the easily confused category group and the prediction results of the easily confused category group in the prediction dataset are fused to obtain the detection results.

4. The method for detecting defects in sewer pipes based on a confusion matrix according to claim 1, characterized in that, It also includes the step of removing duplicate and low-quality prediction boxes from the detection results to obtain the final detection result, which specifically includes: The predicted bounding boxes in the detection results are sorted according to their confidence scores and weights; Starting with the prediction box with the highest confidence, traverse the prediction boxes to obtain all prediction boxes with an overlap higher than a certain threshold. All the predicted bounding boxes found are assigned to the same cluster, and the weighted average coordinates, confidence score, and category are calculated to obtain the final detection result.

5. The method for detecting defects in sewer pipes based on a confusion matrix according to claim 1, characterized in that, The loss function based on the training results and the true labels of the training dataset is calculated as follows: in, This represents the sum of the loss functions of the first-order model and the second-order model. This represents the loss function of the first-order model. This represents the loss function of the second-order model. Represents the classification loss function. Represents the regression loss function. , These represent the balance parameters of the first-order and second-order models, respectively, used to control the relative importance of classification and regression.

6. The method for detecting defects in sewer pipes based on a confusion matrix according to claim 1, characterized in that, The loss function based on the training results of the third-order model and the true labels of the cropped training set is calculated as follows: in, This represents the multi-class focus loss function. Indicates the predicted probability. Indicates the weighting factor. This indicates a regulatory factor.

7. A sewer pipe defect detection system based on a confusion matrix, characterized in that, A method for performing a sewer pipe defect detection method based on a confusion matrix as described in claim 1 includes: The data acquisition module is used to collect images of various types of sewer defects and label them with their categories and locations to obtain training and validation datasets. The model building module constructs a three-stage artificial neural network model based on the first-order model, the second-order model, and the third-order model. The model training module trains and validates the three-stage artificial neural network model based on the training dataset and the validation dataset, and obtains the three-stage artificial neural network model with the highest mean accuracy. The prediction module is used to acquire the prediction dataset and input the prediction dataset into the first-order model and the second-order model with the highest average accuracy to obtain the prediction results of the prediction dataset; The easily confused coarse classification module determines the easily confused category groups based on the multi-class percentage confusion matrix and the prediction results of the prediction dataset; The fine classification module clips the prediction boxes of easily confused category groups in the prediction dataset and inputs them into the third-order model with the highest average accuracy for fine classification, thus obtaining the fine classification results of easily confused category groups. The correction module corrects the prediction results of easily confused category groups in the prediction dataset based on the fine classification results of easily confused category groups, and obtains the detection results; The post-processing module is used to remove duplicate and low-quality prediction boxes from the detection results to obtain the final detection results.

Images