A fine-grained classification method for pipeline cracks based on knowledge distillation
Through knowledge distillation technology, the complex ResNet50 model is transferred to the lightweight student model, solving the problem of limited equipment resources in pipeline defect detection, realizing fast and efficient fine-grain classification of pipeline cracks on embedded devices, and improving detection accuracy and efficiency.
Patent Information
- Application Number
- CN202210490970.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-05-07
AI Technical Summary
The prior art has time-consuming and labor-intensive and error-prone problems in pipeline defect detection. Deep convolutional neural networks are limited in computing and storage resources when training and deploying on embedded devices, making it difficult to achieve fast and accurate fine-grained classification of pipeline cracks.
Using a knowledge-based distillation method, the complex teacher model ResNet50 is transferred to the lightweight student model CMI (combined with MobileNetV2 and Inception networks). Through logits distillation form and temperature regulation, the knowledge of the teacher model is extracted and fusion training is performed to obtain lightweight Level neural networks perform fine-grained classification of pipeline cracks.
It realizes fast and efficient fine-grain classification of pipeline cracks on embedded devices, reduces the computing and storage resource requirements and improves classification accuracy.
Smart Images

Figure CN114758182B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and civil engineering, and in particular to a pipeline crack fine-grained classification method based on knowledge distillation. Background Art
[0002] During the working process, sewer pipes are prone to various potential defects such as corrosion, blockage, cracks, etc., which may cause leakage accidents due to the influence of the working environment and the materials they carry. The pipeline closed-circuit television inspection system (CCTV) is a tool specifically used for underground pipeline inspection. However, the inspection of CCTV is time-consuming, labor-intensive, and error-prone; due to long inspections, operators may feel fatigued, resulting in incorrect assessments of sewer conditions. This brings about the problems of subjective judgment, operator fatigue, and long life cycle costs. Although these problems can be alleviated by proper training and certification of operators and the use of standard condition rating protocols, such as PACP, these limitations cannot be completely eliminated. Therefore, a reliable automatic defect detection and classification system is needed to make up for the lack of subjective interpretation of the actual conditions of buried sewage pipes.
[0003] In recent years, deep learning-based methods have gained widespread attention in the field of pipeline defect detection. Deep learning algorithms, such as convolutional neural networks (CNNs), provide solutions to overcome the difficulties of existing technologies in detecting pipeline defects. However, deep CNNs require more parameters to be trained. Therefore, more computing resources and time are required for training and prediction. When deployed on embedded or edge devices with weak GPUs, deep CNNs with more weight parameters are limited by computing and storage resources. Therefore, in these cases, the simplicity of CNNs is particularly important. Small networks are fast and take up less memory, but it is difficult to get accurate results by directly training on the true value. Therefore, model compression technology has become imperative. And "model distillation" is a method of model compression. Summary of the invention
[0004] Purpose of the invention: In view of the above shortcomings, the present invention discloses a method for fine-grained classification of pipeline cracks based on knowledge distillation. By introducing knowledge distillation technology to compress the convolutional neural network that can complete the task of fine-grained classification of pipeline cracks, a lightweight neural network is obtained, and finally the fine-grained classification of pipeline cracks is quickly obtained.
[0005] Technical solution: To solve the above problems, the present invention adopts a pipeline crack fine-grained classification method based on knowledge distillation, which includes the following steps:
[0006] (1) collecting pipeline images and performing preprocessing to obtain a pipeline crack dataset; the pipeline crack dataset contains manually annotated fine-grained category information of pipeline cracks;
[0007] (2) inputting the pipeline crack data set obtained in step (1) into the teacher model for model training, and obtaining the teacher model with the highest accuracy as the trained teacher model by adjusting the learning rate of the teacher model; the teacher model adopts ResNet50;
[0008] (3) Design a student model based on a lightweight neural network. The structure of the student model is: first pass through the MobileNetV2 network with 4 bottlenecks, and then pass through the Inception network;
[0009] (4) At temperature T, the knowledge of the teacher model trained by distillation is transferred to the student model, and the student model is trained based on the knowledge to obtain a student model with the minimum final loss as the trained student model; the knowledge is the probability that the fine-grained pipeline crack output by the teacher model belongs to a certain category and the category of the fine-grained pipeline crack manually labeled;
[0010] (5) The image to be classified is input into the trained student model to obtain the final crack fine-grained classification result.
[0011] Furthermore, (4.1) uses logits distillation to extract the probability that the fine-grained pipeline crack output by the trained teacher model belongs to a certain category; specifically, it includes: defining the input logits vector of the last fully connected layer of the teacher model as z, and outputting the probability p of belonging to the i-th category i The softmax function is calculated as:
[0012]
[0013] In the formula, z i represents the logit of the i-th category; j represents traversing the entire category, from the first category to the C-th category, where C is the total number of categories;
[0014] (4.2) Introducing temperature T, the probability of step (4.1) is transferred to the student model, then the probability p′ of the fine-grained pipeline gap belonging to the i-th category in the student model is i for:
[0015]
[0016] (4.3) The final loss in the student model is:
[0017]
[0018] Where L is the final loss; α is a hyperparameter, α∈(0.1, 0.5, 0.9); T is the temperature, T∈(5, 10, 20, 50); KLdiv(·) represents the Kullback-Leibler divergence; represents the probability distribution of the fine-grained crack size of the pipeline belonging to a certain category in the student model; represents the probability distribution of the fine-grained pipeline crack belonging to a certain category in the teacher model; Q s represents the probability that the pipeline crack belongs to a certain category in the student model when T=1; y true is a true value;
[0019] (4.4) The probability that the fine-grained pipeline cracks output by the teacher model belong to a certain category and the manually labeled categories of the fine-grained pipeline cracks are used to perform fusion training on the student model. By adjusting the hyperparameters and temperature, the student model with the minimum final loss is obtained as the trained student model.
[0020] Furthermore, the bottleneck is specifically as follows: first, a 1×1 point-by-point convolution is performed to increase the dimension, then a Depthwise convolution is performed, and finally a 1×1 convolution is performed to reduce the dimension.
[0021] Furthermore, the Inception network has a three-layer structure. The first layer includes three branches. The first branch performs a 3×3 BasicConv2d operation. The second branch first performs a 1×1 BasicConv2d operation and then performs two 3×3 BasicConv2d operations. The third branch performs an average pooling operation. The three branches are concat-operated and input into the second layer. The second layer is a Dropout layer, and the third layer is a fully connected layer.
[0022] Furthermore, the accuracy calculation formula in step (2) is:
[0023]
[0024] Where Acc is the accuracy; TP is the true positive class; TN is the true negative class; FP is the false positive class; FN is the false negative class.
[0025] Furthermore, the preprocessing in step (1) includes: extracting frames once every 30 frames, selecting images for manual annotation, and blurring the information label box on the video frame while annotating; performing data enhancement operations on the images to expand the sample size, that is, horizontally flipping, scaling, and randomly rotating the images by β degrees, β∈(0-180).
[0026] Furthermore, the category information in step (1) includes four categories, namely: longitudinal cracks, circumferential cracks, multiple cracks and normal.
[0027] In addition, the present invention also provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the above method. A debugging device, a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.
[0028] Beneficial effect: Compared with the prior art, the pipeline crack fine-grained classification method based on knowledge distillation described in the present invention has the following significant advantages: the complex model ResNet50 is used as the teacher model, and the lightweight network CMI for crack detection is designed as the student model; the large teacher model is extracted through the knowledge distillation technology to obtain the prior dark knowledge and transfer it to the student model, and then the small student model is trained in combination with the ground truth (GT) for classification, thereby improving the performance of a shallow and fast model by learning a cumbersome model. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 Shown is a flow chart of the method of the present invention;
[0030] Figure 2 Shown is a structural diagram of a student model designed by the method of the present invention;
[0031] Figure 3 Shown is the structure of the bottleneck in the student model;
[0032] Figure 4 Shown is a practical application flow chart of the method of the present invention;
[0033] Figure 5 Shown is a diagram of four categories of fine-grained classification of pipeline cracks in the method of the present invention; Figure 5 (a) shows a transverse crack. Figure 5 (b) shows a longitudinal crack. Figure 5 (c) shows multiple cracks. Figure 5 (d) shows the normal class. DETAILED DESCRIPTION
[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.
[0035] like Figure 1 As shown, the pipeline crack fine-grained classification method based on knowledge distillation described in the present invention specifically includes the following steps:
[0036] Step 1: Obtain pipeline crack data set;
[0037] Specifically, the pipeline robot collects images to obtain CCTV video files, and preprocesses the CCTV video files: extract frames every 30 frames, select images and manually annotate the pipeline cracks in the images, annotate the fine-grained and real categories of pipeline cracks, and form a pipeline crack data set; blur the information label box on the video frame to avoid the influence of the label box on crack detection; perform data enhancement operations on the image, horizontally flip, scale, and randomly rotate (0-180) degrees to expand the sample size. Among them, the four annotated categories are: longitudinal cracks, circumferential cracks, multiple cracks, and normal.
[0038] Step 2: Distillation preparation stage: prepare the trained teacher model and design the student model;
[0039] (1) Select the complex network ResNet50 as the teacher model, and use the pipeline crack data set obtained in step 1 to train the teacher model. During the training process, the teacher model with the highest accuracy in predicting the category is obtained by adjusting the hyperparameters such as the network learning rate. The accuracy calculation formula is:
[0040]
[0041] Where Acc is the accuracy; TP is the true positive class; TN is the true negative class; FP is the false positive class; FN is the false negative class.
[0042] (2) The present invention designs a student model based on a lightweight network CMI (Combined MobileNetV2&Inception), which combines the MobileNetV2 network and the Inception network. Figure 2 and Figure 3 As shown in the figure, the structure of the student model is as follows: 4 inverted residual blocks (bottleneck) are used first, and then a deconvolution operation is performed to enlarge the output feature map to make the crack texture more obvious. Specifically, the bottleneck structure is: first, the dimension is increased by 1×1 point-by-point convolution, then the depthwise convolution is performed, and finally the dimension is reduced by 1×1 convolution; among them, when stride=1 and the shape of the input feature matrix is the same as that of the output feature matrix, a shortcut connection is performed. The structure of InceptionB is used again, that is, after three branches, the first branch performs 3×3 BasicConv2d (the structure is through the convolution layer and the BN layer), the second branch first performs 1×1 BasicConv2d, and then performs two 3×3 BasicConv2d, the third branch performs an average pooling operation, and finally the three branches are concat. A Dropout layer and a fully connected layer are added at the end of the network. The total parameter size of CMI is 35.96MB.
[0043] Step 3: Knowledge Distillation
[0044] (1) Use logits distillation to extract the predicted prior dark knowledge from the teacher model trained in step 2. Specifically, define the input logits vector of the last fully connected layer of the teacher model as z, and let z i represents the logit of the i-th category, then the output belongs to the probability p of the i-th category i Calculated by softmax:
[0045]
[0046] In the formula, z i represents the logit of the i-th category; j represents traversing the entire category, from the first category to the C-th category, where C is the total number of categories;
[0047] (2) The above teacher model can obtain the implicit knowledge of soft target prediction, that is, the probability that the predicted fine-grained pipeline crack belongs to a certain category, and transfer it to the student model; at the same time, the temperature T is introduced to control each soft target, and finally the probability p′ of the fine-grained pipeline crack in the student model belonging to the i-th category is i , the formula is:
[0048]
[0049] (3) The final loss in the student model is the combination of the teacher model’s soft target as the distillation loss and the true label’s hard target as the classification loss. The specific formula is:
[0050]
[0051] Where L is the final loss; α is a hyperparameter with a range of (0.1, 0.5, 0.9); T is the temperature with a range of (5, 10, 20, 50); KLdiv(·) represents the Kullback-Leibler divergence; represents the probability distribution of the fine-grained crack size of the pipeline belonging to a certain category in the student model; represents the probability distribution of the fine-grained pipeline crack belonging to a certain category in the teacher model; Q s represents the probability that the fine-grained crack of the pipeline in the student model belongs to a certain category when T=1; y true is the true value, i.e., the numerical value of the fine-grained category label of pipeline cracks;
[0052] The KLdiv(·) equation is:
[0053]
[0054] Where p(x) and q(x) are two probability distributions of discrete random variables;
[0055] E(·) is the cross entropy, and the equation is as follows:
[0056]
[0057] Where C represents the number of categories, p is the true value, and q is the predicted value.
[0058] (4) The student model is trained based on the probability of the fine-grained pipeline cracks output by the teacher model obtained in (1)-(3) belonging to a certain category and the category of the real fine-grained pipeline cracks manually labeled. By adjusting the hyperparameters and temperature, the final loss minimum L is obtained. min The student model of is taken as the trained student model.
[0059] Step 5: Input the image to be classified into the trained student model to obtain the final classification result. In order to further evaluate the classification effect of the student model, the final classification result can be compared with the real label category information to calculate the accuracy.
[0060] Step 6: Figure 4 As shown in the figure, the trained student model is directly installed on the NLE-AI800 embedded development platform to achieve fine-grained classification of pipeline cracks. Figure 5 As shown in the figure, the different gap categories in the pipeline are directly obtained using the NLE-AI800 embedded development platform. Figure 5 (a)- Figure 5 (d) They are horizontal cracks, longitudinal cracks, multiple cracks, and normal.
Claims
1. A pipeline crack fine-grained classification method based on knowledge distillation, characterized in that: The following steps are involved: (1) Collect pipeline images and perform preprocessing to obtain a pipeline crack dataset; the pipeline crack dataset contains fine-grained category information of pipeline cracks that are manually annotated; the preprocessing includes: extracting frames every 30 frames, selecting images for manual annotation, and blurring the information label box on the video frame while annotating; performing data enhancement operations on the images to expand the sample size, that is, horizontally flipping, scaling, and randomly rotating the images by β degrees, β∈(0-180); the category information includes four categories, namely: longitudinal cracks, circumferential cracks, multiple cracks, and normal; (2) The pipeline crack data set obtained in step (1) is input into the teacher model for model training. By adjusting the learning rate of the teacher model, a teacher model with the highest accuracy is obtained as the trained teacher model; the teacher model adopts ResNet50; the accuracy calculation formula is: ; In the formula, is the accuracy rate; TP It is a real class; TN It is a true negative type; FP It is a false positive class; FN It is a false negative class; (3) Design a student model based on a lightweight neural network; the structure of the student model is: first pass through the MobileNetV2 network with 4 bottlenecks, the bottleneck is first 1×1 point-by-point convolution for dimensionality increase, then pass through Depthwise convolution, and finally pass through 1×1 convolution for dimensionality reduction; then pass through the Inception network, the Inception network has a three-layer structure, the first layer includes three branches, the first branch performs a 3×3 BasicConv2d operation, the second branch first performs a 1×1 BasicConv2d operation, and then performs two 3×3 BasicConv2d operations, the third branch performs an average pooling operation, the three branches are concat-operated and input into the second layer, the second layer is the Dropout layer, and the third layer is the fully connected layer; (4) At temperature T, the knowledge of the teacher model trained by distillation is transferred to the student model. The student model is trained based on the knowledge, and the student model with the minimum final loss is obtained as the trained student model. The knowledge is the probability that the fine-grained pipeline crack output by the teacher model belongs to a certain category and the category of the fine-grained pipeline crack manually labeled. (5) Input the image to be classified into the trained student model to obtain the final crack fine-grained classification result.
2. The pipeline crack fine-grained classification method based on knowledge distillation according to claim 1 is characterized in that: Step (4) specifically includes: (4.1) Using logits distillation, extract the probability that the fine-grained pipeline crack output by the trained teacher model belongs to a certain category; specifically, define the input logits vector of the last fully connected layer of the teacher model as z, and output the probability p of belonging to the i-th category i The softmax function is calculated as: ; In the formula, z i represents the logit of the i-th category; j represents traversing the entire category, from the first category to the C-th category, where C is the total number of categories; (4.2) Introducing temperature T, the probability of step (4.1) is transferred to the student model, then the probability p′ of the fine-grained pipeline gap belonging to the i-th category in the student model is i for: ; (4.3) The final loss in the student model is: ; Where L is the final loss; α is a hyperparameter, α∈(0.1, 0.5, 0.9); T is the temperature, T∈(5, 10, 20, 50); KLdiv(·) represents the Kullback-Leibler divergence; represents the probability distribution of the fine-grained pipeline crack belonging to a certain category in the student model; represents the probability distribution of the fine-grained pipeline crack belonging to a certain category in the teacher model; C is the number of categories; represents cross entropy; Q s represents the probability that the pipeline crack belongs to a certain category in the student model when T=1; y true is a true value; (4.4) The probability that the fine-grained pipeline cracks output by the teacher model belong to a certain category and the manually labeled categories of the fine-grained pipeline cracks are used to perform fusion training on the student model. By adjusting the hyperparameters and temperature, the student model with the minimum final loss is obtained as the trained student model.
3. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 2.
4. A debugging device, characterized in that: A memory, a processor, and a program stored and executable on the memory, wherein when the program is executed by the processor, the steps of the method according to any one of claims 1 to 2 are implemented.
