A method for detecting road surface cracks based on unsupervised learning

Abstract

The application provides a kind of road surface crack detection method based on unsupervised learning, including collecting normal road surface data and generating normal road surface data set, also including the following steps: download vgg16 network model pre-trained on Imagenet data set as expert network, and respectively design identification clone network and positioning clone network;Identification clone network and positioning clone network are trained using the data in normal road surface data set;Get actual road surface image;Actual road surface image is respectively imported into expert network and identification clone network, it is judged whether there is crack, if there is crack, then execute next step;Actual road surface image is respectively imported into expert network and positioning clone network, and the positioning of specific position of crack is realized.The application is used for road surface crack identification and positioning, only needs to provide training set constructed by normal road surface picture, does not need to carry out pixel level road surface crack annotation, saves a lot of manpower cost, and also can have higher detection precision and real-time performance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for detecting road surface cracks based on unsupervised learning. Background Technology

[0002] Road surface cracks are one of the most representative road defects, mainly caused by overloading, temperature changes, and road surface aging. These damages reduce road surface performance, shorten road lifespan, and endanger vehicle driving safety, posing a potential threat to road safety. Locating and repairing cracks is a crucial responsibility of transportation maintenance departments to maintain roads in good condition. Early manual inspections were time-consuming and inefficient, and have gradually been replaced by road inspection vehicles. However, locating cracked sections manually from inspection vehicle video footage presents significant challenges. Fast and accurate road surface crack detection helps in timely road maintenance and prevents further deterioration. With the development of computer vision technology, efforts are focused on applying it for automated crack detection. Existing computer vision-based crack detection methods can be broadly categorized into three types: traditional methods, machine learning-based methods, and deep learning-based methods.

[0003] Traditional detection methods do not require learning from datasets, and most mathematical calculations are analytical, offering lightweight computational speed. However, they have weak generalization ability to various random variable factors, requiring parameter adjustments or algorithm redesign if the application scenario or environment changes.

[0004] Machine learning-based algorithms have been successfully applied in pavement crack detection. Most machine learning-based detection models require manual design of pavement crack features. However, given the diverse types of pavement cracks, it is difficult to design a universal feature extraction model to extract these features, resulting in poor detection performance.

[0005] While deep learning-based methods for detecting road cracks can achieve high accuracy, most are supervised algorithms that require extensive pixel-level annotation, consuming significant human and material resources. Furthermore, outliers in road crack detection are low-probability events, resulting in a small sample size and a severe imbalance problem in supervised learning.

[0006] In 2021, Salehi M, Sadjadi N, and Baselizadeh S published a paper titled "Multiresolution Knowledge Distillation for Anomaly Detection" in *Computer Vision and Pattern Recognition* (IEEE). This paper proposed a multiresolution knowledge distillation (MKD) method for anomaly detection based on unsupervised learning. The network is trained using only normal samples, yet the learned features can distinguish between normal and anomalous samples. A drawback of this method is its low accuracy in identifying cracks. Summary of the Invention

[0007] To address the aforementioned technical issues, this invention proposes a road surface crack detection method based on unsupervised learning for road surface crack identification and localization. It only requires a training set constructed from images of normal road surfaces, eliminating the need for pixel-level crack annotation, thus saving significant manpower costs while also achieving high detection accuracy and real-time performance.

[0008] This invention provides a pavement crack detection method based on unsupervised learning, including collecting normal pavement data and generating a normal pavement dataset, and further including the following steps:

[0009] Step 1: Download the pre-trained VGG16 network model on the ImageNet dataset as an expert network, and design a clone recognition network and a clone localization network respectively;

[0010] Step 2: Train the recognition clone network and the localization clone network using data from the normal road surface dataset;

[0011] Step 3: Obtain an image of the actual road surface;

[0012] Step 4: Import the actual road surface image into the expert network and the recognition clone network respectively, and determine whether there are cracks. If there are cracks, proceed to step 5.

[0013] Step 5: Import the actual road surface image into the expert network and the localization clone network respectively to locate the specific location of the crack.

[0014] Preferably, the expert network uses the first 10 layers of the vgg16 network model as the backbone network, where P1, P2, P3, and P4 are the pooling operations after the 2nd, 4th, 7th, and 10th convolutional layers in the network structure, respectively.

[0015] In any of the above schemes, step 1 preferably includes enabling the identification clone network model and the localization clone network model to learn the knowledge of the vgg16 network model by minimizing the differences in the distillation layers.

[0016] In any of the above schemes, it is preferred that during the learning process, the recognition clone network model discards the convolutional layers after the P4 pooling operation.

[0017] In any of the above schemes, it is preferred to minimize the feature map difference at pooling operation P4 to achieve optimal knowledge distillation for the road surface crack identification task.

[0018] In any of the above schemes, it is preferred that, during the learning process, the recognition clone network model changes the number of channels of all convolutional layers except the 10th convolutional layer to 32.

[0019] In any of the above schemes, it is preferred that during the learning process, the localization clone network model discards the convolutional layers after the P3 pooling operation.

[0020] In any of the above schemes, it is preferred to minimize the feature map difference at pooling operation P3 to achieve optimal knowledge distillation for the task of locating pavement cracks.

[0021] In any of the above schemes, it is preferred that, during the learning process, the localization clone network model changes the number of channels in all convolutional layers except the 7th convolutional layer to 32.

[0022] In any of the above schemes, the preferred option is that the difference refers to the loss function of training the unsupervised network, which includes two parts: the magnitude and direction of the knowledge, represented by the root mean square error and the cosine similarity error, respectively.

[0023] In any of the above schemes, the preferred option is that the root mean square error refers to the Euclidean distance between the outputs of the expert network and the clone network, as shown in the formula:

[0024]

[0025] Where, N i This refers to the Pth digit of the cloning network. i The number of elements in the feature map obtained after each pooling operation. It is the Pth expert network i The j-th element in the feature map obtained after pooling operations It is the Pth cloning network i The j-th element in the feature map obtained after pooling operations, where i represents the pooling layer, s represents the expert network, and c represents the recognition clone network and the localization clone network.

[0026] In any of the above schemes, the preferred embodiment is that the cosine similarity error refers to the directional difference between the knowledge of the expert network and the knowledge of the clone network, and the formula is:

[0027]

[0028] Where vec(x) is the vectorization function and T is the matrix transpose.

[0029] In any of the above schemes, the preferred option is that the total loss function is the weighted sum of the root mean square error and the cosine similarity error, as shown in the formula:

[0030] L total =L1+λL2

[0031] Where λ is a scaling factor that adjusts the weights of the two loss functions.

[0032] In any of the above solutions, the crack detection method preferably includes the following sub-steps:

[0033] Step 41: Calculate the total loss of the feature map output after the P3th pooling operation through two forward propagations of the expert network and the localization clone network model;

[0034] Step 42: Use interpretability methods to obtain all pixels whose contribution to the loss is greater than the threshold;

[0035] Step 43: Obtain the crack morphology in the image through filtering.

[0036] In any of the above schemes, step 41 preferably includes outputting the total loss function L after the P3th pooling operation using the expert network and the localization clone network. total Taking the derivative of the input road surface image x, we can obtain the attribution map Λ of the contribution of each pixel in the input image x to the loss, as shown in the formula.

[0037]

[0038] in, To obtain the partial derivative.

[0039] In any of the above schemes, it is preferable to use Gaussian smoothing and image morphology filtering to reduce noise in the attribution graph, as shown in the formula:

[0040] M = g σ (Λ)

[0041]

[0042] Among them, g σ For Gaussian filtering with standard deviation σ, For corrosion operation, For the expansion operation, B is the structuring element of the expansion operation.

[0043] In any of the above schemes, it is preferred that the Gaussian smoothing process is used to reduce Gaussian noise in the attribution graph.

[0044] In any of the above schemes, it is preferred that the erosion operation is used to eliminate other noise in the image.

[0045] In any of the above schemes, it is preferred that the dilation operation is used to restore the original shape of the image.

[0046] This invention proposes a road surface crack detection method based on unsupervised learning. It designs a lightweight knowledge distillation method based on the multi-resolution knowledge distillation (MKD) method, which improves both the accuracy and inference speed of road surface crack detection tasks. Attached Figure Description

[0047] Figure 1 This is a flowchart of a preferred embodiment of the unsupervised learning-based pavement crack detection method according to the present invention.

[0048] Figure 2 This is a schematic diagram of an embodiment of a lightweight LKD network for pavement crack detection based on unsupervised learning according to the present invention.

[0049] Figure 3 This is a schematic diagram of a dataset image according to a preferred embodiment of the unsupervised learning-based pavement crack detection method of the present invention.

[0050] Figure 4 This is a pavement condition identification ROC curve of a preferred embodiment of the pavement crack detection method based on unsupervised learning according to the present invention.

[0051] Figure 5 This is a crack location ROC curve of a preferred embodiment of the unsupervised learning-based pavement crack detection method according to the present invention. Detailed Implementation

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0053] Example 1

[0054] like Figure 1 As shown, step 100 is executed to collect normal road surface data and generate a normal road surface dataset.

[0055] In step 110, a pre-trained VGG16 network model on the ImageNet dataset is downloaded as an expert network, and a recognition clone network and a localization clone network are designed respectively. The expert network uses the first 10 layers of the VGG16 network model as the backbone, where P1, P2, P3, and P4 are the pooling operations after the 2nd, 4th, 7th, and 10th convolutional layers in the network structure, respectively. By minimizing the differences in the distillation layers, the recognition clone network model and the localization clone network model learn the knowledge of the VGG16 network model.

[0056] During the learning process, the recognition clone network model discards the convolutional layer after the P4 pooling operation and minimizes the feature map difference at pooling operation P4 to achieve optimal knowledge distillation for the road crack recognition task. The recognition clone network model changes the number of channels of all convolutional layers except the 10th convolutional layer to 32. The localization clone network model discards the convolutional layer after the P3 pooling operation and minimizes the feature map difference at pooling operation P3 to achieve optimal knowledge distillation for the road crack localization task. The localization clone network model changes the number of channels of all convolutional layers except the 7th convolutional layer to 32.

[0057] The difference refers to the loss function used to train the unsupervised network, which includes the magnitude and direction of the knowledge, represented by the root mean square error (RMSE) and cosine similarity error, respectively. The RMSE is the Euclidean distance between the outputs of the expert network and the clone network, expressed as follows:

[0058]

[0059] Where, N i This refers to the Pth digit of the cloning network. i The number of elements in the feature map obtained after each pooling operation. It is the Pth expert network i The j-th element in the feature map obtained after pooling operations It is the Pth cloning network i The j-th element in the feature map obtained after pooling operations, where i represents the pooling layer, s represents the expert network, and c represents the recognition clone network and the localization clone network.

[0060] The cosine similarity error refers to the directional difference between the knowledge of the expert network and the knowledge of the clone network, and the formula is:

[0061]

[0062] Where vec(x) is the vectorization function and T is the matrix transpose.

[0063] The total loss function is the weighted sum of the root mean square error and the cosine similarity error, and the formula is as follows:

[0064] L total =L1+λL2

[0065] Where λ is a scaling factor that adjusts the weights of the two loss functions.

[0066] Step 120 is performed, whereby the identification clone network model and the localization clone network model are trained using data from the normal road surface dataset.

[0067] Perform step 130 to obtain an image of the actual road surface.

[0068] Step 140 is executed, in which the actual road surface image is imported into the expert network and the recognition clone network respectively.

[0069] Perform step 150 to determine if there are cracks. The method for determining cracks includes the following sub-steps:

[0070] Execute step 151, calculate the total loss of the feature map output after the P3rd pooling operation through two forward propagations using the expert network and the localization clone network model, and output the total loss function L after the P3rd pooling operation using the expert network and the localization clone network. total Taking the derivative of the input road surface image x, we can obtain the attribution map Λ of the contribution of each pixel in the input image x to the loss, as shown in the formula.

[0071]

[0072] in, To obtain the partial derivative.

[0073] Perform step 152 to obtain all pixels whose contribution to the loss is greater than the threshold using an interpretability method.

[0074] Step 153 involves obtaining the crack morphology in the image through filtering. Gaussian smoothing and image morphology filtering are used to reduce noise in the attribution map, as shown in the formula below.

[0075] M = g σ (Λ)

[0076]

[0077] Among them, g σ For Gaussian filtering with standard deviation σ, For corrosion operation, For the dilation operation, B is the structuring element. The Gaussian smoothing process is used to reduce Gaussian noise in the attribution graph, the erosion operation is used to eliminate other noise in the image, and the dilation operation is used to restore the image to its original shape.

[0078] If the actual road surface image shows no cracks, then repeat step 130.

[0079] If the actual road surface image has cracks, proceed to step 160.

[0080] Step 160 involves importing the actual road surface image into the expert network and the localization clone network to locate the specific location of the crack.

[0081] Example 2

[0082] Based on literature review and analysis and numerous comparative experiments, this invention proposes a lightweight knowledge distillation method (LKD) for Pavement Crack Detection, which improves both accuracy and inference speed in pavement crack detection tasks.

[0083] 1. Network Structure

[0084] This invention, through extensive literature review and analysis, reveals that shallow features in deep convolutional neural networks contain more detailed information, while deep features possess higher semantic information. Pixel-level localization of road cracks relies more on the detailed information in shallow feature maps, while the identification of normal and abnormal road surfaces is a classification problem, requiring the understanding and discrimination of high-level semantic information in the image. Therefore, this invention designs lightweight localization clone networks and recognition clone networks, such as... Figure 2 As shown.

[0085] The core is an expert network, using the first 10 layers of VGG16 as the backbone. P1, P2, P3, and P4 are the pooling operations following the 2nd, 4th, 7th, and 10th convolutional layers, respectively. Distilling shallower knowledge makes it difficult for the cloned network to learn the most representative expert network knowledge, while distilling deeper knowledge easily leads to the loss of detailed information or overfitting. Through extensive ablation experiments, the research group discovered that multi-resolution knowledge distillation, unlike the MKD network, is unnecessary; minimizing the feature map difference at pooling operation P3 achieves optimal knowledge distillation for pavement crack localization. The designed localization cloned network, besides discarding the convolutional layer after pooling operation P3 in the MKD algorithm, also changed the number of channels in all convolutional layers except the 7th to 32. This not only reduced the number of parameters in the localization cloned network to 54.3% of the MKD cloned network and improved the inference speed by 4 fps (frames per second), but later experiments showed an 11.6% improvement in the AUROC (Area Under the Receiver Operating Characteristic curve) for pavement crack localization. Recognizing clone networks requires higher semantic information. Extensive ablation experiments show that, without requiring multi-resolution knowledge distillation, minimizing the feature map difference at pooling operation P4 can achieve optimal knowledge distillation for pavement crack recognition tasks. The designed recognition clone network discards the convolutional layer after the P4 pooling operation and changes the number of channels in all convolutional layers except the 10th convolutional layer to 32. Although the number of parameters in the recognition clone network is basically the same as that in the MKD clone network, the loss function only calculates the difference after P4 pooling, improving the inference speed by 2 fps and the recognition AUROC by 3.0%.

[0086] 2. Loss Function

[0087] By minimizing the difference in the distillation layers, the clone network learns the knowledge of the expert network. This difference is the loss function used to train the unsupervised LKD network, which includes the magnitude and direction of the knowledge, represented by the root mean square error and cosine similarity error, respectively. The root mean square error refers to the Euclidean distance between the outputs of the expert network and the clone network, as shown in Equation 1.

[0088]

[0089] Where, N i This refers to the Pth digit of the cloning network. i The number of elements in the feature map obtained after each pooling operation for the localization clone network P i That is P3, where batchsize represents the number of samples selected in one training iteration, and N i The batch size is 16×16×512; for the clone recognition network Pi If it's P4, then N i It is batchsize×8×8×512. and These are the expert network and the clone network, respectively. i The j-th element in the feature map obtained after pooling operations.

[0090] The cosine similarity error represents the directional difference between the knowledge of the expert network and the knowledge of the clone network, as shown in Equation 2.

[0091]

[0092] Here, vec(x) is a vectorization function that transforms a matrix x of arbitrary dimensions into a 1-dimensional vector. In this case, it transforms the P-th matrix of the expert network and the clone network into a 1-dimensional vector. i The feature map output after each pooling operation is converted into a 1-dimensional vector, which makes it easier to calculate the cosine similarity between the two. and These are the expert network and the clone network, respectively. i The feature map output after each pooling operation. The total loss function is a weighted sum of the two loss functions mentioned above, as shown in Equation 3, where λ is a scaling factor that adjusts the weights of the two loss functions.

[0093] L total =L1+λL2 (3)

[0094] 3. Reasoning process

[0095] Identifying whether a road surface image has cracks requires only two forward propagations: an expert network and a recognition clone network. The test image is input into the trained expert network and recognition clone network. If the total loss of the feature map output after the P4th pooling operation is greater than a set threshold, the input image is judged as an abnormal image with cracks; otherwise, it is considered a normal road surface.

[0096] Crack localization essentially involves binary classification of each pixel in the input image. First, it calculates the total loss of the feature map output after the P3th pooling operation through two forward propagations: an expert network and a localization clone network. Then, it utilizes interpretable methods... [24，25] All pixels contributing more than a threshold to the loss are obtained, and finally, the crack morphology in the image is acquired through filtering. Specifically, this involves using the expert network and the localization clone network to output the total loss L after the P3th pooling operation. total By taking the derivative of the input road surface image x, we can obtain the attribution map Λ of the contribution of each pixel of the input image x to the loss, as shown in Equation 4. This allows us to find all pixels that have a significant impact on the loss function, such as crack areas in abnormal samples.

[0097]

[0098] To reduce noise in the attribution graph, Gaussian smoothing and image morphology filtering are used, as shown in Equation (5) and Equation (6), respectively.

[0099] M = g σ (Λ) (5)

[0100]

[0101] Where g represents a Gaussian filter with a standard deviation of σ. and The erosion and dilation operations represent the erosion and dilation operations, respectively. Equation (6) performs an opening operation that involves erosion followed by dilation. Gaussian smoothing can reduce Gaussian noise in the attribution graph; erosion can eliminate other noise in the image, but it may also accidentally delete some pixels from the original binary image; dilation restores the image to its original shape. B is the structuring element of the dilation operation, usually in the shape of an ellipse or a disk. Through the above operations, the attribution graph Λ can obtain a more accurate crack morphology.

[0102] Example 3

[0103] A pavement crack detection method based on unsupervised learning is described below:

[0104] 1. Model Training

[0105] Dataset: The public CFD dataset contains only 118 images of road surface cracks with a resolution of 480×320 pixels, along with corresponding pixel-level annotations. There are no normal road surface images. The annotations are 480×320 resolution black-and-white binary images. All crack images in the dataset were taken on sidewalks in Beijing using an iPhone 5, and contain noise such as uneven lighting, shadows, oil stains, and watermarks, reflecting the actual road surface conditions in the city and increasing the difficulty of crack detection. To achieve unsupervised network model training, the research group took 357 normal road surface images with a resolution of 504×378 pixels using an iPhone 12 and added them to the dataset. 239 normal road surface images were randomly selected as the training set, and the remaining 118 normal road surface images and 118 cracked road surface images from the CFD dataset were used as the test set for crack recognition. The 118 cracked images and corresponding annotated images from the CFD dataset were used as the test set for crack localization. Examples of dataset images are shown below. Figure 3 As shown.

[0106] Experimental conditions: Operating system: CentOS 7.9; Deep learning framework: PyTorch 1.7.1; CPU: Xeon(R) Gold 6230R; Memory: 256G; GPU: NVIDIA GeForce 3090x4.

[0107] The training set contains only images of normal road surfaces, excluding those containing cracks. Our goal is to train two clone networks, and then use the expert network and the two trained clone networks separately for road crack identification and localization. Training each clone network model requires approximately 150 epochs, taking about 3 minutes. In all experiments, we set λ to 0.5 and optimized using the Adam optimizer with a learning rate of 0.001 and a batch size of 64.

[0108] 2. Model Testing

[0109] The test set for crack recognition includes images of normal road surfaces and cracked road surfaces. All samples in the test set are fed into the LKD expert network and recognition clone network. The performance metrics of the network's recognition part are obtained by comparing the prediction results with the input image categories. The test set for crack localization only contains images of road cracks and their corresponding annotations. These are fed into the LKD expert network and localization clone network, and the localization results are compared with the corresponding annotated images to verify the network's localization performance.

[0110] To verify the advantages of the improved model, comparative experiments were conducted on the dataset described in Section 4.1 using both the lightweight LKD network and the MKD network. The ROC curves (receiver-roperating characteristic curves) of the two network models for pavement condition recognition and crack localization are shown below. Figure 4 and Figure 5 As shown.

[0111] from Figure 4 and Figure 5 As can be seen from the ROC curves, the ROC curves of the LKD network proposed in this invention are closer to the upper left corner (0.0, 1.0) than those of the MKD network in both pavement condition identification and crack localization, indicating superior performance. AUROC represents the area under the ROC curve. Quantitative calculations allow for a clearer comparison of the performance of the improved LKD and MKD network models, as shown in Table 1.

[0112] Model Identify AUROC Positioning AUROC Recognition speed (FPS) Positioning speed (FPS) MKD 96.91％ 79.59％ 38 13 LKD 99.86％ 88.81％ 40 17

[0113] Table 1 Comparison of experimental results

[0114] Therefore, compared with the MKD network, the LKD network proposed in this invention not only improves the inference speed by 2fps when identifying road surface conditions, but also improves the identification AUROC by 3.0%; when locating cracks, it improves the inference speed by 4fps and the road surface crack location AUROC by 11.6% compared with the MKD network.

[0115] To better understand this invention, specific embodiments have been described in detail above, but these are not intended to limit the invention. Any simple modifications made to the above embodiments based on the technical essence of this invention still fall within the scope of this invention. Each embodiment in this specification focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to mutually. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A method for detecting pavement cracks based on unsupervised learning, comprising collecting normal pavement data and generating a normal pavement dataset, characterized in that, It also includes the following steps: Step 1: Download the pre-trained VGG16 network model on the ImageNet dataset as an expert network, and design a clone recognition network and a clone localization network respectively; The recognition clone network model discards the convolutional layer after the P4 pooling operation and minimizes the feature map difference at the P4 pooling operation to achieve the best knowledge distillation for the road crack recognition task. The localization clone network model discards the convolutional layer after the P3 pooling operation and minimizes the feature map difference at the P3 pooling operation to achieve the best knowledge distillation for the road crack localization task. Step 2: Train the recognition clone network and the localization clone network using data from the normal road surface dataset; Step 3: Obtain an image of the actual road surface; Step 4: Import the actual road surface image into the expert network and the recognition clone network respectively, and determine whether there are cracks. If there are cracks, proceed to step 5. Step 5: Import the actual road surface image into the expert network and the localization cloning network respectively to locate the specific location of the crack.

2. The pavement crack detection method based on unsupervised learning as described in claim 1, characterized in that, The expert network uses the first 10 layers of the vgg16 network model as the backbone, where P1, P2, P3, and P4 are the pooling operations after the 2nd, 4th, 7th, and 10th convolutional layers in the network structure, respectively.

3. The pavement crack detection method based on unsupervised learning as described in claim 2, characterized in that, Step 1 includes enabling the identification clone network model and the localization clone network model to learn the knowledge of the expert network by minimizing the differences in the distillation layers.

4. The pavement crack detection method based on unsupervised learning as described in claim 3, characterized in that, During the learning process, the recognition clone network model changed the number of channels in all convolutional layers except the 10th convolutional layer to 32.

5. The pavement crack detection method based on unsupervised learning as described in claim 4, characterized in that, During the learning process, the localization clone network model changed the number of channels in all convolutional layers except the 7th convolutional layer to 32.

6. The pavement crack detection method based on unsupervised learning as described in claim 5, characterized in that, The difference refers to the loss function used to train the unsupervised network, which includes the magnitude and direction of the knowledge, represented by the root mean square error and cosine similarity error, respectively.

Images