Asphalt pavement defect detection and identification method based on semi-supervised learning

Abstract

The invention discloses an asphalt pavement defect detection and identification method based on semi-supervised learning, and the method comprises the steps: inputting a to-be-identified asphalt pavement image into a convolutional neural network model obtained through training, and judging whether the to-be-identified asphalt pavement image is intact or has a defect prediction result based on an optimal defect value threshold value; the step of training to obtain the convolutional neural network model comprises the steps of constructing the convolutional neural network model based on a convolutional neural network ResNet-18; constructing a training set based on a pre-acquired intact asphalt pavement image, inputting the training set into a convolutional neural network model, and performing asphalt pavement image feature extraction by the convolutional neural network model to obtain an extracted image patch; flattening the extracted image patch to obtain an image patch space feature vector of the training set; and fusing the image patch space feature vectors of all the training sets and carrying out random sampling. According to the method, a semi-supervised learning strategy is adopted, and only intact asphalt pavement images easy to collect are adopted as a training set.

Description

Technical Field

[0001] The invention relates to a detection and identification method for asphalt pavement defects based on semi-supervised learning, and belongs to the technical field of civil and traffic engineering. Background Art

[0002] With the development of my country's economy and the advancement of urbanization, a large number of expressways and inter-city first-class highways have been built. While these road network hubs provide guarantees for regional economic development, their own health monitoring and maintenance cannot be ignored. Due to some internal and external factors of asphalt roads, internal factors such as asphalt aging over time, external factors such as ambient temperature, rainfall, snowfall, and human factors such as overloading and heavy traffic, these high-grade roads (mainly asphalt pavements) inevitably suffer from surface damage or defects, manifested as cracks, potholes, rutting, subsidence and other symptoms, thus affecting driving comfort and road traffic safety, and causing traffic jams. Therefore, in order to ensure the smooth and safe road traffic, road health monitoring and maintenance are an indispensable part of road engineering. Traditional road monitoring and maintenance often use manual visual inspection, which is time-consuming and labor-intensive, inefficient and easy to miss.

[0003] With the development of computer vision technology, the use of vehicle-mounted cameras or mobile phones to directly drive on the road section to be inspected and record videos or collect images, and then use artificial intelligence models to intelligently identify the collected videos or images, which not only saves time but also improves detection accuracy, thereby improving the efficiency of asphalt pavement damage detection. However, in reality, there are a large number of asphalt pavements that are not damaged and are in normal use, while there are relatively few asphalt pavement sections that are damaged, resulting in asymmetric image samples collected, which brings difficulties to the training of convolutional neural networks. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the prior art, provide a detection and identification method for asphalt pavement defects based on semi-supervised learning, and use the convolutional neural network model ResNet-18 to extract and fuse asphalt pavement image features. The extracted image patches are flattened to form patch spatial feature vectors. These extracted image patch feature vectors are then fused and randomly sampled to reduce the amount of model calculation. The training set of the convolutional neural network model ResNet-18 is only based on intact asphalt pavement images, and the spatial feature vectors trained based on intact asphalt pavement images are multivariate normally distributed; it is used to replace the traditional manual visual inspection of asphalt highway pavement damage to improve the detection efficiency and accuracy of asphalt pavement damage. At present, the artificial intelligence model in this regard is mainly based on the transfer learning strategy, and the artificial intelligence model needs to be retrained to adapt to the needs of asphalt pavement images. Retraining the artificial intelligence model requires a large number of intact asphalt pavement images and damaged asphalt pavement images. However, in reality, damaged asphalt pavement images are less and more difficult to collect. In addition, retraining the artificial intelligence model to adapt to the detection and identification of asphalt pavements also takes a lot of time, and the model efficiency is low. The present invention adopts a semi-supervised learning strategy and uses a trained artificial intelligence model, so there is no need to retrain for asphalt pavement images, thereby saving model training time. In addition, the model of the present invention only uses easy-to-collect intact asphalt pavement images as training sets, which is particularly suitable for use in situations where damaged asphalt images are difficult to collect.

[0005] Preferably, the present invention provides inputting the asphalt pavement image to be identified into the trained convolutional neural network model, the convolutional neural network model extracts the features of the asphalt pavement image, and obtains the image patch of the extracted asphalt pavement image to be identified;

[0006] Flattening the extracted image patch of the asphalt pavement image to be identified to obtain a spatial feature vector of the image patch of the asphalt pavement image to be identified;

[0007] The spatial feature vectors of the image patches of the asphalt pavement image to be identified are fused and randomly sampled;

[0008] Based on the image patch space feature vectors of the training set obtained in advance, a normal distribution of the image patch space feature vectors of the training set is calculated;

[0009] Using Mahalanobis distance as a metric, the gap between the spatial feature vector of each image patch of the asphalt pavement image to be identified and the corresponding normal distribution in the training set is determined;

[0010] The Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve.

[0011] Based on the ROC curve, the maximum Youden index is used to determine the defect value of the asphalt pavement image to be identified;

[0012] Based on a predetermined

[0013] If the defect value of the asphalt pavement image to be identified exceeds a predetermined optimal defect value threshold, the asphalt pavement image to be identified is determined to have defects; if the defect value of the asphalt pavement image to be identified is lower than the optimal defect value threshold, the asphalt pavement image is determined to be intact.

[0014] Prioritize, train a convolutional neural network model, including:

[0015] A calibration set is constructed based on pre-acquired intact asphalt road images;

[0016] Based on the image patch space feature vector of the training set, a normal distribution of the image patch space feature vector is obtained by calculation;

[0017] Using Mahalanobis distance as a metric, the distance between each image patch spatial feature vector in the calibration set and the corresponding normal distribution in the training set is determined;

[0018] The Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve.

[0019] Based on the ROC curve, the maximum Youden index is used to determine the optimal defect value threshold.

[0020] Preferably, the spatial feature vectors of the intact asphalt pavement images of the training set are multivariate normally distributed.

[0021] Preferably, the convolutional neural network model includes a first convolutional layer, a first pooling layer, a first module, a second module, a third module, a fourth module, a second pooling layer and a fully connected layer, which are connected in sequence.

[0022] Preferably, the first module comprises a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer connected in sequence,

[0023] The second module includes the sixth convolutional layer, the seventh convolutional layer, the eighth convolutional layer and the ninth convolutional layer connected in sequence.

[0024] The third module includes the tenth convolutional layer and the eleventh convolutional layer connected in sequence.

[0025] The fourth module includes a twelfth convolutional layer and a thirteenth convolutional layer connected in sequence.

[0026] Preferably, the first convolutional layer has 64 filters, a filter size of 7×7, and a stride of 2; the second, third, fourth, and fifth convolutional layers have 64 filters, a filter size of 3×3, the sixth, seventh, eighth, and ninth convolutional layers have 128 filters, a filter size of 3×3, the tenth and eleventh convolutional layers have 256 filters, a filter size of 3×3, the twelfth and thirteenth convolutional layers have 512 filters, a filter size of 3×3, and the stride of the sixth, tenth, and twelfth convolutional layers is 2.

[0027] Preferably, the Mahalanobis distance M(X ij ) is calculated as:

[0028]

[0029] Where, X ij is the spatial feature vector of the image patch extracted from the asphalt road image; μ ij and Σ ij They represent the mean and covariance of the spatial feature vector of the image patch extracted at the position (i, j) of the intact asphalt road image in all the training sets.

[0030] Preferably, the size of the first asphalt image spatial feature patch output by the first module is adjusted to 56×56, the size of the second asphalt image spatial feature patch output by the second module is adjusted to 28×28, and the size of the third asphalt image spatial feature patch output by the third module is adjusted to 14×14.

[0031] Preferably, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0032] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0033] The beneficial effects achieved by the present invention are:

[0034] The present invention aims to propose an artificial intelligence method for asphalt pavement image processing based on semi-supervised learning. The convolutional neural network ResNet-18 is used to extract the features of the asphalt pavement image, and the extracted image patches are flattened to form patch space feature vectors. Subsequently, these extracted image patch feature vectors are fused and randomly sampled to reduce the amount of model calculation. The training set of the convolutional neural network model ResNet-18 is only used for intact asphalt pavement images, and the spatial feature vectors trained based on intact asphalt pavement images are multivariate normally distributed; then the receiver operating characteristic curve ROC is used to determine the defect value threshold of the asphalt pavement image. The Mahalanobis distance is selected as a metric and converted into a defect value. If the defect value of the asphalt pavement image to be identified exceeds the defect value threshold, it indicates that the asphalt pavement to be identified is damaged; if the defect value of the asphalt pavement image to be identified is lower than the defect value threshold, it indicates that the asphalt pavement is intact and undamaged. The defect value threshold of the asphalt pavement image is determined using an independent asphalt pavement image calibration set. Finally, the convolutional neural network model ResNet-18 was tested using a test set of asphalt road images, and a heat map was used to explain the underlying logic of the neural network model for image classification.

[0035] The present invention is used to replace the traditional manual visual inspection of damaged asphalt road pavement to improve the detection efficiency and accuracy of damaged asphalt pavement. At present, artificial intelligence models in this regard are mainly based on transfer learning strategies, which retrain artificial intelligence models to adapt to the needs of asphalt pavement images and use a large number of intact asphalt pavement images and damaged asphalt pavement images to retrain artificial intelligence models. However, in reality, damaged asphalt pavement images are relatively few and difficult to collect. In addition, it takes a lot of time to retrain artificial intelligence models to adapt to the detection and identification of asphalt pavements, and the model efficiency is low. The present invention adopts a semi-supervised learning strategy and uses a trained artificial intelligence model. Therefore, it does not need to be retrained for asphalt pavement images, thereby saving model training time. In addition, the model of the present invention only uses easy-to-collect intact asphalt pavement images as training sets, which is particularly suitable for use in situations where damaged asphalt images are difficult to collect.

[0036] Based on this actual situation, the present invention proposes a semi-supervised machine learning strategy, which can make full use of the collected normal images of asphalt pavement, that is, intact asphalt pavement images to train the model and detect and identify road damage, while reducing the computational amount of retraining of the neural network model, thereby improving the computational efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 is a flow chart of some embodiments of the present application;

[0039] Figure 2 is a structural diagram of a model in some embodiments of the present application;

[0040] Figure 3 is a schematic diagram of an asphalt road surface image in some embodiments of the present application;

[0041] Figure 4 is a schematic diagram of an asphalt pavement image sample in some embodiments of the present application;

[0042] Figure 5 is a histogram of average defect values ​​of intact asphalt pavement images and damaged asphalt pavement images in some embodiments of the present application;

[0043] Figure 6 is a schematic diagram of a ROC curve in some embodiments of the present application;

[0044] Figure 7 is a schematic diagram of a confusion matrix before adjustment in some embodiments of the present application;

[0045] Figure 8 is a schematic diagram of a true positive sample of an asphalt pavement image in some embodiments of the present application;

[0046] Fig. 9 is a schematic diagram of superimposed heat of a true positive sample of an asphalt pavement image in some embodiments of the present application;

[0047] Fig.10 is a schematic diagram of a true negative sample of an asphalt pavement image in some embodiments of the present application;

[0048] Fig.11 is a schematic diagram of superimposed thermal force of a true negative sample of an asphalt pavement image in some embodiments of the present application;

[0049] Fig.12 is a schematic diagram of an asphalt road surface image classified as a false positive in some embodiments of the present application;

[0050] Fig.13 is a schematic diagram of superimposed heat of true negative samples of asphalt pavement images classified as false positives in some embodiments of the present application;

[0051] Fig.14 is a schematic diagram of an asphalt pavement image of a false negative classification result predicted by a model in some embodiments of the present application;

[0052] Fig.15 is a schematic diagram of an asphalt pavement image superimposed with thermal force of a false negative classification result predicted by a model in some embodiments of the present application;

[0053] Fig.16 It is a schematic diagram of the accuracy of model prediction results in some embodiments of the present application. DETAILED DESCRIPTION

[0054] Embodiment 1

[0055] In recent decades, with economic development and population growth, a high-grade road network including expressways and first-class highways (mainly asphalt pavement) has developed rapidly. With the construction of these infrastructures, the corresponding real-time monitoring of road health and timely maintenance have become an indispensable part. These inspections and tests are carried out regularly to ensure that road facilities can maintain good working condition throughout their service life. However, various reasons such as bad weather, overloading and aging of asphalt itself over time can cause asphalt road pavement to break, which will destroy the comfort of passengers and may bring potential traffic safety hazards and traffic jams. Therefore, it is crucial to monitor the quality of road pavement in real time and promptly detect and solve potential and existing damage defects such as cracks, potholes, rutting and loose aggregates caused by aging and hardening of asphalt. Traditional visual inspection methods are inefficient and prone to missed inspections. With the development of artificial intelligence, more and more road pavement inspections are using artificial intelligence methods. However, due to the diversity of road pavement damage and the interference of factors such as road markings such as lane lines and zebra crossings, image shadows and changes in lighting during shooting, it is extremely difficult to accurately identify and locate road pavement damage using artificial intelligence methods.

[0056] There are many studies in this area, but methods that can efficiently and accurately detect and locate asphalt road pavement defects are still under exploration. According to the technology and equipment used, these studies can be roughly divided into the following three categories: 3D laser scanning to reconstruct asphalt road pavement methods, computer vision-based identification methods, and identification methods based on running vehicle response. In the 3D laser scanning to reconstruct asphalt road pavement methods, the image data of the road surface is usually obtained by a laser scanner installed on the detection vehicle. Due to the high cost of laser scanner equipment, this method is relatively uneconomical. With the rapid development of computer vision technology, more and more studies on road pavement defect detection based on vision technology have been conducted. Compared with manual visual inspection, automated visual identification is more efficient and accurate, and it reduces human errors. Overall, vision-based road pavement defect detection and identification methods are more widely used and have achieved certain success in many fields. However, it is worth noting that these methods are often easily disturbed by image background or image noise, thus lacking robustness. In addition, vision-based asphalt pavement damage identification methods usually require a large amount of asphalt pavement image data to train deep learning models, while the image data of asphalt pavement in a damaged state is relatively small, so it may be difficult to collect a large amount of data. At the same time, labeling these explicit training data is also laborious and time-consuming. Moreover, existing research rarely involves how to interpret the detection results of neural networks to understand the underlying logic of these artificial intelligence models in making decisions, which is crucial to improving the accuracy of existing neural network models. Finally, the vehicle response-based method involves the use of sensors such as smartphones, accelerometers, and gyroscopes to measure the vehicle's response signals. Existing studies have shown that the asphalt pavement damage identification method based on vehicle response is the most cost-effective method, because some sensors can be implemented through smartphone software. However, the connection between the vehicle's response signal and the damage of the highway pavement is complex and will be affected by many factors, such as the type of vehicle, the type of smartphone, the vehicle's driving speed, and the placement and orientation of the sensor. These factors will cause the analysis results to be relatively discrete and unable to accurately reflect the actual road pavement damage condition.

[0057] Semi-supervised learning aims to make full use of implicit data, here refers to intact asphalt road image data, reduce the workload of labeling images, and improve the performance and generalization ability of neural network models. It involves training with a relatively small amount of labeled data (explicit data) and a large amount of unlabeled data (implicit data). Labeled data guides the model to train in the desired direction, while unlabeled data helps the model autonomously explore and learn the underlying structure and patterns in the data. Given the large amount of easily accessible unlabeled data in the real world, semi-supervised learning has significant advantages over supervised learning methods.

[0058] Another issue that needs to be considered is the rationality and basis of the trained neural network model's judgment and decision-making on the given asphalt pavement image. When using artificial intelligence technology, especially the deep convolutional neural network model, the model is similar to a "black box". The image is input and the intelligent model gives the result. The accuracy of the model is then statistically obtained. The higher the accuracy, the better the model. It is rarely considered why the model makes such a judgment. Based on known research results, it is known that the deep convolutional neural network model has the strongest ability and the highest accuracy, but its interpretability is the worst. In contrast, the simple linear regression model is the simplest and easiest to understand computational model. Its prediction results are easy to explain theoretically, but its accuracy is low and the model's ability is the worst. Therefore, it is crucial to try to reasonably explain the results of the deep convolutional neural network model to understand the underlying logic of the model's judgment and adopt corresponding strategies to improve its performance.

[0059] See also Figure 1 , the present application discloses inputting an asphalt pavement image to be identified into a convolutional neural network model obtained through training, the convolutional neural network model extracts features of the asphalt pavement image, and obtains an image patch of the extracted asphalt pavement image to be identified;

[0060] Flattening the extracted image patch of the asphalt pavement image to be identified to obtain a spatial feature vector of the image patch of the asphalt pavement image to be identified;

[0061] The spatial feature vectors of the image patches of the asphalt pavement image to be identified are fused and randomly sampled;

[0062] Based on the image patch space feature vectors of the training set obtained in advance, a normal distribution of the image patch space feature vectors of the training set is calculated;

[0063] Using Mahalanobis distance as a metric, the gap between the spatial feature vector of each image patch of the asphalt pavement image to be identified and the corresponding normal distribution in the training set is determined;

[0064] All the calculated Mahalanobis distance values ​​are converted to the

[01] interval, the Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve.

[0065] Based on the ROC curve, the maximum Youden index is used to determine the defect value of the asphalt pavement image to be identified;

[0066] Based on a predetermined

[0067] If the defect value of the asphalt pavement image to be identified exceeds a predetermined optimal defect value threshold, the asphalt pavement image to be identified is determined to have defects; if the defect value of the asphalt pavement image to be identified is lower than the optimal defect value threshold, the asphalt pavement image is determined to be intact.

[0068] Among them, the convolutional neural network model is trained, including:

[0069] Based on the convolutional neural network ResNet-18, a convolutional neural network model was constructed;

[0070] A training set is constructed based on a previously acquired intact asphalt pavement image, and the training set is input into a convolutional neural network model, which extracts features of the asphalt pavement image to obtain extracted image patches;

[0071] Flatten the extracted image patches to obtain the image patch space feature vectors of the training set;

[0072] The image patch space feature vectors of all training sets are fused and randomly sampled.

[0073] Prioritize, train a convolutional neural network model, including:

[0074] A calibration set is constructed based on pre-acquired intact asphalt road images;

[0075] Based on the image patch space feature vector of the training set, a normal distribution of the image patch space feature vector is obtained by calculation;

[0076] Using Mahalanobis distance as a metric, the distance between each image patch spatial feature vector in the calibration set and the corresponding normal distribution in the training set is determined;

[0077] The Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve.

[0078] Based on the ROC curve, the maximum Youden index is used to determine the optimal defect value threshold.

[0079] The present invention uses a semi-supervised learning method to effectively reduce the workload of collecting and labeling explicit asphalt road pavement image data, and only uses intact asphalt road pavement images to train the neural network model. Secondly, the asphalt road pavement defect values ​​are used to generate a heat map to explain the underlying logic of the judgment made by the trained neural network model. The flowchart of the present invention is shown below.

[0080] ResNet-18 consists of 18 layers (17 convolutional layers + 1 fully connected layer) to learn the features of the input image. The first convolutional layer contains 64 filters, each with a size of 7*7 and a stride of 2, followed by a maximum pooling layer. It is followed by four modules, each with two paired convolutional layer units, followed by an average pooling layer and a fully connected layer. The ResNet-18 architecture is as follows Figure 2 As shown, the core steps of the method based on image patch feature extraction in the present invention are shown on the left side of the figure. In general, ResNet-18 uses shortcut connections to simplify the deep convolutional neural network, and strikes a balance between the high performance of the model and the complexity of the model, so it is the preferred convolutional neural network model of the present invention.

[0081] Figure 2 The main steps of the method used in the present invention are shown on the left. The main advantages of this method can be described in the following two aspects. First, it uses the trained ResNet-18 model to extract image patches from the input asphalt pavement image without retraining the ResNet-18 model for asphalt pavement images, which makes this method time-saving. Second, as a semi-supervised learning method, it only needs to be trained with normal asphalt pavement image data to learn and extract specific patterns from the structure of the training data. In view of these advantages, we only need to fine-tune the hyperparameters of the model to distinguish between defective asphalt pavement images and normal asphalt pavement images.

[0082] like Figure 2 As shown, the first module, the second module and the third module of ResNet-18 are used to extract the first asphalt image spatial feature patch, the second asphalt image spatial feature patch and the third asphalt image spatial feature patch respectively. The first module will output 64 first asphalt image spatial feature patches of size 56*56, the second module will output 128 second asphalt image spatial feature patches of size 28*28, and the third module will produce 256 third asphalt image spatial feature patches of size 14*14. In order to ensure the consistency in size of image patches of different scales output by different modules, an upsampling method is used here to align the image patches. The first asphalt image spatial feature patch, the second asphalt image spatial feature patch and the third asphalt image spatial feature patch are all adjusted to 56*56, as shown Figure 2 As shown. Then, the first asphalt image spatial feature patch, the second asphalt image spatial feature patch and the third asphalt image spatial feature patch are flattened and fused, and adjusted to have 448 channels per image to form a feature vector matrix. This upsampling and fusion process makes full use of the extracted multi-scale resolution image patches, so that the model can effectively capture the overall and detailed features of the asphalt pavement image at the same time.

[0083] As mentioned before, the ResNet-18 model is trained using only the collected normal asphalt road images and learns the distribution pattern of the image patch feature vectors from the training set. Assume that the distribution of the patch feature vectors of the training dataset is a multivariate normal distribution N(μ ij ,Σ ii ), where (i, j) represents the position in the normal asphalt road image. If the spatial feature vector of the extracted normal asphalt road image is represented by X ij , then at the position (i, j) of the normal asphalt road image, the mean and covariance of all spatial feature vectors in the training set can be expressed as:

[0084]

[0085] In the formula, μ ij and Σ ij Respectively represent the mean and covariance of the spatial feature vectors of the image patch extracted at position (i, j) for all normal asphalt pavement images in the training set. I represents the identity matrix, N represents the total number of normal asphalt pavement images in the training set, and k refers to the kth normal asphalt pavement image in the training set, satisfying k∈[1,N]. The last term εI in formula (2) is the added regularization term to ensure that the covariance matrix is ​​full rank and invertible, and the regularization constant is set to ε=0.02.

[0086] The Mahalanobis distance is used to evaluate the difference between the spatial feature vector of each image patch of the asphalt pavement image in the calibration set and the corresponding normal distribution learned from the training set. Unlike the Euclidean distance, the Mahalanobis distance takes into account the effect of the covariance between the feature vectors on the calculation results. If the asphalt pavement image contains potholes, cracks, rutting, or loose surface deslurry, the Mahalanobis distance calculated for the image will be larger than the Mahalanobis distance of a normal asphalt pavement image. The calculation method of the Mahalanobis distance is as follows:

[0087]

[0088] Where X ij It is still the spatial feature vector of the image patch extracted from the asphalt pavement image. The larger the Mahalanobis distance, the higher the probability of defects on the asphalt pavement image. The Mahalanobis distance matrix is ​​then visualized and converted into a defect value map, and finally further converted into a heat map to locate anomalies in the asphalt pavement image and the judgment basis of the interpretation model.

[0089] The purpose of calibration using the asphalt pavement image calibration set is to determine the defect value threshold that distinguishes normal asphalt pavement images from defective asphalt pavement images. The receiver operating characteristic (ROC) curve is used here, and the area under the curve (AUC) value of the heat map is calculated. The larger the AUC value, the better the prediction result of the model. Then, according to the ROC curve, the maximum Youden index is used to select the optimal defect value threshold. The defect value threshold determined by this method has the characteristics of maximizing the true positive rate and minimizing the false positive rate. After the defect value threshold is determined, the threshold can be used to test and classify the test set of asphalt pavement images.

[0090] One disadvantage of the deep convolutional neural network model is its 'black box' feature, that is, it is difficult to explain the underlying logic on which its decisions are based. Although the deep convolutional neural network model can achieve accuracy and high precision in prediction results, its prediction results are also the most difficult to explain. The present invention attempts to use a heat map to explain the results of the deep convolutional neural network model. The heat map is overlaid on the original asphalt pavement image to clearly reveal how the deep convolutional neural network model makes decisions to distinguish between defective asphalt pavement images and normal asphalt pavement images. The heat map uses different colors to highlight the focus of the deep convolutional neural network model on the image when making judgments. Exploring the underlying logic hidden in the deep convolutional neural network model helps to discover the focus of the model and make corrections to improve the performance of the model in future applications.

[0091] In an embodiment of the present application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0092] In an embodiment of the present application, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0093] Embodiment 2

[0094] Asphalt pavement image data were collected in Chongchuan District, Nantong, including the areas along Qingnian Road, Yanhe Road and Tongjing Avenue. Figure 3 These locations were selected because Nantong Metro Line 2 runs along Qingnian Road. Therefore, monitoring the anomalies of Qingnian Road and nearby roads above Metro Line 2 is particularly important for evaluating the dynamic impact of subway operation on adjacent road facilities. The collected asphalt road pavement image dataset includes 1,000 defective asphalt pavement images marked as explicit and 2,000 normal asphalt pavement images without defects marked as implicit, as shown in Figure 1. Figure 3 shown.

[0095] Figure 4 Some samples of asphalt road pavement images are shown. The five images in the first row show normal asphalt road images without defects, and the five images in the second row show asphalt road images with various types of defects. Since these photos were taken at different times, different lighting conditions and some tree shadows are shown in the normal asphalt road images. These factors will inevitably affect the accuracy of the convolutional neural network model's judgment, which we will discuss in subsequent sections.

[0096] The acquired asphalt road pavement image data is divided into three sets, namely, training set, calibration set and test set. Specifically, 300 normal asphalt pavement images are used for training. In the calibration stage, 100 normal asphalt pavement images and 100 defective asphalt pavement images are selected. In the testing stage, 300 defect-free asphalt pavement images and 300 defective asphalt pavement images are used. Finally, 300 normal asphalt pavement images are used for training, 200 average mixed images (100 defective asphalt pavement images and 100 defect-free normal asphalt pavement images) are used for calibration, and 600 average mixed images (300 defective asphalt pavement images and 300 defect-free normal asphalt pavement images) are used for testing. Before training, calibration and testing, all randomly selected images are preprocessed by resizing the images and performing center cropping to obtain standard images of 224*224 pixels.

[0097] As mentioned above, the patch feature vectors of the asphalt road pavement image are extracted through the three layers of the ResNet-18 model with different spatial sizes, so the feature vectors extracted by all modules are adjusted to 56*56 pixels by upsampling operation. These extracted feature vectors are then fused and reshaped into the standard spatial feature vectors of each asphalt road image. In order to avoid information redundancy extracted by the model and reduce the amount of calculation, about 1 / 4 of the total feature vectors are retained by random selection. Then the mean value μ of the spatial normal distribution function of the normal asphalt road image is calculated based on the training set. ij and covariance Σ ij The calibration phase is mainly used to determine the defect value threshold. The data set contains 100 normal asphalt pavement images and 100 asphalt pavement images with defects. The Mahalanobis distance is used to measure the difference between the normal distribution function obtained from the calibration set asphalt pavement images and the training set. The Mahalanobis distance is then used to generate a defect value map for each asphalt pavement image and calculate the average defect value of the image.

[0098] The receiver operating characteristic (ROC) curve is used to determine the defect value threshold. The area under the curve (AUC) is used to evaluate the quality of the model, and the larger the AUC value, the better the model. The defect value threshold is selected according to the maximum Youden index. The defect value threshold corresponds to the maximum distance between the ROC curve and the random line. The test asphalt pavement dataset is 300 normal asphalt pavement images and 300 defective asphalt pavement images randomly selected from the overall dataset. Following the same process as the calibration dataset, the Mahalanobis distance between the feature vector of the test asphalt pavement image and the obtained normal distribution is calculated, and a defect value map of each test asphalt pavement image is generated. The defect value map is then converted into a heat map to visualize the defect values ​​in the image. The heat map is overlaid on the original asphalt pavement image to explain the decision logic of the ResNet-18 model and determine the subsequent improvement direction.

[0099] Figure 5 The distribution of the average defect value of the asphalt pavement image obtained using the calibration set asphalt pavement image data. As mentioned above, the defect value is a reflection of the Mahalanobis distance. Figure 5 It shows that the ResNet-18 model can well distinguish between images of asphalt pavement with defects and images of normal asphalt pavement. Specifically, the average defect value of normal asphalt pavement images is generally lower than 0.08, while the defect value of images of asphalt pavement with defects is usually higher, mostly exceeding 0.08. Figure 5 It can also be seen that there is an overlap between the two classes, that is, the minority samples of both classes of images fall within the range of average defect values ​​of 0.06 to 0.10.

[0100] ROC curve Figure 6 As shown in the figure. The AUC value is 0.977, indicating that the performance of the convolutional neural network model is relatively strong. As mentioned above, the maximum Youden index is used to determine the optimal threshold value of the defect value of 0.0819 to distinguish between defective asphalt pavement images and normal asphalt pavement images. The red dotted line in the figure represents a random line, that is, the probability of occurrence on this line is 50%.

[0101] Another important research content of the present invention is to understand how ResNet-18 makes judgments on defective asphalt pavement images and normal asphalt pavement images. This aspect is often overlooked and there are few research results. However, understanding the operation mode and underlying judgment logic of the convolutional neural network model is crucial to improving the prediction performance of the model. The present invention will display the classification results of some asphalt pavements and the heat map overlaid on them. In order to ensure the consistency of the heat map results, the heat map display range is uniformly constrained for all images in the test set, that is, the minimum value is set to 0 and the maximum value is set to the maximum defect value of all calibration data.

[0102] First, we try to use heatmaps to explain the asphalt pavement samples with true positive test results. These true positive samples indicate that the asphalt pavement images with defects are correctly judged and classified. Figure 8 and Fig. 9 shown.

[0103] Fig. 9 The heat map shows the samples that the convolutional neural network model correctly classified as true positive results, and Figure 8 Corresponding. Fig. 9 It can be seen that through semi-supervised learning, the ResNet-18 neural network model can effectively capture the exact location of defects in the asphalt pavement image, as shown in red. This shows that ResNet-18 can not only classify asphalt pavement images into normal and defective classes, but also accurately locate defects in the image. The average defect values ​​of these three images are 0.0826, 0.0825, and 0.0894, respectively, all exceeding the defect value threshold of 0.0819.

[0104] Next, the present invention analyzes the samples classified as true negative results by ResNet-18 based on semi-supervised learning, such as Fig.10 The true negative result shows that the neural network model correctly identifies the normal asphalt road image. Fig.11 A sample image with a heat map superimposed is shown.

[0105] Fig.11 The heat map shows that it is basically cold, which indicates that the neural network model believes that there are no defects in the asphalt road image. However, we found Fig.11 The first image in the figure shows a warm color in the upper right corner due to the change in lighting. This shows that even if the asphalt road image is classified correctly, the lighting change in the local area still affects the decision of the neural network model. The average defect values ​​of these sample images are 0.0766, 0.0787, and 0.064, respectively, which are all lower than the threshold of 0.0819.

[0106] Samples classified as false positives are as follows Fig.12 As shown in the figure. False positive results refer to the fact that the neural network model ResNet-18 mistakenly classifies an asphalt pavement image without defects as an asphalt pavement image with defects. Studying these misclassified images will help understand the judgment logic of the neural network model and make targeted improvements, which may improve its subsequent performance.

[0107] from Fig.13It can be seen that the first asphalt road image was misclassified mainly due to lighting changes and tree shade. This shows that the neural network model is very sensitive to lighting and tree shade. The average defect values ​​of these images are 0.1123, 0.082, and 0.0889, respectively, all exceeding the defect value threshold of 0.0819. In particular, the first image shows more warm colors and has the highest average defect score, which shows that when collecting normal asphalt road images, pay attention to avoiding shadows and lighting changes to reduce the occurrence of such errors.

[0108] at last, Fig.14 The false negative classification results of the neural network model based on semi-supervised learning are shown. Fig.14 Displays the original asphalt road image, while Fig.15 The corresponding overlay heatmap is shown. False negative results occur when defective asphalt pavement images are incorrectly classified as normal. Fig.14 All three images show cracks, but they are too narrow to be seen. Fig.15 In the middle image, the trained ResNet-18 neural network model still detects the location of the crack despite the classification error, as shown in the warm color. This classification error is due to the selected classification metric (average defect value). Even if the defect value in a local area is high, after averaging the defect value of the entire image, the overall defect value may be lower than the threshold and cause misclassification. The average defect values ​​of these images are 0.0811, 0.0728, and 0.0775, all lower than the threshold of 0.0819.

[0109] Through the above results, it can be inferred that interference factors such as lighting changes, shadows, leaves and road markings (such as zebra crossings) can cause false positive classification errors. Therefore, these factors should be avoided as much as possible when collecting normal asphalt pavement image data. Regarding the false negative misclassification, after analysis, asphalt pavement images with relatively larger and more obvious defects should be obtained to improve the accuracy of the neural network model.

[0110] In the present invention, the classification accuracy of the ResNet-18 neural network model is improved by expanding the asphalt pavement image database. Then, normal asphalt pavement images are collected to expand the training data set (implicit set), and factors such as shadows, leaves, road markings, and lighting changes are avoided during the collection process. In addition, the present invention also tries to collect asphalt pavement images with more obvious cracks and potholes to enrich the data set of asphalt pavement defects (explicit set). Then, the performance of the ResNet-18 neural network model is retested using this expanded asphalt pavement image data set. The confusion matrix calculated after the data set is expanded is as follows: Fig.16 shown.

[0111] Depend on Fig.16 It can be seen that the classification accuracy of the ResNet-18 neural network model increased from 0.868 to 0.887 using the expanded asphalt pavement image dataset. This improvement shows that using heat maps to explain the decision-making process of the neural network model can guide the expansion of the asphalt pavement image dataset in the right direction.

[0112] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0113] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention invented herein. This application is intended to cover any variations, uses or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art that are not invented by the present invention. The specification and examples are intended to be exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0114] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A method for detecting and identifying asphalt pavement defects based on semi-supervised learning, characterized in that: include: Inputting the asphalt pavement image to be identified into the trained convolutional neural network model, the convolutional neural network model extracts features of the asphalt pavement image to obtain an image patch of the extracted asphalt pavement image to be identified; Flattening the extracted image patch of the asphalt pavement image to be identified to obtain a spatial feature vector of the image patch of the asphalt pavement image to be identified; The spatial feature vectors of the image patches of the asphalt pavement image to be identified are fused and randomly sampled; Based on the image patch space feature vectors of the training set obtained in advance, a normal distribution of the image patch space feature vectors of the training set is calculated; Using Mahalanobis distance as a metric, the gap between the spatial feature vector of each image patch of the asphalt pavement image to be identified and the corresponding normal distribution in the training set is determined; The Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve. Based on the ROC curve, the maximum Youden index is used to determine the defect value of the asphalt pavement image to be identified; If the defect value of the asphalt pavement image to be identified exceeds a predetermined optimal defect value threshold, the asphalt pavement image to be identified is determined to have defects; if the defect value of the asphalt pavement image to be identified is lower than the optimal defect value threshold, the asphalt pavement image is determined to be intact.

2. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 1 is characterized in that: The convolutional neural network model is trained, including: Based on the convolutional neural network ResNet-18, a convolutional neural network model was constructed; A training set is constructed based on a previously acquired intact asphalt pavement image, and the training set is input into a convolutional neural network model, which extracts features of the asphalt pavement image to obtain extracted image patches; Flatten the extracted image patches to obtain the image patch space feature vectors of the training set; The spatial feature vectors of all image patches in the training set are fused and randomly sampled; A calibration set is constructed based on pre-acquired intact asphalt road images; Based on the image patch space feature vector of the training set, a normal distribution of the image patch space feature vector is obtained by calculation; Using Mahalanobis distance as a metric, the distance between each image patch spatial feature vector in the calibration set and the corresponding normal distribution in the training set is determined; The Mahalanobis distance is converted into a heat map, and the area under the curve of the heat map is calculated using the receiver operating characteristic curve to obtain the ROC curve. Based on the ROC curve, the maximum Youden index is used to determine the optimal defect value threshold.

3. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 1 is characterized in that: The spatial feature vectors of intact asphalt pavement images in the training set show a multivariate normal distribution.

4. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 1 is characterized in that: The convolutional neural network model includes a first convolutional layer, a first pooling layer, a first module, a second module, a third module, a fourth module, a second pooling layer and a fully connected layer which are connected in sequence.

5. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 4 is characterized in that: The first module includes the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer connected in sequence. The second module includes the sixth convolutional layer, the seventh convolutional layer, the eighth convolutional layer and the ninth convolutional layer connected in sequence. The third module includes the tenth convolutional layer and the eleventh convolutional layer connected in sequence. The fourth module includes a twelfth convolutional layer and a thirteenth convolutional layer connected in sequence.

6. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 5 is characterized in that: The first convolutional layer has 64 filters, the filter size is 7×7, and the stride is 2; the second, third, fourth and fifth convolutional layers have 64 filters, the filter size is 3×3, the sixth, seventh, eighth and ninth convolutional layers have 128 filters, the filter size is 3×3, the tenth and eleventh convolutional layers have 256 filters, the filter size is 3×3, the twelfth and thirteenth convolutional layers have 512 filters, the filter size is 3×3, and the stride of the sixth, tenth and twelfth convolutional layers is 2.

7. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 2 is characterized in that: Mahalanobis distance M(X ij ) is calculated as: In the formula, X ij is the spatial feature vector of the image patch extracted from the asphalt road image; μ ij and Σ ij They represent the mean and covariance of the spatial feature vector of the image patch extracted at the position (i, j) of the intact asphalt road image in all the training sets.

8. The method for detecting and identifying asphalt pavement defects based on semi-supervised learning according to claim 1 is characterized in that: The size of the first asphalt image spatial feature patch output by the first module is adjusted to 56×56, the size of the second asphalt image spatial feature patch output by the second module is adjusted to 28×28, and the size of the third asphalt image spatial feature patch output by the third module is adjusted to 14×14.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images