A pothole road detection system and detection method based on convolutional neural network
By adopting a convolutional neural network-based detection method in the pit detection system, combining image preprocessing and local binary mode feature extraction, the problems of inefficiency and insufficient accuracy in the prior art are solved, and efficient and accurate pothole detection effects are achieved.
Patent Information
- Application Number
- CN202410979809.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2025-04-18
- Estimated Expiration
- 2044-07-22
AI Technical Summary
When processing large-scale and complex road environment data, the detection methods are inefficient and insufficiently accurate, making it difficult to meet the needs of efficient and accurate pothole detection.
A detection system based on convolutional neural network is adopted, including image preprocessing module, PD-CNN module, random forest module and control module. Through standardized preprocessing and local binary mode feature extraction, combining the advantages of deep learning and traditional machine learning, model performance is optimized and overfitting is prevented.
It significantly improves the consistency and robustness of image features, enhances the model's adaptability and classification performance in complex road environments, improves the model's adaptability to images of different formats and quality, and achieves efficient and accurate pothole detection.
Smart Images
Figure CN118710987B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and machine learning, and in particular relates to a pothole road detection system and detection method based on convolutional neural network. Background Art
[0002] In recent years, with the rapid growth of the global economy, the pressure on the transportation system has been increasing day by day. In particular, road maintenance and safety issues have become key issues in public safety management. The pothole problem on the road surface is particularly worthy of attention. It not only increases the cost of vehicle operation, but also directly affects driving safety. Studies have shown that potholes are one of the main factors leading to traffic accidents and vehicle damage. Therefore, developing an efficient and accurate pothole detection technology is of vital importance to improve the service life of roads and ensure driving safety. Traditional pothole detection methods mainly rely on manual visual inspection and basic machine learning techniques such as support vector machines, decision trees or random forests. These methods have many limitations in practical applications, especially when dealing with large-scale and complex road environment data, which are often limited by low efficiency and insufficient accuracy. For example, manual inspection is not only time-consuming, but also easily affected by the subjective judgment of inspectors, resulting in inconsistent detection results. Although machine learning methods have improved the degree of automation of detection, they still have shortcomings in feature extraction and model generalization.
[0003] Therefore, a new technical solution is urgently needed in the prior art to solve this problem. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a pothole road detection system and detection method based on a convolutional neural network, which is used to solve the problems of low efficiency and insufficient accuracy of the detection method in the prior art when processing large-scale and complex road environment data.
[0005] The technical solution adopted by the present invention is to provide a pothole road detection system based on convolutional neural network, and the detection system includes an image preprocessing module, a PD-CNN module, a random forest module and a control module;
[0006] The image preprocessing module is used to perform standardization processing and feature extraction on the road image;
[0007] The PD-CNN module is used to classify and identify the preprocessed images;
[0008] The random forest module is used to verify and further process the classification results;
[0009] The control module is used to coordinate the operations of the image preprocessing module, the PD-CNN module and the random forest module.
[0010] The image preprocessing module adopts a local binary pattern feature extraction method to enhance the system model's processing capability for road images of different formats and qualities.
[0011] The PD-CNN module is built based on the pre-trained VGG16 model and customized by adjusting parameters to optimize model performance, including freezing the weights of certain layers to maintain the feature extraction capabilities of the pre-trained network, while unfreezing the last few layers for fine-tuning to ensure that the model can adapt to specific types of road image features.
[0012] The control module includes a parameter optimization unit for adjusting the parameters of the PD-CNN model during training to achieve optimal classification performance.
[0013] Another technical solution adopted by the present invention is to provide a pothole road detection method based on a convolutional neural network, comprising the following steps:
[0014] (1) Data collection and preprocessing
[0015] Road images are collected from different sources and standardized through the image preprocessing module, including denoising and enhancement. The image preprocessing module performs data enhancement through the Albumentations library, which allows for more precise adjustment of the parameters of each transformation, making the enhanced dataset more suitable for specific pothole road recognition training, including but not limited to various complex transformations such as noise, blur, illumination changes, occlusion simulation, etc. The subsequent image preprocessing module also performs standardization and local binary pattern (LBP) feature extraction on the road images, which greatly enhances the key features in the image.
[0016] (2) Feature extraction
[0017] The local binary pattern feature extraction method is used to extract the features of the road image;
[0018] (3) Model training
[0019] The preprocessed images are fed into the PD-CNN model for training, the weights of certain layers are frozen to maintain the feature extraction capability of the pre-trained network, and the last few layers are unfrozen for fine-tuning to ensure that the model can adapt to the characteristics of specific types of road images; early stopping technology and model checkpoints are introduced to optimize the training process, prevent overfitting and ensure the best performance on the validation set; by using the two callback functions EarlyStopping and ModelCheckpoint, the model is helped to maintain the generalization ability of unseen data, ensuring that the best model obtained each time is saved; in the later stages of training, the high layers (the last two to three convolutional blocks) of the pre-trained model are fine-tuned on the dataset to fine-tune these layers so that the model can better adapt to the specific pothole shape and size, thereby optimizing the model performance;
[0020] (4) Classification and verification
[0021] Use the trained PD-CNN model to classify images, use the random forest module to verify the classification results, and find the best model configuration through parameter tuning of the verified random forest and LBP combination module;
[0022] (5) Result output and analysis
[0023] Based on the classification and verification results, a test report is output.
[0024] The preprocessing method in step (1) is to adjust the input image to a standard size of 224x224 to adapt to the input requirements of the PD-CNN model, then convert the image from the RGB color space to the HSV color space, apply histogram equalization in the HSV space to enhance the contrast of the image, perform gamma correction in the lightness channel with a gamma value of 2.2 to adjust the brightness of the image, and use a 5x5 Gaussian filter for smoothing to remove image noise.
[0025] In the step (2), the PD-CNN model extracts image features through multi-layer convolution and pooling, wherein the convolution layer is used to extract low-level to high-level features, and the pooling layer is used to reduce the dimension of the feature map and reduce the computational complexity; a local binary pattern feature extraction method is used to enhance the model's ability to recognize different textures and impurities in road images; the OpenCV library is used for image processing, including calculating LBP features, and the Matplotlib library is used to visualize the features, further improving the performance of the model in processing complex road environments.
[0026] In the step (3), the preprocessed image is input into the pre-trained PD-CNN model for training. The parameters of the PD-CNN model, such as the learning rate and the batch size, are adjusted to optimize the model performance so that it can adapt to different road image data. During the training process, data enhancement technology is used to improve the generalization ability of the model to ensure that the model can maintain high-precision classification and recognition performance in various actual road environments.
[0027] The method also includes a comprehensive test of the model through different evaluation indicators, including precision, recall, F1 score and ROC curve.
[0028] A computer-readable storage medium stores a computer program, wherein the program implements any of the above methods when executed by a computer.
[0029] Through the above design scheme, the present invention can bring the following beneficial effects:
[0030] The present invention combines the advantages of deep learning and traditional machine learning, and can provide stronger feature extraction capabilities and higher model generalization capabilities through standardized preprocessing and LBP feature extraction methods. Through standardized preprocessing, the image is uniformly adjusted to a standard size of 224x224, color space conversion and histogram equalization are performed to improve contrast and consistency, and a Gaussian filter is used to remove noise and enhance the robustness of the model. LBP feature extraction generates binary numbers by comparing the intensity of pixels with the surrounding neighborhood, which is particularly suitable for processing texture information and enhancing the model's ability to recognize different textures and impurities in road images. Compared with other methods, standardized preprocessing reduces interference factors in feature extraction, and LBP has more advantages in texture feature processing. Combining these two methods, the consistency and robustness of image features are significantly improved, the adaptability and classification performance of the model in complex road environments are enhanced, and the model is more efficient and accurate than other technologies, and the adaptability of the model to images of different formats and qualities is improved. In the model training process, by introducing early stopping technology (EarlyStopping) and model checkpoint (ModelCheckpoint), the overfitting of the model is effectively prevented, ensuring the best performance on the validation set. These techniques help the model maintain its ability to generalize to unseen data and ensure that the best model for each training is saved. In addition, by freezing the weights of certain layers of the pre-trained model (VGG16) to maintain its feature extraction capabilities, while unfreezing the last few layers for fine-tuning, the model can adapt to specific types of road image characteristics. In the later stages of training, by fine-tuning the high layers of the pre-trained model (the last two to three convolutional blocks) on the dataset, these layers are fine-tuned to make the model better adapt to specific pothole shapes and sizes, further optimizing model performance. Experimental results show that the PD-CNN model has an accuracy of 98.339% on the training set and 99.003% on the validation set, showing extremely high classification performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A flow chart of a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0032] Figure 2 A schematic diagram of the network structure of a PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network according to the present invention;
[0033] Figure 3 It is a second convolutional layer feature map of feature extraction of a PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network of the present invention;
[0034] Figure 4It is a fifth convolutional layer feature map of feature extraction of a PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network of the present invention;
[0035] Figure 5 A texture feature map of an image of a road in a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0036] Figure 6 The present invention is a schematic diagram of a random forest model of a pothole road detection system and detection method based on a convolutional neural network;
[0037] Figure 7 Grayscale color histogram feature 1 of a normal road of a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0038] Figure 8 Grayscale color histogram feature 2 of a normal road of a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0039] Fig. 9 Grayscale color histogram feature 1 of a pothole road of a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0040] Fig.10 Grayscale color histogram feature 2 of a pothole road of a pothole road detection system and detection method based on a convolutional neural network of the present invention;
[0041] Fig.11 A schematic diagram of the change in accuracy function value of the PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network of the present invention;
[0042] Fig.12 A schematic diagram of the change in loss function value of a PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network of the present invention;
[0043] Fig.13 A confusion matrix of a random forest model of a pothole road detection system and detection method based on a convolutional neural network according to the present invention;
[0044] Fig.14 The present invention provides a convolutional neural network-based pothole road detection system and a random forest model for the detection method. DETAILED DESCRIPTION
[0045] The present invention is further described below with reference to the accompanying drawings and specific embodiments:
[0046] Figure 2 The present invention is a network structure diagram of a PotholeDetectorCNN model of a pothole detection system and detection method based on a convolutional neural network. The PotholeDetectorCNN (PD-CNN for short) module is pre-trained and constructed based on the VGG16 model, and the model performance is optimized by adjusting parameters. The PD-CNN model is customized, including freezing the weights of certain layers to maintain the feature extraction capability of the pre-trained network, while unfreezing the last few layers for fine-tuning to ensure that the model can adapt to specific types of road image features. Schematic diagram of the network structure. The figure shows a convolutional neural network structure for image classification. The input layer is a color image of size 224x224x3. The next few layers include a two-dimensional convolutional layer and a maximum pooling layer, which gradually extracts the features of the image and reduces the size of the feature map through a series of convolution operations (using a 3x3 convolution kernel) and pooling operations (using a 2x2 pooling kernel). After the first layer of convolution, a feature map of 224x224x64 is obtained, which is then reduced to 112x112x64 by pooling. This process is repeated in subsequent layers, with the feature map size gradually decreasing and the number of features gradually increasing, from 112x112x128 to 28x28x512. Each layer of convolution operation extracts more complex features, and the pooling layer reduces the amount of computation and overfitting. The feature map after convolution and pooling operations is flattened into a one-dimensional vector and input to the fully connected layer. The fully connected layer performs high-level feature combination and finally outputs the classification result through a fully connected layer of size 1000. The entire model extracts image features through multiple layers of convolution and pooling, and finally classifies through the fully connected layer. This structure is very effective in the classification task of complex images, and achieves accurate classification by extracting and combining features layer by layer.
[0047] Different environmental conditions and different times may cause certain differences between images, so it is necessary to perform Normalization and Standardization on the training data before building the model. The present invention unifies the format, color, noise, and other characteristic factors of each picture involved in the training. After data processing, traditional machine learning models and deep learning models are established respectively, and their model accuracy and various indicators are explored and compared. The main parameters for preprocessing in the experiment adjust the input image to a standard size of 224x224 to adapt to the input requirements of the PD-CNN model. The image is then converted from the RGB color space to the HSV color space. Histogram equalization is applied in the HSV space to enhance the contrast of the image. Then, gamma correction is performed in the brightness channel with a gamma value of 2.2 to adjust the brightness of the image. Finally, a 5x5 Gaussian filter is used for smoothing to remove image noise. Through the above preprocessing steps, the format, color and other characteristic factors of the image can be unified to adapt to subsequent feature extraction and model training.
[0048] Use the above PD-CNN model for feature extraction: Use Keras with the pre-trained PD-CNN model to extract and visualize the feature maps of specific convolutional layers, and use the matplotlib library to display these feature maps. Select the second and fifth convolutional layers of a normal road image to extract the extracted feature maps as shown below: Figure 3 , Figure 4 As shown. The PD-CNN model extracts image features through multiple layers of convolution and pooling, and finally classifies through a fully connected layer. The specific steps are as follows: Use Keras to load the pre-trained PD-CNN model. Load the PD-CNN model pre-trained based on the ImageNet dataset by calling the VGG16 function in keras.applications and setting the weights parameter to 'imagenet'. Input the preprocessed image into the PD-CNN model to extract the feature map of a specific convolution layer. By defining a new model with the input layer of PD-CNN as input and the second and fifth convolution layers as output, use the model to predict the preprocessed image and obtain the feature map. Use the matplotlib library to visualize the extracted feature map. The feature map shows the response of the image at a specific convolution layer, which can help understand how the model extracts and processes image features.
[0049] To further improve the robustness of the features, we use the local binary pattern (LBP) feature extraction method. LBP is a simple and effective texture descriptor that generates a binary number by comparing the intensity of a pixel with its surrounding neighborhood. This method is particularly suitable for processing texture information and can enhance the model's ability to recognize different textures and impurities in road images. Use OpenCV and Matplotlib libraries to extract the texture features of the road image, such as Figure 5 As shown. The specific steps are as follows: Use OpenCV's LBP function to calculate the LBP features of the image. The LBP method compares each pixel with its neighboring pixels. If the surrounding pixel value is greater than the central pixel value, the position is recorded as 1, otherwise it is recorded as 0. Finally, these binary arrays are synthesized into a decimal number as the LBP value of the pixel. The calculated LBP features are normalized so that the feature value is between 0 and 1. Divide the LBP features by the maximum value. Use the matplotlib library to visualize the extracted LBP features. The LBP feature map is displayed by the imshow function to help understand the texture information of the image. In order to improve the generalization ability and robustness of the model, the present invention uses the Albumentations library for data enhancement. Specifically, the Albumentations library is used to perform enhancement operations such as horizontal flipping, random brightness contrast adjustment, translation, scaling, rotation and blurring on the training images. These enhanced data sets can effectively improve the generalization ability of the model and reduce overfitting.
[0050] During the model training process, early stopping technology (EarlyStopping) and model checkpoint (ModelCheckpoint) are introduced to prevent overfitting of the model and ensure the best performance on the validation set. These technologies help the model maintain its generalization ability for unseen data and ensure that the best model for each training is saved. In addition, by freezing the weights of certain layers of the pre-trained model (PD-CNN) to maintain its feature extraction capabilities, while unfreezing the last few layers for fine-tuning, the model can adapt to specific types of road image features. In the later stages of training, by fine-tuning the high layers of the pre-trained model (the last two to three convolutional blocks) on the dataset, these layers are fine-tuned to make the model better adapt to specific pothole shapes and sizes, further optimizing model performance.
[0051] The comparison group of this experiment uses the random forest algorithm in the supervised learning algorithm to classify the road surface type. Random forest is a (parallel) ensemble algorithm composed of decision trees, belonging to the bagging type. By combining multiple weak classifiers, the final result is obtained by voting or taking the average, so that the result of the overall model has high accuracy and generalization performance, and also has good stability. It is widely used in various business scenarios. The architecture diagram of its model is as follows Figure 6 shown.
[0052] Randomly select two normal road pictures and two bumpy road pictures as comparisons, and use OpenCV's Python interface to read and process the two sets of road images. Use enumerate(color_channels) to iteratively process each color channel of the image (blue B, green G, red R), because OpenCV uses the BGR format by default. The histogram calculation specifies the current processed color channel, the size of the histogram, and the range of pixel values. By using matplotlib to draw the histogram of each color channel, blue is replaced by a grayscale thin solid line, green is replaced by a grayscale thin dashed line, and red is replaced by a grayscale thick solid line to visualize the image data. The color histogram features 1 and 2 of the normal road are respectively Figure 7 and Figure 8 As shown in Figure 1, the color histogram features 1 and 2 of the pothole road are Fig. 9 and Fig.10 As shown in the figure. Different types of road images are collected to ensure that the dataset is representative and diverse. The dataset includes normal road images and pothole road images. The collected images are preprocessed, and the steps include image resizing, color space conversion, and standardization. The resize and cvtColor functions of OpenCV are used for resizing and color conversion.
[0053] 301 pictures of different roads were selected for training, and the ratio of normal roads to bumpy roads was about 10:1. The pictures participating in the training were normalized and standardized. Under the same training conditions, the Albumentations library was used to perform horizontal flipping, random brightness and contrast adjustment, translation, scaling, rotation, blur and other enhancement operations on the training images. A large number of picture data sets were regenerated based on the original 301 pictures of different roads, and then the PD-CNN model and random forest model were used for training respectively. The accuracy function of the PD-CNN model for testing is shown as follows: Fig.11 As shown, the loss function value of the PD-CNN model changes as follows Fig.12 shown.
[0054] The random forest model in the control group of this experiment usually does not use Epoch (training cycle) to measure the training process, because random forest is a tree-based integration method and does not involve the optimization of accuracy and loss function in the iterative process. Random forest improves the classification accuracy and stability of the model by constructing multiple decision trees and voting or averaging their results. During the training process, the random forest model uses training data to construct multiple decision trees. Each tree is trained by randomly selecting features to enhance the generalization ability of the model. When evaluating the performance of the random forest model, the evaluation index function and confusion matrix are mainly used. The evaluation index functions include accuracy, precision, recall and F1 score. These indicators can comprehensively measure the classification performance of the model. The confusion matrix is as follows: Fig.13 As shown in Figure 1, the performance of the random forest model in the classification task is shown, including the number of correctly classified and misclassified samples. In addition, the ROC curve is also an important tool for evaluating the performance of the classification model. The ROC curve of the random forest model is shown in Figure 1. Fig.14 As shown in the figure, the classification performance of the model at different thresholds is shown, and its AUC value reflects the model's ability to distinguish between positive and negative samples. Through these evaluation methods, we can fully understand the performance of the random forest model in the road recognition detection task, and further optimize the model parameters and training strategies based on the evaluation results to improve the classification accuracy and robustness of the model.
[0055] The implementation methods of the present invention are not limited to the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A pothole road detection method based on convolutional neural network, using a pothole road detection system based on convolutional neural network, the detection system includes an image preprocessing module, a PD-CNN module, a random forest module and a control module; the image preprocessing module is used to perform standardization processing and feature extraction on road images, the image preprocessing module adopts a local binary pattern feature extraction method to enhance the system model's processing capability for road images of different formats and qualities; the PD-CNN module is used to classify and identify the preprocessed images, the PD-CNN module is constructed based on VGG16 model pre-training, and optimizes model performance by adjusting parameters, the PD-CNN module is customized, including freezing the weights of certain layers to maintain the feature extraction capability of the pre-trained network, while unfreezing the last few layers for fine-tuning to ensure that the model adapts to specific types of road image features; the random forest module is used to verify and further process the classification results; the control module is used to coordinate the operations of the image preprocessing module, the PD-CNN module and the random forest module, the control module includes a parameter optimization unit, which is used to adjust the parameters of the PD-CNN model during training to achieve optimal classification performance, characterized in that: The following steps are involved: (1) Data collection and preprocessing Road images are collected from different sources and standardized through the image preprocessing module, including denoising and enhancement. The image preprocessing module performs data enhancement through the Albumentations library, which allows for more precise adjustment of the parameters of each transformation, making the enhanced dataset more suitable for specific pothole road recognition training, including but not limited to various complex transformations such as noise, blur, illumination changes, and occlusion simulation. The subsequent image preprocessing module also performs standardization and local binary pattern (LBP) feature extraction on the road images, which enhances the key features in the image. (2) Feature extraction The local binary pattern feature extraction method is used to extract the features of the road image; (3) Model training The preprocessed images are fed into the PD-CNN model for training, freezing the weights of certain layers to maintain the feature extraction capability of the pre-trained network, while unfreezing the last few layers for fine-tuning to ensure that the model adapts to the characteristics of a specific type of road image; Early stopping technology and model checkpoints are introduced to optimize the training process, prevent overfitting and ensure the best performance on the validation set; by using the two callback functions EarlyStopping and ModelCheckpoint, the model is helped to maintain its generalization ability for unseen data and ensure that the best model obtained each time is saved; in the later stages of training, the high layers of the pre-trained model, that is, the last two to three convolutional blocks, are fine-tuned on the dataset to make the model better adapt to the specific pothole shape and size, thereby optimizing model performance; (4) Classification and verification Use the trained PD-CNN model to classify images, use the random forest module to verify the classification results, and find the best model configuration through parameter tuning of the verified random forest and LBP combination module; (5) Result output and analysis Based on the classification and verification results, a test report is output.
2. The pothole detection method based on convolutional neural network according to claim 1, characterized in that: The preprocessing method in step (1) is to adjust the input image to a standard size of 224x224 to adapt to the input requirements of the PD-CNN model, then convert the image from the RGB color space to the HSV color space, apply histogram equalization in the HSV space to enhance the contrast of the image, perform gamma correction in the lightness channel with a gamma value of 2.2 to adjust the brightness of the image, and use a 5x5 Gaussian filter for smoothing to remove image noise.
3. The pothole detection method based on convolutional neural network according to claim 1, characterized in that: In the step (2), the PD-CNN model extracts image features through multi-layer convolution and pooling, wherein the convolution layer is used to extract low-level to high-level features, and the pooling layer is used to reduce the dimension of the feature map and reduce the computational complexity; a local binary pattern feature extraction method is used to enhance the model's ability to recognize different textures and impurities in road images; the OpenCV library is used for image processing, including calculating LBP features, and the Matplotlib library is used to visualize the features, further improving the performance of the model in processing complex road environments.
4. The pothole detection method based on convolutional neural network according to claim 1, characterized in that: In the step (3), the preprocessed image is input into the pre-trained PD-CNN model for training. The parameters of the PD-CNN model, such as the learning rate and the batch size, are adjusted to optimize the model performance so that it can adapt to different road image data. During the training process, data enhancement technology is used to improve the generalization ability of the model to ensure that the model can maintain high-precision classification and recognition performance in various actual road environments.
5. The pothole detection method based on convolutional neural network according to claim 1, characterized in that: The method also includes a comprehensive test of the model through different evaluation indicators, including precision, recall, F1 score and ROC curve. 6 . A computer-readable storage medium having a computer program stored thereon, wherein the program implements the method according to claim 1 when executed by a computer.