An Unsupervised Road Damage Detection Method for Smart City Construction

Abstract

The present invention relates to an unsupervised road surface damage detection method for smart city construction, including: performing data augmentation on the non-damaged road surface images in the training set; constructing an encoder and a decoder based on a deep convolutional neural network, and combining the two to form an autoencoder, training the autoencoder to learn the mapping relationship between the data-augmented non-damaged road surface images and the original non-damaged road surface images after reconstruction; inputting the damaged road surface images in the test set to the autoencoder for testing, outputting a reconstructed recovery result map, comparing it with the damaged road surface images, obtaining a reconstruction error map, and performing post-processing to obtain the road surface damage detection result corresponding to the damaged road surface image; inputting the actual road surface image into the autoencoder after training and testing to obtain the actual road surface damage detection result. Compared with the prior art, the present invention saves the manpower, material resources and time costs required for manual annotation, and can effectively improve the detection efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of smart city construction, and particularly to an unsupervised road damage detection method for smart city construction. Background Art

[0002] In the field of smart city construction, intelligent road damage detection is an important task to ensure road quality, safety, smoothness and comfort. Any road damage that is not detected and repaired in time will not only bring inconvenience to driving comfort, but also pose a great threat to vehicle conditions, traffic safety and human lives, resulting in irreparable personal injuries and property losses. At present, the discovery of road damage still relies on manual daily maintenance inspections. Structural engineers and certified inspectors regularly inspect the roads and report the locations of the damage. However, this form of manual detection is usually affected by factors such as vehicle speed, staff experience, and concentration, and there are problems such as low maintenance efficiency and high operating costs. Moreover, the road damage detection results obtained manually are usually not objective enough because such detection results only depend on the personal experience and subjective judgment of the inspectors. Therefore, in the situation where artificial intelligence has been developing vigorously in recent years, as a leading task in the field of smart city construction, researchers from companies such as Baidu, Alibaba Cloud, SenseTime, and Hisense Network Technology are all committed to developing intelligent road damage detection algorithms to promote the implementation of the artificial intelligence road inspection system. And road damage detection is not only an infrastructure maintenance issue, but also a function of the advanced driver assistance system (ADAS) embedded in many automotive companies' L3 / L4 autonomous vehicles. Emerging autonomous driving systems often have higher requirements for road maintenance standards.

[0003] Classic intelligent road damage detection algorithms have been studied for more than a decade. Most of these algorithms are based on traditional two-dimensional image processing methods, which can be generally summarized as including four stages: image preprocessing, image segmentation, damage area extraction, and post-processing of detection results. These algorithms first use methods such as median filtering, Gaussian filtering, bilateral filtering, and morphological filtering to filter out redundant information on the image and highlight the road damage area; then the preprocessed road image is divided into foreground (damaged road area) and background (undamaged road area); then, the damaged road area is extracted from the segmented foreground according to geometric and texture assumptions; finally, post-processing is performed on the extracted damaged road area to further improve the road damage detection results. Although these classic methods can achieve results in some simple scenarios, they are usually computationally intensive and sensitive to various environmental factors, especially not very robust to different lighting and weather conditions. In addition, road damage usually has an irregular shape, making the geometric assumptions made in these classic algorithms sometimes infeasible.

[0004] To solve the above problems, with the latest progress in machine / deep learning, deep convolutional neural networks (DCNNs) have become the mainstream pavement damage detection technology considered by researchers. Different from the classical algorithms that segment pavement images by explicitly setting parameters and manually setting thresholds, existing DCNN algorithms usually use a large number of manually annotated road data for backpropagation training. They can be regarded as data-driven algorithm types. They do not require manual parameter setting, do not assume the shape of pavement damage, and are more robust to various environmental factors. The above data-driven pavement damage detection algorithms can be mainly divided into three categories: (1) image classification networks; (2) object detection networks; (3) semantic segmentation networks. Among them, the first type of image classification network algorithm trains the network to classify undamaged pavement data and damaged pavement data through data; the second type of object detection network algorithm realizes instance-level pavement damage detection by training an object detection network, and its output result generally uses a bounding box to label the pavement damage area; the third type of semantic segmentation network algorithm segments road images (RGB images or disparity / depth maps) by training a semantic segmentation network to achieve pixel-level / semantic-level road damage detection.

[0005] Although the above DCNN algorithms have achieved good results in the three tasks of image classification, object detection, and semantic segmentation of pavement damage, these data-driven supervised algorithm-based networks require a large number of manually annotated datasets during training. Preparing such a complete and usable dataset is usually very labor-intensive. And because pavement damage is not everywhere in reality, it often takes a lot of manpower, material resources, and time costs to collect pavement damage images. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide an unsupervised pavement damage detection method for smart city construction. Based on the autoencoder (AutoEncoder) and data augmentation technology, using the idea of reconstruction and image difference, it realizes pixel-level segmentation of pavement damage areas and can efficiently and accurately detect pavement damage.

[0007] The purpose of the present invention can be achieved through the following technical solutions: An unsupervised pavement damage detection method for smart city construction includes the following steps:

[0008] S1. Obtain a dataset including a training set and a test set, and use a data augmentation method of stripe-shaped and rasterized filling to perform data augmentation on the undamaged pavement images in the training set;

[0009] S2. Construct an encoder based on a deep convolutional neural network to extract the information and features of the undamaged pavement images after data augmentation, and generate feature maps of various sizes.

[0010] S3. Construct a decoder based on a deep convolutional neural network to reconstruct and restore the original undamaged pavement images according to the feature maps.

[0011] S4. Combine the encoder and the decoder to form an encoder-decoder based on a deep convolutional neural network, that is, an AutoEncoder model. Use the undamaged pavement images after data augmentation and the reconstructed original undamaged pavement images to train the AutoEncoder to learn the mapping relationship between the two.

[0012] S5. Use the AutoEncoder trained in step S4 to input the damaged pavement images in the test set for testing, and output the corresponding reconstructed and restored result images.

[0013] S6. Compare and process the damaged pavement images and the reconstructed and restored result images to obtain the corresponding reconstruction error images.

[0014] S7. Post-process the reconstruction error images, filter the redundant information on the images, perform pavement damage segmentation, and output the pavement damage detection results corresponding to the damaged pavement images.

[0015] S8. Collect actual pavement images, input the actual pavement images into the AutoEncoder after training and testing, and output the pavement damage detection results corresponding to the actual pavement images.

[0016] Further, the data augmentation method using stripe-like and rasterized filling in step S1 specifically uses stripe-like fillers of multiple different scales and randomly selected rasterized fillers to achieve different data augmentation effects.

[0017] Further, step S2 specifically uses five first convolutional blocks based on a deep convolutional neural network to form the encoder, and the structures of the five first convolutional blocks all include convolutional operation - batch normalization operation - ReLU activation - pooling operation.

[0018] Further, the input image size of the encoder is 224 pixels in height * 224 pixels in width. After passing through five first convolutional blocks, feature maps of 112 pixels in height * 112 pixels in width, 56 pixels in height * 56 pixels in width, 28 pixels in height * 28 pixels in width, 14 pixels in height * 14 pixels in width, and 7 pixels in height * 7 pixels in width are generated in sequence.

[0019] Further, step S3 specifically uses five second convolutional blocks based on a deep convolutional neural network to form a decoder. The structures of the five second convolutional blocks all include an upsampling operation - a transposed convolution operation - a batch normalization operation - a ReLU activation, and sequentially restore the feature map with a size of 7 pixels in height * 7 pixels in width to a feature map with a size of 112 pixels in height * 112 pixels in width. Among the five second convolutional blocks, the output of the last second convolutional block is a new reconstructed original undamaged road surface image with a size of 224 pixels in height * 224 pixels in width.

[0020] Further, the model loss function of the autoencoder in step S4 is specifically:

[0021]

[0022] where y rec is the reconstructed original undamaged road surface image, y aug is the undamaged road surface image after data augmentation, and bs is the batch size selected each time for each round of training.

[0023] Further, step S5 specifically includes the following steps:

[0024] S51. Save the parameters of each layer of the trained autoencoder model;

[0025] S52. In the test phase, freeze the parameters of each layer of these saved autoencoder models, use the damaged road surface images in the test set as the input of the autoencoder, and output the corresponding reconstructed and restored result images.

[0026] Further, the parameters of each layer of the autoencoder model include the weights and bias values of each neuron.

[0027] Further, step S6 specifically subtracts the damaged road surface image and the restored result image pixel by pixel to obtain the corresponding reconstruction error image.

[0028] Further, step S7 specifically performs an image filtering algorithm and an image segmentation algorithm on the reconstruction error image in sequence to filter redundant information on the image and perform road damage segmentation. The image filtering algorithm includes, but is not limited to, Gaussian filtering, bilateral filtering, and morphological filtering. The image segmentation algorithm includes, but is not limited to, Otsu’s thresholding, triangle thresholding, and adaptive thresholding.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] I. The present invention proposes a novel unsupervised intelligent road damage detection algorithm for smart city construction. Aiming at the problem that existing intelligent road damage detection algorithms in smart city construction require a large amount of manually labeled training data, through data augmentation processing of the training set and combined with the constructed autoencoder, an unsupervised road damage detection method is realized. Based on the idea of reconstruction and image difference, pixel-level segmentation of road damage areas is achieved, effectively solving the problem that supervised intelligent road damage detection algorithms generally require a large amount of manually labeled data sets, saving the manpower, material resources and time costs required for manual labeling, avoiding the problem of inaccurate manual labeling caused by subjectivity, and being able to improve the detection efficiency and accuracy.

[0031] II. The present invention comprehensively uses data augmentation methods of stripe-shaped and rasterized filling, trains the neural network to learn the reconstruction and recovery mapping relationship and ability from the data-augmented image to the original image, greatly enhancing the image reconstruction ability of the neural network and being able to reliably guarantee the accuracy of subsequent detection.

[0032] III. The present invention designs a simple and easy-to-implement autoencoder framework based on a deep convolutional neural network. This network structure does not require additional training strategies and supervised labeling, and can further save manpower, material resources and time costs, and effectively improve the detection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic flowchart of the method of the present invention;

[0034] Figure 2 is a flowchart framework of the method of the present invention;

[0035] Figure 3 is a schematic diagram of the data augmentation method of stripe-shaped and rasterized filling adopted in the present invention;

[0036] Figure 4 is a schematic diagram of the autoencoder model structure in the present invention;

[0037] Figure 5 is a sample of an undamaged road surface image used during training in the embodiment;

[0038] Figure 6 is a sample of a damaged road surface image used during testing in the embodiment;

[0039] Figure 7 is the road surface image reconstructed and recovered by the autoencoder model during testing in the embodiment;

[0040] Figure 8 is Figure 6 and Figure 7 the corresponding reconstruction error image obtained after contrast processing. Detailed implementation mode

[0041] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] Embodiment

[0043] As Figure 1 shown, an unsupervised road surface damage detection method for smart city construction includes the following steps:

[0044] S1. Obtain a data set including a training set and a test set, and use a data augmentation method of stripe-shaped and rasterized filling to perform data augmentation on the non-damaged road surface images in the training set;

[0045] S2. Construct an encoder based on a deep convolutional neural network to extract the information and features of the input data-augmented non-damaged road surface images, and generate feature maps of various sizes;

[0046] S3. Construct a decoder based on a deep convolutional neural network to reconstruct and restore the original non-damaged road surface image according to the feature maps;

[0047] S4. Combine the encoder and the decoder to form an encoder-decoder based on a deep convolutional neural network, that is, an autoencoder model. Use the data-augmented non-damaged road surface images and the reconstructed original non-damaged road surface images to train the autoencoder to learn the mapping relationship between the two;

[0048] S5. Use the autoencoder trained in step S4 to input the damaged road surface images in the test set for testing, and output the corresponding reconstructed result images;

[0049] S6. Compare and process the damaged road surface images and the reconstructed result images to obtain the corresponding reconstruction error images;

[0050] S7. Post-process the reconstruction error images, filter the redundant information on the images, perform road surface damage segmentation, and output the road surface damage detection results corresponding to the damaged road surface images;

[0051] S8. Collect actual road surface images, input the actual road surface images into the autoencoder after training and testing, and output the road surface damage detection results corresponding to the actual road surface images.

[0052] Specifically, the data augmentation method of stripe-shaped and rasterized filling proposed in step S1 will perform corresponding processing after inputting the non-damaged road surface images in the training stage, including stripe-shaped fillers of various different scales and randomly selected rasterized fillers, and finally achieve different data augmentation effects as the input of the encoder.

[0053] In step S2, an encoder consisting of five convolutional blocks based on a deep convolutional neural network is built to extract the information and features of the input data - augmented undamaged road surface images. The structures of the five convolutional blocks all include convolutional operation - batch normalization operation - ReLU activation - pooling operation. The size of the input image is 224 pixels in height * 224 pixels in width. After passing through the five convolutional blocks, feature maps with sizes of 112 pixels in height * 112 pixels in width, 56 pixels in height * 56 pixels in width, 28 pixels in height * 28 pixels in width, 14 pixels in height * 14 pixels in width, and 7 pixels in height * 7 pixels in width are generated in sequence.

[0054] In step S3, a decoder consisting of five convolutional blocks based on a deep convolutional neural network is built to reconstruct a new undamaged road surface image. The structures of the five convolutional blocks are all upsampling operation - transposed convolution operation - batch normalization operation - ReLU activation. The feature map with a size of 7 pixels in height * 7 pixels in width is restored into a feature map with a size of 112 pixels in height * 112 pixels in width in sequence; the output of the last convolutional block is the reconstructed undamaged road surface image with a size of 224 pixels in height * 224 pixels in width;

[0055] In step S4, the network models in steps S2 and S3 are combined to form an encoder - decoder based on a deep convolutional neural network, that is, an AutoEncoder model. The purpose of this auto - encoder is to learn the reconstruction and recovery mapping relationship from the data - augmented undamaged road surface image to the original undamaged road surface image. Let the former be y_aug and the latter be y_rec. The batch size selected each time for each round of training is bs (batch size). Then the loss function of the designed auto - encoder model can be expressed as:

[0056]

[0057] In step S5, first, the parameters of each layer of the trained auto - encoder model are saved, including the weights and bias values of each neuron. Then, in the test phase, the weights and bias values of these saved neurons are frozen, and the damaged road surface images in the test set of the data set are used as the input of the auto - encoder, and the corresponding reconstruction and recovery results are output. In principle, because in the training phase, this auto - encoder only learns how to reconstruct and recover the mapping relationship from the data - augmented undamaged road surface image to the original undamaged road surface image, so in the test phase, this auto - encoder cannot recover the damaged areas on the images in the test set.

[0058] In step S6, during the testing phase, the undamaged road surface image after reconstruction and restoration by the trained autoencoder is obtained. Since the autoencoder cannot restore the damaged areas on the image, the reconstructed and restored image is compared with the original image. By performing a pixel-by-pixel subtraction operation, the reconstructed error image is obtained, and thus the damaged areas on the road surface in the image can be highlighted.

[0059] In step S7, post-processing is performed on the reconstructed error image obtained in step S6, including using one of methods such as Gaussian filtering, bilateral filtering, and morphological filtering to filter out redundant information on the image, and then using one of methods such as Otsu’s thresholding, triangle thresholding, and adaptive thresholding to segment the damaged areas, and finally obtaining the damage detection result corresponding to the input test road surface image.

[0060] In step S8, an existing road inspection vehicle can be used. After a lightweight modification, relying on inspection and sensing devices such as cameras deployed on ordinary vehicles, the problem of portability of maintenance equipment can be solved, and the above method is used to realize intelligent detection of road surface damage, which can detect various types of road surface damage such as cracks, block cracks, longitudinal cracks, transverse cracks, subsidence, ruts, potholes, and looseness, automatically discover and master the accurate positions of road diseases, and can also be confirmed through video linkage; in addition, by accumulating background reports of road surface damage, data sharing among systems such as road construction, management, and maintenance can be further realized.

[0061] This embodiment applies the above technical solution and combines it with the SDNET2018 dataset to verify the effectiveness of the technical solution. Among them, the framework schematic diagram is as Figure 2 shown, mainly including:

[0062] First, a data augmentation method using striped and rasterized filling is used to perform data augmentation on the undamaged road surface images in the training set;

[0063] Second, an encoder composed of five convolutional blocks based on deep convolutional neural networks is built, and the structure is convolutional operation - batch normalization operation - ReLU activation - pooling operation, which is used to extract the information and features of the input data-augmented undamaged road surface images and generate feature maps of various sizes;

[0064] Third, a decoder composed of five convolutional blocks based on deep convolutional neural networks is built, and the structure is upsampling operation - transposed convolution operation - batch normalization operation - ReLU activation, which is used to reconstruct and restore the original image of the undamaged road surface image according to the feature maps;

[0065] IV. Combine the network models in Steps II and III to form an encoder-decoder based on a deep convolutional neural network, namely an AutoEncoder model. Train this autoencoder to learn the reconstruction and recovery mapping relationship from the data-augmented undamaged pavement images to the original undamaged pavement images;

[0066] V. Use the model trained in Step IV to input the damaged pavement images in the test set of the dataset for testing, and output the corresponding reconstruction and recovery result images;

[0067] VI. Subtract the damaged pavement images and the reconstructed and recovered pavement images pixel by pixel to obtain the corresponding reconstruction error images;

[0068] VII. Successively perform an image filtering algorithm and an image segmentation algorithm on the reconstruction error images to filter out redundant information on the images and achieve pavement damage segmentation;

[0069] VIII. In the construction of a smart city, with the data collected by sensing devices such as cameras on patrol vehicles, perform unsupervised intelligent detection of pavement damage to automatically discover and master the precise locations of road diseases.

[0070] In Step I of this embodiment, first input the undamaged pavement images in the training set into the algorithm framework, and then convert the image data into a numpy format that is convenient for processing in python. To input into the neural network built with pytorch, we further convert the numpy-formatted image data into a tensor format in pytorch. Then, we use the data augmentation methods of stripe-like and rasterized filling shown in Figure 3 (a) and (b) respectively select random-sized forms to perform corresponding data augmentation processing on the input undamaged pavement images in the training set. The two designed data augmentation methods both remove some regions in the image and then let the network learn to generate the original input image. The design principles are: (1) The image regions should have the same probability of being removed; (2) The removed regions should have multiple scales; (3) The shapes of the removed regions should have multiple directions.

[0071] In Step I, based on these three design requirements, this embodiment designs several multi-scale stripe-like occlusion maps to indicate the regions to be deleted, sets the pixel values of these regions to 0, uses black to represent the regions to be removed, and sets the ratio between the white and black regions to 1:1; obtains complementary stripe-like occlusion maps by swapping the black and white regions; obtains stripe-like occlusion maps of different scales by changing the widths of the black and white stripes; changes the directions of the stripes to obtain stripe-like occlusion maps in the vertical and horizontal directions. In addition, three scales of rasterized filling occlusion maps are designed, and the randomly divided grids are filled and their pixel values are set to 0.

[0072] In the second step of this embodiment, as Figure 4 the structure shown in the first half, an encoder composed of five convolutional blocks, each consisting of a convolutional operation - batch normalization operation - ReLU activation - pooling operation, is built based on the PyTorch framework. The Sequential command provided by PyTorch is used to establish a multi-layer feed-forward neural network, where each component in Sequential is an object inherited from the nn.Module module of PyTorch. During training, the tensor matrix size of the input image is (3 * 224 * 224). Here, "3" represents that the input image is a three-layer color image composed of R (Red), G (Green), and B (Blue). After being processed by each convolutional block, the information and features of the image are extracted, and feature maps with sizes of (64 * 112 * 112), (128 * 56 * 56), (256 * 28 * 28), (512 * 14 * 14), and (512 * 7 * 7) are output in sequence.

[0073] In the second step, convolution is a mathematical concept defined as the serial integral operation of a convolution kernel (also called a filter) function on an input signal. The convolution operation is actually a multi-step process. In the first step, the n * n-sized convolution kernel takes the inner product with the first n * n region in the upper left corner of the original image, that is, multiplies the corresponding positions and then adds these numbers together. The result will be output as the first pixel value in the upper left corner of the feature map. In the second step, the n * n-sized convolution kernel takes the inner product with the second n * n region in the upper left corner of the original image, that is, multiplies the values in the convolution kernel with the pixel values of the original image at each position and adds the final results together to obtain the value of the second pixel on the feature map. After that, the convolution kernel continuously moves from left to right and from top to bottom, performs corresponding operations with the next corresponding original image, and obtains the output of the next pixel on the feature map until the original image is covered, and the calculation of the feature map through the convolution operation is completed.

[0074] In the second step, considering that in the actual operation process, it is difficult for the encoder and decoder to be trained only through convolution operations. Because for a neural network based on deep convolution, repeated iteration is likely to cause a large numerical drift in the output result and generate a relatively large variance. Furthermore, in this technical solution, by adding a batch normalization operation to each layer of the network, the output value of this layer can be limited within a given range, thereby avoiding excessive fluctuations. Usually, the batch normalization technique is an operation node added before the non-linear activation function of each layer of the neural network. The operation performed is to normalize the input values and then map them to a set range, that is:

[0075]

[0076] Among them, α and β are parameters for learning, which will be automatically updated when the backpropagation algorithm is executed. μ b and σ b are respectively the mean and variance of the input values in a batch. Therefore, no matter how large the variation range of the input data is, the output result after batch normalization operation will always be limited within the range of [β - α, β + α], and α and β can be learned. Thus, it not only limits the variation range of the output values of the neural network but also does not lose flexibility.

[0077] In step two, the full name of the ReLU activation function is the Linear Rectification Function, also known as the Rectified Linear Unit, which is a commonly used non-linear activation function in artificial neural networks. This function is exactly an identity function when the input is greater than 0, which is equivalent to not performing any calculation. Therefore, compared with the traditional neural network activation function Sigmoid, it has excellent features such as fast calculation and convenient backpropagation of errors. At the same time, since it is divided into two discontinuous parts at the position of 0, it has the same non-linear characteristics as the Sigmoid function. Therefore, it is more suitable for deep feedforward neural networks. In the encoder built in this technical solution, the pooling operation used is max pooling. The role of pooling is to obtain coarse-grained information, which can simulate the process of humans observing things from a more macroscopic scale, that is, overlooking the entire forest rather than individual trees. Mathematically, the operation is to compress the original image of multiple squares into one pixel point, thus ignoring information. In this technical solution, the max pooling operation is adopted, from left to right and from top to bottom, to complete the calculation of obtaining the maximum value of the window pixels by the n * n pooler.

[0078] In step three of this embodiment, as Figure 4 shown in the latter half of the structure, five convolutional blocks composed of upsampling operation - transposed convolution operation - batch normalization operation - ReLU activation are built to form a decoder. The size of the tensor matrix of the input feature map is (512 * 7 * 7). After being processed by each convolutional block, it is successively restored into feature maps with sizes of (512 * 14 * 14), (256 * 28 * 28), (128 * 56 * 56), and (64 * 112 * 112), and finally the original image of the undamaged road surface with a size of (3 * 224 * 224) is reconstructed and restored.

[0079] In step three, the transposed convolution operation can be regarded as the mirror image form of the convolution operation, and each convolution operation has a corresponding deconvolution operation. First, convert the convolution kernel of the convolution operation into the convolution kernel of the deconvolution operation, which is an operation of "upside down and left-right flipped" relative to the original convolution kernel; second, pad the input image corresponding to the deconvolution with 0s on both sides to form a larger image, so that after applying the deconvolution convolution kernel to this padded image, the output image obtained is an image of the same size as the input image of the convolution operation; third, perform convolution on the padded input image with the deconvolution convolution kernel to obtain the convolution operation result of the deconvolution kernel.

[0080] In step four of this embodiment, as Figure 4 shown as a whole, combine the network models in steps two and three to form an encoder-decoder based on a deep convolutional neural network, that is, an AutoEncoder model. Train this autoencoder to learn the mapping relationship of reconstructing and restoring the non-damaged road surface image after data augmentation to the original non-damaged road surface image. There are 12,000 non-damaged road surface images used in the training of this embodiment, with a total of 10 scenarios. Some schematic diagrams of these images are as Figure 5 shown. The graphics card used is NVIDIA RTX 3090Ti. A total of 200 epochs were trained, the batchsize was set to 64, the learning rate was set to 0.0001, and it was trained for 3 hours. The value of the loss function finally dropped to 0.010.

[0081] In step five of this embodiment, first use the torch.save() function of pytorch to save the parameters of each layer of the model with the best effect on the validation set during the training stage, including the weights and bias values of each neuron; then, in the test stage, use the torch.load() function of pytorch to load the saved best model parameters, and input the test set in the dataset SDNET2018, as Figure 6 shown, that is, the road surface image with damage, as the input of the autoencoder, and output the corresponding reconstructed and restored result image, as Figure 7 shown. Because during the training stage, this autoencoder only learned how to reconstruct and restore the mapping relationship from the non-damaged road surface image after data augmentation to the original non-damaged road surface image, so in the test stage, this autoencoder cannot restore the damaged area on the image in the test set;

[0082] In step six of this embodiment, subtract the road surface image with damage and the reconstructed and restored road surface image pixel by pixel to obtain the corresponding reconstruction error image, as Figure 8As shown below. First, use the numpy.transpose() function in the numpy library to convert the matrix obtained by the neural network into a matrix format of (224, 224, 3). Then multiply all values by 255, and use the numpy.clip function in the numpy library to limit the values of all pixels in the image to the range of (0 - 255). Finally, use the numpy.absolute function in the numpy library to ensure that the result of subtracting the damaged road surface image and the reconstructed road surface image pixel by pixel is greater than 0, and use the cv2.imwrite function in the cv2 library to save the corresponding reconstruction error map.

[0083] In step seven of this embodiment, the principle of Gaussian filtering is to perform weighted averaging on the entire image. The value of each pixel point is obtained by weighted averaging of itself and other pixel values in the neighborhood. Bilateral filtering is a non-linear filtering method, which is a compromise treatment combining the spatial proximity and pixel value similarity of the image, considering both spatial domain information and gray similarity. Morphological filtering includes erosion, dilation, opening operation, closing operation, etc., which are used to filter out redundant information on the image.

[0084] In step seven, Otsu’s thresholding, also known as the maximum inter-class variance method, the basic idea is to use a threshold to divide the data in the image into two categories. The gray levels of the pixel points in one category of the image are all less than this threshold, and the gray levels of the pixel points in the other category of the image are all greater than or equal to this threshold. Triangle thresholding is a method for automatically solving the image segmentation threshold. This method uses histogram data and finds the optimal threshold based on a pure geometric method. Adaptive thresholding does not calculate the global threshold of the image, but calculates its local threshold according to the brightness distribution of different regions of the image, and can adaptively calculate different thresholds for different regions of the image.

[0085] In step eight of this embodiment, for the construction of a smart city, after lightly modifying the existing road inspection vehicles and deploying inspection and sensing devices such as cameras, the developed unsupervised intelligent road damage detection algorithm can be executed in the background to detect multiple types of road damages in real time, such as cracks, block cracks, longitudinal cracks, transverse cracks, subsidence, ruts, potholes, looseness, etc. Automatically discover and master the exact location of road diseases, confirm through video linkage, further automatically frame the road damages and perform automatic calculation and annotation of parameters, and realize the visualization of damage detection results.

[0086] In step eight, damage reports of the road surface can be accumulated in the background system to achieve data sharing among systems such as road construction, management, and maintenance. In addition, uploading the damage result data together with the construction system data, road network operation data, etc. to the background enables further processing such as data integration, data mining, and AI inference, forming a complete and scientific road surface damage database, establishing a disease prediction model, and making the evolution of road surface damage "measurable". Furthermore, a reasonable maintenance plan can be formulated to predict maintenance in advance and reduce the maintenance cycle. Relying on the unsupervised road surface damage intelligent detection algorithm proposed in this technical solution, maintenance management personnel can complete the detection under complex road surface conditions, minimizing the impact of uncertain factors such as weather and environment on the detection accuracy, and achieving controllability of highway inspection in multiple scenarios.

[0087] In summary, this technical solution proposes a new unsupervised road surface damage intelligent detection algorithm for smart city construction, effectively solving the problem that existing mainstream supervised road surface damage intelligent detection algorithms require a large amount of manually labeled data during training, saving the manpower and material resources required for manual labeling, avoiding the problem of inaccurate manual labeling caused by subjectivity, and being more suitable for the implementation of smart city construction tasks. Two data augmentation methods, namely striped and rasterized filling, are proposed, which can effectively enhance the reconstruction and recovery ability of the built autoencoder. During the training stage, through the constraint learning of the loss function, the built autoencoder can comprehensively extract the information and features of the input undamaged road surface image during the encoding stage and perfectly reconstruct and restore the original undamaged road surface image during the decoding stage; during the testing stage, the trained autoencoder cannot perfectly reconstruct and restore the damaged area on the input damaged road surface image, making the obtained reconstruction error map able to highlight the damaged area well. This embodiment conducted experiments on the SDNET2018 dataset, Figure 8 showing the effect of the finally obtained reconstruction error map. It can be seen that the output result of the proposed algorithm is relatively ideal. After post-processing, the reconstruction error map obtained during the testing stage can achieve an output effect no less than that of the supervised road surface damage intelligent detection algorithm. Figure 6 and Figure 7 comprehensively showing the trained autoencoder, which cannot restore the damaged area on the damaged road surface image. It can be found that, Figure 6 and Figure 7 the comparison effect is very obvious. The reconstructed image visually fills in the potholes automatically, which is also the core discovery of this technical solution and the mechanism for the algorithm to work. This technical solution designs a simple and feasible autoencoder framework based on a deep convolutional neural network. The network structure does not require additional training strategies and supervised annotations and can be used as the basic framework for this method. Subsequent researchers can continue to develop based on this.

[0088] Therefore, compared with the existing supervised intelligent road damage detection algorithms, this technical solution has unique advantages and plays a very good guiding role in future research. Considering the needs of today's smart city construction as a national strategy, the development of various functions is booming, and it is even more necessary for researchers to start from practical problems and find the difficulties in implementing the intelligent road damage recognition algorithm into the full-automatic AI road inspection. This technical solution is conducive to the subsequent application of the intelligent road damage recognition algorithm in the emerging fields of digital twins and smart city construction.

Claims

1. An unsupervised road damage detection method for smart city construction, characterized in that, It includes the following steps: S1. Obtain a dataset containing a training set and a test set, and use a data augmentation method of striped and rasterized filling to perform data augmentation on the non-damaged road surface images in the training set; S2. Construct an encoder based on a deep convolutional neural network to extract the information and features of the input data-augmented non-damaged road surface images and generate feature maps of various sizes; S3. Construct a decoder based on a deep convolutional neural network to reconstruct and restore the original non-damaged road surface image according to the feature maps; S4. Combine the encoder and the decoder to form an encoder-decoder based on a deep convolutional neural network, that is, an autoencoder model. Use the data-augmented non-damaged road surface images and the reconstructed original non-damaged road surface images to train the autoencoder to learn the mapping relationship between the two; S5. Use the autoencoder trained in step S4 to input the damaged road surface images in the test set for testing and output the corresponding reconstructed result images; S6. Compare and process the damaged road surface images and the reconstructed result images to obtain the corresponding reconstruction error images; S7. Post-process the reconstruction error images, filter the redundant information on the images, perform road damage segmentation, and output the road damage detection results corresponding to the damaged road surface images; S8. Collect actual road surface images, input the actual road surface images into the autoencoder after training and testing, and output the road damage detection results corresponding to the actual road surface images; Specifically, step S2 is to use five first convolutional blocks based on a deep convolutional neural network to form an encoder. The structures of the five first convolutional blocks all include convolutional operation - batch normalization operation - ReLU activation - pooling operation; Specifically, step S3 is to use five second convolutional blocks based on a deep convolutional neural network to form a decoder. The structures of the five second convolutional blocks all include upsampling operation - transposed convolution operation - batch normalization operation - ReLU activation, and sequentially restore the feature maps with a size of 7 pixels in height * 7 pixels in width to feature maps with a size of 112 pixels in height * 112 pixels in width; among the five second convolutional blocks, the output of the last second convolutional block is a new reconstructed original non-damaged road surface image with a size of 224 pixels in height * 224 pixels in width.

2. The unsupervised road damage detection method for smart city construction according to claim 1, wherein The data augmentation method of striped and rasterized filling used in step S1 specifically uses various striped fillers with different scales and randomly selected rasterized fillers to achieve different data augmentation effects.

3. The unsupervised road damage detection method for smart city construction according to claim 1, characterized in that, The input image size of the encoder is 224 pixels in height * 224 pixels in width. After passing through five first convolutional blocks, feature maps with sizes of 112 pixels in height * 112 pixels in width, 56 pixels in height * 56 pixels in width, 28 pixels in height * 28 pixels in width, 14 pixels in height * 14 pixels in width, and 7 pixels in height * 7 pixels in width are generated in sequence.

4. A method for unsupervised road surface damage detection for smart city construction according to claim 1, characterized in that The model loss function of the autoencoder in step S4 is specifically: Among them, y rec is the original undamaged road surface image after reconstruction, and y aug is the undamaged road surface image after data augmentation. bs is the batch size selected each time for each round of training.

5. The unsupervised road damage detection method for smart city construction according to claim 1, wherein, Step S5 specifically includes the following steps: S51. Save the parameters of each layer of the autoencoder model after training is completed; S52. During the testing phase, freeze the parameters of each layer of these saved autoencoder models, and use the damaged pavement images in the test set as the input of the autoencoder to output the corresponding reconstructed and restored result images.

6. The unsupervised road damage detection method for smart city construction according to claim 5, wherein, The parameters of each layer of the autoencoder model include the weights and bias values of each neuron.

7. The unsupervised road damage detection method for smart city construction according to claim 1, characterized in that, Specifically, step S6 is to subtract the damaged pavement image and the restored result image pixel by pixel to obtain the corresponding reconstruction error image.

8. A method for unsupervised road surface damage detection for smart city construction according to claim 1, characterized in that, Specifically, step S7 is to sequentially perform an image filtering algorithm and an image segmentation algorithm on the reconstruction error image to filter redundant information on the image and perform pavement damage segmentation. The image filtering algorithm includes, but is not limited to, Gaussian filtering, bilateral filtering, and morphological filtering. The image segmentation algorithm includes Otsu’s thresholding, triangle thresholding, and adaptive thresholding.

Images