A road crack detection method based on deep learning

Abstract

The application aims to provide a road crack detection method based on deep learning, comprising the following steps: collecting actual road crack picture images, the collected actual road crack picture images being original images, marking the crack positions in the original images, training a crack detection network model to obtain a feature map corresponding to the crack; marking a target region from the whole image, and outputting a crack circumscribed rectangle and a crack width according to the marked target region. Two branches of CNN and Encoder-Decoder are used for training respectively, the respective advantages of CNN fast and Transformer accurate are combined, the shortage of uneven sample quantity underfitting is avoided, the crack detection accuracy is greatly improved, the problems of low calculation efficiency and low detection accuracy caused by the complex network model in the prior art are solved, efficient and high-accuracy crack detection is realized, and the crack width (pixel number) can be accurately given.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically a road crack detection method based on deep learning. Background Technology

[0002] my country has a large number of highways, long distances, and wide distribution, making it difficult to quickly and accurately assess the damage to different road sections. Traditional manual inspection methods are slow, costly, and lack standardized statistical criteria.

[0003] Crack detection is a challenging task due to the poor continuity and low contrast of cracks. Current mainstream detection frameworks include SSD, YOLO, and Mask R-CNN. SSD and YOLO can only detect the bounding rectangle of the crack and cannot accurately identify important indicators such as crack width. Mask R-CNN models are complex, have low inference efficiency, and low accuracy.

[0004] For example, application number CN202210909472.3, "Road Crack Detection Method and System Based on Deep Learning," describes a method and system for detecting road cracks based on deep learning. The method includes preprocessing photos of the target road to obtain a prediction set and preprocessing images of existing cracks to obtain a training set. It constructs a deep convolutional neural network model including feature extraction, region proposal generation, and classification / regression structures, and trains the network model using the training set. The prediction set is then input into the trained network model to obtain road crack detection results. The system includes a data acquisition module and an intelligent detection module. The data acquisition module acquires photos of the target road, and the intelligent detection module obtains road crack detection results based on the acquired photos. This method can achieve automatic identification, improve detection efficiency and accuracy, and is applicable to roads with different crack conditions. However, this patent only uses the FastRCNN method, which can only detect the circumscribed rectangle of the crack, lacking crucial information such as crack width and length.

[0005] Application number CN202211342286.2 discloses a real-time road crack detection method integrating CNN and Transformer. The method involves acquiring original road images and labeling road cracks in these images using LabelImg to obtain binary labels for the original road images. An original road image dataset is created based on the original road images and their binary labels. A road crack detection model is then created, comprising a CNN convolutional coding module, a Transformer coding module, a feature fusion module, and a segmentation prediction module. The image dataset is used as training samples to train the road crack detection model. Road images are acquired in real-time and input into the trained model. The result is then output as the road image detection result. This patent's network design uses a serial approach combining CNN and Transformer networks for model training, resulting in low efficiency and slow speed. Summary of the Invention

[0006] The purpose of this invention is to provide a road crack detection method based on deep learning, so as to solve the problems of low computational efficiency and low detection accuracy caused by the complexity of network models in the prior art. This invention achieves high-efficiency and high-accuracy crack detection, and can also accurately provide the crack width (number of pixels).

[0007] The objective of this invention is achieved through the following technical means: a road crack detection method based on deep learning, comprising the following steps:

[0008] Collect actual road crack images. The collected actual road crack images are the original images. Mark the crack locations in the original images to obtain the coordinate point file of the corresponding crack locations in the images.

[0009] The actual road crack images were normalized and used as input images for the network.

[0010] The crack detection network model is trained by feeding the normalized image and the corresponding crack location coordinates file into the network. The first image branch uses Fast R-CNN to perform whole-image convolution on the image to obtain the crack outline, and then uses RestNet50 to extract crack image features to obtain the feature map. The second branch uses an Encoder-Decoder network to complete crack feature extraction and obtain the corresponding crack feature map.

[0011] Feature fusion: The features obtained from branch one and branch two are fused to obtain the first fused feature map;

[0012] Based on the first fusion feature map, cracks in the labeled crack images in the training dataset are identified, crack detection coordinates are obtained, and the branch fusion is optimized according to the loss function until the loss value meets the requirements. The first branch and the second branch are fused to obtain the crack detection network model and the second fusion feature map is obtained.

[0013] Image recognition involves taking an input image and, based on the second fused feature map, evaluating each pixel of the input image. If a pixel's value is greater than a set threshold, it is considered a target region; otherwise, it is considered a non-target region. This process marks the target region from the entire image.

[0014] Based on the labeled target area, output the outer rectangle of the crack and the crack width.

[0015] The specific method for normalizing the training dataset is as follows: images of different resolutions in different scenes are scaled proportionally to a size of 640*640. If a short side is not large enough for scaling, padding is applied by adding 0 pixels to complete the image to a size of 640*640. The pixel values ​​of the image are then normalized by subtracting the mean from the pixel value and dividing by the variance. Finally, a three-channel 640*640 resolution image is used as the network input.

[0016] The specific steps for fusing branch one and branch two are as follows: ω = λω1 + (1-λ)ω2, where ω is the fused feature map, ω1 is the feature map of branch one, ω2 is the feature map of branch two, and λ is an adjustment parameter with a value of [0,1].

[0017] The loss function is, If the crack detection coordinates are consistent with the actual crack coordinates in the crack location coordinate point file, then Pt is 1; otherwise, it is not 1. The γ coefficient takes the value [0,5].

[0018] The crack detection network model is transformed into an ONNX model, and the ONNX model is simplified by scissoring. This simplified ONNX model is then used as the final crack detection network model to detect road cracks.

[0019] The specific steps of the first branch are as follows: The FastRCNN network uniformly samples the input image, that is, it directly performs whole-image convolution on the original input image; it extracts the crack contour corresponding to each pre-selected region through region proposals, where each region proposal is defined by a four-tuple array, i.e. (r,c,h,w), which is the left-right row and column coordinates, height and width of the window. These coordinates are all corresponding to the original image. It is also necessary to map the coordinate system of the original image onto feature maps to complete the crack contour ROI detection task; for each pre-selected crack contour, i.e., ROI region, the corresponding crack image features are extracted through the backbone network RestNet50, and the feature map is output.

[0020] The specific steps of the second branch are as follows: Encode classifies and analyzes the low-level local pixel values ​​of the input image, classifies crack pixels, and assigns a value of 0 or 1 to crack pixels through a mask, that is, 1 is a crack pixel and 0 is not a crack pixel. Decoder collects the basic pixel information of cracks and maps the same crack to the corresponding pixel to obtain the feature map.

[0021] When the training dataset is insufficient, non-target regions of images in the training dataset are randomly selected as backgrounds and target regions as foregrounds. A set of foreground and background regions is randomly selected and combined to form a new set of training data. A new training dataset is synthesized on a large scale and added to the original training dataset.

[0022] The beneficial effects of this invention are as follows: by using two branches, CNN and Encoder-Decoder, for separate training, it combines the advantages of CNN's speed and Transformer's accuracy, and avoids the shortcomings of uneven sample size and underfitting, which greatly improves the accuracy of the task. It solves the problems of low computational efficiency and low detection accuracy caused by the complexity of network models in the prior art, and achieves high-efficiency and high-accuracy crack detection, while also providing a relatively accurate crack width (number of pixels). Attached Figure Description

[0023] Figure 1 Here is a flowchart of a deep learning-based road crack detection method;

[0024] Figure 2 Image showing the crack identification results;

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0026]

Example 1

[0027] like Figure 1 As shown, a road crack detection method based on deep learning includes the following steps:

[0028] The process involves acquiring actual road crack images, which serve as the original images. The locations of the cracks in the original images are then marked to obtain the coordinate point file corresponding to the crack locations. These acquired images are then distinguished from the normalized network input images.

[0029] When the training dataset is insufficient, non-target regions of images in the training dataset are randomly selected as backgrounds and target regions as foregrounds. A set of foreground and background regions is randomly selected and combined to form a new set of training data. A new training dataset is synthesized on a large scale and added to the original training dataset.

[0030] like Figure 1 As shown, firstly, images of actual road cracks are collected as samples for training the detection model. Samples are collected from multiple scenes and time periods from multiple angles. Each image requires the road surface to be basically intact and clear, with obvious cracks. For example, cameras can be mounted on the roof of a car, mounted on a small mobile vehicle, or mounted on a low-flying drone. Various acquisition devices ensure different acquisition angles, such as top-down, frontal, and side views. The algorithm trained from this data can ensure that the equipment has strong generalization ability when used in the future.

[0031] Cracks were manually marked using bounding boxes and plotting points to form the training dataset. During calibration, mature calibration tools such as LabelImg were used to outline the crack locations and mark the crack lines, generating a sequence of pixel coordinates of key points on the crack lines, i.e., the crack location coordinate file. This coordinate file was stored in an XML file in JSON data format and served as the basic dataset for future algorithm training. The crack location coordinate file included the crack's bounding rectangle and crack width. By using the crack location coordinate file and the image as input, the trained model could identify cracks and obtain the corresponding crack's bounding rectangle and crack width.

[0032] The actual road crack images were normalized and used as input images for the network.

[0033] The specific method for normalizing the training dataset is as follows: images of different resolutions in different scenes are scaled proportionally to a size of 640*640. If a short side is not large enough for scaling, padding is applied by adding 0 pixels to complete the image to a size of 640*640. The pixel values ​​of the image are then normalized by subtracting the mean from the pixel value and dividing by the variance. Finally, a three-channel 640*640 resolution image is used as the network input.

[0034] The collected images of actual road cracks can be enhanced using various data augmentation methods, such as horizontal stretching, rotation, small-angle vertical rotation transformation, adding noise, and chromaticity changes, to further expand the dataset, enhance cracks of different sizes and angles in the dataset, and improve generalization ability.

[0035] After normalization, the data is input into the training network.

[0036] When the number of samples is insufficient, non-target regions of images in the training dataset can be randomly selected as backgrounds and target regions as foregrounds. A set of foreground and background data can be randomly selected and combined to form a new training dataset. This new training dataset can be synthesized on a large scale and added to the original training dataset.

[0037] The crack detection network model is trained by feeding the normalized image and the corresponding crack location coordinates file into the network. The first image branch uses Fast R-CNN to perform whole-image convolution on the image to obtain the crack outline, and then uses RestNet50 to extract crack image features to obtain the feature map. The second branch uses an Encoder-Decoder network to complete crack feature extraction and obtain the corresponding crack feature map.

[0038] The specific steps of the first branch are as follows: The FastRCNN network uniformly samples the input image, that is, it directly performs whole-image convolution on the original input image; it extracts the crack contour corresponding to each pre-selected region through region proposals, where each region proposal is defined by a four-tuple array, i.e. (r,c,h,w), which is the left-right row and column coordinates, height and width of the window. These coordinates are all corresponding to the original image. It is also necessary to map the coordinate system of the original image onto feature maps to complete the crack contour ROI detection task; for each pre-selected crack contour, i.e., ROI region, the corresponding crack image features are extracted through the backbone network RestNet50, and the feature map is output.

[0039] The specific steps of the second branch are as follows: the Encoder classifies and analyzes the low-level local pixel values ​​of the input image, classifies crack pixels, and assigns a value of 0 or 1 to crack pixels through a mask, that is, 1 is a crack pixel and 0 is not a crack pixel. The Decoder collects the basic pixel information of cracks and maps the same crack to the corresponding pixel to obtain the feature map.

[0040] After the input image samples are processed, the training of the branch model begins.

[0041] We employ two branches: FastRCNN and Encoder-Decoder. We train the feature maps using the two branch networks separately, and then fuse the training results of the two branches.

[0042] Branch 1:

[0043] FastRCNN samples the input image (i.e., the normalized image) uniformly and performs full-image convolution directly on the original input image, without performing region or partial convolution.

[0044] Crack contours corresponding to each pre-selected region are extracted through region proposals. Each region proposal is defined by a quad array, i.e. (r,c,h,w), which represents the top left row and column coordinates, height, and width of the window. These coordinates correspond to the original image. The coordinate system of the original image also needs to be mapped onto the feature maps. This step is mainly to complete the crack contour ROI detection task.

[0045] For each pre-selected crack contour, i.e. ROI region, the corresponding crack image features are extracted through the backbone network RestNet50 to obtain the feature map of branch one.

[0046] Features are the characteristic points of a crack, which may be vectors or matrices. A feature map is a two-dimensional image that presents the features. Features and feature maps are the same thing.

[0047] Branch Two:

[0048] An Encoder-Decoder network structure is used to complete the task of feature extraction from crack images and output a feature map.

[0049] The encoder is composed of multiple convolutional networks, mainly consisting of convolutional layers, pooling layers, and batch normalization layers. The convolutional layers are responsible for capturing local features of the image, the pooling layers downsample the image and pass scale-invariant features to the next layer, and the batch normalization layer normalizes the distribution of the training images to accelerate learning.

[0050] The Encoder categorizes and analyzes the low-level local pixel values ​​of the original image, classifies crack pixels, and assigns a value of 0 or 1 to crack pixels through a Mask, that is, 1 is a crack pixel and 0 is not a crack pixel.

[0051] The Decoder collects basic pixel information of the crack and maps the same crack to the corresponding pixel. It upsamples the scaled-down feature image and then performs convolution processing on the upsampled crack image to improve the geometry of the crack and make up for the loss of detail caused by the pooling layer in the Encoder scaling down the object.

[0052] Feature fusion: The features obtained from branch one and branch two are fused to obtain the first fused feature map;

[0053] The specific steps for fusing branch one and branch two are as follows: ω = λω1 + (1-λ)ω2, where ω is the fused feature map, ω1 is the feature map of branch one, ω2 is the feature map of branch two, and λ is an adjustment parameter with a value of [0, 1].

[0054] After branch one and branch two are trained separately, they are fused using ω = λω1 + (1-λ)ω2 to obtain the fused feature map. λ can be adjusted according to the statistical distribution of the samples, taking values ​​[0, 1].

[0055] Based on the first fusion feature map, cracks in the labeled crack images in the training dataset are identified, crack detection coordinates are obtained, and the branch fusion is optimized according to the loss function until the loss value meets the requirements. The first branch and the second branch are fused to obtain the crack detection network model and the second fusion feature map is obtained.

[0056] The loss function is, If the crack detection coordinates are consistent with the actual crack coordinates in the crack location coordinate point file, then Pt is 1; otherwise, it is not 1. The γ coefficient takes the value [0, 5], and is generally 2.

[0057] After fusion, the Focal Loss function is used to confirm whether the fused model and output feature map can be used to detect cracks.

[0058] Using the first feature fusion map, crack detection coordinates are obtained from the labeled images. These coordinates are then compared with the actual crack coordinates in the corresponding crack location coordinate file. If they match, Pt is set to 1; otherwise, it is not set to 1. This process continues until the loss value, or crack error, is determined. If the value is less than the set threshold, it means that the fused model can be used as a crack detection network model for detection, and the feature map output by the crack detection network model for detection is used as the second fused feature map.

[0059] If the loss value does not meet the target, the learning rate is adjusted, and features are obtained again through branch one and branch two. The features are then fused again to obtain a new first feature fusion map. The new first feature fusion map is then tested using the loss function until the loss rate meets the target.

[0060] If multiple models meet the criteria, the network parameters with the best training effect and the smallest loss value will be selected as the crack detection network model for detection.

[0061] After obtaining the crack detection network model, it can be converted into an ONNX model, namely the Open Neural Network Exchange Format, to facilitate cross-platform use in the future. The ONNX model is then pruned and simplified by merging mergeable network layers and removing unnecessary side branches, and used as the final crack detection network model to detect road cracks.

[0062] The crack detection network model has now been trained.

[0063] The first and second fused feature maps were obtained using the same method; the first and second were simply used to distinguish whether they had been verified by the loss function.

[0064] Image recognition involves taking an input image and, based on the second fused feature map, evaluating each pixel of the input image. If a pixel's value is greater than a set threshold, it is considered a target region; otherwise, it is considered a non-target region. This process marks the target region from the entire image.

[0065] Generally, if the foreground pixel is greater than the threshold, it indicates that it is the foreground (i.e., the target area with cracks) and is set to 1; if it is less than the threshold, it indicates that it is the background (non-target area without cracks) and is set to 0.

[0066]

[0067] like Figure 2 As shown, based on the labeled target area, the output crack circumscribed rectangle and crack width are calculated. A model inference program was designed using C++ / C and CUDA languages, and detection (three-channel 640*640 resolution image) was implemented on a Tesla P4 with a single-frame latency of 20 milliseconds and memory usage of less than 400MB. Figure 2 As shown, the recognition score is obtained, which represents the reliability of crack recognition.

[0068] In practical use, based on the second fusion feature map, the target area is marked by determining whether each pixel of the acquired image is foreground or background. Since the crack location coordinate point file is also sent into the training during model training, the output result includes the output crack bounding rectangle and crack width.

Claims

1. A road crack detection method based on deep learning, characterized in that, Includes the following steps: Collect actual road crack images. The collected actual road crack images are the original images. Mark the crack locations in the original images to obtain the coordinate point file of the corresponding crack locations in the images. The actual road crack images were normalized and used as input images for the network. The normalized image and the corresponding crack location coordinates are fed into the network as input. The first image branch uses Fast R-CNN to perform full-image convolution on the image to obtain the crack outline, and then uses RestNet50 to extract crack image features to obtain a feature map. The second branch uses an Encoder-Decoder network to complete crack feature extraction and obtain the corresponding crack feature map. The specific steps of the first branch are as follows: The FastRCNN network uniformly samples the input image, that is, it directly performs whole-image convolution on the original input image; it extracts the crack contour corresponding to each pre-selected region through region proposals, where each region proposal is defined by a four-element array, i.e. (r, c, h, w), which is the left-top row and column coordinates, height and width of the window. These coordinates are all corresponding to the original image. It is also necessary to map the coordinate system of the original image onto feature maps to complete the crack contour ROI detection task; for each pre-selected crack contour, i.e., ROI region, the corresponding crack image features are extracted through the backbone network RestNet50, and the feature map is output. Feature fusion: The features obtained from branch one and branch two are fused to obtain the first fused feature map; Based on the first fusion feature map, cracks in the labeled crack images in the training dataset are identified, crack detection coordinates are obtained, and the branch fusion is optimized according to the loss function until the loss value meets the requirements. The first branch and the second branch are fused to obtain the crack detection network model and the second fusion feature map is obtained. Image recognition involves taking an input image and, based on the second fused feature map, evaluating each pixel of the input image. If a pixel's value is greater than a set threshold, it is considered a target region; otherwise, it is considered a non-target region. This process marks the target region from the entire image. Based on the labeled target area, output the outer rectangle of the crack and the crack width.

2. The road crack detection method based on deep learning according to claim 1, characterized in that: The specific method for normalizing the training dataset is as follows: images of different resolutions in different scenes are scaled proportionally to a size of 640*640. If a short side is not large enough for scaling, padding is applied by adding 0 pixels to complete the image to a size of 640*640. The pixel values ​​of the image are then normalized by subtracting the mean from the pixel value and dividing by the variance. Finally, a three-channel 640*640 resolution image is used as the network input.

3. The road crack detection method based on deep learning according to claim 1, characterized in that: The specific method for merging branch one and branch two is as follows: , To fuse feature maps, It is the feature map of branch one. It is the feature map of branch two. The parameter is set to [0, 1].

4. The road crack detection method based on deep learning according to claim 1, characterized in that: The loss function is, If the crack detection coordinates match the actual crack coordinates in the crack location coordinate file, then Pt is 1; otherwise, it is not 1. The coefficient takes values ​​of [0, 5].

5. The road crack detection method based on deep learning according to claim 1, characterized in that: The crack detection network model is transformed into an ONNX model, and the ONNX model is simplified by scissoring. This simplified ONNX model is then used as the final crack detection network model to detect road cracks.

6. The road crack detection method based on deep learning according to claim 1, characterized in that: The specific steps of the second branch are as follows: the Encoder classifies and analyzes the low-level local pixel values ​​of the input image, classifies crack pixels, and assigns a value of 0 or 1 to crack pixels through a mask, that is, 1 is a crack pixel and 0 is not a crack pixel. The Decoder collects the basic pixel information of cracks and maps the same crack to the corresponding pixel to obtain the feature map.

7. The road crack detection method based on deep learning according to claim 1, characterized in that: When the training dataset is insufficient, non-target regions of images in the training dataset are randomly selected as backgrounds and target regions as foregrounds. A set of foreground and background regions is randomly selected and combined to form a new set of training data. A new training dataset is synthesized on a large scale and added to the original training dataset.

Images