A road crack detection method and system based on improved YOLO

Abstract

The present application relates to the technical field of image processing, and more particularly to a road crack detection method and system based on improved YOLO, comprising data enhancement and crack labeling to obtain a pre-training data set and a training data set; a MobileNetV3 backbone feature extraction network module is used, an Attention mechanism is added before the feature splicing part of the enhanced feature extraction network, and a ConvLSTM module is added between the prediction network and the enhanced feature extraction network; pre-training weights are generated, and re-training is performed through the training data set. The present application takes into account the characteristics of single road crack category and large information correlation between video images, adopts a MobileNetV3 backbone feature extraction network, adds an Attention mechanism before the feature splicing part of the enhanced feature extraction network module, and adds a ConvLSTM module between the prediction network and the enhanced feature extraction network.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a road crack detection method and system based on an improved YOLO. Background Technology

[0002] The detection of cracks in concrete pavements has been a hot research topic in recent years. Concrete pavements are highly susceptible to cracking due to factors such as severe weather, road erosion, heavy loads, and road aging. Cracks affect the service life of roads, and under conditions of insufficient maintenance and overloading, they can accelerate the deterioration and collapse of concrete pavements, causing serious safety accidents and threatening people's lives and property.

[0003] Traditional algorithms have good feature extraction capabilities for cracks with obvious features, but when detecting small cracks or crack images that are severely affected by noise, they will produce pseudo-segmentation due to noise, which will have a significant impact on the crack detection effect. To address these issues, convolutional neural networks have emerged. In crack detection, Unet networks are mostly used to segment and detect road surfaces. However, Unet networks are relatively slow and cannot meet the requirements of real-time processing.

[0004] Existing deep learning-based road defect detection methods in patents use the original YOLO network. This method first requires a large number of training samples, resulting in slow training speed; secondly, it does not filter the feature information of crack images, making it unable to effectively learn crack-related features and causing underfitting; in addition, it does not consider the temporal continuity of crack images, which may lead to missed detections. Summary of the Invention

[0005] To address the shortcomings of existing algorithms, this invention takes into account the characteristics of road cracks being of a single type and the high correlation between information between video images. Based on the YOLO network framework, the original backbone feature extraction network is changed to a MobileNetV3 network. Attention mechanism is added before the feature concatenation part of the enhanced feature extraction network module, and a ConvLSTM module is added between the prediction network and the enhanced feature extraction network.

[0006] The technical solution adopted in this invention is: a road crack detection method and system based on an improved YOLO, comprising the following steps:

[0007] Step 1: Collect publicly available crack datasets and construct a dataset of road surfaces to be detected. Augment the data and label the cracks to obtain a pre-training dataset and a training dataset.

[0008] Further data enhancement includes: inversion, rotation, contrast enhancement, and brightness enhancement;

[0009] Step 2: Construct an improved YOLO network model: Replace the original backbone feature extraction network module with MobileNetV3 network, add an Attention mechanism before the feature concatenation part of the enhanced feature extraction network, add a ConvLSTM module between the prediction network and the enhanced feature extraction network, and adjust the prediction network model according to the characteristics of the cracks.

[0010] Furthermore, the improved YOLO network model includes: the first layer performs a convolutional block operation with a stride of 2 followed by a convolutional block operation with a stride of 1, transferring the feature map output from the first layer to the second layer; the second layer performs a convolutional block operation with a stride of 2 followed by a convolutional block operation with a stride of 1, transferring the feature map output from the second layer to the third layer; the third layer performs a convolutional block operation with a stride of 2 followed by two convolutional blocks with a stride of 1, transferring the feature map output from the third layer to the fourth layer; the fourth layer performs a convolutional block operation with a stride of 2 followed by five convolutional blocks with a stride of 1, transferring the feature map output from the fourth layer to the fifth layer; the fifth layer performs a convolutional block operation with a stride of 2 followed by five convolutional blocks with a stride of 1.

[0011] Furthermore, the enhanced feature extraction network includes: convolving the fifth layer to obtain the sixth layer, upsampling the sixth layer and passing it to the seventh layer, concatenating and convolving the seventh layer with the fourth layer to obtain the output of the seventh layer; downsampling the output of the seventh layer and passing it to the eighth layer, concatenating and convolving the eighth layer with the sixth layer to obtain the output of the eighth layer.

[0012] Furthermore, the prediction network includes performing convolution operations on the seventh and eighth layers to obtain the final network output.

[0013] Furthermore, the convolution operations on the seventh and eighth layers are performed using a ConvLSTM module with depth separable convolution.

[0014] Step 3: Input the pre-training dataset into the improved YOLO network model for pre-training, generate pre-training weights, and then retrain using the training dataset.

[0015] The system based on the improved YOLO road crack detection method includes a camera module, a Raspberry Pi detection device, and a GPS positioning module. The camera module is used to acquire video of road cracks and transmit it to the Raspberry Pi detection device, and the GPS positioning module is used to transmit the location of the crack to the Raspberry Pi detection device when the network detects a crack.

[0016] The beneficial effects of this invention are:

[0017] 1. Using MobileNetV3 as the backbone feature of YOLO reduces the number of network model parameters, lowers model complexity, and enables training with a small number of sample datasets, while also preventing overfitting.

[0018] 2. Use the Attention mechanism to select more important features from the input features to train the model, making the model training results accurate and effective.

[0019] 3. Use the ConvLSTM module to combine the feature information from the previous time step with the information from the current time step, and comprehensively consider the temporal continuity of the crack image.

[0020] 4. First, use the public crack dataset to train the network model and extract effective features, and then use real road cracks for transfer learning to reduce the amount of training data.

[0021] 5. When the network detects cracks in the road surface, it will automatically record the GPS location of the cracks for easy maintenance. Attached Figure Description

[0022] Figure 1 This is a flowchart of the road crack detection method based on the improved YOLO of the present invention;

[0023] Figure 2 This is the overall network structure diagram of the road crack detection method based on the improved YOLO of the present invention;

[0024] Figure 3 This is a diagram of the ConvLSTM network structure of the present invention;

[0025] Figure 4 This is a schematic diagram of the depth-separable convolution operation of the present invention;

[0026] Figure 5 This is the road joint marking diagram of the present invention;

[0027] Figure 6 This is a diagram showing the effect of road joint detection according to the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0029] Crack detection is challenging due to interference from lighting and road debris. Small cracks on roads are often overlooked, leading to sudden expansion and potential safety hazards. To address these issues, we first integrated publicly available datasets and augmented them with data augmentation. We then labeled the datasets to obtain a pre-training dataset. To reduce model parameters and improve accuracy, we considered temporal information by adding attention mechanisms and temporal convolutions to the network model. The model was trained using the pre-training dataset, and transfer learning was performed on the trained model using collected real road cracks to obtain even more accurate results.

[0030] like Figure 1 As shown, a road crack detection method and system based on an improved YOLO includes the following steps:

[0031] Step 1: Collect publicly available crack datasets and construct a dataset of road surfaces to be detected. Expand the datasets by rotating, enhancing contrast, and enhancing brightness, and label the datasets to obtain pre-training datasets and training datasets.

[0032] Rotation refers to rotating an image around its center at a certain angle; contrast enhancement refers to stretching or compressing the brightness value range in an image to the brightness display range specified by the display system, thereby improving the overall or partial contrast of the image; brightness enhancement refers to magnifying or reducing the brightness value in an image according to a certain value.

[0033] The annotation uses a broad-based approach, meaning that a single crack is annotated using several bounding boxes, such as... Figure 5 As shown, this is done to improve the effective feature ratio of the target box and speed up the network's extraction of effective features;

[0034] Step 2: Construct a YOLO object detection network model (improved YOLO network), such as... Figure 2 As shown, the original backbone feature extraction network module is replaced by the MobileNetV3 network. An Attention mechanism is added before the feature concatenation part of the enhanced feature extraction network module. A ConvLSTM module is added between the prediction network and the enhanced feature extraction network. The prediction network model is adjusted according to the characteristics of the crack.

[0035] The improved YOLO network includes: a backbone feature extraction network, an enhanced feature extraction network, and a prediction network. The backbone feature extraction network extracts useful information by reducing the feature map size and increasing the number of feature map channels. The enhanced feature extraction network integrates the semantic and spatial information of the backbone feature extraction network. The prediction network is trained by calculating the loss between the prediction results and the label image.

[0036] The backbone feature extraction process is as follows: The MobileNetV3 network is used to extract features from the input image. In the first layer, convolutional block operations with stride 2 and stride 1 are performed. The feature map output from the first layer is then transferred to the second layer. In the second layer, a convolutional block operation with stride 2 is performed first, followed by a convolutional block operation with stride 1. The feature map output from the second layer is then transferred to the third layer. In the third layer, a convolutional block operation with stride 2 is performed first, followed by two convolutional block operations with stride 1. The feature map output from the third layer is then transferred to the fourth layer. In the fourth layer, a convolutional block operation with stride 2 is performed first, followed by five convolutional block operations with stride 1. The feature map output from the fourth layer is then transferred to the fifth layer. In the fifth layer, a convolutional block operation with stride 2 is performed first, followed by five convolutional block operations with stride 1.

[0037] The enhanced feature extraction network is specifically implemented as follows: convolution is performed on the fifth layer to obtain the sixth layer; the sixth layer is upsampled and passed to the seventh layer; the seventh layer is concatenated and convolved with the fourth layer to obtain the output of the seventh layer; the output of the seventh layer is downsampled and passed to the eighth layer; the eighth layer is concatenated and convolved with the sixth layer to obtain the output of the eighth layer. The enhanced feature extraction network integrates the semantic and spatial information of the backbone feature extraction network.

[0038] The backbone feature extraction network uses convolutional blocks with a stride of 2 instead of pooling operations, aiming to utilize the feature information extracted by the network as much as possible while halving the height and width of the feature map.

[0039] Attention mechanisms include channel attention and spatial attention. Channel attention involves performing global max pooling on a single input feature layer, then processing it with a fully connected layer, activating the result with a sigmoid function, and multiplying the result by the original input feature layer. Spatial attention involves taking the maximum and average values ​​for each channel of the input feature map, concatenating the two results, adjusting the number of channels with a single convolution, applying a sigmoid function, and then multiplying the result by the original input feature layer.

[0040] The prediction network specifically involves performing convolution operations on the seventh and eighth layers to obtain the final network output.

[0041] like Figure 3 The ConvLSTM module includes: a forget gate, an input gate, a state gate, an output gate, a cell state, and a cell output. The forget gate concatenates the cell output from the previous time step with the cell input from the current time step to generate the input information. The input information is then subjected to convolution and sigmoid activation to generate the forget gate information, with the formula f. t =Sigmoid(W f [ht-1 ,x t ]+b f Input gate: The input information is obtained by performing convolution and sigmoid activation on the input information separately. The formula is i. t =Sigmoid(W i [h t-1 ,x t-1 ]+b i State gate: The state gate information is obtained by separately performing convolution transformation and Tanh activation on the input information, and the formula is s. t =Tanh(W s [h t-1 ,x t-1 ]+b s Output gate: The input information is convolved to generate the output gate information, and the formula is o. t =Sigmoid(W o [h t-1 ,x t-1 ]+b o ); Unit state: Multiply the forget gate information by the previous unit state, multiply the input gate by the state gate, and then add the two together to obtain the unit state, with the formula being c. t =f t *c t-1 +i t *s t Unit output: After activating the unit state, multiply it by the output gate, as shown in the formula h. t =o t *Tanh(c t );

[0042] Since ConvLSTM has a large number of parameters, it is easy to cause insufficient training. Therefore, depthwise separable convolution is used instead of the original convolution, and it is used between the prediction network and the enhanced feature extraction network, which have less computation. Considering that ConvLSTM involves multiplication, which can easily cause the feature map parameters to become smaller, it is standardized after the model is convolved.

[0043] like Figure 4 This is a schematic diagram of depthwise separable convolution operation. Since the ConvLSTM module has a large number of parameters, the number of parameters is reduced after depthwise separable convolution.

[0044] A normal convolution is: using out_channel convolutional kernels with in_channel number of convolutional layers to convolve the input feature layer, with the parameter count being out_channel(in_channel*kernel_size^2);

[0045] Depthwise separable convolution is performed by using in_channel convolutional kernels of number 1 to convolve each layer of the input feature layer to obtain the convolution result, and then using out_channel 1*1 convolutions of number in_channel to fuse the convolution result. The number of parameters is: out_channel*kernel_size^2+out_channel*in_channel*1^2.

[0046] The adjustment to the prediction network model specifically involves deleting the original small target prediction network and retaining the medium and large target detection networks.

[0047] The pre-trained dataset is fed into the model for pre-training, and pre-trained weights are generated to facilitate transfer learning.

[0048] Pre-training was performed as follows: the model was first loaded with the official MobileNetV3 weights, the backbone network was frozen and only the prediction network was trained, and then the backbone network was unfrozen and the entire network was trained. Focal loss was used to prevent imbalance between positive and negative samples. The SGD algorithm was selected to optimize the improved YOLO, and the cosine learning rate descent method was selected to adjust the learning rate for each epoch. The initial learning rate was 0.01, and the momentum parameter was set to 0.9.

[0049] Importing the training dataset into a network model loaded with pre-trained weights for retraining improves the model's accuracy and applicability.

[0050] Retraining involves loading the pre-trained weights and training dataset into the network. Since both the pre-trained and training datasets consist of crack images, the network only needs to be frozen during training.

[0051] Deploying the YOLO network to mobile devices enables real-time on-site detection of road cracks. When a crack is detected, the GPS module records the real-time location of the crack, facilitating future road maintenance.

[0052] The mobile device includes a camera module, a Raspberry Pi inspection device, and a GPS positioning module. The camera module is used to capture video of road cracks and transmit it to the Raspberry Pi inspection device. The inspection device has a pre-trained YOLO model embedded in it to detect cracks in the video. The detection results are as follows: Figure 5 The GPS positioning module is used to transmit the location of the crack to the Raspberry Pi detection device when the network detects a crack.

[0053] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A road crack detection method based on an improved YOLO, characterized in that, Includes the following steps: Step 1: Collect publicly available crack datasets and construct a dataset of the road surface to be detected. Augment the data and annotate the cracks to obtain a pre-training dataset and a training dataset. Step 2: Using the MobileNetV3 backbone feature extraction network module, an Attention mechanism is added before the feature concatenation part of the enhanced feature extraction network, and a ConvLSTM module is added between the prediction network and the enhanced feature extraction network to build an improved YOLO network model. The enhanced feature extraction network includes: convolving the fifth layer to obtain the sixth layer, upsampling the sixth layer and passing it to the seventh layer, concatenating and convolving the seventh layer with the fourth layer to obtain the output of the seventh layer; downsampling the output of the seventh layer and passing it to the eighth layer, concatenating and convolving the eighth layer with the sixth layer to obtain the output of the eighth layer. Step 3: Input the pre-training dataset into the improved YOLO network model for pre-training, generate pre-training weights, and then retrain using the training dataset.

2. The road crack detection method based on the improved YOLO according to claim 1, characterized in that, The improved YOLO network model includes the following steps: The first layer performs a convolutional block operation with a stride of 2 followed by a convolutional block operation with a stride of 1, transferring the feature map output from the first layer to the second layer; the second layer performs a convolutional block operation with a stride of 2 followed by a convolutional block operation with a stride of 1, transferring the feature map output from the second layer to the third layer; the third layer performs a convolutional block operation with a stride of 2 followed by two convolutional blocks with a stride of 1, transferring the feature map output from the third layer to the fourth layer; the fourth layer performs a convolutional block operation with a stride of 2 followed by five convolutional blocks with a stride of 1, transferring the feature map output from the fourth layer to the fifth layer; the fifth layer performs a convolutional block operation with a stride of 2 followed by five convolutional blocks with a stride of 1.

3. The road crack detection method based on the improved YOLO according to claim 1, characterized in that, The prediction network consists of performing convolution operations on layers 7 and 8 to obtain the final network output.

4. The road crack detection method based on the improved YOLO according to claim 3, characterized in that, Convolution operations on the seventh and eighth layers are performed using the ConvLSTM module with depth-separable convolution.

5. A system employing the road crack detection method based on the improved YOLO according to any one of claims 1-4, characterized in that, It includes a camera module, a Raspberry Pi inspection device, and a GPS positioning module. The camera module is used to capture video of road cracks and transmit it to the Raspberry Pi inspection device, while the GPS positioning module is used to transmit the location of the crack to the Raspberry Pi inspection device when the network detects the crack.

Images