A method for detecting road surface cracks based on deep learning

Abstract

The application discloses a kind of based on deep learning's road surface crack detection method, based on industrial grade target detection algorithm YOLOv5 proposes improved road surface crack detection model YOLOv5-Crack, and through light weight, it is deployed to Web end, it is convenient for user to upload picture, real-time video etc., while also can locate the map information of crack, it is convenient for maintenance unit to repair.The YOLOv5 technology used can learn a large number of road surface image data through neural network model, extract the feature information therein, to realize the automated road surface crack detection, it can not only improve the accuracy and efficiency of detection, but also avoid subjective error and inconsistency in the process of artificial detection.In addition, the model trained by YOLOv5 can also be continuously iterated and optimized, gradually improve the accuracy and robustness of the model, so that it can adapt to different road surface environments and different types of cracks, to improve the efficiency and safety of road maintenance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a road surface crack detection method based on deep learning. BACKGROUND

[0002] With the development of society, the highway traffic network is becoming more and more huge. A large number of freight and passenger transport demand makes the daily traffic volume increase continuously, which leads to the difficulty of maintenance of the highway network in China to increase dramatically, and further leads to the continuous rise of the highway maintenance financial expenditure. If the road surface cracks cannot be repaired in time, it will lead to more serious damage to the road surface, increase the risk of driving, affect traffic safety, and also increase the cost of vehicle damage and repair.

[0003] At present, most highway departments mainly conduct on-site inspection and measurement by artificial means, which is low in efficiency, low in precision, high in cost, and easy to be disturbed by human factors, and is difficult to meet the detection needs of large-scale road network. Therefore, the exploration of a high detection precision and real-time road surface crack target detection method can not only improve the driving comfort, but also further help to improve the traffic safety and road service life, optimize the road network maintenance cost and management efficiency, save the national financial expenditure, and realize the sustainable development of road traffic. The target detection technology based on deep learning and computer vision has been widely developed and applied due to its excellent feature extraction capability, and its accuracy and noise resistance are better than traditional image morphological methods.

[0004] Zhao Fang et al. improved the Canny operator, which provided experimental basis for real-time road crack detection, but there were still deficiencies in the identification of small cracks on the road surface. Quintana et al. used support vector machines to train a classifier based on local feature descriptors for the identification of road cracks. Dihao A et al. proposed to use pixel intensity and multi-scale field information to solve the problems of poor light environment and shadow. K. Fernandes et al. proposed to use the picture features of the cracks to effectively describe the road cracks. These road crack detection methods have high requirements for image quality, and the extracted crack features are not accurate enough, which leads to low detection accuracy and cannot meet the real-time requirements.

[0005] Li Liangfu et al. used the improved window sliding algorithm to divide the crack image into smaller cracks and background images, and then established a DBCC (Deep Bridge Crack Classify) classification mode of convolutional neural network, which enhanced the effectiveness and generalization of recognition. Kai Li et al. proposed to combine the pixel-based adaptive weighted cross-entropy (Weighted Cross-entropy, WCE) loss with the Jaccard distance, and then apply it to the two-stage algorithm, which promotes high-quality pixel-level pavement crack detection. These pavement crack detection algorithms run too many selected areas, although they can ensure a certain accuracy, but due to the large amount of calculation, it is difficult to guarantee real-time performance.

[0006] Vishal Mandal et al. used YOLOv2 technology to realize automatic detection of pavement cracks, but the precision was insufficient. DUO MA et al. combined YOLOv3 with PCGAN adversarial network, and combined with the median flow algorithm, which accelerated the detection speed and also improved the accuracy to a certain extent, but the detection effect of the algorithm for the crack-like crack was not good. Hao Junming et al. introduced the Ghost module in the YOLOv4 backbone network, and then introduced the ECA (Efficient Channel Attention) attention mechanism at the output end, which improved the detection speed while ensuring a certain accuracy, but there was still a problem of overall low accuracy. SUMMARY

[0007] The present application aims to solve the problem of low overall accuracy in existing pavement crack detection methods, and provides a pavement crack detection method based on deep learning.

[0008] To solve the above problems, the present application is realized by the following technical scheme:

[0009] A pavement crack detection method based on deep learning, comprising the following steps:

[0010] Step 1, constructing a pavement crack detection model;

[0011] The pavement crack detection model is composed of 5 layer convolution modules, 4 CA attention mechanism adding modules, 1 efficient spatial pyramid pooling module, 4 ghost convolution modules, 2 up-sampling modules, 4 fusion modules, 2 three-convolution modules with a step of 2, 2 three-convolution modules with a step of 1, and 3 two-dimensional convolution modules.

[0012] The input of the first layer convolution module forms the input of the pavement crack detection model, the output of the first layer convolution module is connected to the input of the second layer convolution module, the output of the second layer convolution module is connected to the input of the first CA attention mechanism addition module, the output of the first CA attention mechanism addition module is connected to the input of the third layer convolution module, the output of the third layer convolution module is connected to the input of the second CA attention mechanism addition module, the output of the second CA attention mechanism addition module is connected to the input of the fourth layer convolution module, the output of the fourth layer convolution module is connected to the input of the third CA attention mechanism addition module, the output of the third CA attention mechanism addition module is connected to the input of the fifth layer convolution module, the output of the fifth layer convolution module is connected to the input of the fourth CA attention mechanism addition module, and the output of the fourth CA attention mechanism addition module is connected to the input of the high-efficiency spatial pyramid pooling module.

[0013] The output of the high-efficiency spatial pyramid pooling module is connected to the input of the first ghost shuffle convolution module, the output of the first ghost shuffle convolution module is connected to the input of the first upsampling module, the output of the first upsampling module and the output of the third CA attention mechanism addition module are simultaneously connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the first three-convolution module with a step of 2, the output of the first three-convolution module with a step of 2 is connected to the input of the second ghost shuffle convolution module, the output of the second ghost shuffle convolution module is connected to the input of the second upsampling module, the output of the second upsampling module and the output of the second CA attention mechanism addition module are simultaneously connected to the input of the second fusion module, and the output of the second fusion module is connected to the input of the second three-convolution module with a step of 2; the output of the second three-convolution module with a step of 2 is connected to the input of the third ghost shuffle convolution module, the output of the third ghost shuffle convolution module and the output of the second ghost shuffle convolution module are simultaneously connected to the input of the third fusion module, the output of the third fusion module is connected to the input of the first three-convolution module with a step of 1, the output of the first three-convolution module with a step of 1 is connected to the input of the fourth ghost shuffle convolution module, the output of the fourth ghost shuffle convolution module and the output of the first ghost shuffle convolution module are simultaneously connected to the input of the fourth fusion module, and the output of the fourth fusion module is connected to the input of the second three-convolution module with a step of 1.

[0014] The input of the first two-dimensional convolution module is connected to the output of the second three-convolution module with a step of 2, the input of the second two-dimensional convolution module is connected to the output of the first three-convolution module with a step of 1, the input of the third two-dimensional convolution module is connected to the output of the second three-convolution module with a step of 1, and the output of the first two-dimensional convolution module, the output of the second two-dimensional convolution module and the output of the third two-dimensional convolution module jointly form the output of the pavement crack detection model.

[0015] Step 2, training the road crack detection model constructed in step 1 using a set of road sample images labeled with road cracks to obtain a trained road crack detection model.

[0016] Step 3, sending the road image to be detected into the trained road crack detection model obtained in step 2, and the trained road crack detection model outputs the segmented image.

[0017] The CA attention mechanism module is composed of a step-1 triple convolution module and a CA attention mechanism module; the input of the step-1 triple convolution module forms the input of the CA attention mechanism module, the output of the step-1 triple convolution module is connected to the input of the CA attention mechanism module, and the output of the CA attention mechanism module forms the output of the CA attention mechanism module.

[0018] The high-efficiency spatial pyramid pooling module is composed of two Mish convolution modules, one maximum pooling module, two average pooling modules, and one fusion module; the input of the first Mish convolution module forms the input of the high-efficiency spatial pyramid pooling module, the output of the first Mish convolution module is connected to the input of the first maximum pooling module, the input of the first average pooling module, and the input of the second average pooling module, the output of the first Mish convolution module, the output of the first maximum pooling module, the output of the first average pooling module, and the output of the second average pooling module are simultaneously connected to the input of the fusion module, the output of the fusion module is connected to the input of the second Mish convolution module, and the output of the second Mish convolution module forms the output of the high-efficiency spatial pyramid pooling module.

[0019] The Mish convolution module is composed of a two-dimensional convolution module, a normalization module, and a Mish activation function module; the input of the two-dimensional convolution module forms the input of the Mish convolution module, the output of the two-dimensional convolution module is connected to the input of the normalization module, the output of the normalization module is connected to the input of the Mish activation function module, and the output of the Mish activation function module forms the output of the Mish convolution module.

[0020] The ghost shuffle convolution module is composed of two graph layer convolution modules, two fusion modules, and a slicing module; the input of the first graph layer convolution module forms the input of the ghost shuffle convolution module, the output of the first graph layer convolution module is connected to the input of the second graph layer convolution module, the output of the first graph layer convolution module and the output of the second graph layer convolution module are simultaneously connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the slicing module, the output of the slicing module is connected to the input of the second fusion module, and the output of the second fusion module forms the output of the ghost shuffle convolution module.

[0021] The loss function Loss of the road crack detection modelSIoU is:

[0022]

[0023] wherein, IoU is the intersection over union loss, A is the area of the ground crack real boundary box, and B is the area of the ground crack predicted boundary box;

[0024] Delta is the distance loss, is the horizontal coordinate of the centroid of the ground crack real boundary box, is the vertical coordinate of the centroid of the ground crack real boundary box, is the horizontal coordinate of the centroid of the ground crack predicted boundary box, is the vertical coordinate of the centroid of the ground crack predicted boundary box, c w is the width of the minimum circumscribed rectangle of the ground crack real boundary box and the ground crack predicted boundary box, c h is the height of the minimum circumscribed rectangle of the ground crack real boundary box and the ground crack predicted boundary box;

[0025] Omega is the shape loss, Theta is the cost of the shape, w is the width of the ground crack predicted boundary box, h is the height of the ground crack predicted boundary box, w gt is the width of the ground crack real boundary box, h gt is the height of the ground crack real boundary box;

[0026] Lambda is the angle loss, Sigma is the Euclidean distance between the centroid of the ground crack real boundary box and the centroid of the ground crack predicted boundary box.

[0027] Compared with the prior art, the improved pavement crack detection model YOLOv5-Crack is proposed based on an industrial target detection algorithm YOLOv5, and is deployed to the Web end through lightening, which is convenient for users to upload pictures, real-time videos and the like, and can also locate the map information of the cracks, so as to facilitate the maintenance units to carry out maintenance. The YOLOv5 technology used can learn a large amount of pavement image data through a neural network model, extract the feature information therein, and thus realize automatic pavement crack detection, which can not only improve the accuracy and efficiency of detection, but also avoid subjective errors and inconsistencies in the artificial detection process. In addition, the model trained by YOLOv5 can be continuously iteratively optimized to gradually improve the precision and robustness of the model, so that it can adapt to different pavement environments and different types of cracks, thereby improving the efficiency and safety of pavement maintenance. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1Structure diagram of a pavement crack detection model (YOLOv5-Crack).

[0029] Figure 2 Structure diagram of a CA attention mechanism plus module (CA-plus).

[0030] Figure 3 Pseudo-code implementation diagram of a CA attention mechanism plus module (CA-plus).

[0031] Figure 4 Structure diagram of an efficient spatial pyramid pooling module (ESPP).

[0032] Figure 5 Structure diagram of a ghost shuffling convolution module (GSConv).

[0033] Figure 6 Principle diagram of a SIoU loss.

[0034] Figure 7 Pavement crack dataset.

[0035] Figure 8 Comparison diagram of loss convergence curves.

[0036] Figure 9 Comparison diagram of precision convergence curves.

[0037] Figure 10 Comparison of heat maps.

[0038] Figure 11 Comparison diagram of experimental results. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to specific examples.

[0040] A pavement crack detection method based on deep learning includes the following steps:

[0041] Step 1, constructing a pavement crack detection model.

[0042] The present application proposes an improved pavement crack detection model YOLOv5-Crack based on the industrial-grade target detection algorithm YOLOv5, as shown in Figure 1The pavement crack detection model YOLOv5-Crack is composed of 5 layer convolution modules (Conv), 4 CA attention mechanism plus modules (CA-plus), 1 efficient spatial pyramid pooling module (ESPP), 4 fusion modules (Concat), 2 upsample modules (Upsample), 4 ghost shuffle convolution modules (GSConv), 2 triple convolution modules with a step of 2 (C3-S2), 2 triple convolution modules with a step of 1 (C3), and 3 two-dimensional convolution modules (Conv2d).

[0043] In the backbone network: the input of the first layer convolution module forms the input of the pavement crack detection model, the output of the first layer convolution module is connected to the input of the second layer convolution module, the output of the second layer convolution module is connected to the input of the first CA attention mechanism plus module, the output of the first CA attention mechanism plus module is connected to the input of the third layer convolution module, the output of the third layer convolution module is connected to the input of the second CA attention mechanism plus module, the output of the second CA attention mechanism plus module is connected to the input of the fourth layer convolution module, the output of the fourth layer convolution module is connected to the input of the third CA attention mechanism plus module, the output of the third CA attention mechanism plus module is connected to the input of the fifth layer convolution module, the output of the fifth layer convolution module is connected to the input of the fourth CA attention mechanism plus module, and the output of the fourth CA attention mechanism plus module is connected to the input of the efficient spatial pyramid pooling module.

[0044] In the neck network (Neck): the output of the high-efficiency spatial pyramid pooling module is connected to the input of the first ghost shuffle convolution module, the output of the first ghost shuffle convolution module is connected to the input of the first upsampling module, the output of the first upsampling module and the output of the third CA attention mechanism plus module are simultaneously connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the first three-convolution module with a step of 2, the output of the first three-convolution module with a step of 2 is connected to the input of the second ghost shuffle convolution module, the output of the second ghost shuffle convolution module is connected to the input of the second upsampling module, the output of the second upsampling module and the output of the second CA attention mechanism plus module are simultaneously connected to the input of the second fusion module, the output of the second fusion module is connected to the input of the second three-convolution module with a step of 2; the output of the second three-convolution module with a step of 2 is connected to the input of the third ghost shuffle convolution module, the output of the third ghost shuffle convolution module and the output of the second ghost shuffle convolution module are simultaneously connected to the input of the third fusion module, the output of the third fusion module is connected to the input of the first three-convolution module with a step of 1, the output of the first three-convolution module with a step of 1 is connected to the input of the fourth ghost shuffle convolution module, the output of the fourth ghost shuffle convolution module and the output of the first ghost shuffle convolution module are simultaneously connected to the input of the fourth fusion module, and the output of the fourth fusion module is connected to the input of the second three-convolution module with a step of 1.

[0045] In the neck network (Neck): the input of the first two-dimensional convolution module is connected to the output of the second three-convolution module with a step of 2, the input of the second two-dimensional convolution module is connected to the output of the first three-convolution module with a step of 1, the input of the third two-dimensional convolution module is connected to the output of the second three-convolution module with a step of 1, and the output of the first two-dimensional convolution module, the output of the second two-dimensional convolution module and the output of the third two-dimensional convolution module together form the output of the pavement crack detection model.

[0046] In the prediction network (Prediction): due to the hidden nature and multiplicity of the pavement cracks, the target detection algorithm often cannot accurately observe the information of the specific cracks. The attention mechanism can make the network focus on local information and find useful crack information. The traditional channel attention pays less attention to position information, and the spatial attention pays less attention to channel information. The CA attention mechanism integrates the position information while paying attention to the channel content of the image, so that the network can capture the long-range dependence in the spatial direction on the basis of expanding the global receptive field. In order to capture the accurate position information of the pavement cracks, the CA attention mechanism plus module is used to extract the pavement crack feature information in the backbone network. The CA attention mechanism plus module (CA-plus) is as follows: Figure 2As shown, the CA-plus is composed of a stride-1 triple convolution module (C3) and a CA attention mechanism module (CA). The input of the stride-1 triple convolution module forms the input of the CA attention mechanism plus module, the output of the stride-1 triple convolution module is connected to the input of the CA attention mechanism module, and the output of the CA attention mechanism module forms the output of the CA attention mechanism plus module. The specific implementation pseudo code of the CA-plus is as follows Figure 3 As shown.

[0047] The CA attention mechanism only encodes the height H and width W two dimensions. After the input X passes through the residual module, one-dimensional adaptive average pooling is performed in the horizontal and vertical directions respectively to obtain two features in different spatial directions. The output of the cth channel at the height h is as shown in equation (1):

[0048]

[0049] Similarly, the output of the cth channel at the width w is as shown in equation (2):

[0050]

[0051] For the two one-dimensional features, a splicing operation is performed on the width, and then a convolution, normalization processing, and nonlinear regression are performed to obtain a feature s containing horizontal and vertical spatial information, which is defined as shown in equation (3):

[0052] s = δ (F ([z h ,z w ])) (3)

[0053] Then the feature s is split in the channel to obtain two feature maps through Conv2d convolution and h_sigmoid activation function, so that the dimensions are the same as the input X:

[0054] y h = h σ (s h (s h ) ) (4)

[0055] y w = h σ (s w (s w ) ) (5)

[0056] The two feature maps are complementarily applied to the input feature map through certain weight adjustment to enhance the attention to the road cracks, that is, the original feature is fused with the feature with weight in the width and height directions, and the output o is defined as shown in equation (6):

[0057]

[0058] The application embeds a triple convolution module on the basis of the CA attention mechanism module to form a CA attention mechanism plus module, that is, the feature map extracted by each feature layer is processed, and finally a feature map with a large receptive field and independent perception direction is obtained.

[0059] In YOLOv5, a fast spatial pyramid pooling module (SPPF) is used as a structure for feature fusion, and the feature maps obtained by downsampling are sequentially input into three 5*5 pooling layers, and the maximum pooling results of each layer are spliced, which effectively avoids the image distortion problem caused by image region cropping and scaling operation, realizes the scale uniformity of the input feature map before the neck network layer, and can better guarantee the accuracy of the result. However, the maximum pooling operation in the fast spatial pyramid pooling module only considers the position of the maximum value on the feature map, which may ignore some crack feature information, increase the risk of overfitting, and also introduce noise to cause crack information loss. Therefore, in the backbone network, the original fast spatial pyramid pooling module is replaced by an efficient spatial pyramid pooling module. As shown in Figure 4 The efficient spatial pyramid pooling module (ESPP) is composed of two Mish convolution modules (MConv), one maximum pooling module (MaxPool), two average pooling modules (AvgPool) and one fusion module (Concat). The input of the first Mish convolution module forms the input of the efficient spatial pyramid pooling module, the output of the first Mish convolution module is connected to the input of the first maximum pooling module, the input of the first average pooling module and the input of the second average pooling module, the output of the first Mish convolution module, the output of the first maximum pooling module, the output of the first average pooling module and the output of the second average pooling module are connected to the input of the fusion module, the output of the fusion module is connected to the input of the second Mish convolution module, and the output of the second Mish convolution module forms the output of the efficient spatial pyramid pooling module.

[0060] The Mish convolution module (MConv) is composed of a two-dimensional convolution module (Conv2d), a normalization module (BatchNorm) and a Mish activation function module (Mish). The input of the two-dimensional convolution module forms the input of the Mish convolution module, the output of the two-dimensional convolution module is connected to the input of the normalization module, the output of the normalization module is connected to the input of the Mish activation function module, and the output of the Mish activation function module forms the output of the Mish convolution module. Since the SiLU activation function adopted in the layer convolution module in YOLOv5 has a large negative half-axis value, the regularization effect and generalization ability generated during the training process are poor, so the present application proposes a new Mish convolution module which replaces the SiLU activation function in the layer convolution module with a Mish activation function. Since the Mish activation function is non-saturated without upper bound, it can effectively speed up the training speed, and the Mish activation function has the properties of continuous differentiability and non-monotonicity, which can effectively avoid singular points. The excellent performance of the Mish convolution module makes the precision higher, the speed faster and the robustness stronger when extracting crack features by downsampling. Therefore, the efficient spatial pyramid pooling module replaces the layer convolution module with the Mish convolution module on the basis of the fast spatial pyramid pooling module. The efficient spatial pyramid pooling module using the Mish convolution module has better smoothness, so that the feature information can better enter the feature fusion network. In addition, the efficient spatial pyramid pooling module also improves the connection method of the pooling of the fast spatial pyramid pooling module. In the small receptive field layer, the efficient spatial pyramid pooling module still uses the maximum pooling. While in the large receptive field layer, the efficient spatial pyramid pooling module uses 9x9 and 13x13 average pooling layers. The average pooling operation will average the values on the feature map, which is less affected by noise and outliers, and is more suitable for complex road crack detection. At the same time, average pooling can better preserve spatial information and reduce the risk of partial overfitting.

[0061] In the convolutional neural network, all feedback images need to transmit the extracted features to the channel in the backbone network. During the transmission process, the compression of the width and height of the feature map and the expansion of the channel will cause the loss of local crack image semantic information. Since road crack detection has certain requirements for real-time performance, it is necessary to speed up the calculation while ensuring accuracy, so the present application introduces a lightweight ghost shuffle convolution module in the neck network to replace the layer convolution module for upsampling. The ghost shuffle convolution module (GSConv) is as follows Figure 5As shown, it is composed of 2 layer convolution modules (Conv), 2 fusion modules (Concat) and 1 slice module (Y[0] / Y[1]). The input of the first layer convolution module forms the input of the ghost shuffle convolution module, the output of the first layer convolution module is connected to the input of the second layer convolution module, the output of the first layer convolution module and the output of the second layer convolution module are simultaneously connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the slice module, the output of the slice module is connected to the input of the second fusion module, and the output of the second fusion module forms the output of the ghost shuffle convolution module. The ghost shuffle convolution module maintains semantic information as much as possible, that is, it saves the hidden connection between each channel as much as possible while greatly reducing the computational complexity. Let the number of input channels be C inp , and the number of output channels be C outp . After passing through the first convolution, the input will produce a feature tensor with a channel number of C outp / 2, then pass through another convolution to obtain another feature tensor with a channel number of C outp / 2, then concatenate the two feature tensors, then perform channel shuffle on the data size, that is, rearrange the original feature map channel arrangement order, and finally perform a concatenation operation.

[0062] If GSConv is used to replace Conv in each stage of the YOLOv5 network, it will cause a large increase in the number of network layers, thereby increasing the resistance of information flow propagation, so only a part of GSConv can be used to replace Conv in the network. Since the feature information in the neck network is fused through the efficient spatial pyramid pooling module, the channel dimension reaches the maximum, the width and height dimensions reach the minimum, and the redundant information at this moment is the least, and the use of GSConv will be better, therefore, the present application only uses GSConv to replace Conv in the neck network, and does not use GSConv to replace Conv in the backbone network.

[0063] For a target detection algorithm, the most important thing is to predict the target to be detected through the specific position data of the bounding box.

[0064] The CIoU (Complete Intersection over Union) loss function is used in the traditional YOLOv5, which is composed of classification loss, positioning loss and confidence loss through weighted addition, and its definition is as formula (7)-(10):

[0065]

[0066]

[0067]

[0068]

[0069] where A and B represent the area of the real and predicted bounding box respectively, b and b gt represent the centroid of the predicted and real bounding box respectively, p represents the Euclidean distance between the two bounding boxes, a is the weight function, and v is used to represent the similarity of the aspect ratio.

[0070] However, the calculation of CIoU relies on the aggregation of the bounding box regression indicators, without taking into account the direction mismatch between the required bounding box and the predicted bounding box, thus greatly reducing the training speed and prediction accuracy of the network. Therefore, the present application uses a more comprehensive SIoU (Scylla Intersection over Union) loss function, which is composed of four parts of loss, namely angle, distance, shape and IoU loss.

[0071] The angle loss is defined as formula (11)-(12):

[0072]

[0073]

[0074] where s is the Euclidean distance between the centroid of the real ground crack bounding box and the centroid of the predicted ground crack bounding box, c h is the height of the minimum circumscribed rectangle of the real ground crack bounding box and the predicted ground crack bounding box. The calculation of this angle loss can make the predicted box better regress to the same horizontal or vertical line as the real box, accelerating the model convergence, as shown in Figure 6 .

[0075] The distance loss is defined as formula (13)-(16):

[0076]

[0077]

[0078]

[0079] g = 2 - a (16)

[0080] wherein, is the centroid coordinate of the real bounding box, is the centroid coordinate of the predicted bounding box, c w is the width of the minimum circumscribed rectangle of the real ground crack bounding box and the predicted ground crack bounding box, c h is the height of the minimum circumscribed rectangle of the real ground crack bounding box and the predicted ground crack bounding box.

[0081] The shape loss is defined as equations (17)-(19):

[0082]

[0083]

[0084]

[0085] where θ defines the cost of shape and is unique for each dataset, obtained by genetic algorithm, w is the width of the ground crack prediction bounding box, h is the height of the ground crack prediction bounding box, w gt is the width of the ground crack real bounding box, h gt is the height of the ground crack real bounding box.

[0086] In summary, SIoU introduces the angle, distance and shape loss required for regression, reduces the degree of freedom of regression compared to CIoU, and makes the network converge faster, thereby improving the accuracy of regression.

[0087] Step 2, using the road surface sample image set labeled with road surface cracks to train the road surface crack detection model constructed in step 1 to obtain a trained road surface crack detection model.

[0088] Step 3, sending the road surface image to be detected into the trained road surface crack detection model obtained in step 2, and the trained road surface crack detection model outputs a segmented picture.

[0089] The professional road surface crack dataset GRDDC 2020 (Global Road Damage Detection Challenge 2020) is used below to verify the effect of the present application. The dataset covers a variety of road surface crack pictures from highways, rural roads, etc. in three different countries (Japan, Czech Republic, India), and the cracks are D00 (longitudinal cracks), D10 (transverse cracks), D20 (crack cracks) and D40 (pit cracks), which are highly professional and practical. The dataset has a total of 21,041 pictures, which are divided into training set, test set and validation set in the form of 6:2:2. The official also gives two test sets Test1 and Test2, with picture quantities of 2631 and 2664 respectively. The specific crack pictures are shown in Fig. 7.

[0090] (1) Attention module comparison experiment

[0091] In order to verify the effectiveness of the CA attention module for crack detection, SE, ECA and CBAM attention modules are introduced for comparison experiment, and the experimental results are shown in Table 1.

[0092] Table 1 Attention module comparison experiment

[0093]

[0094] From the attention module comparison experiment in Table 1, it can be seen that, except for CA, other attention modules have more or all reduced indicators, and the CA attention module has improved mAP@0.5 by up to 0.7%, and the calculation amount is also the lowest, therefore, the introduction of CA attention module has practical significance for crack detail information extraction.

[0095] In order to verify the effectiveness of the CA-plus module, an experiment was conducted, and the results are shown in Table 2.

[0096] Table 2 Attention structure comparison experiment

[0097]

[0098] From the attention structure comparison experiment in Table 2, it can be seen that the CA-plus structure improves the evaluation index value by 0.1 GFLOPs of calculation amount. In summary, the CA-plus structure proposed in this paper has a role in improving detection performance.

[0099] (2) ESPP module comparison experiment

[0100] In order to verify the effectiveness of the ESPP structure for crack detection, the original YOLOv5 model and the YOLOv5l-ES model with ESPP are compared, and the results are shown in Table 3.

[0101] Table 3 ESPP comparison experiment

[0102]

[0103] From the verification experiment in Table 3, it can be seen that due to the superiority of MConv module and average pooling, its use in ESPP can reduce the overall calculation amount of the model and improve the regression speed. At the same time, the detection accuracy is also improved, which proves that the feature information is better into the fusion network, and the generalization ability of the model is improved. The training time is reduced by 7.2%, which also verifies the effectiveness of ESPP in improving the training speed.

[0104] (3) GSConv module comparison experiment

[0105] In order to verify the effectiveness of GSConv, this paper uses YOLOv5l model and YOLOv5l-GS model with GSConv replacing Conv in Neck for comparison experiment, and the results are shown in Table 4.

[0106] Table 4 GSConv comparison experiment

[0107]

[0108] From the verification experiment of Table 4, it can be seen that after replacing Conv with GSConv, the calculation amount will decrease slightly, at the cost of reducing R value, which can improve P value and mAP@0.5 value.

[0109] (4) Loss function comparison experiment

[0110] To verify the replacement of the CIoU loss function calculation in YOLOv5 with the SIoU loss function, a comparison experiment was conducted, and the experimental results are shown in Table 5.

[0111] Table 5 Loss function comparison experiment

[0112]

[0113] From Table 5, by replacing the loss function calculation method in the model, the recall rate R of the trained model is greatly improved by 1.5% under the condition of unchanged calculation amount, which proves that changing SIoU has an effect on improving regression accuracy.

[0114] (5) Curve comparison analysis

[0115] In the self-test data set, the loss convergence curve and precision convergence curve of YOLOv5-Crack and Baseline are respectively as Figure 8 and Figure 9 .

[0116] From the loss convergence curve comparison chart, it can be seen that the loss of YOLOv5-Crack decreases more smoothly than the previous Baseline, and its loss value is lower in the final value, indicating that YOLOv5-Crack has better stability in learning the features of cracks and can better generalize test data.

[0117] From the precision convergence curve comparison chart, it can be seen that the average precision value of YOLOv5-Crack is significantly higher than that of Baseline, and the convergence time of the former is later, without obvious overfitting phenomenon, indicating that it has strong generalization ability and can adapt to a wider road crack scene, with better practical application value.

[0118] (6) Ablation experiment

[0119] Table 6 Ablation experiment

[0120]

[0121] The ablation experiment is performed through the official test sets Test1 and Test2, and the F1 score is taken as the evaluation index. As can be seen from the experimental results in Table 6, compared with the baseline Baseline, the serial number 5 improves both CA-plus and ESPP, and the F1 scores are increased by 2.13% and 1.21% on the two test sets respectively, proving the effectiveness of the simultaneous improvement of CA-plus and ESPP; in the serial number 6, the GSConv is used to replace the Conv on the basis of CA-plus, and the F1 score is increased more than the single item, proving that the GSConv has effect on improving the detection accuracy and speeding up the detection; in the serial number 7, the loss function is modified to SIoU on the basis of the original CA-plus, and the FPS is increased by 0.8 compared with the serial number 1, proving that the SIoU loss function is helpful for improving the regression accuracy and also improves the detection speed to some extent. For the final model YOLOv5-Crack, the calculation amount is reduced by 7.24%, the F1 score is increased by 4.05% and 3.93% respectively, and the detection speed FPS is reduced by only 1.3, wherein the reduction of the calculation speed is mainly due to the CA-plus module, but the effect obtained by combining the four improvements is better than that of each module added alone, that is, better detection effect is obtained in the road crack detection.

[0122] (7) Main algorithm comparison experiment

[0123] In order to verify the superiority of the YOLOv5-Crack model, the YOLOv5-Crack model is compared with YOLOv4, YOLOv5m, YOLOv7, EfficientDet and Faster R-CNN, and the experimental results are shown in Table 7.

[0124] Table 7 Main algorithm comparison experiment

[0125]

[0126] As can be seen from Table 7, compared with the mainstream two-stage target detection algorithm and a plurality of mainstream one-stage target detection algorithms, the algorithm proposed in the application has more superior performance in F1 score.

[0127] In order to make the improvement effect more obvious, the present application performs heat map visualization analysis on the model before and after improvement, and the experimental results are shown in Figure 10 Through the analysis of the heat map, the heat map coverage of the improved YOLOv5-Crack model is more obvious, and the influence on the prediction output result is greater, that is, the positioning accuracy of the crack target is greatly improved, which means that the improved algorithm pays more attention to the feature target of the crack, thereby proving the effectiveness of the improvement. At the same time, in order to make the comparison result more intuitive, the prediction effect of the algorithm model is shown in Figure 11shown.

[0128] It should be noted that, although the above embodiments of the present application are illustrative, they are not limiting of the present application, and therefore the present application is not limited to the specific embodiments described above. Any other embodiments obtained by persons skilled in the art under the teachings of the present application without departing from the principles of the present application are considered to be within the scope of the present application.

Claims

1. A method for detecting a road surface crack based on deep learning, characterized by, The steps comprise the following: Step 1, constructing a pavement crack detection model; The pavement crack detection model is composed of 5 layer convolution modules, 4 CA attention mechanism adding modules, 1 high-efficiency spatial pyramid pooling module, 4 ghost shuffle convolution modules, 2 up-sampling modules, 4 fusion modules, 2 three-convolution modules with a step of 2, 2 three-convolution modules with a step of 1, and 3 two-dimensional convolution modules; The input of the first layer convolution module forms the input of the pavement crack detection model, the output of the first layer convolution module is connected to the input of the second layer convolution module, the output of the second layer convolution module is connected to the input of the first CA attention mechanism adding module, the output of the first CA attention mechanism adding module is connected to the input of the third layer convolution module, the output of the third layer convolution module is connected to the input of the second CA attention mechanism adding module, the output of the second CA attention mechanism adding module is connected to the input of the fourth layer convolution module, the output of the fourth layer convolution module is connected to the input of the third CA attention mechanism adding module, the output of the third CA attention mechanism adding module is connected to the input of the fifth layer convolution module, the output of the fifth layer convolution module is connected to the input of the fourth CA attention mechanism adding module, and the output of the fourth CA attention mechanism adding module is connected to the input of the high-efficiency spatial pyramid pooling module; The output of the high-efficiency spatial pyramid pooling module is connected to the input of the first ghost shuffle convolution module, the output of the first ghost shuffle convolution module is connected to the input of the first up-sampling module, the output of the first up-sampling module and the output of the third CA attention mechanism adding module are simultaneously connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the first three-convolution module with a step of 2, the output of the first three-convolution module with a step of 2 is connected to the input of the second ghost shuffle convolution module, the output of the second ghost shuffle convolution module is connected to the input of the second up-sampling module, the output of the second up-sampling module and the output of the second CA attention mechanism adding module are simultaneously connected to the input of the second fusion module, the output of the second fusion module is connected to the input of the second three-convolution module with a step of 2, the output of the second three-convolution module with a step of 2 is connected to the input of the third ghost shuffle convolution module, the output of the third ghost shuffle convolution module and the output of the second ghost shuffle convolution module are simultaneously connected to the input of the third fusion module, the output of the third fusion module is connected to the input of the first three-convolution module with a step of 1, the output of the first three-convolution module with a step of 1 is connected to the input of the fourth ghost shuffle convolution module, the output of the fourth ghost shuffle convolution module and the output of the first ghost shuffle convolution module are simultaneously connected to the input of the fourth fusion module, and the output of the fourth fusion module is connected to the input of the second three-convolution module with a step of 1. The output of the first two-dimensional convolution module is connected to the output of the second triple convolution module with a step size of 2, the output of the second two-dimensional convolution module is connected to the output of the first triple convolution module with a step size of 1, the output of the third two-dimensional convolution module is connected to the output of the second triple convolution module with a step size of 1, and the output of the first two-dimensional convolution module, the output of the second two-dimensional convolution module and the output of the third two-dimensional convolution module jointly form the output of the pavement crack detection model; Step 2, using the pavement sample image set labeled with pavement cracks to train the pavement crack detection model constructed in step 1 to obtain a trained pavement crack detection model; Step 3, sending the pavement image to be detected into the trained pavement crack detection model obtained in step 2, and the trained pavement crack detection model outputs a segmented picture.

2. The method of claim 1, wherein the CA The attention mechanism adding module is composed of a triple convolution module with a step size of 1 and a CA attention mechanism module; The input of the triple convolution module with a step size of 1 forms the input of the CA attention mechanism adding module, the output of the triple convolution module with a step size of 1 is connected to the input of the CA attention mechanism module, and the output of the CA attention mechanism module forms the output of the CA attention mechanism adding module.

3. The method of claim 1, wherein the method is characterized by, The efficient spatial pyramid pooling module is composed of two Mish convolution modules, one maximum pooling module, two average pooling modules and one fusion module; The input of the first Mish convolution module forms the input of the efficient spatial pyramid pooling module, the output of the first Mish convolution module is connected to the input of the first maximum pooling module, the input of the first average pooling module and the input of the second average pooling module, the output of the first Mish convolution module, the output of the first maximum pooling module, the output of the first average pooling module and the output of the second average pooling module are connected to the input of the fusion module, the output of the fusion module is connected to the input of the second Mish convolution module, and the output of the second Mish convolution module forms the output of the efficient spatial pyramid pooling module.

4. The method of claim 3, wherein the method is characterized by, The Mish convolution module is composed of one two-dimensional convolution module, one normalization module and one Mish activation function module; The input of the two-dimensional convolution module forms the input of the Mish convolution module, the output of the two-dimensional convolution module is connected to the input of the normalization module, the output of the normalization module is connected to the input of the Mish activation function module, and the output of the Mish activation function module forms the output of the Mish convolution module.

5. The method of claim 1, wherein the method is characterized by, The ghost shuffle convolution module is composed of two graph layer convolution modules, two fusion modules and one slicing module; The input of the first graph layer convolution module forms the input of the ghost shuffle convolution module, the output of the first graph layer convolution module is connected to the input of the second graph layer convolution module, the output of the first graph layer convolution module and the output of the second graph layer convolution module are connected to the input of the first fusion module, the output of the first fusion module is connected to the input of the slicing module, the output of the slicing module is connected to the input of the second fusion module, and the output of the second fusion module forms the output of the ghost shuffle convolution module.

6. The method of claim 1, wherein the method is characterized by, Loss function of the road surface crack detection model Loss SIoU is: In the formula, IoU is an intersection over union loss, A is an area of a ground crack real boundary box, and B is an area of a ground crack predicted boundary box. Δ is a distance loss, is a horizontal coordinate of the centroid of the ground crack real bounding box, is a vertical coordinate of the centroid of the ground crack real bounding box, is a horizontal coordinate of the centroid of the ground crack predicted bounding box, is a vertical coordinate of the centroid of the ground crack predicted bounding box, w is a width of the minimum circumscribed rectangle of the ground crack real bounding box and the ground crack predicted bounding box, h is a height of the minimum circumscribed rectangle of the ground crack real bounding box and the ground crack predicted bounding box. Ω is a shape loss, θ is a cost of shape, w is a width of a ground crack prediction bounding box, h is a height of the ground crack prediction bounding box, is a width of a ground crack real bounding box, is a height of the ground crack real bounding box; A is an angle loss, σ is the Euclidean distance between the centroid of the ground crack real bounding box and the centroid of the ground crack predicted bounding box.

Images