Pavement defect detection method based on YOLO multi-scale attention

Abstract

The invention relates to the technical field of pavement defect detection, and discloses a pavement defect detection method based on YOLO multi-scale attention, and the method comprises the following steps: 1, obtaining and processing data; 2, constructing a model for training; and step 3, defect detection. According to the road surface defect detection method based on YOLO multi-scale attention, a global attention mechanism and a multi-scale dynamic feature aggregation module are integrated, so that the perception capability of defect features is enhanced, the spatial feature extraction and channel information fusion process is decoupled, the cooperative improvement of the detection precision and the calculation efficiency is realized, and the detection accuracy and the calculation efficiency are improved. The problems of low efficiency and low precision in actual pavement defect detection can be solved, and the method has universality.

Description

Technical Field

[0001] This invention relates to the field of pavement defect detection technology, and specifically to a pavement defect detection method based on YOLO multi-scale attention. Background Technology

[0002] With the rapid development of road construction and the continuous increase in the mileage of roads open to traffic, the rate of road surface damage is accelerating. Therefore, real-time detection of road surface defects is crucial. Traditional road surface defect detection mainly relies on manual visual inspection, which suffers from low efficiency and high cost, and is no longer sufficient to meet the requirements. As alternatives, machine vision methods, such as Gaussian mixture models, support vector machines, and random forests, are often unsuccessful in terms of accuracy on large-scale datasets. In recent years, deep learning methods based on convolutional neural networks (CNNs) have significantly improved the efficiency of road surface defect identification by automatically extracting image features through multi-level convolution and pooling operations.

[0003] However, standard CNNs still suffer from low detection accuracy when faced with surface defects in complex textures and under varying lighting conditions. Therefore, incorporating modules such as compressed excitation networks and coordinate attention has become a research hotspot due to its ability to better focus on and capture the characteristic regions of defects.

[0004] Because asphalt is commonly used for road surfaces, the long-term effects of natural factors and traffic loads cause a variety of complex defects, which appear as "larger in the foreground and smaller in the background" in the defect images. At the same time, road surface cracks often vary greatly in size.

[0005] As a target detection network in deep learning, YOLO is improved in this invention by designing its internal structure, proposing a pavement defect detection method based on YOLO multi-scale attention (Global attention Mechanism for Deeply Separable YOLO, GMD-YOLO). This method utilizes the deep collaboration between global contextual information and local details to accurately capture minute defects, and then fully integrates multi-scale features. Furthermore, traditional convolution is decoupled into an efficient combination of depthwise convolution and pointwise convolution, significantly reducing computational complexity while ensuring rich representation capabilities, thus meeting the needs of real-time industrial detection. Finally, the effectiveness and reliability of the model are verified during pavement defect detection. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a rationally designed road surface defect detection method based on YOLO multi-scale attention.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] A pavement defect detection method based on YOLO multi-scale attention includes the following steps:

[0009] Step 1: Data Acquisition and Integration

[0010] The dataset of road surface defect images collected on-site was divided, collected, and integrated, and then divided into training set, validation set, and test set according to proportions.

[0011] Step 2: Build and train the model

[0012] A road surface defect detection model based on YOLO multi-scale attention was constructed and trained.

[0013] Step 2.1: Input image data and obtain feature maps according to the convolution process, introducing a dual-module GAM (channel-spatial attention).

[0014] First, the channel attention module identifies the importance of each channel in the input feature map and weights them accordingly. The specific calculation method is as follows:

[0015]

[0016] Where X and X' are the input and permuted feature maps, respectively. For the entire dataset, C is the number of channels, H is the height of the feature map, W is the width of the feature map, and permute means replacing the dimensions of the three-dimensional data from C×H×W to H×W×C;

[0017] The input is then fed into a two-layer perceptron to obtain the channel attention-weighted feature map:

[0018] X reduced =ReLU(Linear(X')) (2)

[0019] X channel_att =Linear(X) reduced (3)

[0020] M c =σ(X) channel_att (4)

[0021] X channel =M c ⊙X channel_att (5)

[0022] Where ReLU represents the activation function, used to introduce nonlinearity. It is the entire dataset, and Linear represents a linear layer; in formula (2), the linear layer reduces the number of channels of the input feature from C to C / r, where r is the reduction rate; the linear layer in formula (3) restores the number of channels to C, X reduced Feature map representing reduced channel count; X channel_att M represents the channel attention weight; c X represents the channel attention weights, σ ​​represents the Sigmoid activation function, and X represents the channel attention weights. channel This represents the feature map after channel attention weighting, and ⊙ represents the element-wise multiplication operation.

[0023] Step 2.2: Put X channel The input to the spatial attention module first passes through a convolutional layer, and the specific calculation formula is as follows:

[0024] X spatial_conv =ReLU(Conv2D(X) channel (6)

[0025] Where Conv2D represents a two-dimensional convolution operation with C / r as the number of output channels, X spatial_conv This represents the feature map after the first convolution;

[0026] Following batch normalization and ReLU activation, the channel count is restored through the same convolutional layer and batch normalized again. Finally, the spatial attention weights M are obtained by normalization using the Sigmoid activation function. s Then, combined with the spatial attention-weighted feature X spatial_att Multiplication yields the final spatial attention-weighted feature X. spatial :

[0027] X spatial_restored =ReLU(BatchNorm2D(X) spatial_conv (7)

[0028] X spatial_att =BatchNorm2D(X spatial_restored (8)

[0029] M s =σ(X) spatial_att (9)

[0030] X spatial =M s ⊙X spatial_att (10)

[0031] Among them, X spatial_restored This represents the feature map after the second convolution, and BatchNorm2D represents the two-dimensional batch normalization operation.

[0032] Step 2.3: Combine large convolutional kernel attention (LKA) to design a multi-scale attention fusion module, inputting feature map F. C That is, X from the previous step spatial Furthermore, cascaded one-dimensional separable convolutional kernels are used to alleviate the high computational cost of large kernel depthwise convolutions. The calculation formula is as follows:

[0033]

[0034] Where * represents convolution, d is the dilation rate, k is the kernel size, and W C Indicates a deep convolution kernel. This indicates a round-down operation. and Z C This represents the output of each depthwise convolution layer;

[0035] Then use a 1×1 convolution kernel W 1×1 Perform convolution to obtain attention map A C The final output feature map Obtained by the Hadamard product of the attention map and the input feature map:

[0036] A C =W 1×1 *Z C (13)

[0037]

[0038] in, Represents the Hadamah altar;

[0039] Step 2.4: Decompose the standard convolution using depthwise separable convolution DSConv:

[0040] N=D C +P C (15)

[0041] D C =D i ×D i ×C in (16)

[0042] P C =C in ×C out (17)

[0043] Where N represents standard convolution, D C and P C D represents depthwise convolution and pointwise convolution, respectively. i C represents the kernel size in the depthwise convolution process. in and C outThese represent the number of input and output channels, respectively.

[0044] Step 2.5: Calculate the error between the detection result and the actual situation using the loss function (Loss) to achieve backpropagation.

[0045] Loss=α1L box +α2L cls (18)

[0046] Where α1 and α2 represent the weighting factors of the corresponding losses, L box and L cls These represent the localization loss and classification loss, respectively.

[0047] Step 3: Defect Detection

[0048] The trained model is used to perform inference in stages to identify specific defect information and assess the confidence level of the results.

[0049] Furthermore, the process of step 1 is as follows:

[0050] Step 1.1: Construct a dataset from the road damage images collected by the vehicle-mounted inspection equipment, defining 7 defect types: transverse cracks, alligator cracks, potholes, block cracks, block repairs, crack repairs, and longitudinal cracks.

[0051] Step 1.2: Use the image annotation tool LabelImg to obtain the location and category labels required for training, and save them in txt format to meet the format requirements of the YOLO input dataset;

[0052] Step 1.3: The dataset is then divided into training set, validation set and test set according to the set ratio.

[0053] Furthermore, step 3 is as follows:

[0054] Step 3.1: Save the weight file after training, and then set the threshold and other parameters required for detection to facilitate subsequent detection;

[0055] Step 3.2: Input the test image dataset into the trained model to perform road surface defect detection, and finally visualize the defect category information and its predicted probability.

[0056] The beneficial effects of this invention are mainly reflected in the following aspects: This invention proposes a road surface defect detection method based on YOLO multi-scale attention. By integrating the global attention mechanism and the multi-scale dynamic feature aggregation module, it enhances the perception ability of defect features and decouples the spatial feature extraction and channel information fusion process, thereby achieving a synergistic improvement in detection accuracy and computational efficiency. It can solve the problems of low efficiency and low accuracy in actual road surface defect detection and has universality and versatility. Attached Figure Description

[0057] Figure 1 This invention is a road surface defect detection framework based on YOLO multi-scale attention;

[0058] Figure 2 This is a schematic diagram of the structure of the GAM module of the present invention;

[0059] Figure 3 This is a schematic diagram of the structure of the multi-scale attention fusion module of the present invention;

[0060] Figure 4 This is a schematic diagram of the structure of the DSConv module of the present invention. Detailed Implementation

[0061] The present invention will be further described below with reference to the accompanying drawings.

[0062] A pavement defect detection method based on YOLO multi-scale attention includes the following steps:

[0063] Step 1: Data Acquisition and Processing

[0064] The data set of road surface defects collected on site was divided, collected, and integrated.

[0065] Step 2: Build and train the model

[0066] A road surface defect detection model based on YOLO multi-scale attention was constructed and trained. The overall framework is shown in the appendix. Figure 1 .

[0067] Step 3: Defect Detection

[0068] The trained model is used to perform inference in stages to identify specific defect information and assess the confidence level of the results.

[0069] Furthermore, the process of step 1 is as follows:

[0070] Step 1.1: Filter the image data collected on site, remove low-quality data such as blurry images, and then integrate them into a defect dataset.

[0071] Step 1.2: Use image augmentation methods to augment the data, and then divide it into training set, validation set and test set according to a set ratio.

[0072] Furthermore, the process of step 2 is as follows:

[0073] Step 2.1: Since traditional convolutional neural networks focus on local features and neglect the interaction between channels and space within the network, a channel-space attention dual module (Global Attention Mechanism, GAM) is introduced (see Appendix). Figure 2 First, the channel attention module weights the input feature map by recognizing the importance of each channel. The specific calculation formula is as follows:

[0074]

[0075] Where X and X' are the input and permuted feature maps, respectively. For the entire dataset, C is the number of channels, H is the height of the feature map, and W is the width of the feature map. `Permute` indicates that the dimensions of the 3D data are replaced from C×H×W to H×W×C. This is then input into a two-layer perceptron.

[0076] X reduced =ReLU(Linear( (2)

[0077] X channel_att =Linear(X) reduce (3)

[0078] In the first perceptron layer, ReLU represents the activation function, used to introduce non-linearity. Linear represents a linear layer; the linear layer in equation (2) reduces the number of channels in the input feature from C to C / r, where r is the reduction rate. The linear layer in equation (3) restores the number of channels to C. reduced Feature map representing a reduced number of channels. X channel_att This represents the channel attention weight.

[0079] Then, the channel attention weights M are obtained by normalization using the Sigmoid activation function. c Finally, it is compared with the original input feature map X. channel_att Multiply to obtain the channel attention-weighted feature X. channel :

[0080] M c =σ(X) channel_att (4)

[0081] X channel =M c ⊙X channe (5)

[0082] Where σ represents the Sigmoid activation function, and ⊙ represents the element-wise multiplication operation.

[0083] Step 2.2: Next, take X after channel attention... channelThe input to the spatial attention module first passes through a convolutional layer, reducing the number of channels from C to C / r to maintain the spatial dimension of the feature map. The specific calculation formula is as follows:

[0084] X spatial_conv =ReLU(Conv2D(X) cha (6)

[0085] Here, Conv2D represents a two-dimensional convolution operation with C / r as the number of output channels. spatial_conv This represents the feature map after the first convolution.

[0086] Following batch normalization and ReLU activation, the channel count is restored through the same convolutional layer and batch normalized again. Finally, the spatial attention weights M are obtained by normalization using the Sigmoid activation function. s Then, combined with the spatial attention-weighted feature X spatial_att Multiplication yields the final spatial attention-weighted feature X. spatial :

[0087] X spatial_restored =ReLU(BatchNorm2D(X) spatial_c (7)

[0088] X spatial_att =BatchNorm2D(X spatial_restored (8)

[0089] M s =σ(X) spatial_att (9)

[0090] X spatial =M s ⊙X spatial_att (10)

[0091] Among them, X spatial_restored This represents the feature map after the second convolution. BatchNorm2D represents the two-dimensional batch normalization operation.

[0092] Step 2.3: To effectively capture multi-scale features, a multi-scale attention fusion module is designed by combining Large Kernel Attention (LKA), as shown in the appendix. Figure 3 Input feature map F C That is, X from the previous step spatial Furthermore, cascaded one-dimensional separable convolutional kernels are used to alleviate the high computational cost of large kernel depthwise convolutions. The calculation formula is as follows:

[0093]

[0094] Among them, * and These represent convolution and Hadamard product, respectively. d is the dilation rate. k is the kernel size, representing the maximum receptive field. W C Indicates a deep convolution kernel. This indicates a floor operation, used for convolution operations on the input feature map. and Z C This represents the output of a depthwise convolution.

[0095] Then use a 1×1 convolution kernel W 1×1 Perform convolution to obtain attention map A C The final output feature map Obtained by the Hadamard product of the attention map and the input feature map:

[0096] A C =W 1×1 *Z C (13)

[0097]

[0098] in, It represents the Hadamaji.

[0099] Step 2.4: To reduce the number of computational parameters and improve training efficiency, an efficient depthwise separable convolution (DSConv) is adopted (see appendix). Figure 4 The standard convolution process can be broken down as follows:

[0100] N=D C +P C (15)

[0101] Where N represents standard convolution. D C and P C These represent depthwise convolution and pointwise convolution, respectively, used to extract spatial and channel features from each input feature map.

[0102] D C =D i ×D i ×C in (16)

[0103] P C =C in ×C out (17)

[0104] Among them, D i This represents the kernel size for depthwise convolution, used for spatial feature extraction in each channel. (C) in and C outThese represent the number of input and output channels, respectively. All channels are merged into an output feature map. This achieves the goal of reducing computational load and improving computational efficiency.

[0105] Step 2.5: To measure the accuracy of the detection, a loss function (Loss) is used to calculate the error between the detection result and the actual situation, thus implementing backpropagation:

[0106] Loss=α1L box +α2L cls (18)

[0107] Where α1 and α2 represent the weighting factors of the corresponding losses. box and L cls These represent localization loss and classification loss, respectively.

[0108] Furthermore, the process of step 3 is as follows:

[0109] Step 3.1: Save the weight file after training, and then set the threshold and other parameters required for detection to facilitate subsequent detection.

[0110] Step 3.2: Input the test image dataset into the trained model to perform road surface defect detection, and finally visualize the defect category information and its predicted probability.

[0111] Example:

[0112] A pavement defect detection method based on YOLO multi-scale attention is described below:

[0113] Step 1: Obtain the road surface defect dataset

[0114] Step 1.1: The ZOYON-RTM vehicle-mounted pavement defect detection equipment was used to acquire pavement image data. Its two-dimensional pavement damage detection module consists of a linear array camera with a resolution of 2048 pixels / line and an infrared laser pavement auxiliary lighting system, ensuring all-weather, day and night detection of pavement defects. Typical and frequently occurring pavement defects of Class I, II, and III highways in Hangzhou and Jiaxing were collected, and a dataset of pavement damage images was constructed, defined into 7 defect types: 1442 transverse crack images, 614 alligator crack images, 756 pothole images, 1242 block crack images, 684 block repair images, 794 strip repair images, and 1097 longitudinal crack images.

[0115] Step 1.2: Since the images used for model training must be of consistent size, the image annotation tool LabelImg is used to obtain the location and category labels required for training, and the data is saved in txt format to meet the format requirements of the YOLO input dataset.

[0116] Step 1.3: The dataset is then divided into training, validation and test sets in a 6:2:2 ratio.

[0117] Step 2: Train the road surface defect detection model

[0118] Step 2.1: Obtain feature maps from the original image through a convolution process. Identify the importance of each channel in the input feature map based on the channel attention mechanism, assign greater weights to important channel features, and finally normalize the channel feature weight values ​​to around 0.015.

[0119] Step 2.2: Determine the importance of each spatial location in the feature map based on the spatial attention mechanism (spatial features are highlighted in the weight range of -0.02 to 0.02) in order to focus on defective areas of the image.

[0120] Step 2.3: In the multi-scale attention fusion module, large convolutional kernels and attention mechanisms are combined to capture defects based on global information and effectively aggregate multi-scale defect features.

[0121] Step 2.4: Optimize the fusion convolution module, reduce running parameters, and improve model computation efficiency.

[0122] Step 2.5: Calculate the error based on the loss function and perform iterative optimization, reducing the error from 3.38 to 0.67.

[0123] Step 3: Use the test set data for defect detection

[0124] Step 3.1: Input the test image dataset into the trained model for detection and obtain the detection results.

[0125] Step 3.2: Use the original single-stage detection model as a baseline for comparison. To quantify the model performance, mean average precision (mAP) is used as the evaluation metric.

[0126]

[0127] Where AP represents the detection precision of a single category, P(r) represents the precision corresponding to a specific recall rate r, mAP represents the mean precision of all categories, and C represents the total number of categories.

[0128] The specific results are shown in Table 1 below;

[0129] Table 1. mAP results for each model regarding pavement defect measurement.

[0130]

[0131] The method of this invention adopts a road defect detection method based on YOLO multi-scale attention, which improves the detection performance of the model and can solve the problems of low efficiency and low accuracy in actual road defect detection. It has universality and versatility.

[0132] The embodiments described in this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered as limited to the specific forms stated in the embodiments. The scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A pavement defect detection method based on YOLO multi-scale attention, characterized in that, Includes the following steps: Step 1: Data Acquisition and Integration The dataset of road surface defect images collected on-site was divided, collected, and integrated, and then divided into training set, validation set, and test set according to proportions. Step 2: Build and train the model A road surface defect detection model based on YOLO multi-scale attention was constructed and trained. Step 2.1: Input image data and obtain feature maps according to the convolution process, introducing a dual-module GAM (channel-spatial attention). First, the channel attention module identifies the importance of each channel in the input feature map and weights them accordingly. The specific calculation method is as follows: Where X and X' are the input and permuted feature maps, respectively. For the entire dataset, C is the number of channels, H is the height of the feature map, W is the width of the feature map, and permute means replacing the dimensions of the three-dimensional data from C×H×W to H×W×C; The input is then fed into a two-layer perceptron to obtain the channel attention-weighted feature map: X reduced =ReLU(Linear(X')) (2) X channel_att =Linear(X reduced ) (3) M c =σ(X channel_att ) (4) X channel =M c ⊙X channel_att (5) Where ReLU represents the activation function, used to introduce nonlinearity. It is the entire dataset, and Linear represents a linear layer; in formula (2), the linear layer reduces the number of channels of the input feature from C to C / r, where r is the reduction rate; the linear layer in formula (3) restores the number of channels to C, X reduced Feature map representing reduced channel count; X channel_att M represents the channel attention weight; c X represents the channel attention weights, σ ​​represents the Sigmoid activation function, and X represents the channel attention weights. channel This represents the feature map after channel attention weighting, and ⊙ represents the element-wise multiplication operation. Step 2.2: Put X channel The input to the spatial attention module first passes through a convolutional layer, and the specific calculation formula is as follows: X spatial_conv =ReLU(Conv2D(X channel )) (6) Where Conv2D represents a two-dimensional convolution operation with C / r as the number of output channels, X spatial_conv This represents the feature map after the first convolution; Following batch normalization and ReLU activation, the channel count is restored through the same convolutional layer and batch normalized again. Finally, the spatial attention weights M are obtained by normalization using the Sigmoid activation function. s Then, combined with the spatial attention-weighted feature X spatial_att Multiplication yields the final spatial attention-weighted feature X. spatial : X spatial_restored <ReLU(BatchNorm2D(X spatial_conv )) (7) X spatial_att =BatchNorm2D(X spatial_restored ) (8) M s =σ(X spatial_att ) (9) X spatial =M s ⊙X spatial_att (10) Among them, X spatial_restored This represents the feature map after the second convolution, and BatchNorm2D represents the two-dimensional batch normalization operation. Step 2.3: Combine large convolutional kernel attention (LKA) to design a multi-scale attention fusion module, inputting feature map F. C That is, X from the previous step spatial Furthermore, cascaded one-dimensional separable convolutional kernels are used to alleviate the high computational cost of large kernel depthwise convolutions. The calculation formula is as follows: Where * represents convolution, d is the dilation rate, k is the kernel size, and W C Indicates a deep convolution kernel. This indicates a round-down operation. and Z C This represents the output of each depthwise convolution layer; Then use a 1×1 convolution kernel W 1×1 Perform convolution to obtain attention map A C The final output feature map Obtained by the Hadamard product of the attention map and the input feature map: AND C =In 1×1 *WITH C (13) in, Represents the Hadamah altar; Step 2.4: Decompose the standard convolution using depthwise separable convolution DSConv: N=D C +P C (15) D C =D i ×D i ×C in (16) P C =C in ×C out (17) Where N represents standard convolution, D C and P C D represents depthwise convolution and pointwise convolution, respectively. i C represents the kernel size in the depthwise convolution process. in and C out These represent the number of input and output channels, respectively. Step 2.5: Calculate the error between the detection result and the actual situation using the loss function (Loss) to achieve backpropagation. Loss=α1L box +α2L cls (18) Where α1 and α2 represent the weighting factors of the corresponding losses, L box and L cls These represent the localization loss and classification loss, respectively. Step 3: Defect Detection The trained model is used to perform inference in stages to identify specific defect information and assess the confidence level of the results.

2. The road surface defect detection method based on YOLO multi-scale attention as described in claim 1, characterized in that, The process of step 1 is as follows: Step 1.1: Construct a dataset from the road damage images collected by the vehicle-mounted inspection equipment, defining 7 defect types: transverse cracks, alligator cracks, potholes, block cracks, block repairs, crack repairs, and longitudinal cracks. Step 1.2: Use the image annotation tool LabelImg to obtain the location and category labels required for training, and save them in txt format to meet the format requirements of the YOLO input dataset; Step 1.3: The dataset is then divided into training set, validation set and test set according to the set ratio.

3. The road surface defect detection method based on YOLO multi-scale attention as described in claim 1, characterized in that, The process of step 3 is as follows: Step 3.1: Save the weight file after training, and then set the threshold and other parameters required for detection to facilitate subsequent detection; Step 3.2: Input the test image dataset into the trained model to perform road surface defect detection, and finally visualize the defect category information and its predicted probability.

Images