Multi-scale pavement disease detection method based on YOLOv8
By introducing the hollow space pyramid pooling structure and the space-time cross-attention mechanism in the YOLOv8 detection method, combined with the weighted two-way feature pyramid network, the problem of insufficient multi-scale feature extraction in the existing technology is solved, and more efficient pavement disease detection is achieved.
Patent Information
- Application Number
- CN202411798041.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-09
- Publication Date
- 2025-03-18
AI Technical Summary
The prior art is difficult to effectively deal with multi-scale features in road defect detection, resulting in inaccurate positioning and incomplete identification.
Multi-scale pavement disease detection method based on YOLOv8 is adopted to extract multi-scale features through the hollow space pyramid pooling structure, and combine the space-time cross attention mechanism and weighted bidirectional feature pyramid network to achieve multi-scale integration and fusion of features.
It improves detection accuracy and efficiency, can quickly and accurately identify and locate road surface diseases, and provides road management departments with more reliable road health data.
Smart Images
Figure CN119649058A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of road surface monitoring, and in particular relates to a multi-scale road surface disease detection method based on YOLOv8. Background Art
[0002] The safety of vehicle driving is largely affected by the condition of the road surface. Road defects can lead to further deterioration of the road surface, which has an adverse impact on vehicle safety and road durability. Traditionally, road defect detection relies on manual inspection, traditional image processing methods, and machine learning methods. However, these methods often lack efficiency and reliability.
[0003] With the advancement of information technology and deep learning technology, convolutional neural networks (CNNs) have made great progress in the field of image recognition. In recent years, deep learning has been widely used in the field of road defect detection. However, current research often only focuses on a small number of defect types, without fully considering the diversity and scale differences of road defects in the real world. The multi-scale characteristics of road defects often expose the limitations of existing detection networks in multi-scale feature extraction, resulting in inaccurate positioning and incomplete recognition. Summary of the invention
[0004] The purpose of this invention is to provide a multi-scale pavement disease detection method based on YOLOv8 to improve detection accuracy and efficiency and provide more reliable road health data for highway management departments.
[0005] The purpose of the present invention is to achieve the following technical means: a multi-scale pavement disease detection method based on YOLOv8, comprising the following steps:
[0006] Image box selection: cluster the input image data to obtain the aiming point box, i.e. the real box;
[0007] Backbone feature extraction: Through the backbone feature extraction network of the YOLOv8 algorithm, the features of the input image are compressed in high dimensions to obtain a compressed feature map;
[0008] The pooling module uses a dilated spatial pyramid pooling structure to extract multi-scale feature maps from the compressed feature maps through dilated convolutions of different scales, and then concatenates, integrates and reduces the dimensions of feature maps of different scales to obtain pooled feature maps.
[0009] The spatiotemporal cross attention module convolves the spatiotemporal cross attention of the pooled feature map with the pooled feature map through the spatiotemporal cross attention mechanism to obtain weighted multi-scale features, thereby capturing the complex temporal and spatial dependencies of the pooled feature map.
[0010] Feature fusion network: inputs weighted multi-scale features into weighted bidirectional feature pyramid network, sets learnable weights for each input feature, and transmits bidirectional information from top to bottom and bottom to top to obtain fused multi-scale features;
[0011] Prediction box selection: YOLOv8’s detection head classifies and regresses the fused multi-scale features output by the weighted bidirectional pyramid network to generate a prediction box.
[0012] Loss function, using SIoU loss function, compares the predicted box and the real box, so as to regress the bounding box of the predicted box. The loss function is used as the objective function in the direction of model parameter optimization. When the loss value reaches the minimum and converges, it means that the predicted box is closer to the real box.
[0013] Image prediction uses prediction parameters to extract the input image backbone features, pooling module, spatiotemporal cross attention module, feature fusion network and prediction box selection steps to obtain the prediction box.
[0014] Specifically, the pooling module uses 1x1 convolution to extract fine-grained features of the compressed feature map, and then uses 3x3 convolution in sequence; and sets the expansion rates to 6, 12, and 18 respectively to capture features of different scales of the compressed feature map; then, the global context information of the compressed feature map is obtained through the global average pooling layer; finally, these feature maps of different scales are spliced, and integrated and reduced in dimension through a 1x1 convolution layer to generate the final pooled feature map.
[0015] The spatiotemporal cross attention module is specifically:
[0016] Temporal attention: The input pooled feature map is expanded and processed by a linear layer, and then mapped to the query, key, and value space. The attention weight is calculated by the Softmax function. The attention weight is convolved with the value space to obtain the temporal attention to capture the dependency between different time points.
[0017] Spatial attention: convolution operation is performed on the input pooled feature map to generate attention map directly on the spatial scale to highlight the key spatial features in the image;
[0018] Temporal attention and spatial attention are multiplied to obtain spatiotemporal cross attention, which is multiplied with the original input, i.e., the pooled feature map, to obtain weighted multi-scale features.
[0019] Specifically, the feature fusion network strengthens the semantic information of low-level features from the top-down path for the input multi-scale features, and supplements the spatial detail information for high-level features through the bottom-up path, and assigns dynamic learnable weight parameters to the multiple input features, including the input weight W in , output weight Wout And the bidirectional fusion weight W sd , where W in Controls the contribution of each input feature to the fusion result, W out determines the impact of fusion features on subsequent layers, while W sd It is used to balance the interaction of two-way information in the fusion process.
[0020] The prediction box is selected by convolving and normalizing the fused multi-scale features output by the feature fusion network, and then classifying and regressing the real box. The classification branch inside the detection head predicts the target probability, and the regression branch predicts the position of the bounding box. Finally, the prediction box with the largest target probability is screened out as the final prediction box through non-maximum suppression.
[0021] The loss function is,
[0022]
[0023] Among them, δ controls the network's attention to shape loss;
[0024]
[0025] (b cx , b cy ) represents the center point coordinates of the real box and the predicted box; c w 、c h Respectively represent the width and height of the minimum outer rectangle of the two boxes;
[0026]
[0027] ∧ represents the vector angle loss; h represents the height difference between the center points of the real box and the predicted box; l represents the distance between the center points of the two boxes.
[0028] The ground truth boxes are annotated using the K-means++ clustering algorithm.
[0029] The real frame is specifically marked as,
[0030] (1) Select the first center point and randomly select a data point from the data set as the first center point u1;
[0031] (2) Select the subsequent center point, for each data point x i , excluding the selected center point, calculate its distance D(x i ), which is defined as follows
[0032]
[0033] Use the square of the distance as the weight and select the next center point according to the probability distribution, which is defined as follows
[0034]
[0035] Where: P(x i ) is to select the data point x i The probability of being the next center point.
[0036] (3) Repeat the selection of center points and step (2) until K center points are selected;
[0037] (4) Perform standard K-means clustering using K initial center points and iteratively update the center points until convergence.
[0038] The beneficial effects of the present invention are: using the YOLOv8 algorithm as the main framework, using ASPP (atrous spatial pyramid pooling) as the pooling structure, extracting multi-scale features through atrous convolutions of different scales; realizing effective interaction and enhancement of time domain and spatial domain information through the spatiotemporal cross attention mechanism; using the weighted bidirectional feature pyramid network as the feature fusion network to realize multi-scale feature integration between different levels; using the K-means++ algorithm to generate accurate anchor boxes; in terms of loss function, using SIoU (smoothed intersection over union) as the loss function to improve the accuracy and robustness of target box prediction. In actual road inspections, the present invention can quickly and accurately identify and locate road surface diseases, provide accurate road health information for highway management agencies, and assist managers in making scientific and efficient maintenance decisions, thereby improving the safety of vehicle driving, extending the service life of roads, and reducing highway maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a schematic diagram of the structure of a multi-scale target detection algorithm based on YOLOv8;
[0040] Figure 2 This is a schematic diagram of the hollow space pyramid pooling structure;
[0041] Figure 3 Schematic diagram of the spatiotemporal cross attention module structure;
[0042] Figure 4 Schematic diagram of the weighted bidirectional feature pyramid network structure
[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. DETAILED DESCRIPTION
[0044] [Example 1]
[0045] like Figure 1As shown, a multi-scale pavement disease detection method based on YOLOv8 includes the following steps:
[0046] Image box selection: cluster the input image data to obtain the aiming point box, i.e. the real box;
[0047] First, you need to annotate the image to be recognized and mark the aiming point box as the true box for subsequent loss function regression.
[0048] The ground truth boxes are annotated using the K-means++ clustering algorithm.
[0049] The K-means++ algorithm first randomly selects K data points as the initial center points. Next, it determines which cluster each data point should be assigned to by calculating the distance from each data point to the cluster to which the nearest center point belongs. Then, it calculates the mean of all data points in each cluster and updates the mean as the new center point. This process is repeated until the position of the center point changes very little or the preset maximum number of iterations is reached. After clustering, the K-means++ algorithm provides the aspect ratios of the K aiming boxes, which represent the target objects detected by the algorithm at different scales.
[0050] The real frame is specifically marked as,
[0051] (1) Select the first center point and randomly select a data point from the data set as the first center point u1;
[0052] (2) Select the subsequent center point, for each data point x i , excluding the selected center point, calculate its distance D(x i ), which is defined as follows
[0053]
[0054] Use the square of the distance as the weight and select the next center point according to the probability distribution, which is defined as follows
[0055]
[0056] Where: P(x i ) is to select the data point x i The probability of being the next center point.
[0057] (3) Repeat the selection of center points and step (2) until K center points are selected;
[0058] (4) Perform standard K-means clustering using K initial center points and iteratively update the center points until convergence.
[0059] Backbone feature extraction: Through the backbone feature extraction network of the YOLOv8 algorithm, the features of the input image are compressed in high dimensions to obtain a compressed feature map;
[0060] The input image without a true box is sent to the YOLOv8 algorithm, and the image is compressed in high dimension to obtain a compressed feature map. Specifically, the unknown image is used as the input of the model, and the backbone feature extraction network compresses the input image scale in multiple channel dimensions through a series of convolutions with different convolution kernel sizes, so that each compressed feature point can accurately respond to the unknown feature area, and the multiple channels formed can improve the utilization rate of the compressed features.
[0061] like Figure 2 As shown in the figure, the pooling module uses the atrous spatial pyramid pooling structure (ASPP) to extract multi-scale feature maps through atrous convolutions of different scales for the compressed feature maps, and then concatenates, integrates and reduces the dimensions of the feature maps of different scales to obtain the pooled feature maps;
[0062] ASPP (Atrous Spatial Pyramid Pooling) is the pooling structure of the algorithm. It extracts multi-scale features through atrous convolutions of different scales and enhances the contextual information of feature maps.
[0063] Specifically, the ASPP module processes the feature map using multiple dilated convolutions with different expansion rates to extract feature information of different scales. These feature maps of different scales are concatenated to form a rich multi-scale feature representation. The main purpose of ASPP is to capture global context information while maintaining the resolution of the feature map and enhance the model's ability to detect objects of various scales.
[0064] Specifically, the pooling module uses 1x1 convolution (conv) to extract fine-grained features of the compressed feature map, and then uses 3x3 convolution in sequence; and sets the expansion rate (rate) to 6, 12, and 18 respectively to capture features of different scales of the compressed feature map; then, the global context information of the compressed feature map is obtained through the global average pooling layer (image pooling); these five steps are parallel steps, and finally, these feature maps of different scales are spliced, and integrated and reduced in dimension through a 1x1 convolution layer to generate the final pooled feature map.
[0065] The spatiotemporal cross attention module convolves the spatiotemporal cross attention of the pooled feature map with the pooled feature map through the spatiotemporal cross attention mechanism to obtain weighted multi-scale features, thereby capturing the complex temporal and spatial dependencies of the pooled feature map.
[0066] The spatiotemporal cross attention mechanism can effectively capture complex dependencies in time and space, and capture complex multi-scale feature dependencies by modeling the interaction between feature graphs in time and space. By modeling the spatiotemporal features of feature graphs, the ability to capture complex multi-scale features is significantly improved, thereby enhancing the algorithm's attention to important features during the detection process.
[0067] like Figure 3 As shown, the spatiotemporal cross attention module is specifically:
[0068] Temporal attention: The input pooled feature map is flattened and processed by a linear layer, and then mapped to the query, key, and value space to calculate the temporal attention (QK**B / sqrt(c)). The attention weight is calculated by the Softmax function.
[0069] The attention weight is spatially convolved with the value (v) to obtain temporal attention to capture the dependencies between different time points;
[0070] Spatial attention: convolution operation is performed on the input pooled feature map to generate attention map directly on the spatial scale to highlight the key spatial features in the image;
[0071] The temporal attention and the spatial attention are multiplied (conv) to obtain the spatiotemporal cross attention, which is multiplied with the original input, i.e., the pooled feature map, to obtain the weighted multi-scale features (where B, H, W, and C are the dimensions of the features).
[0072] like Figure 4 As shown in the figure, the feature fusion network inputs the weighted multi-scale features into the weighted bidirectional feature pyramid network, sets a learnable weight for each input feature, and transmits bidirectional information from top to bottom and bottom to top to obtain fused multi-scale features;
[0073] The core idea of using a weighted bidirectional feature pyramid network is to optimize the fusion process between multi-scale features and realize bidirectional information transfer by introducing learnable weights based on the traditional feature pyramid network (FPN). Specifically, P1 to P5 are high-level features and low-level features from the backbone feature extraction network, respectively. The lower direction indicates the transfer of high-level features, and the upper direction indicates the transfer of low-level features. In the fusion process, features of different resolutions contribute differently to the fusion effect. Therefore, a learnable weight W is set for each input to jointly determine the result of feature fusion. This method will be dynamically adjusted according to the training of the network to ensure that features of different scales can be reasonably weighed in the fusion process, thereby improving the final detection performance;
[0074] Specifically, the feature fusion network strengthens the semantic information of low-level features from the top-down path for the input multi-scale features, and supplements the spatial detail information for high-level features through the bottom-up path, and assigns dynamic learnable weight parameters to the multiple input features, including the input weight W in , output weight W out And the bidirectional fusion weight W sd , where W in Controls the contribution of each input feature to the fusion result, W out determines the impact of fusion features on subsequent layers, while W sd It is used to balance the interaction of two-way information in the fusion process.
[0075] P1 to P5 represent multi-scale feature maps from the backbone feature extraction network, where P1 represents high-resolution but low-semantic features, and P5 represents low-resolution but high-semantic features. The network transmits information in both directions, strengthening the semantic information of low-level features from the top-down path (from P5 to P1), and supplementing the spatial detail information of high-level features through the bottom-up path (from P1 to P5). The core fusion process is to assign dynamic learnable weight parameters to multiple input features, including input weight W in , output weight W out And the bidirectional fusion weight W sd , where W in Controls the contribution of each input feature to the fusion result, W out determines the impact of fusion features on subsequent layers, while W sd It is used to balance the interaction of bidirectional information in the fusion process. Through network training, the weight parameters can be dynamically adjusted according to task requirements to ensure the rationality of feature fusion. In particular, through the bidirectional pyramid design, full utilization of features of different resolutions can be achieved, which solves the limitation that the fixed weighting method in traditional FPN cannot adapt to multiple tasks. In addition, this dynamic fusion strategy shows significant performance advantages in small target detection and cross-scale target detection tasks because it can effectively weigh the detailed expression of high-resolution features with the global semantic information of low-resolution features. This design enables the weighted bidirectional feature pyramid network to show higher robustness and better detection effect in target detection tasks, especially in scenarios with strong multi-scale fusion requirements, the performance improvement is particularly obvious.
[0076] Specifically, Figure 4 As shown in Figure 1, the spatial detail information is supplemented for the high-level features through a bottom-up path (from P1 to P5). The calculation method of the output of branches P1 to P5 is similar. Taking branch P2 as an example, P1 and the input weight W 1in Multiply, the input of P2 and the weight W 2inMultiply them together, and then multiply the result of the two multiplications to get W 2sd ; W 2sd Upward, that is, P3 transfer;
[0077] The top-down path (from P5 to P1) strengthens the semantic information of low-level features, W 4sd With W 5in After convolution, we get W 5out , and pass it downward, that is, W 4in , W 4sd and W 5out Convolution gets W 4out , p2 and p3 are the same, since there is no W in P1 1sd , directly W 1in and W 2out Convolution gets W 1out .
[0078] Prediction box selection: YOLOv8’s detection head classifies and regresses the fused multi-scale features output by the weighted bidirectional pyramid network to generate a prediction box.
[0079] The prediction box is selected by selecting the prediction box output by the detection head and finally selecting the prediction box with the highest target probability as the final prediction box through non-maximum suppression.
[0080] The loss function uses the SIoU loss function to compare the predicted box with the true box, and then regress the predicted box. The loss function is the objective function in the direction of model parameter optimization. When the loss value reaches the minimum and converges, it means that the predicted box is closer to the true box.
[0081] The loss function uses the SIoU loss function. On the basis of considering the overlap between the real box and the predicted box, the distance between the center points of the two boxes, and the aspect ratio, the vector angle between the two boxes will affect the accurate and fast regression of the predicted box, improve the regression accuracy and efficiency of the predicted box, compare the predicted box with the real box, and regress the bounding box of the predicted box to make the predicted box closer to the real box until the accuracy requirements are met. The parameters of the backbone feature extraction, pooling module, spatiotemporal cross attention module, feature fusion network and prediction box selection are used as prediction parameters;
[0082] The loss function is,
[0083]
[0084] Among them, δ controls the network's attention to shape loss;
[0085]
[0086] (bcx , b cy ) represents the center point coordinates of the real box and the predicted box; c w 、c h Respectively represent the width and height of the minimum outer rectangle of the two boxes;
[0087]
[0088] ∧ represents the vector angle loss; h represents the height difference between the center points of the real box and the predicted box; l represents the distance between the center points of the two boxes.
[0089] IoU stands for Intersection over Union. IoU is a measure of the accuracy of detecting corresponding objects in a specific data set. Any task that results in a predicted bounding box in the output can be measured using IoU.
[0090] Image prediction uses prediction parameters to extract the input image backbone features, pooling module, spatiotemporal cross attention module, feature fusion network and prediction box selection steps to obtain the prediction box.
[0091] Finally, the parameters used for this prediction are adopted, and the above steps are followed (no need to mark the real frame), namely backbone feature extraction, pooling module, spatiotemporal cross attention module, feature fusion network and prediction frame selection steps. The same parameters are used to predict the prediction frame for the input image, and the pavement disease detection result can be obtained.
Claims
1. A multi-scale pavement disease detection method based on YOLOv8, characterized by: The following steps are included: Image box selection: cluster the input image data to obtain the aiming point box, i.e. the real box; Backbone feature extraction: Through the backbone feature extraction network of the YOLOv8 algorithm, the features of the input image are compressed in high dimensions to obtain a compressed feature map; The pooling module uses a dilated spatial pyramid pooling structure to extract multi-scale feature maps from the compressed feature maps through dilated convolutions of different scales, and then concatenates, integrates and reduces the dimensions of feature maps of different scales to obtain pooled feature maps. The spatiotemporal cross attention module convolves the spatiotemporal cross attention of the pooled feature map with the pooled feature map through the spatiotemporal cross attention mechanism to obtain weighted multi-scale features, thereby capturing the complex temporal and spatial dependencies of the pooled feature map. Feature fusion network: inputs weighted multi-scale features into weighted bidirectional feature pyramid network, sets learnable weights for each input feature, and transmits bidirectional information from top to bottom and bottom to top to obtain fused multi-scale features; Prediction box selection: YOLOv8’s detection head classifies and regresses the fused multi-scale features output by the weighted bidirectional pyramid network to generate a prediction box. Loss function, using SIoU loss function, compares the predicted box and the real box, so as to regress the predicted box. The loss function is used as the objective function in the direction of model parameter optimization. When the loss value reaches the minimum and converges, it means that the predicted box is closer to the real box. Image prediction uses prediction parameters to extract the input image backbone features, pooling module, spatiotemporal cross attention module, feature fusion network and prediction box selection steps to obtain the prediction box.
2. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: Specifically, the pooling module uses 1x1 convolution to extract fine-grained features of the compressed feature map, and then uses 3x3 convolution in sequence; and sets the expansion rates to 6, 12, and 18 respectively to capture features of different scales of the compressed feature map; then, the global context information of the compressed feature map is obtained through the global average pooling layer; finally, these feature maps of different scales are spliced, and integrated and reduced in dimension through a 1x1 convolution layer to generate the final pooled feature map.
3. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: The spatiotemporal cross attention module is specifically: Temporal attention: The input pooled feature map is expanded and processed by a linear layer, and then mapped to the query, key, and value space. The attention weight is calculated by the Softmax function. The attention weight is convolved with the value space to obtain the temporal attention to capture the dependency between different time points. Spatial attention: convolution operation is performed on the input pooled feature map to generate attention map directly on the spatial scale to highlight the key spatial features in the image; Temporal attention and spatial attention are multiplied to obtain spatiotemporal cross attention, which is multiplied with the original input, i.e., the pooled feature map, to obtain weighted multi-scale features.
4. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: Specifically, the feature fusion network strengthens the semantic information of low-level features from the top-down path for the input multi-scale features, and supplements the spatial detail information for high-level features through the bottom-up path, and assigns dynamic learnable weight parameters to the multiple input features, including the input weight W in , output weight W out And the bidirectional fusion weight W sd , where W in Controls the contribution of each input feature to the fusion result, W out determines the impact of fusion features on subsequent layers, while W sd It is used to balance the interaction of two-way information in the fusion process.
5. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: The prediction box is selected by convolving and normalizing the fused multi-scale features output by the feature fusion network, and then classifying and regressing the real box. The classification branch inside the detection head predicts the target probability, and the regression branch predicts the position of the bounding box. Finally, the prediction box with the largest target probability is screened out as the final prediction box through non-maximum suppression.
6. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: The loss function is, Among them, δ controls the network's attention to shape loss; (b cx , b cy ) represents the center point coordinates of the real box and the predicted box; c w 、c h Respectively represent the width and height of the minimum outer rectangle of the two boxes; ∧ represents the vector angle loss; h represents the height difference between the center points of the real box and the predicted box; l represents the distance between the center points of the two boxes.
7. The multi-scale pavement damage detection method based on YOLOv8 according to claim 1 is characterized in that: The ground truth boxes are annotated using the K-means++ clustering algorithm.
8. The multi-scale pavement damage detection method based on YOLOv8 according to claim 7 is characterized in that: The real frame is specifically marked as, (1) Select the first center point and randomly select a data point from the data set as the first center point u1; (2) Select the subsequent center point, for each data point x i , excluding the selected center point, calculate its distance D(x i ), which is defined as follows Use the square of the distance as the weight and select the next center point according to the probability distribution, which is defined as follows Where: P(x i ) is to select the data point x i The probability of being the next center point. (3) Repeat the selection of center points and step (2) until K center points are selected; (4) Perform standard K-means clustering using K initial center points and iteratively update the center points until convergence.