A method for detecting road surface defects in rapid highway images based on deep fusion convolutional neural networks.

Abstract

This invention proposes a method for detecting pavement defects in highway fast-detection images based on a deep fusion convolutional neural network, comprising: constructing a pavement defect image annotation dataset for highway fast-detection images; constructing a basic model for pavement defect detection using a simple YOLOv8-RSDD-HRII model and a deep fusion model using YOLOv8-CBAM-RSDD-HRII; constructing a secondary fusion network model for pavement defect detection using Mobile-Net-YOLOv8-CBAM-RSDD-HRII; obtaining the detection box coordinates; and constructing an efficiency network for pavement defect detection using a deep fusion convolutional neural network. The Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model is used to obtain the detection box coordinates and construct a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for pavement defect detection in highway rapid inspection images. The detection box coordinates generated by the sub-fusion network model are obtained in parallel and then integrated and learned to construct a deep fusion convolutional neural network model DFCNN-RSDD-HRII for pavement defect detection in highway rapid inspection images. This model can more accurately identify pavement defects in highway rapid inspection images and effectively improve highway maintenance efficiency.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent transportation and road maintenance, and specifically relates to a method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks. Background Technology

[0002] With the continuous improvement of my country's highway network and the expansion of road scale, more and more road maintenance problems have emerged, the most common of which is road surface defects. After the road surface is damaged, repeated vehicle traffic will cause serious damage to the road and even damage the roadbed. Therefore, the maintenance and management of road surface is particularly important. If road surface defects can be detected and reported in a timely manner, the efficiency of highway maintenance in my country can be effectively improved, maintenance costs can be reduced, and the safety and service life of my country's highways can be improved.

[0003] In the field of highway pavement distress detection, traditional manual or semi-automated methods rely primarily on road maintenance personnel for exploration, measurement, and evaluation analysis. This approach is time-consuming, inaccurate, and susceptible to human error, increasing the safety of maintenance workers during on-site operations. However, with the continuous development and innovation of computer science and technology, machine learning and deep learning technologies can be trained on large amounts of highway pavement distress image data to automatically learn and detect different types of pavement distress. This improves the accuracy and speed of pavement distress detection. Furthermore, video sensors and other equipment can be installed on vehicles to collect pavement data in real time, further enhancing the timeliness of pavement distress detection.

[0004] Compared to single convolutional neural networks, deeply fused convolutional neural networks can achieve higher accuracy and significantly improve the overall performance of the model. Therefore, this invention proposes a method for detecting pavement defects in highway rapid detection images based on deeply fused convolutional neural networks. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing technology, this invention provides a method for detecting pavement defects in highway rapid detection images based on deep fusion convolutional neural networks. Based on the original YOLOv8 network model, the method first improves its feature extraction network by fusing and integrating it, and then uses deep fusion convolutional neural networks to detect pavement defects in highway rapid detection images. This method can provide certain technical support for the identification and classification of pavement defect information.

[0006] Technical Solution: To achieve the above objectives, this invention provides a method for detecting pavement defects in highway rapid detection images based on deep fusion convolutional neural networks, comprising the following steps:

[0007] S1: Construct a dataset of labeled road surface defects from rapid highway inspection images;

[0008] S2: Construct a simple YOLOv8-RSDD-HRII model for detecting pavement defects in highway rapid detection images;

[0009] S3: Construct a YOLOv8-CBAM-RSDD-HRII deep fusion basic model for road surface defect detection in highway rapid detection images;

[0010] S4: Construct a Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes.

[0011] S5: Construct an Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in rapid highway image detection, and obtain the coordinates of the detection boxes.

[0012] S6: Construct a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes.

[0013] S7: Integrate the coordinates of the detection boxes generated by the three sub-fusion network models to construct a deep fusion convolutional neural network model DFCNN-RSDD-HRII for detecting pavement defects in highway rapid detection images.

[0014] Furthermore, the specific method for constructing the highway rapid detection image pavement defect image annotation dataset in step S1 is as follows:

[0015] S1-1: First, acquire the original images of the road surface for rapid inspection. The collected image data are images taken by the data collection vehicle on the Baolong G5XY section of the highway from August 18 to 21, 2022, at a constant speed, with a fixed shooting time interval, and the angle between the area array video sensor and the horizontal ground is 30°. The resolution of the collected road surface images for rapid inspection is 2064*1544.

[0016] S1-2: From the 18,713 original highway pavement images collected, images with obvious pavement distress characteristics were selected, mainly including transverse cracks and longitudinal cracks. Images with unclear pavement distress characteristics or overly complex and messy pavement distress characteristics were deleted, resulting in a total of 4,000 effective highway pavement distress images.

[0017] S1-3: LabelImg software was used to annotate the road surface defects images of the highway rapid detection. A segmented annotation method was adopted to ensure that the road surface defect information in each segment was consistent. The dataset label name is: {Transverse,Longitudinal}.

[0018] S1-4: After annotation, a dataset of 4000 highway pavement defect annotations based on highway fast detection images was obtained, including 11892 transverse cracks and 3495 longitudinal cracks, with a total of 15387 labels. The 4000 annotated highway pavement images were then randomly divided into training, validation, and test sets in a ratio of 8:1:1.

[0019] Furthermore, a simple YOLOv8-RSDD-HRII model for detecting pavement defects in highway rapid detection images is constructed in step S2.

[0020] Furthermore, the specific method for constructing the YOLOv8-CBAM-RSDD-HRII deep fusion basic model for road surface defect detection in step S3 is as follows: the attention mechanism can be used to optimize the backbone, i.e., the feature extraction part, of the convolutional neural network, thereby increasing the detection effect and efficiency of the target detection model. The more and more significant the defect features contained in the road surface image of the highway rapid detection, the higher the accuracy of the network model in detection. Based on this, a convolutional block attention module is adopted. The YOLOv8 network is optimized using the CBAM module to construct a deep fusion foundation model, YOLOv8-CBAM-RSDD-HRII, for road surface defect detection in highway rapid detection images. The specific optimization steps are as follows: Based on the original YOLOv8-RSDD-HRII network model, a CBAM module is added after the C2f module in its backbone. Depending on the feature map, the CBAM module after the first C2f module has parameters of 128×7, the second C2f module has parameters of 256×7, and the third C2f module has parameters of 512×7. The first variable is the number of channels in the feature map of that layer, and the second variable is the size of the convolution kernel of the spatial attention mechanism in the CBAM attention module. For example, the CBAM module with parameters of 128×7 has a convolution kernel size of 7×7 and 128 channels. Other parts are consistent with the YOLOv8-RSDD-HRII model.

[0021] Furthermore, in step S4, a Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model is constructed for detecting minor road surface defects in highway rapid detection images to obtain the coordinates of the detection boxes. The specific method is as follows: Feature fusion is performed using the Backbone network of Mobile-Net and YOLOv8-CBAM-RSDD-HRII. The last 1×1 pointwise convolutional layer and the last average pooling layer of the Mobile-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Mobile-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0022] Furthermore, in step S5, an Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model is constructed for detecting minor road surface defects in highway rapid detection images to obtain the coordinates of the detection boxes. The specific method is as follows: feature fusion is performed between the Efficient-Net network and the Backbone network of the YOLOv8-CBAM-RSDD-HRII model. The last stage of the Efficient-Net network is removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of the Efficient-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0023] Furthermore, in step S6, a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model is constructed for detecting minor road surface defects in highway rapid detection images to obtain the coordinates of the detection boxes. The specific method is as follows: feature fusion is performed using the Backbone network of the Res-Net and YOLOv8-CBAM-RSDD-HRII models. The last average pooling and fully connected layers of the Res-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Res-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0024] Furthermore, in step S7, the specific method for integrating and learning the detection box coordinates generated by the three sub-fusion network models to construct the deep fusion convolutional neural network model DFCNN-RSDD-HRII (Deep Fusion Convolutional Neural Network for Road Surface Disease Detection using Highway Rapid Inspection Images) is as follows: the three sub-fusion network models are connected in parallel, and the detection results of the three sub-fusion network models are fed together into the detection box fusion module for integrated learning, thereby constructing the deep fusion convolutional neural network model DFCNN-RSDD-HRII and simultaneously generating the detection box information of the deep fusion convolutional neural network DFCNN-RSDD-HRII.

[0025] Furthermore, the specific content of the detection box fusion module for ensemble learning is as follows: If a road surface defect in the input highway fast detection image is detected by a certain fusion network model, then the final coordinates of the DFCNN-RSDD-HRII model detection box are:

[0026]

[0027] DB stands for Detection Box. Referring to the DFCNN-RSDD-HRII model, Refers to each sub-net in the DFCNN-RSDD-HRII model. For the ground truth detection bounding boxes, variables i = 1, 2, 3 represent the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, and the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, respectively; variables j = 1, 2, 3, 4 represent the vertex coordinates of the four vertices of the detection bounding box. If a certain sub-fusion network model fails to detect a target, then the data in that sub-network model is considered dead. S =0; For example, if you want to find the coordinates of the first vertex of a bounding box, the formula is to first calculate 0; Assume the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model satisfies but Therefore, we get variable i = 2, and the coordinates of the final detection box are DBS0,1 = DBS2,1. Similarly, we calculate the coordinates of the other three vertices of the detection box to obtain the detection box of the target in the DFCNN-RSDD-HRII model. If none of the three sub-fusion network models detect a target, the target is calculated as a negative sample, and the target in the final DFCNN-RSDD-HRII model is also considered not detected. After all the detection boxes of the three sub-fusion network models have been processed according to the above formula, the newly generated detection box is taken as the detection result of the DFCNN-RSDD-HRII model. Attached Figure Description

[0028] Figure 1 Schematic diagram of a highway rapid inspection image acquisition vehicle;

[0029] Figure 2 The structure diagram of the YOLOv8-CBAM-RSDD-HRII basic model is shown below;

[0030] Figure 3 This is a schematic diagram of the CBAM attention module.

[0031] Figure 4 The sub-converged network architecture is Mobile-Net-YOLOv8-CBAM-RSDD-HRII.

[0032] Figure 5 The network structure is Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network;

[0033] Figure 6 The network structure is a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network.

[0034] Figure 7 It is a DFCNN-RSDD-HRII deep fusion convolutional neural network structure;

[0035] Figure 8 Box plots of the F1-score for the five models. Detailed Implementation

[0036] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0037] This invention provides a method for detecting pavement defects in highway rapid detection images based on deep fusion convolutional neural networks, comprising the following steps:

[0038] S1: Construct a dataset of labeled road surface defects from rapid highway detection images. The specific method is as follows:

[0039] S1-1: First, acquire the original images of the highway surface for rapid inspection. The collected image data consists of images taken by the data acquisition vehicle on the Baolong G5XY section highway from August 18th to 21st, 2022, at a constant speed, with fixed shooting intervals, and the angle between the area array video sensor and the horizontal ground at 30°. The highway rapid inspection image data acquisition vehicle is as follows: Figure 1 As shown, it mainly includes an area array video sensor and storage device, and the resolution of the collected highway rapid inspection road surface images is 2064*1544.

[0040] S1-2: From the 18,713 original highway pavement images collected, images with obvious pavement distress characteristics were selected, mainly including transverse cracks and longitudinal cracks. Images with unclear pavement distress characteristics or overly complex and messy pavement distress characteristics were deleted, resulting in a total of 4,000 effective highway pavement distress images.

[0041] S1-3: LabelImg software was used to annotate the road surface defects images of the highway rapid detection. A segmented annotation method was adopted to ensure that the road surface defect information in each segment was consistent. The dataset label name is: {Transverse,Longitudinal}.

[0042] S1-4: After annotation, a dataset of 4000 highway pavement defect annotations based on highway fast detection images was obtained, including 11892 transverse cracks and 3495 longitudinal cracks, with a total of 15387 labels. The 4000 annotated highway pavement images were then randomly divided into training, validation, and test sets in a ratio of 8:1:1.

[0043] S2: Construct a simple YOLOv8-RSDD-HRII model for detecting pavement defects in highway rapid detection images.

[0044] S3: Construct a deep fusion foundation model, YOLOv8-CBAM-RSDD-HRII, for pavement defect detection in rapid highway image inspection. The specific method is as follows: Attention mechanisms can be used to optimize the backbone (feature extraction part) of the convolutional neural network, thereby increasing the detection performance and efficiency of the target detection model. The more numerous and significant the defect features contained in a rapid highway image inspection, the higher the accuracy of the network model's detection. Based on this, a Convolutional Block Attention Module (CBAM) is used to optimize the YOLOv8 network, constructing the deep fusion foundation model YOLOv8-CBAM-RSDD-HRII for pavement defect detection in rapid highway image inspection. Its structure is as follows: Figure 2 As shown, the specific optimization steps are as follows: The CBAM module consists of two sub-modules: the Channel Attention Module (CAM) and the Spatial Attention Module (SAM). The CBAM module generates new sub-feature maps along both the channel and spatial dimensions. The sub-feature maps are multiplied by the original feature maps to obtain the optimized new feature maps, as shown below. Figure 3 As shown, based on the original YOLOv8-RSDD-HRII network model, a CBAM module is added after the C2f module in its backbone. The parameters of the CBAM module after the first C2f module are 128×7, the second C2f module is 256×7, and the third C2f module is 512×7, depending on the feature map. The first variable is the number of channels in the feature map of that layer, and the second variable is the size of the convolution kernel of the spatial attention mechanism in the CBAM attention module. For example, the CBAM module with parameters of 128×7 has a convolution kernel size of 7×7 and 128 channels. The other parts are consistent with the YOLOv8-RSDD-HRII model.

[0045] S4: Construct a Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes. The specific method is as follows: Feature fusion is performed using a Backbone network of Mobile-Net and YOLOv8-CBAM-RSDD-HRII. The sub-fusion network model structure is as follows: Figure 4As shown, the last 1×1 pointwise convolutional layer and the last average pooling layer of the Mobile-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Mobile-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0046] S5: Construct an Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in rapid highway image detection, and obtain the coordinates of the detection boxes. The specific method is as follows: Feature fusion is performed using the Efficient-Net network and the Backbone network of the YOLOv8-CBAM-RSDD-HRII model. The sub-fusion network model structure is as follows: Figure 5 As shown, the last stage of the Efficient-Net network is removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Efficient-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0047] S6: Construct a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes. The specific method is as follows: Feature fusion is performed using the Backbone network of the Res-Net and YOLOv8-CBAM-RSDD-HRII models. The sub-fusion network model structure is as follows: Figure 6As shown, the last average pooling and fully connected layers of the Res-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Res-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model.

[0048] S7: The specific method for constructing a deep fusion convolutional neural network model DFCNN-RSDD-HRII (Deep Fusion Convolutional Neural Network for Road Surface Disease Detection using Highway Rapid Inspection Images) by integrating and learning the detection box coordinates generated by the three sub-fusion network models is as follows: The three sub-fusion network models are connected in parallel. If a road surface defect in the input highway rapid inspection image is detected by a certain sub-fusion network model, the final detection box coordinates of the DFCNN-RSDD-HRII model are:

[0049]

[0050] DB stands for Detection Box. Referring to the DFCNN-RSDD-HRII model, Refers to each sub-net in the DFCNN-RSDD-HRII model. For the ground truth detection bounding boxes, variables i = 1, 2, 3 represent the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, and the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, respectively; variables j = 1, 2, 3, 4 represent the vertex coordinates of the four vertices of the detection bounding box. If a certain sub-fusion network model fails to detect a target, then the data in that sub-network model is considered dead. S =0; For example, if you want to find the coordinates of the first vertex of a bounding box, the formula is to first calculate 0; Assume the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model satisfies but Therefore, we get variable i = 2, and the final coordinates of the detection box are... Similarly, the coordinates of the other three vertices of the detection box are calculated to obtain the detection box of the target in the DFCNN-RSDD-HRII model, thus constructing the deep fusion convolutional neural network model DFCNN-RSDD-HRII. The network model structure is as follows: Figure 7 As shown; if none of the three sub-fusion network models detect a certain target, the target is calculated as a negative sample, and thus the target in the final DFCNN-RSDD-HRII model is also considered not detected; after all the detection boxes of the three sub-fusion network models have been processed according to the above formula, the newly generated detection boxes are taken as the detection result of the DFCNN-RSDD-HRII model.

[0051] To verify the superiority of the fusion model DFCNN-RSDD-HRII, a comparative experiment was conducted. Based on the constructed highway fast-detection image pavement defect annotation dataset, and randomly divided into training, validation, and test sets in an 8:1:1 ratio, four highway fast-detection image pavement defect detection models—YOLOv8-CBAM-RSDD-HRII, Mobile-Net-YOLOv8-CBAM-RSDD-HRII, Efficient-Net-YOLOv8-CBAM-RSDD-HRII, and Res-Net-YOLOv8-CBAM-RSDD-HRII—were trained respectively. The training parameters are shown in Table 6.

[0052] Table 6 Training parameters for four models

[0053]

[0054] After the model training was completed, the trained model was applied to the test set, and the detection results of different models on the test set were compared and analyzed. The results are shown in Tables 7-9. It can be seen that the overall detection performance of the deep fusion convolutional neural network DFCNN-RSDD-HRII model is better than the other four models, indicating that the DFCNN-RSDD-HRII model is more suitable for detecting small road surface defects in highway rapid detection image scenarios.

[0055] Table 7 Experimental Results of Five Models

[0056]

[0057] Table 8. Results of transverse crack detection for five models

[0058]

[0059] Table 9. Results of longitudinal crack detection for five models.

[0060]

[0061] To further demonstrate the effectiveness of the fusion model DFCNN-RSDD-HRII, 20 cross-validation experiments were conducted. In each experiment, 100 images of road surface defects detected by rapid road inspection were randomly selected from the test sample library for testing. The detection accuracy of the five models was statistically analyzed, and their F1-Score box plots are shown below. Figure 8 As shown, the DFCNN-RSDD-HRIII model outperforms the other four models, and its F1-Score distribution range is smaller.

[0062] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for detecting pavement defects in highway images based on deep fusion convolutional neural networks, characterized by: Includes the following steps: S1: Construct a dataset of labeled road surface defects from rapid highway inspection images; S2: Construct a simple YOLOv8-RSDD-HRII model for road surface defect detection in highway rapid detection images. The specific steps are as follows: Construct the YOLOv8 model YOLOv8-RSDD-HRII for road surface defect detection in highway rapid detection images. Here, Resolution is the scale of the variables in each Stage, Channels is the number of channels for each Stage variable, Layers is the number of times the Operator operation is repeated in each Stage, Kernels is the size of the convolution kernel for the convolution operation, CBL is the layer where the variables are first convolved, then normalized, and activated using the LeakyReLU activation function; C2f is the C2f network structure layer, SPPF 5×5 is the 5×5 kernel size in the SPPF structure, Unsample is the upsampling operation on the variables, Concat is the concatenation operation on the variables, and the concatenation level is the connected C2f module and the C2f module of the same size in the Backbone; Detect_X is the output module, and the output dimension is related to the number of target categories. S3: Construct a YOLOv8-CBAM-RSDD-HRII deep fusion basic model for road surface defect detection in highway rapid detection images; the specific steps are as follows: on the basis of the original YOLOv8-RSDD-HRII network model, add a CBAM module after the C2f module in its backbone. The parameters of the CBAM module after the first C2f module are 128×7, the parameters of the CBAM module after the second C2f module are 256×7, and the parameters of the CBAM module after the third C2f module are 512×7, depending on the feature map. The first variable is the number of channels of the feature map of this layer, and the second variable is the size of the convolution kernel of the spatial attention mechanism in the CBAM attention module. The CBAM module with parameters of 128×7 means that the convolution kernel size is 7×7 and the number of channels is 128. The other parts are consistent with the YOLOv8-RSDD-HRII model. S4: Construct a Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes. The specific steps are as follows: Feature fusion is performed using the Backbone network of Mobile-Net and YOLOv8-CBAM-RSDD-HRII. The last 1×1 pointwise convolutional layer and the last average pooling layer of the Mobile-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Mobile-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent network of the YOLOv8-CBAM-RSDD-HRII model to obtain the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model. S5: Construct an Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in rapid highway image detection, and obtain the coordinates of the detection boxes. The specific steps are as follows: Feature fusion is performed between the Efficient-Net network and the Backbone network of the YOLOv8-CBAM-RSDD-HRII model. The last stage of the Efficient-Net network is removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of the Efficient-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model. S6: Construct a Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model for detecting minor road surface defects in highway rapid image detection, and obtain the coordinates of the detection boxes. The specific steps are as follows: Feature fusion is performed using the Backbone network of the Res-Net and YOLOv8-CBAM-RSDD-HRII models. The last average pooling and fully connected layers of the Res-Net network are removed, and its output feature map is fused and concatenated with the feature map output by the Backbone in the YOLOv8-CBAM-RSDD-HRII model. This ensures that the output dimension of Res-Net is consistent with the output dimension of the Backbone in the YOLOv8-CBAM-RSDD-HRII model. The feature maps are then superimposed and fed into the subsequent networks of the YOLOv8-CBAM-RSDD-HRII model to obtain the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model. S7: Integrate the coordinates of the detection boxes generated by the three sub-fusion network models to construct a deep fusion convolutional neural network model DFCNN-RSDD-HRII for detecting pavement defects in highway rapid detection images.

2. The method for detecting road surface defects in highway images based on deep fusion convolutional neural networks according to claim 1, characterized in that: The specific method for constructing the highway rapid detection image pavement defect image annotation dataset in step S1 is as follows: S1-1: First, acquire the original image of the road surface for rapid inspection. The acquired image data is the image taken by the data acquisition vehicle when it is traveling at a constant speed on a certain section of the road, the shooting time interval is fixed, and the angle between the area array video sensor and the horizontal ground is 30°. The resolution of the acquired road surface image for rapid inspection is 2064*1544. S1-2: From the 18,713 original highway pavement images collected, images with obvious pavement distress characteristics were selected, including transverse and longitudinal cracks. Images with unclear pavement distress characteristics or overly complex and messy pavement distress characteristics were deleted. Finally, a total of 4,000 effective highway pavement distress images were obtained. S1-3: LabelImg software was used to annotate the road surface defects images of the highway rapid detection. A segmented annotation method was adopted to ensure that the road surface defect information in each segment was consistent. The dataset label name is: {Transverse, Longitudinal}. S1-4: After the annotation was completed, a total of 4,000 images of highway pavement disease annotation dataset based on highway fast detection images were obtained, including 11,892 transverse cracks, 3,495 longitudinal cracks, and a total of 15,387 labels. The 4,000 labeled highway pavement images were then randomly divided into training, validation, and test sets in a ratio of 8:1:

1.

3. The method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks according to claim 1, characterized in that: The specific method for constructing the YOLOv8-CBAM-RSDD-HRII deep fusion basic model for road surface defect detection in step S3 is as follows: The attention mechanism can be used to optimize the backbone of the convolutional neural network, i.e., the feature extraction part, thereby increasing the detection effect and efficiency of the target detection model. The more and more significant the defect features contained in the road surface image of the highway rapid detection, the higher the accuracy of the network model in detection. Based on this, a convolutional attention module is used to optimize the YOLOv8 network and construct the YOLOv8-CBAM-RSDD-HRII deep fusion basic model for road surface defect detection in highway rapid detection images. Here, Resolution is the scale of the variable in each stage, Channels is the number of channels of the variable in each stage, Layers is the number of times the Operator operation is repeated in each stage, Kernels is the size of the convolution kernel of the convolution operation, CBL is to first perform convolution operation on the variable of this layer, then perform standardization operation, and use the LeakyReLU activation function for activation; C2f is the C2f network structure layer, and the Kernel of the CBAM module is... 7×7 refers to the 7×7 kernel size in the CBAM module used, SPPF 5×5 refers to the 5×5 kernel size in the SPPF structure, Unsample is the upsampling operation on the variables, Concat is the concatenation operation on the variables, the concatenation level is the C2f module connected to it and the C2f module of the same size in the Backbone, Detect_X is the output module, the output dimension is related to the number of target categories. In the road surface image of the highway quick detection in this paper, the road surface defects are divided into two categories, so the number of output channels is 3×(2+5)=21.

4. The method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks according to claim 1, characterized in that: In the structural parameters of the Mobile-Net network in step S4, Channels is the number of channels. The number of repetitions for this layer structure is given. Conv2d is a regular 2D convolution, bottleneck is an inverse residual structure, and Avgpool is average pooling.

5. The method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks according to claim 1, characterized in that: In the structural parameters of the Efficient-Net network in step S5, Resolution is the size of the variable in each Stage, Channels is the number of channels for the variable in each Stage, and Layers is the number of times the Operator is repeated in each Stage.

6. The method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks according to claim 1, characterized in that: In the structural parameters of the Res-Net network in step S6, conv1 is a normal convolution operation, conv2_x to conv5_x are convolution operations with residual structures, outputsize is the size of the output variable after passing through a certain layer, and 18-layer, 34-layer, 50-layer, 101-layer and 152-layer represent Res-Net networks with different residual structures.

7. The method for detecting road surface defects in highway rapid detection images based on deep fusion convolutional neural networks according to claim 1, characterized in that: In step S7, the coordinates of the detection boxes generated by the three sub-fusion network models are integrated and learned to construct the deep fusion convolutional neural network model DFCNN-RSDD-HRII for road surface defect detection in highway rapid detection images. The specific method is as follows: the three sub-fusion network models are connected in parallel, and the detection results of the three sub-fusion network models are sent together to the detection box fusion module for integrated learning, thereby constructing the deep fusion convolutional neural network model DFCNN-RSDD-HRII and simultaneously generating the detection box information of the deep fusion convolutional neural network DFCNN-RSDD-HRII.

8. The method for detecting pavement defects in highway images based on deep fusion convolutional neural networks according to claim 7, characterized in that: The specific content of the detection box fusion module for ensemble learning is as follows: If a road surface defect in the input highway fast detection image is detected by a certain fusion network model, then the final coordinates of the DFCNN-RSDD-HRII model detection box are: （9-1） in, It is an abbreviation for Detection Box. Referring to the DFCNN-RSDD-HRII model, Refers to each sub-net in the DFCNN-RSDD-HRII model. For the true detection box, variables These refer to the Mobile-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, and the Res-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model, respectively; variables , where represents the vertex coordinates of the four vertices of the detection box. If the fused network model fails to detect something in a certain iteration, then that iteration of the network model... If the coordinates of the first vertex of a bounding box are required, the formula is to first calculate... Assume that the Efficient-Net-YOLOv8-CBAM-RSDD-HRII sub-fusion network model satisfies ,but Thus, we obtain the variable. Then the coordinates of the final detection box are Similarly, the coordinates of the other three vertices of the detection box are calculated to obtain the detection box of the target in the DFCNN-RSDD-HRII model. If none of the three sub-fusion network models detect a target, the target is calculated as a negative sample, and the target in the final DFCNN-RSDD-HRII model is also considered not detected. After all the detection boxes of the three sub-fusion network models have been processed according to the above formula, the newly generated detection box is taken as the detection result of the DFCNN-RSDD-HRII model.

Images