A road damage detection method based on deep learning

Abstract

The invention discloses a road damage detection technology field, specifically a road damage detection method based on deep learning, including the steps of using RDDA algorithm for detection, specifically as follows: road damage data set construction, image feature extraction module construction, design of deep separable parameter-free attention mechanism convolution module, extraction of feature map deep features, design of spatial pyramid adaptive pooling fusion module, use of FPN module for feature fusion, design of NWD‑EIoU loss function, use of Head detection head for detection, model training and verification, and application of the model. The invention effectively solves the problems of large number of parameters in current road damage detection algorithm, insufficient accuracy of small target disease detection, and poor effect of road disease detection under complex background, thereby making the algorithm parameter number of the invention lower, being suitable for the deployment of edge devices, and meeting the lightweight and real-time requirements of road damage detection.

Description

Technical Field

[0001] The present invention relates to the technical field of road damage detection, and specifically to a road damage detection method based on deep learning. Background Art

[0002] Highways are one of the basic transportation carriers in my country. With the continuous development of urban transportation and the continuous expansion of road construction, the detection and repair of road diseases have become increasingly important. Road diseases, such as cracks, potholes and cracks, not only affect driving safety, but may also cause vehicle damage and traffic congestion. Therefore, fast and accurate detection of road diseases is crucial to maintaining road safety and improving traffic efficiency. However, the types of diseases are complex, and the characteristics of each disease from cracks to cracks are not obvious. In addition, the complexity of the urban environment, such as traffic signs, trees and buildings, leads to increased background interference, making disease detection more difficult. Traditional target detection algorithms can no longer meet the needs of road disease detection, and it is urgent to improve the accuracy of road disease detection.

[0003] With the continuous development of computer vision, more and more machine vision detection technologies are being applied to industrial inspection. At present, the main target detection algorithms used in road disease detection include the one-stage detection algorithm YOLO, SSD algorithm and the two-stage algorithm R-CNN, Faster-RCNN. However, these two types of algorithms have the following disadvantages:

[0004] 1. Low model detection accuracy: Due to the variety of road damage types and the complex road backgrounds in different environments, the feature extraction of road damage is relatively complex. The existing target detection algorithm cannot capture the effective features of road damage well, and the detection accuracy of road damage is low.

[0005] 2. Difficulty in detecting small target defects: Road defects include small-sized targets such as road pollution and cracks. Existing detection algorithms have poor detection performance for small target defects, and are prone to missed detection and false detection.

[0006] 3. The number of model parameters is too large: For road damage detection, the deployment of edge devices is an essential condition, which can greatly reduce the cost of road damage detection. The existing model parameters are too large to be deployed on edge devices.

[0007] Therefore, a road damage detection method based on deep learning is invented. Summary of the invention

[0008] In view of the above problems and / or the existing problems in a road damage detection method based on deep learning, the present invention is proposed.

[0009] Therefore, the purpose of the present invention is to provide a road damage detection method based on deep learning, which can solve the above-mentioned existing problems.

[0010] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0011] A road disease detection method based on deep learning includes the following steps of using the RDDA algorithm for detection:

[0012] S1: Construction of road damage dataset;

[0013] S2: build image feature extraction module;

[0014] S3: Design a deep separable parameter-free attention mechanism convolution module;

[0015] S4: Extract deep features of feature maps;

[0016] S5: Design spatial pyramid adaptive pooling fusion module;

[0017] S6: Use FPN module for feature fusion;

[0018] S7: Design NWD-EIoU loss function;

[0019] S8: Use Head detection;

[0020] S9: Model training and validation;

[0021] S10: Apply the model.

[0022] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S1 is as follows:

[0023] S11: Use drones to photograph road disease sections and collect road disease images as a data set;

[0024] S12: Preprocess the data set, screen qualified images, balance the number of road disease samples in each category, and annotate the collected data set using annotation software.

[0025] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S2 is as follows:

[0026] S21: Input the road damage image in the data set into the Conv2D_BN_LeakyRelu module for processing. In the Conv2D_BN_LeakyRelu module, the input image is first subjected to a standard convolution operation with a convolution kernel of 3×3 and a step size of 2, and then enhanced by the nonlinear activation function LeakyReLU, and then subjected to batch normalization processing to generate an initial feature map F1. The size of the feature map F1 is assumed to be C×H×W, where C represents the number of channels, H represents the height, and W represents the width.

[0027] S22: Input the feature map F1 to the Conv2D_BN_LeakyRelu module. In the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 3×3 and a step size of 2 on F1, then perform a nonlinear activation function LeakyReLU enhancement, and then perform Batch Normalization to generate a feature map F2. The size of the feature map F2 is 2C×H / 2×W / 2.

[0028] S23: Input feature map F2 to the ELAN module, and output feature map F3 after convolution and concatenation operations. The size of F3 is 2C×H / 2×W / 2;

[0029] S24: The feature map F3 is further input into the MaxPool module. In the MaxPool module, F3 is subjected to maximum pooling to finally obtain a feature map F4, whose size is 2C×H / 4×W / 4.

[0030] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S3 is as follows:

[0031] S31: The road disease feature map F4 with a size of 2C×H / 4×W / 4 obtained in S2 is used as the input feature map. The first branch F4 is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F5 with a size of 2C×H / 4×W / 4.

[0032] S32: The second branch F4 performs a DSC convolution with a convolution kernel of 1×1 and a step size of 1, and two DSC convolutions with a convolution kernel of 3×3 and a step size of 1. In the ELAN-SimAM-D module, the output of each DSC convolution is used as the input of the next DSC convolution, which is repeated three times in sequence. Finally, the three DSC convolutions output feature maps F4′, feature map F4″, and feature map F4″′, respectively. The feature maps F4′, F4″, and F4″′ are all 2C×H / 4×W / 4.

[0033] S33: concatenate the feature map F4′, the feature map F4″, the feature map F4″′ and the feature map F5 to obtain a feature map F6 with a size of 8C×H / 4×W / 4;

[0034] S34: Input the feature map F6 into the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. The Conv2D_BN_LeakyRelu module outputs a feature map F7 with a size of 4C×H / 4×W / 4. Then the feature map F7 is input to the SimAM layer, and three-dimensional weights are assigned to the key features of the small target. The complex background area and the small target area are divided into pixels. After passing through the SimAM module, the feature map F7' is finally obtained, and the size of F7' is 4C×H / 4×W / 4;

[0035] Among them, the calculation formula of F7' is shown in formula (1)-(4);

[0036]

[0037]

[0038] Among them, t is the target neuron, λ is the regularization term, is the energy function, The smaller the value of , the greater the difference between the characteristics of the neuron and its neighboring neurons, and the more important t is. represents the mean value of all neurons in each channel, is the variance of the neuron in each channel, x i are the other neurons in the i-th channel, N is the number of all neurons on a certain channel, E is the sum of the energy functions in all channels and spatial dimensions, the sigmoid function ensures that the value of E is not too large, and ⊙ represents the Hadamard product.

[0039] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S4 is as follows:

[0040] S41: The feature map F7' is further input to the MaxPool module, and after convolution and concatenation operations, the feature map F8 is output. The size of the feature map F8 is 4C×H / 8×W / 8;

[0041] S42: The feature map F8 is further input to the ELAN-SimAM-D module to obtain the final output feature map F1′1 of the ELAN-SimAM-D module. The size of the feature map F1′1 is 8C×H / 8×W / 8;

[0042] S43: The feature map F1′1 is further input to the MaxPool module, and the feature map F is output after convolution and concatenation operations. 12 , feature map F 12 The dimensions are 8C×H / 16×W / 16;

[0043] S44: The feature map F 12 Continue to input into the ELAN-SimAM-D module, and in the ELAN-SimAM-D module, obtain the final output feature map F1′5 of the ELAN-SimAM-D module. The size of the feature map F1′5 is 16C×H / 16×W / 16.

[0044] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S5 is as follows:

[0045] S51: The road disease feature map F1′5 with a size of 16C×H / 16×W / 16 is used as the input feature map. The first branch is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16. 16 ;

[0046] S52: The second branch F1′5 inputs the Conv2D_BN_LeakyRelu module, first performs a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then performs a nonlinear activation function LeakyReLU enhancement, and then performs BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16 17 ;

[0047] S53: Feature map F 17The first three branches are processed with adaptive pooling of sizes 5×5, 9×9, 13×13 and step size 1, respectively, to obtain the feature map F 17 ', feature map F 17 "、Feature graph F 17 ”', the fourth branch does not process the input feature map, and the feature map sizes output by the four branches are all 8C×H / 16×W / 16;

[0048] Among them, the feature map F 17 ', feature map F 17 "、Feature graph F 17 The specific calculation process of adaptive pooling of ” is as shown in formula (5) to formula (8);

[0049]

[0050] in, is the adaptive pooling output, represents smooth approximate maximum pooling, represents smooth approximate average pooling, β = {0, ..., 1} is related to the downsampling volume The weight mask of the same size, R represents the kernel neighborhood, i and j are the coordinate sets of R, Based on the inverse of the distance between each activation and the average activation, AdaPool can calculate the gradient for each kernel vector during the back-propagation process, which can adaptively capture details and make pooling more regionally adaptive;

[0051] S54: The feature graph F1′7, the feature graph F1′7′, the feature graph F1′7″, the feature graph F1′7″ 17 Splice and finally get the feature map F 18 , feature map F 18 The dimensions are 32C×H / 16×W / 16;

[0052] S55: The feature map F 18 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H×W 19 ;

[0053] S56: The feature map F 19 With the feature map F 16 After concatenation, we finally get a feature map F with a size of 16C×H / 16×W / 16 20 ;

[0054] S57: The feature map F 20 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F of size 8C×H / 16×W / 16 21 .

[0055] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S6 is as follows:

[0056] S61: Make feature map F7, feature map F 11 , feature map F 21 As the input of FPN module;

[0057] S62: FPN module inputs feature map F7, feature map F 11 , feature map F 21 Perform feature fusion and finally output feature map F 22 , feature map F 23 , feature map F 24 .

[0058] As a preferred solution of the road damage detection method based on deep learning described in the present invention, the specific process of S7 is as follows: combining the EIoU loss function and the NWD loss function to form a new road damage detection bounding box positioning loss function NWD-EIoU;

[0059] Its NWD-EIoU loss function L NWD_EIoU The calculation formula is shown in formula (9);

[0060] L NWD-EIoU =α·L EIoU +(1-α)·L NWD (9)

[0061] Where a represents a regular term, the value is from 0 to 1, L EIoU Represents the value of the EIoU loss function, L NWD It represents the value of the NWD loss function. The NWD-CIoU loss function combines the two loss functions, adjusts the weights of the two parameters through the regularization term a, and determines the most appropriate value of a during the training process;

[0062] In addition, L EIoU and L NWDThe specific calculation formulas are shown in formulas (10), (11), (12), and (13) respectively;

[0063]

[0064] Among them, IOU represents the intersection of the predicted box and the real box, I represents the intersection of the predicted box and the real box, U represents the union of the predicted box and the real box, b and b gt Represent the center points of the predicted box and the real box, w and w respectively gt Represent the width of the predicted box and the real box, h and h respectively gt Represents the height of the predicted box and the real box, w c and h c is the width and height of the minimum bounding rectangle of the predicted bounding box and the true bounding box, ρ is the Euclidean distance between the two points, EIOU splits the aspect ratio loss into the difference between the predicted width and height and the minimum bounding box width and height, which accelerates the convergence of the model and improves the regression accuracy. C represents the number of types detected in the data set, and W2 2 (N a ,N b ) is a second-order Wasserstein distance definition, where N a and N b Respectively represent the bounding box A=(cx a ,cy a ,w a ,h a ) and bounding box B = (cx b ,cy b ,w b ,h b ), ||·|| F is the Frobenius norm.

[0065] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S8 is as follows:

[0066] S81: subject feature maps from different scales to several layers of convolution operations to further extract features;

[0067] S82: Generate multiple anchor boxes for bounding box prediction through the predefined anchor box mechanism. Each anchor box outputs the center coordinates of the bounding box, the adjustment amount of width and height, and the confidence score, which indicates the probability of containing the object in the box.

[0068] S83: The network outputs the classification score of each anchor box and removes redundant detection boxes through non-maximum suppression to generate the final detection result.

[0069] As a preferred solution of the road disease detection method based on deep learning described in the present invention, the specific process of S9 is as follows:

[0070] S91: Initialize all neural network parameters and set related hyperparameters;

[0071] S92: Divide the road damage data set processed by S1 into a training set, a validation set and a test set. The training set and the validation set are divided into multiple batches. Each time, a batch of training set data is input into the road damage detection model for training, and the training loss value loss of the batch is calculated;

[0072] S93: After completing a round of training for all batches of data in the entire training set, the validation set is input into the road damage detection model in batches to obtain a corresponding batch loss value batch_loss;

[0073] S94: During the training and verification process, the road damage detection model will automatically learn and adjust parameters according to the loss and batch_loss conditions each time. The training process will continue for one or more rounds until the batch_loss value converges, indicating that the road damage detection model training is completed;

[0074] The specific process of S10 is as follows:

[0075] S101: input the collected road damage pavement image data into the detection model, and the model will automatically detect the image to obtain the detection result of the road damage image, and the detection result includes the category, location and detection confidence information of the road damage;

[0076] S102: Display the detection result on the image.

[0077] Compared with existing technologies:

[0078] The present invention uses deep separable convolution and combines the SimAM attention mechanism to design a deep separable parameter-free attention mechanism convolution module, thereby improving the model's ability to detect road damage, reducing the interference of complex backgrounds on road damage features, and improving the network's feature extraction capability. At the same time, deep separable convolution can effectively reduce the number of model parameters; the present invention introduces adaptive pooling to design a spatial pyramid adaptive pooling fusion module, so that pooling is more regionally adaptive, and can better extract and retain important features of road damage, improve the model's detection accuracy, and reduce the model's false detection; by combining the Swarthastan distance NWD and EIoU to design the NWD-EIoU loss function, the model's convergence speed is accelerated, and the regression accuracy is improved. Based on this, the present invention effectively solves the problems of large number of parameters in the current road damage detection algorithm, insufficient detection accuracy of small target damage, and poor detection effect of road damage under complex backgrounds, thereby making the algorithm parameter number of the present invention lower, suitable for the deployment of edge devices, and meeting the lightweight and real-time requirements of road damage detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the process of the present invention;

[0080] Figure 2 It is the structural diagram of RDDA algorithm of the present invention;

[0081] Figure 3 This is the structural diagram of the ELAN-SimAM-D module of the present invention;

[0082] Figure 4 This is the structural diagram of the SPPAdaF module of the present invention;

[0083] Figure 5 This is a diagram showing an example of applying the model of the present invention. DETAILED DESCRIPTION

[0084] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0085] The present invention provides a road disease detection method based on deep learning, please refer to Figure 1-Figure 5 , including the steps of using RDDA algorithm for detection, as follows:

[0086] S1: Construction of road damage dataset;

[0087] Among them: The specific process of S1 is as follows:

[0088] S11: Use drones to photograph road disease sections and collect road disease images as a data set;

[0089] S12: preprocess the data set, screen qualified images, balance the number of road disease samples in each category, and annotate the collected data set using annotation software (labelme software can be used as annotation software);

[0090] In this process, since the resolution of the road disease dataset is generally large, the images in the dataset are processed using a sliding window algorithm. By adjusting the size and step size of the sliding window, the dataset is reduced in resolution while retaining valid features, while ensuring that the images in the dataset do not lose features.

[0091] S2: build image feature extraction module;

[0092] Among them: The specific process of S2 is as follows:

[0093] S21: Input the road damage image in the data set into the Conv2D_BN_LeakyRelu module for processing. In the Conv2D_BN_LeakyRelu module, the input image is first subjected to a standard convolution operation with a convolution kernel of 3×3 and a step size of 2, and then enhanced by the nonlinear activation function LeakyReLU, and then subjected to batch normalization processing to generate an initial feature map F1. The size of the feature map F1 is assumed to be C×H×W, where C represents the number of channels, H represents the height, and W represents the width.

[0094] Among them: Conv2D_BN_LeakyRelu module (referred to as CBL module) is a standard convolution module, which first performs standard convolution operation on the input feature map, then performs nonlinear activation function LeakyReLU enhancement, and then performs BatchNormalization batch normalization processing; In addition, the feature map is the output result of the middle layer of the convolutional neural network (CNN) in deep learning; In CNN, the feature map is obtained by applying convolution operation and activation function to the input image; In the convolution layer of CNN, each convolution kernel can be regarded as an extractor of a specific type of feature of the input image, and the output generated by each convolution kernel is the corresponding feature map;

[0095] S22: Input the feature map F1 to the Conv2D_BN_LeakyRelu module. In the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 3×3 and a step size of 2 on F1, then perform a nonlinear activation function LeakyReLU enhancement, and then perform Batch Normalization to generate a feature map F2. The size of the feature map F2 is 2C×H / 2×W / 2.

[0096] S23: Input feature map F2 to the ELAN module, and output feature map F3 after convolution and concatenation operations. The size of F3 is 2C×H / 2×W / 2;

[0097] Among them: The ELAN module is an effective network structure. By adjusting the length of the gradient path, the network can capture more features and improve robustness. Its main function is to improve the effect of target detection through efficient feature fusion and multi-scale feature extraction;

[0098] S24: The feature map F3 is further input to the MaxPool module. In the MaxPool module, F3 is subjected to maximum pooling to finally obtain a feature map F4, the size of which is 2C×H / 4×W / 4.

[0099] Among them: MaxPool module is the maximum pooling module, which performs maximum pooling processing on the input feature map;

[0100] S3: Design a deep separable parameter-free attention mechanism convolution module;

[0101] Among them: The specific process of S3 is as follows:

[0102] S31: The road disease feature map F4 with a size of 2C×H / 4×W / 4 obtained in S2 is used as the input feature map. The first branch F4 is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F5 with a size of 2C×H / 4×W / 4.

[0103] S32: The second branch F4 performs a DSC convolution with a convolution kernel of 1×1 and a step size of 1, and two DSC convolutions with a convolution kernel of 3×3 and a step size of 1. In the ELAN-SimAM-D module, the output of each DSC convolution is used as the input of the next DSC convolution, which is repeated three times in sequence. Finally, the three DSC convolutions output feature maps F4′, feature map F4″, and feature map F4″′, respectively. The feature maps F4′, F4″, and F4″′ are all 2C×H / 4×W / 4.

[0104] Among them: Depthwise separable convolution (DSC) consists of two parts: depthwise convolution and pointwise convolution; depthwise convolution is used to extract spatial features, while pointwise convolution is used to extract channel features; the convolution operation is grouped in the feature dimension, and depthwise convolution is performed independently on each channel, and then a 1x1 convolution is used to aggregate all channels together;

[0105] S33: concatenate the feature map F4′, the feature map F4″, the feature map F4″′ and the feature map F5 to obtain a feature map F6 with a size of 8C×H / 4×W / 4;

[0106] S34: Input the feature map F6 into the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. The Conv2D_BN_LeakyRelu module outputs a feature map F7 with a size of 4C×H / 4×W / 4. Then the feature map F7 is input to the SimAM layer, and three-dimensional weights are assigned to the key features of the small target. The complex background area and the small target area are divided into pixels. After passing through the SimAM module, the feature map F7' is finally obtained, and the size of F7' is 4C×H / 4×W / 4;

[0107] Among them, the calculation formula of F7' is shown in formula (1)-(4);

[0108]

[0109] Among them, t is the target neuron, λ is the regularization term, is the energy function, The smaller the value of , the greater the difference between the characteristics of the neuron and its neighboring neurons, and the more important t is. represents the mean value of all neurons in each channel, is the variance of the neuron in each channel, x i are other neurons in the i-th channel, N is the number of all neurons on a channel, E is the sum of the energy functions in all channels and spatial dimensions, the sigmoid function ensures that the value of E is not too large, ⊙ represents the Hadamard product;

[0110] in:

[0111] (1) The sigmoid function is a commonly used activation function and is widely used in neural networks;

[0112] (2) Definition of energy function: In physics, energy is defined as the ability of an object to perform work; in neural networks, the state of a neuron can be analogized to an energy state, which can be described by an energy function; the energy function is a function that maps the state of a neuron to a real value, which evaluates the stability and reliability of the current state of the neuron;

[0113] (3) The SimAM attention mechanism is a publicly available parameter-free attention mechanism that is used to enhance the model’s preservation of important information about target features. It can assign three-dimensional weights to small target features and effectively divide pixels between features and background.

[0114] The implementation example of this step is as follows: suppose the size of the road damage feature map is C×H×W, where C is the number of channels of the feature map, H is the height of the feature map, and W is the width of the feature map; Example: For the road damage target feature map F4 with an input size of 64×160×160, two branches are processed. The first branch performs a CBL convolution operation with a convolution kernel of 1×1 and a step size of 1. Finally, the CBL module outputs a feature map F5 with a size of 64×160×160; the second branch continuously performs a DSC convolution with a convolution kernel of 3×3 and a step size of 1, and two DSC convolutions with a convolution kernel of 3×3 and a step size of 1. In the ELAN-SimAM-D module, the output of each DSC convolution is used as the next DSC convolution. The output of the three DSC convolutions is feature map F4', feature map F4", and feature map F4''; feature map F4', feature map F4", feature map F4'' and feature map F5 are concatenated to obtain feature map F6 with a size of 256×160×160; feature map F6 is subjected to a CBL convolution operation with a convolution kernel of 3×3 and a step size of 1, and the output size is a feature map F7 with a size of 128×160×160; feature map F7 is input into the SimAM layer, a three-dimensional weight is assigned to the key features of the small target, and the complex background area and the small target area are divided into pixels, and finally a feature map F7' with a size of 128×160×160 is obtained; F7' is the output feature map of ELAN-SimAM-D;

[0115] In this process, in order to improve the accuracy of road damage detection tasks and make the algorithm suitable for the deployment of edge devices, a deep separable parameter-free attention mechanism convolution module (ELAN-SimAM-D) is designed; the road damage information obtained by the ELAN-SimAM-D module is very rich, and can effectively retain important information of road damage characteristics. The SimAM attention mechanism can give appropriate three-dimensional weights to key features, divide complex background images and target features, and reduce the impact of complex road damage on the extraction of road damage target features; at the same time, deep separable convolution can effectively reduce parameters compared to ordinary convolution, making the model more lightweight; the ELAN-SimAM-D module is very suitable for road damage detection tasks under such complex backgrounds and can meet the deployment of edge devices;

[0116] The ELAN-SimAM-D module includes a CBL module, a DSC layer, and a SimAM layer. The DSC layer is a depth-separable convolution layer, in which a depth-separable convolution operation with a convolution kernel of 1×1 and a step size of 1 is first performed, and then a nonlinear activation function LeakyReLU enhancement is performed, and then a batch normalization process is performed. Compared with ordinary convolution layers, the number of model parameters and the amount of calculation can be reduced. The SimAM layer mainly adds three-dimensional weights of features to reduce the interference of background on features and improve the detection accuracy of the model.

[0117] The process of the ELAN-SimAM-D module is as follows: the ELAN-SimAM-D module will receive the feature map F4 obtained in S2. For the input feature map F4, the depth-separable parameter-free attention mechanism convolution module processes the input feature map in two different branches. The first branch inputs the CBL module to obtain the feature map F5, and the other branch inputs the three-layer DSC layer. The output of each DSC layer is used as the input of the next DSC layer, and this is repeated three times in sequence. The three DSC layers output feature maps F4′, F4″, and F4″′ respectively; the feature maps F4′, F4″, and F4″′ are concatenated with the unprocessed feature map F5 to obtain feature map F6; the feature map F6 is input into the CBL module to obtain the feature map F7, and the feature map F7 is input into the SimAM layer to add three-dimensional weights to the features to obtain the final output feature map F7′ of the ELAN-SimAM-D module; the ELAN-SimAM-D module is as follows Figure 3 As shown;

[0118] S4: Extract deep features of feature maps;

[0119] Among them: The specific process of S4 is as follows:

[0120] S41: The feature map F7' is further input to the MaxPool module, and after convolution and concatenation operations, the feature map F8 is output. The size of the feature map F8 is 4C×H / 8×W / 8;

[0121] S42: The feature map F8 is further input to the ELAN-SimAM-D module to obtain the final output feature map F1′1 of the ELAN-SimAM-D module. The size of the feature map F1′1 is 8C×H / 8×W / 8;

[0122] S43: The feature map F1′1 is further input to the MaxPool module, and the feature map F is output after convolution and concatenation operations. 12 , feature map F 12 The dimensions are 8C×H / 16×W / 16;

[0123] S44: The feature map F 12 Continue to input into the ELAN-SimAM-D module, in the ELAN-SimAM-D module, obtain the final output feature map F1′5 of the ELAN-SimAM-D module, and the size of the feature map F1′5 is 16C×H / 16×W / 16;

[0124] S5: Design spatial pyramid adaptive pooling fusion module;

[0125] Among them: The specific process of S5 is as follows:

[0126] S51: The road disease feature map F1′5 with a size of 16C×H / 16×W / 16 is used as the input feature map. The first branch is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16. 16 ;

[0127] S52: The second branch F1′5 inputs the Conv2D_BN_LeakyRelu module, first performs a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then performs a nonlinear activation function LeakyReLU enhancement, and then performs BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16 17 ;

[0128] S53: Feature map F 17The first three branches are processed with adaptive pooling of sizes 5×5, 9×9, 13×13 and step size 1, respectively, to obtain the feature map F 17 ', feature map F 17 "、Feature graph F 17 ”', the fourth branch does not process the input feature map, and the feature map sizes output by the four branches are all 8C×H / 16×W / 16;

[0129] Among them, the feature map F 17 ', feature map F 17 "、Feature graph F 17 The specific calculation process of adaptive pooling of ” is as shown in formula (5) to formula (8);

[0130]

[0131] in, is the adaptive pooling output, represents smooth approximate maximum pooling, represents smooth approximate average pooling, β = {0, ..., 1} is related to the downsampling volume The weight mask of the same size, R represents the kernel neighborhood, i and j are the coordinate sets of R, Based on the inverse of the distance between each activation and the average activation, AdaPool can calculate the gradient for each kernel vector during the back-propagation process, which can adaptively capture details and make pooling more regionally adaptive;

[0132] S54: The feature graph F1′7, the feature graph F1′7′, the feature graph F1′7″, the feature graph F1′7″ 17 Splice and finally get the feature map F 18 , feature map F 18 The dimensions are 32C×H / 16×W / 16;

[0133] S55: The feature map F 18 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H×W 19 ;

[0134] S56: The feature map F 19 With the feature map F 16 After concatenation, we finally get a feature map F with a size of 16C×H / 16×W / 16 20 ;

[0135] S57: The feature map F 20 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F of size 8C×H / 16×W / 16 21 .

[0136] The embodiment of this step is as follows: Assume that the size of the road damage feature map is C×H×W, where C is the number of channels of the feature map, H is the height of the feature map, and W is the width of the feature map; Example: For a road damage target feature map F with an input size of 512×20×20 15 The first branch performs a CBL convolution operation with a convolution kernel of 1×1 and a step size of 1 on the feature map F 15 Convolution is performed to reduce the dimension, and finally the CBL module outputs a feature map F with a size of 256×20×20 16 ; The second branch performs a CBL convolution operation with a convolution kernel of 1×1 and a step size of 1 on the feature map F 15 Convolution is performed to reduce the dimension, and finally the CBL module outputs a feature map F with a size of 256×20×20 17 ; Feature map F 17 The first three branches are processed with AdaPool of size 5×5, 9×9, 13×13 and step size 1, respectively, to obtain the feature map feature map F 17 ', feature map F 17 "、Feature graph F 17 ”', the fourth branch does not process the input feature map; the feature map F 17 ', feature map F 17 "、Feature graph F 17 ”', feature map F 17 After concatenation, we finally get a feature map F with a size of 1024×20×20 18 ; Feature map F 18 Perform a CBL convolution operation with a convolution kernel of 1×1 and a step size of 1 to reduce the dimension of the feature map F4. Finally, the CBL module outputs a feature map F with a size of 256×20×20. 19 ; Then the feature map F 19 With the feature map F 16 After concatenation, we get a feature map F with a size of 512×20×20 20 ; The feature map F 20Perform a CBL convolution operation with a convolution kernel of 1×1 and a step size of 1 on the feature map F 20 Perform convolution to reduce dimension and finally get the feature map F 21 , F 21 The size is 256×20×20;

[0137] In this process, in order to enhance the detection capability of irregular road defects, a spatial pyramid adaptive pooling fusion module (SPPAdaF module) is designed. By introducing adaptive pooling (AdaPool), the model can retain important information of feature maps and improve the model's ability to extract features.

[0138] The spatial pyramid adaptive pooling fusion module (SPPAdaF module) includes the CBL module and the adaptive pooling layer (called AdaPool layer). AdaPool is an adaptive pooling layer. In this network layer, the input features are adaptively pooled with different convolution kernel sizes, strides, and padding, which can retain important information of the features and improve the detection accuracy of the model.

[0139] The specific process of the SPPAdaF module is as follows: For the input feature map F1′5, the SPPAdaF module processes the input feature map in two different branches, and inputs the CBL module in the first branch to obtain the feature map F 16 ; Input the CBL module in another branch to obtain the feature map F 17 ; Then the feature map F 17 The first three branches are processed with adaptive pooling of sizes 5×5, 9×9, 13×13 and step size 1, respectively, to obtain feature maps F1′7, feature map F1′7′, feature map F1′7″; feature maps F1′7, feature maps F1′7′, feature maps F1′7″, feature maps F1′7″, feature maps F1′7′ 17 Concatenate to get the feature map F 18 ; Feature map F 18 Input the CBL module to get the feature map F 19 , feature map F 19 and feature map F 16 Splice and get the feature map F 20 ; Final feature map F 20 Input the CBL module to get the final output feature map F 21 ;SPPAdaF module such as Figure 4 As shown;

[0140] S6: Use the FPN module for feature fusion (FPN is a pyramid feature fusion module used to fuse the feature maps of different scales obtained previously);

[0141] Among them: The specific process of S6 is as follows:

[0142] S61: Make feature map F7, feature map F 11 , feature map F 21 As the input of FPN module;

[0143] S62: FPN module inputs feature map F7, feature map F 11 , feature map F 21 Perform feature fusion and finally output feature map F 22 , feature map F 23 , feature map F 24 ;

[0144] In this process, the FPN module is a structure used for target detection, which aims to extract rich features from different scales. It builds a pyramid structure at different levels of the convolutional network to fuse low-resolution high-level features and high-resolution low-level features, thereby enhancing the detection ability of targets of various sizes.

[0145] S7: Design NWD-EIoU loss function;

[0146] Among them: The specific process of S7 is as follows: In order to improve the model's detection ability for small target diseases, the EIoU loss function and the NWD loss function are combined to form a new road disease detection bounding box positioning loss function NWD-EIoU; The NWD-EIoU loss function can improve the accuracy of the small target disease positioning box, thereby reducing the false detection and missed detection of small target diseases; Compared with the traditional loss function, the NWD-EIoU loss function can better consider the overlap between targets, handle targets of unconventional shapes, and split the aspect ratio loss into the difference between the predicted width and height and the minimum external frame width and height, which accelerates the convergence speed of the model and improves the regression accuracy; The detection of road diseases can effectively maintain scale invariance, so that the model has better robustness in detecting road diseases, and at the same time improves the detection accuracy of small targets;

[0147] Its NWD-EIoU loss function L NWD_EIoU The calculation formula is shown in formula (9);

[0148] L NWD-EIoU =α·L EIoU +(1-α)·L NWD (9)

[0149] Where a represents a regular term, the value is from 0 to 1, L EIoU Represents the value of the EIoU loss function, L NWDIt represents the value of the NWD loss function. The NWD-CIoU loss function combines the two loss functions, adjusts the weights of the two parameters through the regularization term a, and determines the most appropriate value of a during the training process;

[0150] In addition, L EIoU and L NWD The specific calculation formulas are shown in formulas (10), (11), (12), and (13) respectively;

[0151]

[0152] Among them, IOU represents the intersection of the predicted box and the real box, I represents the intersection of the predicted box and the real box, U represents the union of the predicted box and the real box, b and b gt Represent the center points of the predicted box and the real box, w and w respectively gt Represent the width of the predicted box and the real box, h and h respectively gt Represents the height of the predicted box and the real box, w c and h c is the width and height of the minimum bounding rectangle of the predicted bounding box and the true bounding box, ρ is the Euclidean distance between the two points, EIOU splits the aspect ratio loss into the difference between the predicted width and height and the minimum bounding box width and height, which accelerates the convergence speed of the model and improves the regression accuracy. C represents the number of types detected in the dataset. is a second-order Wasserstein distance definition, where N a and N b Respectively represent the bounding box A=(cx a ,cy a ,w a ,h a ) and bounding box B = (cx b ,cy b ,w b ,h b ), ||·|| F is the Frobenius norm;

[0153] S8: Use Head detection;

[0154] Among them: The specific process of S8 is as follows:

[0155] S81: subject feature maps from different scales to several layers of convolution operations to further extract features;

[0156] S82: Generate multiple anchor boxes for bounding box prediction through the predefined anchor box mechanism. Each anchor box outputs the center coordinates of the bounding box, the adjustment amount of width and height, and the confidence score, which indicates the probability of containing the object in the box.

[0157] S83: The network outputs the classification score of each anchor box and removes redundant detection boxes through non-maximum suppression to generate the final detection result;

[0158] In this process, the Head detection head is the last part of the network, responsible for generating the final target detection results, including the bounding box position, confidence and category;

[0159] S9: Model training and validation;

[0160] Among them: The specific process of S9 is as follows:

[0161] S91: Perform parameter training and update on the road damage detection model constructed in steps S2-S8, initialize all neural network parameters, and set related hyperparameters, such as training rounds, batch size, optimizer, learning rate, and maximum value of gradient clipping strategy;

[0162] S92: Divide the road damage data set processed by S1 into a training set, a validation set and a test set. The training set and the validation set are divided into multiple batches. Each time, a batch of training set data is input into the road damage detection model for training, and the training loss value loss of the batch is calculated;

[0163] S93: After completing one round of training for all batches of data in the entire training set (there may be multiple rounds of training), the validation set is input into the road damage detection model in batches to obtain the corresponding batch loss value batch_loss;

[0164] S94: During the training and verification process, the road damage detection model will automatically learn and adjust parameters according to the loss and batch_loss conditions each time. The training process will continue for one or more rounds until the batch_loss value converges, indicating that the road damage detection model training is completed;

[0165] In the RDDA algorithm, the ELAN-SimAM-D module is designed by introducing the deep separable convolution and SimAM attention mechanism; after the standard convolution, the parameter quantity Params conv And the amount of calculation Flops conv And the number of parameters after depth-separable convolution Params dsc And the amount of calculation Flops dsc As shown in formulas (14), (15), (16), and (17);

[0166] Params conv =M×K×K×N (14)

[0167] Flops conv =F×F×M×K×K×N (15)

[0168] Params dsc =M×(K×K+N) (16)

[0169] Flops dsc =F×F×M×(K×K+N) (17)

[0170] Among them, K×K represents the size of the convolution kernel, M represents the input channel, N represents the output channel, and F×F represents the size of the output feature map. Therefore, through calculation, it can be seen that the depth-separable convolution can effectively reduce the number of parameters and the amount of calculation of the model. Through verification, RDDA also has fewer parameters and calculations than traditional algorithms.

[0171] At the same time, the RDDA algorithm uses adaptive pooling (AdaPool) to design the SPPAdaF module and a new loss function NWD-EIoU loss function. It has been verified that compared with traditional detection algorithms, the RDDA algorithm has a better detection effect on road diseases, and the missed detection and false detection are lower.

[0172] S10: Applying the model (using the trained road damage detection model to detect road surface images to automatically detect road damage on the road surface);

[0173] Among them: The specific process of S10 is as follows:

[0174] S101: input the collected road damage pavement image data into the detection model, and the model will automatically detect the image to obtain the detection result of the road damage image, and the detection result includes the category, location and detection confidence information of the road damage;

[0175] S102: Display the detection result on the image, see the example Figure 5 .

[0176] In the whole process, the structure diagram of RDDA algorithm is as follows Figure 2As shown in the figure, the main modules of the RDDA algorithm include Conv2D_BN_LeakyRelu (CBL) module, ELAN module, ELAN-SimAM-D module, MaxPool module, SPP DaF module, FPN module, and Head module; Conv2D_BN_LeakyRelu module is a convolution module, in which the input feature map is first subjected to standard convolution operation, and then enhanced by the nonlinear activation function LeakyReLU, and then subjected to batch normalization; ELAN module is a feature extraction module, in which standard convolution, feature splicing and other operations are performed on the input feature map. The ELAN-SimAM-D module is a new feature extraction module. In this module, the input feature map is extracted through operations such as deep separable convolution, standard convolution, and feature splicing. The extracted feature map will be assigned three-dimensional weights by the channel-free attention mechanism SimAM, and finally the feature map will be output; the ELAN-SimAM-D module uses deep separable convolution for feature extraction, which greatly reduces the number of parameters and calculations compared to ordinary convolution modules. At the same time, the dimensional weights that can be assigned by the SimAM attention mechanism can effectively reduce the interference of complex backgrounds on features and improve the detection accuracy of the model; the MaxPool module is a maximum pooling module, and its main functions include Downsampling and increasing the receptive field can reduce the dimension and size of the data, thereby reducing the number of network parameters and the amount of computation, and improving the efficiency of the model; the SPP DaF module is a feature fusion module, in which the features are pooled by introducing adaptive pooling to effectively retain the important information of the features, thereby reducing the missed detection rate and false detection rate of the model and improving the detection accuracy of the model; the FPN module is an enhanced feature extraction module, which fuses the extracted feature maps of different scales and then sends the fused features to the Head module for detection; the Head module is the detection head of the model, which detects the input feature map and outputs the detection result of the image; in the training of the RDDA algorithm, the NWD-EIoU loss function is used As the loss function of the RDDA algorithm, the NWD-EIoU loss function can maintain scale invariance, better consider the overlap between targets, and better handle targets of unconventional shapes, so that the model can improve the detection accuracy of road defects while enhancing the robustness of the model; among them, unconventional shapes usually refer to irregular shapes or unusual shapes; unconventional shapes can refer to those shapes that do not belong to standard geometric shapes (such as circles, rectangles, triangles, etc.), such as ellipses, pentagons, irregular polygons, etc.; among road defects, such as cracking, the appearance of irregular cracks on the road surface may be like a spider web, or a twisted, nonlinear shape, which is an unconventional shape defect.

[0177] In addition, embodiments of the present invention are as follows:

[0178] A road damage dataset of 2912 images with 9426 labeled objects covering 6 categories was used to train a road damage detection model.

[0179] Dataset annotation: The collected road damage dataset was annotated, including target category and location information;

[0180] Dataset division: The dataset is randomly divided into training set, validation set and test set in a ratio of 6:2:2; specifically, the training set contains 1748 images, while the validation set and test set each contain 582 images;

[0181] Model training: The data in the training set is input into the road damage detection model for training; the entire training process is set to 300 epochs, and the parameters in the model are adjusted according to the loss value in each iteration;

[0182] Training results: After adjusting the relevant parameters, the accuracy of the road damage detection model finally reached a stable state. On the test set, the average accuracy of the model fluctuated between 80% and 88%.

[0183] experiment:

[0184] Table 1 Comparison of experimental results.

[0185]

[0186]

[0187] The experimental results show that the YOLOV5 algorithm has a mAP value of 78.79%, but the number of parameters is large and the recall rate is low, which requires high hardware for the deployed equipment. The YOLOV7 algorithm has poor detection effect, and the number of parameters reaches 142.1M, which cannot meet the detection requirements for road diseases. The YOLOV8 detection algorithm has a mAP value of 80.14, a parameter number of 42.6, and a recall rate of 79.24. Although it is slightly higher than the average accuracy of YOLOV7-tiny, the number of parameters is too large to meet the needs of road disease detection. The RDDA algorithm has a mAP@0.5 of 82.14%, a detection speed of 60.41 FPS, a recall rate of 83.70%, and a model parameter number of 7.74M. Compared with traditional detection algorithms, the detection accuracy is higher, and the number of parameters is reduced, which meets the deployment of edge devices. The FPS of the RDDA algorithm reached 60.41, which can meet the real-time needs of road disease detection.

[0188] result:

[0189] In the present invention, deep separable convolution (DSC) and SimAM attention mechanism are used to enhance the model's ability to detect road damage, reduce the interference of complex background, improve feature extraction efficiency, and effectively reduce model parameters.

[0190] In the present invention, adaptive pooling AdaPool is introduced through the SPPAdaF module, so that the pooling process is more regionally adaptive, which helps to better extract and retain the key features of road diseases, thereby improving the detection accuracy and reducing false detections.

[0191] In this invention, the normalized Gauss-Wasserstein distance (NWD) is combined with EIoU to construct a new loss function NWD-EIoU, thereby improving the localization accuracy of small target diseases in complex backgrounds, better considering the overlap between targets, and being able to handle targets of unconventional shapes. The aspect ratio loss is split into the difference between the predicted width and height and the minimum bounding box width and height, which accelerates the convergence speed of the model and improves the regression accuracy.

[0192] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A road disease detection method based on deep learning, characterized in that: The steps of using the RDDA algorithm for detection are as follows: S1: Construction of road damage dataset; S2: build image feature extraction module; S3: Design a deep separable parameter-free attention mechanism convolution module; S4: Extract deep features of feature maps; S5: Design spatial pyramid adaptive pooling fusion module; S6: Use FPN module for feature fusion; S7: Design NWD-EIoU loss function; S8: Use Head detection; S9: Model training and validation; S10: Apply the model; The specific process of S2 is as follows: S21: Input the road damage image in the data set into the Conv2D_BN_LeakyRelu module for processing. In the Conv2D_BN_LeakyRelu module, the input image is first subjected to a standard convolution operation with a convolution kernel of 3×3 and a step size of 2, and then enhanced by the nonlinear activation function LeakyReLU, and then subjected to batch normalization processing to generate an initial feature map F1. The size of the feature map F1 is assumed to be C×H×W, where C represents the number of channels, H represents the height, and W represents the width. S22: Input the feature map F1 to the Conv2D_BN_LeakyRelu module. In the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 3×3 and a step size of 2 on F1, then perform a nonlinear activation function LeakyReLU enhancement, and then perform Batch Normalization to generate a feature map F2. The size of the feature map F2 is 2C×H / 2×W / 2. S23: Input feature map F2 to the ELAN module, and output feature map F3 after convolution and concatenation operations. The size of F3 is 2C×H / 2×W / 2; S24: The feature map F3 is further input to the MaxPool module. In the MaxPool module, F3 is subjected to maximum pooling to finally obtain a feature map F4, the size of which is 2C×H / 4×W / 4. The specific process of S3 is as follows: S31: The road disease feature map F4 with a size of 2C×H / 4×W / 4 obtained in S2 is used as the input feature map. The first branch F4 is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F5 with a size of 2C×H / 4×W / 4. S32: The second branch F4 performs a DSC convolution with a convolution kernel of 1×1 and a step size of 1, and two DSC convolutions with a convolution kernel of 3×3 and a step size of 1. In the ELAN-SimAM-D module, the output of each DSC convolution is used as the input of the next DSC convolution, which is repeated three times in sequence. Finally, the three DSC convolutions output feature maps F′4, F″4, and F4″′ respectively. The feature maps F′4, F″4, and F4″′ are all 2C×H / 4×W / 4. S33: concatenate the feature map F′4, the feature map F4″, the feature map F4″′ and the feature map F5 to obtain a feature map F6 with a size of 8C×H / 4×W / 4; S34: Input the feature map F6 into the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform BatchNormalization batch normalization. The Conv2D_BN_LeakyRelu module outputs a feature map F7 with a size of 4C×H / 4×W / 4. Then the feature map F7 is input to the SimAM layer, and three-dimensional weights are assigned to the key features of the small target. The complex background area and the small target area are divided into pixels. After passing through the SimAM module, the feature map F7' is finally obtained, and the size of F7' is 4C×H / 4×W / 4; Among them, the calculation formula of F7' is shown in formula (1)-(4); Among them, t is the target neuron, λ is the regularization term, is the energy function, The smaller the value of , the greater the difference between the characteristics of the neuron and its neighboring neurons, and the more important t is. represents the mean value of all neurons in each channel, is the variance of the neuron in each channel, x i are other neurons in the i-th channel, N is the number of all neurons on a channel, E is the sum of the energy functions in all channels and spatial dimensions, the sigmoid function ensures that the value of E is not too large, ⊙ represents the Hadamard product; The specific process of S4 is as follows: S41: The feature map F7' is further input to the MaxPool module, and after convolution and concatenation operations, the feature map F8 is output. The size of the feature map F8 is 4C×H / 8×W / 8; S42: The feature map F8 is further input to the ELAN-SimAM-D module to obtain the final output feature map F1′1 of the ELAN-SimAM-D module. The size of the feature map F1′1 is 8C×H / 8×W / 8; S43: The feature map F1′1 is further input to the MaxPool module, and the feature map F is output after convolution and concatenation operations. 12 , feature map F 12 The dimensions are 8C×H / 16×W / 16; S44: The feature map F 12 Continue to input into the ELAN-SimAM-D module, in the ELAN-SimAM-D module, obtain the final output feature map F1′5 of the ELAN-SimAM-D module, and the size of the feature map F1′5 is 16C×H / 16×W / 16; The specific process of S5 is as follows: S51: The road disease feature map F1′5 with a size of 16C×H / 16×W / 16 is used as the input feature map. The first branch is input into the Conv2D_BN_LeakyRelu module. First, a standard convolution operation with a convolution kernel of 1×1 and a step size of 1 is performed. Then, the nonlinear activation function LeakyReLU is enhanced, and then batch normalization is performed. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16. 16 ; S52: The second branch F1′5 inputs the Conv2D_BN_LeakyRelu module, first performs a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then performs a nonlinear activation function LeakyReLU enhancement, and then performs Batch Normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H / 16×W / 16 17 ; S53: Feature map F 17 The first three branches are processed with adaptive pooling of sizes 5×5, 9×9, 13×13 and step size 1, respectively, to obtain the feature map F 17 ', feature map F 17 "、Feature graph F 17 ”', the fourth branch does not process the input feature map, and the feature map sizes output by the four branches are all 8C×H / 16×W / 16; Among them, the feature map F 17 ', feature map F 17 "、Feature graph F 17 The specific calculation process of adaptive pooling of ” is as shown in formula (5) to formula (8); in, is the adaptive pooling output, represents smooth approximate maximum pooling, represents smooth approximate average pooling, β = {0, ..., 1} is related to the downsampling volume The weight mask of the same size, R represents the kernel neighborhood, i and j are the coordinate sets of R, Based on the inverse of the distance between each activation and the average activation, AdaPool can calculate the gradient for each kernel vector during the back-propagation process, which can adaptively capture details and make pooling more regionally adaptive; S54: The feature graph F1′7, the feature graph F1′7′, the feature graph F1′7″, the feature graph F1′7″ 17 After concatenation, we finally get the feature map F 18 , feature map F 18 The dimensions are 32C×H / 16×W / 16; S55: The feature map F 18 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F with a size of 8C×H×W 19 ; S56: The feature map F 19 With the feature map F 16 After concatenation, we finally get a feature map F with a size of 16C×H / 16×W / 16 20 ; S57: The feature map F 20 Input the Conv2D_BN_LeakyRelu module, first perform a standard convolution operation with a convolution kernel of 1×1 and a step size of 1, then perform the nonlinear activation function LeakyReLU enhancement, and then perform batch normalization. Finally, the Conv2D_BN_LeakyRelu module outputs a feature map F of size 8C×H / 16×W / 16 21 ; The specific process of S6 is as follows: S61: Make feature map F7, feature map F 11 , feature map F 21 As the input of FPN module; S62: FPN module inputs feature map F7, feature map F 11 , feature map F 21 Perform feature fusion and finally output feature map F 22 , feature map F 23 , feature map F 24 ; The RDDA algorithm includes: Conv2D_BN_LeakyRelu module, ELAN module, ELAN-SimAM-D module, MaxPool module, SPP DaF module, FPN module, Head module; The Conv2D_BN_LeakyRelu module is used to perform a standard convolution operation on the input feature map, and then perform a nonlinear activation function LeakyReLU enhancement, and then perform a Batch Normalization batch normalization process; The ELAN module is used to extract features from the input feature map through standard convolution and feature concatenation operations; The ELAN-SimAM-D module is used to extract features from the input feature map through depth-separable convolution, standard convolution, and feature concatenation operations. The extracted feature map is assigned a three-dimensional weight by the channel-free attention mechanism SimAM, and finally the feature map is output; The MaxPool module is used to reduce the dimension and size of the data, reduce the number of network parameters and the amount of calculation, and improve the efficiency of the model; The SPPAdaF module is used to pool features by introducing adaptive pooling, effectively retaining important information of features, thereby reducing the missed detection rate and false detection rate of the model and improving the detection accuracy of the model; The FPN module is used to fuse the extracted feature maps of different scales, and then send the fused features to the Head module for detection; The Head module is used to detect the input feature map and output the detection result of the image.

2. The road damage detection method based on deep learning according to claim 1, characterized in that: The specific process of S1 is as follows: S11: Use drones to photograph road disease sections and collect road disease images as a data set; S12: Preprocess the data set, screen qualified images, balance the number of road disease samples in each category, and annotate the collected data set using annotation software.

3. The road damage detection method based on deep learning according to claim 1, characterized in that: The specific process of S7 is as follows: combining the EIoU loss function and the NWD loss function to form a new road disease detection bounding box positioning loss function NWD-EIoU; Its NWD-EIoU loss function L NWD_EIoU The calculation formula is shown in formula (9); L NWD-EIoU =α·L EIoU +(1-a)·L NWD (9) Where a represents a regular term, the value is from 0 to 1, L EIoU Represents the value of the EIoU loss function, L NWD It represents the value of the NWD loss function. The NWD-CIoU loss function combines the two loss functions, adjusts the weights of the two parameters through the regularization term a, and determines the most appropriate value of a during the training process; In addition, L EIoU and L NWD The specific calculation formulas are shown in formulas (10), (11), (12), and (13) respectively; Among them, IOU represents the intersection of the predicted box and the real box, I represents the intersection of the predicted box and the real box, U represents the union of the predicted box and the real box, b and b gt Represent the center points of the predicted box and the real box, w and w respectively gt Represent the width of the predicted box and the real box, h and h respectively gt Represents the height of the predicted box and the real box, w c and h c is the width and height of the minimum bounding rectangle of the predicted bounding box and the true bounding box, ρ is the Euclidean distance between the two points, EIOU splits the aspect ratio loss into the difference between the predicted width and height and the minimum bounding box width and height, which accelerates the convergence of the model and improves the regression accuracy. C represents the number of types detected in the data set, and W2 2 (N a ,N b ) is a second-order Wasserstein distance definition, where N a and N b Respectively represent the bounding box A=(cx a ,cy a ,w a ,h a ) and bounding box B = (cx b ,cy b ,w b ,h b ), ||·|| F is the Frobenius norm.

4. The road damage detection method based on deep learning according to claim 1, characterized in that: The specific process of S8 is as follows: S81: subject feature maps from different scales to several layers of convolution operations to further extract features; S82: Generate multiple anchor boxes for bounding box prediction through the predefined anchor box mechanism. Each anchor box outputs the center coordinates of the bounding box, the adjustment amount of width and height, and the confidence score, which indicates the probability of containing the object in the box. S83: The network outputs the classification score of each anchor box and removes redundant detection boxes through non-maximum suppression to generate the final detection result.

5. The road damage detection method based on deep learning according to claim 1, characterized in that: The specific process of S9 is as follows: S91: Initialize all neural network parameters and set related hyperparameters; S92: Divide the road damage data set processed by S1 into a training set, a validation set and a test set. The training set and the validation set are divided into multiple batches. Each time, a batch of training set data is input into the road damage detection model for training, and the training loss value loss of the batch is calculated; S93: After completing a round of training for all batches of data in the entire training set, the validation set is input into the road damage detection model in batches to obtain a corresponding batch loss value batch_loss; S94: During the training and verification process, the road damage detection model will automatically learn and adjust parameters according to the loss and batch_loss conditions each time. The training process will continue for one or more rounds until the batch_loss value converges, indicating that the road damage detection model training is completed; The specific process of S10 is as follows: S101: input the collected road damage pavement image data into the detection model, and the model will automatically detect the image to obtain the detection result of the road damage image, and the detection result includes the category, location and detection confidence information of the road damage; S102: Display the detection result on the image.

Images