Road defect detection method based on DRR module and SDFM

Abstract

The invention discloses a road defect detection method based on a DRR module and an SDFM. The method comprises the following steps: S1, data acquisition and processing; s2, constructing a model: replacing a DWR module with a DRB module, and replacing a BottleNeck residual module of a backbone network part C2f through the DRR module; downsampling of a backbone network and a neck is replaced by an LDConv convolution module, a neck structure of an SDFM improved model is used, and features of different scales are fused to serve as input of a detection head; s3, model training: training the improved network model by using the made training set by setting training hyper-parameters; s4, model verification: setting verification parameters to verify the trained model; and deploying the verified model to a visual interface to realize real-time detection of road defects. According to the method, the model is improved, so that the detection algorithm has higher accuracy and robustness, and can adapt to different road defects and environmental conditions.

Description

Technical Field

[0001] The invention belongs to the technical field of geo-statistics and spatial analysis, and in particular relates to a road defect detection method based on a DRR module and SDFM. Background Art

[0002] In recent years, with the development of the economy, the number of vehicles has increased dramatically, and the impacts of overloading of large trucks, road degradation, bad weather, etc. will cause damage to the road structure, resulting in cracks, cracks, potholes, collapses and other defects. If these road damages are not discovered and repaired in time, it will pose a great safety hazard to future driving safety. Therefore, it is crucial to discover and repair road damage in a timely manner.

[0003] The traditional method of road defect detection is mainly manual inspection, but the number of roads in the country is huge, the manual inspection method is time-consuming and labor-intensive, and the safety of the inspectors cannot be guaranteed.

[0004] With the advancement of technology, automated inspection is mainly based on multifunctional road inspection vehicles. Compared with manual inspection, multifunctional road inspection vehicles mainly rely on integrated sensors such as GPS, cameras, laser profilers, and ground penetrating radars to achieve convenient and accurate detection of road defects. The construction cost of road measurement vehicles is expensive and is still not suitable for large-scale promotion. Therefore, the current automated inspection system still needs to be improved in terms of the accuracy of road defect identification and the real-time response capability of the algorithm.

[0005] Chinese patent document CN118115444A discloses a road defect target detection method and system based on SLCD-YOLO network, and the specific steps of the method are as follows: S1, select a data set of road damage and perform preprocessing; S2, configure the training environment; S3, improve the YOLOv8 structure to obtain the SLCD-YOLO network model; S4, integrate the SLCD-YOLO network model obtained in step S3 into the training environment configured in step S2, and add the set parameter file to the network model; use the divided training set and verification set images to train and verify the model. The present invention is suitable for real-time and efficient road defect monitoring and identification. In order to improve the inference speed of the model, this road defect detection method introduces SimSPPF in Backbone. The inference speed of a single ConvBNReLU module in SimSPPF is 18% faster than that of ConvBNSiLU in SPPF. At the same time, in order to further reduce the computational complexity of the model, a lightweight cross-scale feature fusion module CCFM module is introduced in Neck. CCFM downsamples the feature map extracted by Backbone in the channel dimension, reducing the number of channels of the output feature map, effectively reducing the computational complexity of the model. In order to solve the accuracy problem of small target defect detection, the LSKA feature selection module is introduced in Backbone. The LSKA feature selection module dynamically selects appropriate convolution kernels by considering the local information of the input feature map to adapt to the contextual information of various target types. At the same time, in order to better obtain multi-scale information, the DWRSeg module is introduced in Neck and fused with C2f to form a new C2f_DWRSeg module. C2f_DWRSeg uses multi-rate depth-direction dilated convolution to effectively extract multi-scale contextual information. In order to further improve the defect detection problem at multiple angles, a lightweight upsampling operator CARAFE is introduced in Neck, which retains more original low-level feature information such as texture, color and structure during the sampling process.

[0006] With the rapid development of deep learning, object detection technology has also been applied to road defects. Object detection technology is mainly divided into two categories. The first category is a two-stage detection model based on regions, which is mainly divided into two processes. The first step is to propose several regions that may contain objects, and the second step is to run a classification network on the proposed regions to obtain the object category in each region. Common two-stage algorithms include region-based fast convolutional neural network (Fast R-CNN), region-based full convolutional network (R-FCN), and mask region-based convolutional neural network (Mask R-CNN). The second is a one-step detection method based on regression, which directly separates specific categories and regresses boundaries. It is faster than the two-stage detection method, but the accuracy is slightly lower. The above methods generally have the disadvantages of large number of parameters and slow detection speed, which are not suitable for real-time detection.

[0007] The above fall detection methods have some disadvantages: the manual detection method relies on manual inspection, and there are many roads, which is not only time-consuming and laborious, but also subject to subjective influence of personnel and may be dangerous. The multifunctional road detection vehicle based on automation relies on certain sensors, which limits the deployment and use of the system, and the detection range and coverage are usually limited by the sensor's perception range. In addition, the detection vehicle is expensive, which puts forward requirements for the real-time and stability of the system. The target detection algorithm based on deep learning requires a clear camera field of view and good lighting conditions to accurately capture road information. In complex or harsh environments, the performance of the algorithm may decline. And a large amount of data is required to learn features before detection. In contrast, the target detection method based on deep learning has better flexibility and generalization ability. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a road defect detection method based on a DRR module and SDFM. The deep learning method in the method can automatically learn and extract features and adapt to different road defects and environmental conditions. The present invention improves the existing algorithm so that its detection algorithm has higher accuracy and robustness. It also overcomes the problems of poor robustness and low accuracy caused by road defects that are too large or too small in the prior art. The yolov8 model architecture is used, which has the advantages of simple structure and relatively few parameters.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is: the road defect detection method based on the DRR module and the SDFM specifically comprises the following steps:

[0010] S1 Data collection and processing: Collect road disease pictures, screen the pictures, and then process the data; after annotation, generate a data set and divide it into a training set and a validation set;

[0011] Construction of the S2 model: First, replace the DWR module with the DRB module, and replace the BottleNeck residual module of the backbone network part C2f with the DRR module; secondly, replace the downsampling of the backbone network and the neck with the LDConv convolution module, and finally, use the surface detail fusion module SDFM to improve the neck structure of the model and fuse features of different scales as the input of the detection head;

[0012] S3 model training: By setting training hyperparameters, the improved network model is trained using the prepared training set;

[0013] S4 Model Verification: Set verification parameters to verify the trained model; deploy the verified model to the visualization interface to achieve real-time detection of road defects.

[0014] Using the above technical solution, in the data processing stage, the data set is downloaded from the website and filtered and integrated by crawling a part of the data, and then data enhancement is selected to double the original data, and divided into training set and validation set by 9:1. In the model improvement stage, firstly, the dilated convolution of D-3 and D-5 branches in the DWR module (Dilation-wise Residual) is replaced with the dilated reparameter module, named DRR (Dilated Reparam Residual), and the BottleNeck residual module of the backbone network part C2f is replaced by the designed DRR module, named C2f_DRR, to obtain more scale features; secondly, the downsampling of the backbone network and the neck is replaced by the LDConv convolution module, and the sampling shape is flexibly adjusted while the growth of the convolution parameters is kept linear; finally, the surface detail fusion module SDFM is used to improve the neck structure of the model, fuse the backbone network features and the neck features, and make full use of the shallow feature detail structure information and deep feature semantic information. It overcomes the problems of poor robustness and low accuracy caused by road defects that are too large or too small in the existing technology, and uses the yolov8 model architecture, which has the advantages of simple structure and relatively few parameters.

[0015] Preferably, the specific steps in step S1 are:

[0016] S11: Download the road disease dataset and crawl the images from the web, and filter the images obtained; select 1,500 images from the road disease dataset and 500 images crawled by the crawler;

[0017] S12: Perform data enhancement on the selected images, perform data enhancement once for each image, and then randomly scale and crop them to obtain a data set; the data set contains 4,000 images in total, including eight categories: horizontal cracks, vertical cracks, block cracks, cracks, pits, repaired mesh cracks, repaired cracks, and repaired pits;

[0018] S13: Label the data in the dataset, that is, use the open source software LabelImg to collect the transmission line foreign body data, and use Make Sense for semi-automatic labeling, and randomly divide the processed dataset into a training set and a validation set in a ratio of 9:1. The training set contains 3,600 images and the validation set contains 400 images.

[0019] Preferably, the specific steps of step S2 are:

[0020] S21: First, the dilated convolutions of the D-3 and D-5 branches in the DWR module (Dilation-wise Residual) are replaced with the DRB module (Dilated Reparam Block), named DRR (Dilated Reparam Residual), and the BottleNeck residual module of the backbone network C2f is replaced with the designed DRR module, named C2f_DRR, to obtain features of more scales; for the residual module of the DRR module, a two-step method is used to efficiently draw multi-scale features and obtain multi-scale context information, reducing the difficulty of acquisition;

[0021] S22: Replace the downsampling of the backbone network and neck with the LDConv convolution module, adjust the initial sampling shape by the offset, and add an offset to any irregular convolution shape, so that the parameter growth becomes linear growth, greatly reducing the parameters;

[0022] S23: A shallow feature fusion module (SDFM) based on a channel-spatial attention mechanism is used to fuse backbone network features and neck features as the detection head input.

[0023] Using the above technical solution, first, the D-3 and D-5 branches in the DWR module, that is, the extended convolution branches with expansion rates of 3 and 5, are replaced with DRB, that is, the extended re-parameter module, and named DRR, so that multi-scale context information can be efficiently obtained in a two-step method inside the residual, and more effective neighbor information can be extracted and a lot of parameters can be reduced by extending the re-parameter module. Finally, we replace the bottleneck in the last two C2f of the backbone network with the designed DRR module and name it C2f_DRR, while maintaining lightweight, fusing multi-scale feature maps; replace the downsampling method of the backbone network and the neck with LDConv, design the corresponding convolution shape through the change of the target feature, and dynamically adapt to the change of the object through the learnable offset, choose to generate the initial sampling coordinates to complete the feature extraction process with any number of parameters, and correct the growth of the number of deformable Conv parameters to linear growth. The convolution shape is designed based on prior knowledge and the network performance is improved. SDFM is used to improve the neck structure of the model. Shallow features contain rich detailed structural information, and deep features contain rich semantic information. SDFM is used to fuse the backbone information and neck information as the input of the detection head, which can better fuse the two parts of information.

[0024] Preferably, the specific steps of efficiently drawing multi-scale features using the two-step method in step S21 are:

[0025] S211 regional residualization: This is achieved through a common 3×3 convolution, combined with a batch normalization BN and ReLU layer. ReLU activates regional features in the regional residual and makes it more concise and efficient. Through the above parts, a series of feature maps with regions of different sizes are generated as input for morphological filtering.

[0026] S212 semantic residualization: morphological filtering is performed on the regional features of different sizes obtained in step S211 by using multi-rate extended depth convolution respectively, and the D-3 and D-5 branches here, i.e., the extended convolutions with the hole rates of 3 and 5, are replaced with extended re-parameterized convolution modules; and the extended re-parameterized convolution modules are then converted into a large kernel conversion layer for signal advancement;

[0027] S213 Aggregate Output: After finally obtaining the multi-scale context information for drawing, multiple outputs are aggregated; that is, all feature maps are connected together, and then the features are merged through BN and point-by-point convolution to form the final residual; finally, the final residual is added to the input feature map to build a stronger and more comprehensive feature representation.

[0028] Preferably, the specific steps of LDConv feature extraction in step S22 are:

[0029] S221: According to the size of N, the irregular convolution initial coordinate P is generated by the generation algorithm n ; Therefore, the original coordinate at position P0 is P n +P0; The focus of this step is to generate the corresponding sampling coordinates on the feature map for a convolution kernel with N parameters;

[0030] S222: Obtain the offset of the corresponding kernel through convolution operation, whose dimension is (B, 2N, H, W), and then add the offset to the original coordinate to generate a new sampling coordinate corresponding to the convolution; the focus of this step is to generate different sample shapes for convolution at different positions of the feature map;

[0031] S223: The features of the corresponding position are obtained by interpolation and resampling, and then the corresponding convolution operation is applied to extract the features, thereby completing the feature extraction. This step is mainly to extract the features of the corresponding position. Through the above three steps, LDConv can complete the convolution operation of any size to extract features.

[0032] Preferably, the specific steps of step S23 are:

[0033] S231: The second C2f output of the backbone network and the second C2f output of the neck are sent to the first SDFM fusion module as the first detection head input; the output of the third C2f of the backbone network, that is, the output of C2f_DRR and the output of the third C2f of the neck are sent to the second SDFM fusion module as the second detection head input; the output of the backbone network through SPPF and the output of the fourth C2f of the neck are sent to the third SDFM fusion module as the input of the third detection head;

[0034] S232: Concatenate the features of the backbone network and the neck features in the channel dimension, and then feed the concatenated features into a channel attention module consisting of convolution and pooling operations to generate attention weights; the attention weights are then applied to weight the original features by element-wise multiplication, and the resulting features are added to the original features from another branch to enhance the representation of the original features.

[0035] Preferably, the formula for the feature enhancement process in step S232 is expressed as:

[0036]

[0037] Among them, F a ' is the backbone network feature enhancement representation, F b ' is the neck network feature enhancement representation, F a is the backbone network feature, F b is the neck network feature, where is element-by-element addition, It is element-by-element multiplication, Pw-Conv n (·) represents n cascaded point-by-point convolutional layers, C(·) represents the channel size of the connection operation, δ(·) represents the sigmoid function, and GAP(·) represents the global average pooling; then, the enhanced features are concatenated in the channel dimension and input into the parallel channel attention and spatial attention modules to generate the final fusion weights.

[0038] Preferably, the formula for generating the fusion weight in step S232 is expressed as:

[0039] Alpha c =Pw-Conv n (GAP(Pw-Conv(C(F a ′,F b ′))));

[0040] Alpha s =Pw-Conv n (Pw-Conv(C(F a ',F b ')));

[0041]

[0042] Among them, Α c is the channel attention, Α s is the spatial attention, W is the generated fusion weight, and F a ' is the backbone network feature enhancement representation, F b ' is the enhanced representation of the neck network feature; the generated weight is used for one of the partial enhanced features, and the fusion weight of the other partial enhanced feature can be expressed as 1-W; therefore, the SDFM fusion process can be expressed as:

[0043]

[0044] Preferably, the specific steps of step S3 are: by setting the training hyperparameters, the improved model in step S3 is trained using the training set in step S1 to obtain a road defect detection model based on the DRR module and SDFM. For the model training hyperparameters, the training platform is BlueCloud, the test hardware platform environment is Python3.8, CUDA11.8, the GPU graphics card is NVIDIA RTX3090, and the memory is 30G; the training parameters are set as follows: the number of training times is 300 times, the BatchSize is set to 8, the SGD optimizer is used, and the input image size is 640*640.

[0045] Preferably, the specific steps of step S4 are: setting verification parameters to verify the road defect detection model based on the DRR module and SDFM trained in step S3 on the verification set, taking mAP@0.5, Precision, Recall, Parameters, and Gflops as evaluation indicators, where mAP@0.5 is the average precision of the detection results of each type, Precision is the proportion of correct targets among the targets detected by the model, Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; finally, the verified road defect detection model based on the DRR module and SDFM is deployed to the visual interface to realize real-time detection of road defects. The calculation formulas of each indicator are:

[0046]

[0047] Among them, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, FN is the number of missed targets, P(r) represents the smooth precision-recall curve, and the area under the smooth curve obtained by integrating it is AP; N represents the number of detected object categories, corresponding to 8 different road defects.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) High accuracy: The improved network further improves the detection accuracy by improving and adding different modules;

[0050] (2) Real-time performance: YOLOv8 is an object detection algorithm designed for real-time applications, and our improved model has fewer parameters, so the entire system can run efficiently in real-time scenarios and meet the needs of real-time road defect detection;

[0051] (3) Lightweight: By changing the downsampling method to LDConv, the convolution parameters increase linearly, reducing a lot of parameters. In addition, the three improved modules all have the effect of reducing parameters, making the model more lightweight.

[0052] (4) Universality: A road defect detection method based on the DRR module and SDFM can be applied to road defect detection tasks under various types of cameras and scenarios, and has strong universality and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the process of the road defect detection method based on the DRR module and the SDFM of the present invention;

[0054] Figure 2 It is a structural diagram of an improved model in the road defect detection method based on the DRR module and the SDFM of the present invention;

[0055] Figure 3 It is a comparison diagram of C2f_DRR and C2f structures in the road defect detection method based on DRR module and SDFM of the present invention;

[0056] Figure 4 It is a DRR structure diagram in the road defect detection method based on the DRR module and SDFM of the present invention;

[0057] Figure 5 The structure diagram of the extended re-parameterized convolution module (DRB) in the road defect detection method based on the DRR module and the SDFM of the present invention;

[0058] Figure 6 It is a structural diagram of LDConv in the road defect detection method based on DRR module and SDFM of the present invention;

[0059] Figure 7 It is a structural diagram of SDFM in the road defect detection method based on DRR module and SDFM of the present invention. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0061] Embodiment: The known technical content of the present invention - YOLOv8 target detection algorithm, specifically:

[0062] YOLOv8 technical features: YOLOv8 is a single-stage target detection algorithm. This algorithm adds some new improvements based on the previous version, which greatly improves its speed and accuracy. The improvements of the YOLOv8 target detection algorithm are as follows:

[0063] Input: Change the input image size to 640*640*3;

[0064] Benchmark network: mainly includes CBS, C2f, and SPPF;

[0065] The CBS module is a composite module composed of two-dimensional convolution (Conv2d), batch normalization (BN), and nonlinear activation function SiLU. The convolution layer extracts feature information by applying a set of learnable filters to the input data for convolution operations. In order to enhance the expressiveness of the network, the ELAN design concept is adopted. This module can obtain rich gradient flow information while ensuring the lightweight of the model. It consists of components such as Conv, Split, and BottleNeck, and adjusts the number of channels for models of different scales. Compared with C3, it has more skip-layer connections and additional Split operations.

[0066] Serial Parallel Pooling Fusion (SPPF) is a method based on Spatial Pyramid Pooling (SPP) that aims to expand the receptive field. It is implemented by serializing multiple 5×5 maximum pooling layers, thereby reducing the number of parameters and significantly reducing the amount of computation. The SPPF module accepts the feature map as input, which is then processed by the ConvBNSiLU module and performs a maximum pooling downsampling operation. Finally, the different downsampling results are concatenated to form the output feature map. By using SPPF, the receptive field can be effectively expanded with fewer parameters;

[0067] The application of YOLOv8's C2f structure and SPPF structure enables the model to better handle targets of different scales, while improving the accuracy and robustness of target detection while maintaining lightweight;

[0068] Neck network: The Neck part is responsible for multi-scale feature fusion. Its structure is PANet. The PANet structure consists of two parts: feature pyramid FPN and path aggregation network PAN. The combination of the two structures fully realizes the fusion of upper and lower feature information flows, and passes the three output branches to the Head structure through upsampling and channel splicing fusion. Compared with YOLOv5, YOLOv8 replaces the C3 in the neck with C2f, and removes the 1×1 convolution before upsampling;

[0069] FPN (Feature Pyramid Network) is a network structure for processing multi-scale features. It generates a series of feature maps with different scales by constructing feature pyramids at different levels of the backbone network. These feature maps contain rich semantic and spatial information and can capture the details and context of objects of different scales. The FPN structure uses upsampling and downsampling operations to fuse these feature maps to obtain a series of feature pyramids with different resolutions to meet the detection requirements of objects of different scales.

[0070] The PAN (Path Aggregation Network) structure is a further improvement on the basis of FPN. It introduces lateral connections and top-down pathways to better fuse and aggregate feature information. Lateral connections are used to combine high-resolution shallow features with low-resolution deep features to enrich and enhance feature representation. The top-down pathway combines lower-resolution feature maps with high-resolution feature maps through upsampling operations to obtain richer contextual information;

[0071] Head output layer: mainly adopts BCE Loss, CIOU_Loss, DFL_Loss, DIOU_nms, TAL label allocation strategy, and decoupling head structure design;

[0072] YOLOv8 uses BCE Loss as classification loss and CIOU_Loss+DFL_Loss as regression loss;

[0073] YOLOv8 abandons the Anchor-Base method and uses the Anchor-Free method, and finds a matching method to replace the side length ratio, TaskAligned. That is, the positive samples are selected based on the classification and regression scores as weighted scores. To match with NMS, the Anchor allocation of training samples needs to meet the following two rules: Normally aligned Anchors should be able to predict high classification scores and have precise positioning; misaligned Anchors should have low classification scores and be suppressed in the NMS stage. Based on the above two goals, TaskAligned designs a new Anchor alignment metric to measure the level of Task-Alignment at the Anchor level. In addition, the Alignment metric is integrated into the sample allocation and loss function to dynamically optimize the prediction of each Anchor;

[0074] In addition, YOLOv8 predictive box filtering introduces DIOU_nms. DIOU_nms considers the distance and IoU value between bounding boxes in the non-maximum suppression (NMS) process to select the most representative bounding boxes. It suppresses redundant prediction boxes while retaining the most relevant target boxes by comprehensively considering the position and shape information of the bounding boxes. This screening method can further improve the accuracy and recall rate of target detection;

[0075] The head part is used to obtain the category and location information of the target object based on feature maps of different sizes. It adopts a decoupled head structure design, which can separate the regression and prediction branches and enhance the generalization performance of the model.

[0076] like Figure 1 As shown, the road defect detection method based on the DRR module and the SDFM specifically includes the following steps:

[0077] S1 Data collection and processing: Collect road disease pictures, screen the pictures, and then process the data; after annotation, generate a data set and divide it into a training set and a validation set;

[0078] The specific steps in step S1 are:

[0079] S11: Download the road disease dataset and crawl the web to obtain pictures, and screen the obtained pictures. In this embodiment, the road disease dataset of the Global Campus Artificial Intelligence Algorithm Competition is downloaded, and pictures are crawled from the web. 1500 pictures of the road disease dataset are selected, and 500 pictures crawled by the crawler are selected.

[0080] S12: Perform data denoising and enhancement on the selected images, perform data enhancement once for each image, and then randomly scale and crop them to obtain a data set; in this implementation, the data set has a total of 4,000 images, including eight categories: horizontal cracks, vertical cracks, block cracks, cracks, pits, repaired mesh cracks, repaired cracks, and repaired pits;

[0081] S13: labeling the data in the data set, i.e., using the open source software LabelImg to collect the transmission line foreign body data, and using Make Sense to perform semi-automatic labeling, and randomly dividing the processed data set into a training set and a validation set in a ratio of 9:1; in this embodiment, the training set contains 3600 images and the validation set contains 400 images;

[0082] Construction of the S2 model: First, replace the DWR module with the DRB module, and replace the BottleNeck residual module of the backbone network part C2f with the DRR module; secondly, replace the downsampling of the backbone network and the neck with the LDConv convolution module, flexibly adjust the sampling shape while maintaining a linear growth in the number of convolution parameters; finally, use the surface detail fusion module SDFM to improve the neck structure of the model, and fuse features of different scales as the input of the detection head; improve the module based on YOLOv8 to obtain the improved model; the improvements to the YOLOv8 model are as follows: Figure 2 As shown; the specific steps of step S2 are:

[0083] S21: First, the dilated convolutions of the D-3 and D-5 branches in the DWR module (Dilation-wise Residual) are replaced with the DRB module (Dilated Reparam Block), named DRR (Dilated Reparam Residual), and the BottleNeck residual module of the backbone network part C2f is replaced by the designed DRR module, named C2f_DRR, to obtain features of more scales; for the residual module of the DRR module, a two-step method is used to efficiently draw multi-scale features, reduce the difficulty of acquisition, and obtain multi-scale context information; to obtain more scale context information, the comparison chart of C2f and C2f_DRR is shown in the figure below. Figure 3 As shown;

[0084] The DRR structure diagram is as follows Figure 4 As shown, the specific steps of using the two-step method to efficiently draw multi-scale features in step S21 are:

[0085] S211 regional residualization: Figure 4The RR part in the above is realized by a common 3×3 convolution, combined with a batch normalization BN and ReLU layer. ReLU activates the regional features in the regional residual and makes it more concise and efficient. Through the above parts, a series of feature maps with regions of different sizes are generated as the input of morphological filtering.

[0086] S212 semantic residualization: morphological filtering is performed on the regional features of different sizes obtained in step S211 by using multi-rate extended depth convolution respectively, and the D-3 and D-5 branches here, i.e., the extended convolutions with the hole rates of 3 and 5, are replaced with extended re-parameterized convolution modules; and the extended re-parameterized convolution modules are then converted into a large kernel conversion layer for signal advancement;

[0087] Specifically, the extended re-parameterized convolution module is as follows: Figure 5 As shown, large kernel convolution is used together with parallel small kernel convolution to help capture small-scale patterns during training; their outputs are added after two respective batch normalization layers. After training, we use the structural reparameterization method to merge the BN layer into the conv layer, so that the small kernel conv can be equivalently merged into the large kernel conv for inference, and a part of the parameters are reduced. In order to obtain neighboring information of interest, the dilated repair convolution utilizes the above idea and replaces each convolution with an extended layer, that is, it uses a non-dilated small kernel and multiple dilated small kernel layers to enhance a non-dilated large kernel conversion layer. Its hyperparameters include the size K of the large kernel, the kernel size k of the parallel conversion layer, and the dilation rate r. Figure 5 The case of 4 parallel layers shown is represented by K=9, r=(1,2,3,4), k=(5,5,3,3). For larger K, we can use more dilated layers with larger kernel sizes or dilation rates. The only constraint on the kernel size and dilation rate of the parallel branches is (k-1)r+1≤K. To convert an extended reparameter module into a large kernel transformation layer for inference, we first merge each BN into the previous transformation layer and add up all the resulting kernels with appropriate zero padding. In this experiment, the D-3 branch is replaced with an extended reparameter block with K=5, r=(1,2), k=(3,3), and the D-5 branch is replaced with an extended reparameter block with K=7, r=(1,2,3), k=(5,3,3). The two-step region residualization-semantic residualization method makes the learning process orderly and can more effectively obtain multi-scale context information;

[0088] S213 Aggregate Output: After finally obtaining the multi-scale context information for drawing, multiple outputs are aggregated; that is, all feature maps are connected together, and then the features are merged through BN and point-by-point convolution to form the final residual; finally, the final residual is added to the input feature map to build a stronger and more comprehensive feature representation;

[0089] S22: Replace the downsampling of the backbone network and neck with the LDConv convolution module, adjust the initial sampling shape by the offset, and add an offset to any irregular convolution shape, so that the parameter growth becomes linear growth, greatly reducing the parameters;

[0090] The LDConv structure is as follows Figure 6 As shown; the process of LDConv feature extraction is mainly three steps; the specific steps of LDConv feature extraction in step S22 are:

[0091] S221: According to the size of N, the irregular convolution initial coordinate P is generated by the generation algorithm n ; Therefore, the original coordinate at position P0 is P n +P0; The focus of this step is to generate the corresponding sampling coordinates on the feature map for a convolution kernel with N parameters;

[0092] S222: Obtain the offset of the corresponding kernel through convolution operation, whose dimension is (B, 2N, H, W), and then add the offset to the original coordinate to generate a new sampling coordinate corresponding to the convolution; the focus of this step is to generate different sample shapes for convolution at different positions of the feature map;

[0093] S223: The features of the corresponding positions are obtained by interpolation and resampling, and then the corresponding convolution operation is applied to extract the features, thereby completing the feature extraction; this step is mainly to extract the features of the corresponding positions; through the above three steps, LDConv can complete the convolution operation of any size to extract features;

[0094] The common point between LDConv and deformable convolution is that both convolutions adjust the initial sampling shape by offset. The difference between them is that LDConv completes the feature extraction process with an arbitrary number of parameters by generating initial sampling coordinates. Because of this, it corrects the growth of the number of convolution parameters to linear growth. This provides flexibility in adjusting the number of parameters and network computational overhead and device memory consumption. And the deformable convolution operation cannot extract the features corresponding to the irregular convolution;

[0095] S23: Use SDFM to improve the neck structure of the model. Considering that shallow features contain rich detail information and structural information, a shallow feature fusion module (SDFM) based on the channel-space attention mechanism is used to fuse the backbone network features and neck features as the input of the detection head, so as to make full use of the detail information of shallow features and the semantic information of deep features.

[0096] The specific steps of step S23 are:

[0097] S231: The second C2f output of the backbone network and the second C2f output of the neck are sent to the first SDFM fusion module as the first detection head input; the output of the third C2f of the backbone network, that is, the output of C2f_DRR and the output of the third C2f of the neck are sent to the second SDFM fusion module as the second detection head input; the output of the backbone network through SPPF and the output of the fourth C2f of the neck are sent to the third SDFM fusion module as the input of the third detection head; the architecture of SDFM is as follows Figure 7 As shown;

[0098] S232: Concatenate the features of the backbone network and the neck features in the channel dimension, and then feed the concatenated features into a channel attention module consisting of convolution and pooling operations to generate attention weights; the attention weights are then applied to weight the original features by element-wise multiplication, and the resulting features are added to the original features from another branch to enhance the representation of the original features;

[0099] The formula of the feature enhancement process in step S232 is expressed as:

[0100]

[0101] Among them, F a ' is the backbone network feature enhancement representation, F b ' is the neck network feature enhancement representation, F a is the backbone network feature, F b is the neck network feature, where is element-by-element addition, It is element-by-element multiplication, Pw-Conv n (·) represents n cascaded point-by-point convolutional layers, C(·) represents the channel size of the connection operation, δ(·) represents the sigmoid function, and GAP(·) represents the global average pooling; then, the enhanced features are concatenated in the channel dimension and input into the parallel channel attention and spatial attention modules to generate the final fusion weights.

[0102] The formula for generating the fusion weight in step S232 is expressed as:

[0103] Alpha c =Pw-Conv n (GAP(Pw-Conv(C(F a ',F b '))));

[0104] Alpha s =Pw-Conv n (Pw-Conv(C(F a ',F b ')));

[0105]

[0106] Among them, Α c is the channel attention, Α s is the spatial attention, W is the generated fusion weight, and F a ' is the backbone network feature enhancement representation, F b ' is the enhanced representation of the neck network feature; the generated weight is used for one of the partial enhanced features, and the fusion weight of the other partial enhanced feature can be expressed as 1-W; therefore, the SDFM fusion process can be expressed as:

[0107]

[0108] S3 model training: By setting training hyperparameters, the improved network model is trained using the prepared training set;

[0109] The specific steps of step S3 are: by setting the training hyperparameters, the improved model in step S3 is trained with the training set in step S1 to obtain a road defect detection model based on the DRR module and SDFM; in this embodiment, for the model training hyperparameters, the training platform is BlueCloud, the test hardware platform environment is Python3.8, CUDA11.8, the GPU graphics card is NVIDIA RTX3090, and the memory is 30G; the training parameters are set as follows: the number of training times is 300 times, the BatchSize is set to 8, the SGD optimizer is used, and the input image size is 640*640;

[0110] S4 Model Verification: Set verification parameters to verify the trained model; deploy the verified model to the visualization interface to achieve real-time detection of road defects;

[0111] The specific steps of step S4 are: setting verification parameters to verify the road defect detection model based on the DRR module and SDFM trained in step S3 on the verification set, taking mAP@0.5, Precision, Recall, Parameters, and Gflops as evaluation indicators, wherein mAP@0.5 is the average precision of the detection results of each type, Precision is the proportion of correct targets among the targets detected by the model, Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; finally, the verified road defect detection model based on the DRR module and SDFM is deployed to the visual interface to realize real-time detection of road defects; the calculation formulas of each indicator are as follows:

[0112]

[0113]

[0114] Among them, TP is the number of correctly identified targets, FP is the number of incorrectly identified targets, FN is the number of missed targets, P(r) represents the smooth precision-recall curve, and the area under the smooth curve obtained by integrating it is AP; N represents the number of detected object categories, corresponding to 8 different road defects.

[0115] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A road defect detection method based on DRR module and SDFM, characterized in that: The specific steps include: S1 Data collection and processing: Collect road damage pictures, screen the pictures, and then process the data; after annotation, generate a data set and divide it into a training set and a validation set; Construction of the S2 model: First, replace the DWR module with the DRB module, and replace the BottleNeck residual module of the backbone network part C2f with the DRR module; secondly, replace the downsampling of the backbone network and the neck with the LDConv convolution module, and finally, use the surface detail fusion module SDFM to improve the neck structure of the model and fuse features of different scales as the input of the detection head; S3 model training: By setting training hyperparameters, the improved network model is trained using the prepared training set; S4 Model Verification: Set verification parameters to verify the trained model; deploy the verified model to the visualization interface to achieve real-time detection of road defects.

2. The road defect detection method based on DRR module and SDFM according to claim 1 is characterized in that: The specific steps in step S1 are: S11: Download the road disease dataset and crawl the images online, and filter the obtained images; S12: Perform data enhancement on the selected images, perform data enhancement on each image once, and then scale and crop them to obtain a data set; S13: Label the data in the dataset and divide the processed dataset into a training set and a validation set in a ratio of 9:

1.

3. The road defect detection method based on DRR module and SDFM according to claim 1, characterized in that: The specific steps of step S2 are: S21: First, the extended convolution of the D-3 and D-5 branches in the DWR module is replaced with the DRB module, named DRR, and the BottleNeck residual module of the backbone network part C2f is replaced by the DRR module, named C2f_DRR, to obtain features of more scales; for the residual module of the DRR module, a two-step method is used to efficiently draw multi-scale features and obtain multi-scale context information; S22: Replace the downsampling of the backbone network and neck with the LDConv convolution module, adjust the initial sampling shape by the offset, and add an offset to the irregular convolution shape to make the parameter growth linear; S23: A shallow feature fusion module based on the channel-spatial attention mechanism is used to fuse the backbone network features and neck features as the detection head input.

4. The road defect detection method based on DRR module and SDFM according to claim 3 is characterized in that: The specific steps of using the two-step method to efficiently draw multi-scale features in step S21 are: S211 region residualization: This is achieved through a 3×3 convolution combined with a batch normalization BN and ReLU layer to generate a series of feature maps with regions of different sizes as input for morphological filtering; S212 semantic residualization: morphological filtering is performed on the regional features of different sizes obtained in step S211 by using multi-rate extended depth convolution respectively, and the D-3 and D-5 branches here, i.e., the extended convolutions with the hole rates of 3 and 5, are replaced with extended re-parameterized convolution modules; and the extended re-parameterized convolution modules are then converted into a large kernel conversion layer for signal advancement; S213 Aggregate output: After finally obtaining the multi-scale context information for drawing, multiple outputs are aggregated.

5. The road defect detection method based on DRR module and SDFM according to claim 4 is characterized in that: The specific steps of LDConv feature extraction in step S22 are: S221: According to the size of N, the irregular convolution initial coordinate P is generated by the generation algorithm n ; Therefore, the original coordinate at position P0 is P n +P0; S222: Obtain the offset of the corresponding kernel through the convolution operation, whose dimension is (B, 2N, H, W), and then add the offset to the original coordinate to generate a new sampling coordinate corresponding to the convolution; S223: The features of the corresponding positions are obtained by interpolation and resampling, and then the corresponding convolution operation is applied to extract the features, thereby completing the feature extraction.

6. The road defect detection method based on DRR module and SDFM according to claim 5, characterized in that: The specific steps of step S23 are: S231: The second C2f output of the backbone network and the second C2f output of the neck are sent to the first SDFM fusion module as the first detection head input; the output of the third C2f of the backbone network, that is, the output of C2f_DRR and the output of the third C2f of the neck are sent to the second SDFM fusion module as the second detection head input; the output of the backbone network through SPPF and the output of the fourth C2f of the neck are sent to the third SDFM fusion module as the input of the third detection head; S232: Concatenate the features of the backbone network and the neck features in the channel dimension, and then feed the concatenated features into a channel attention module consisting of convolution and pooling operations to generate attention weights; the attention weights are then applied to weight the original features by element-wise multiplication, and the resulting features are added to the original features from another branch to enhance the representation of the original features.

7. The road defect detection method based on DRR module and SDFM according to claim 6 is characterized in that: The formula of the feature enhancement process in step S232 is expressed as: Among them, F′ a is the backbone network feature enhancement representation, F′ b is the neck network feature enhancement representation, F a is the backbone network feature, F b is the neck network feature, where is element-by-element addition, It is element-by-element multiplication, Pw-Conv n (·) represents n cascaded point-by-point convolutional layers, C(·) represents the channel size of the connection operation, δ(·) represents the sigmoid function, and GAP(·) represents the global average pooling; then, the enhanced features are concatenated in the channel dimension and input into the parallel channel attention and spatial attention modules to generate the final fusion weights.

8. The road defect detection method based on DRR module and SDFM according to claim 7, characterized in that: The formula for generating the fusion weight in step S232 is expressed as: Α c =Pw-Conv n (GAP(Pw-Conv(C(F′ a ,F′ b )))); Α s =Pw-Conv n (Pw-Conv(C(F a ',F b '))); Among them, Α c is the channel attention, Α s is the spatial attention, W is the generated fusion weight, F′ a is the backbone network feature enhancement representation, F b ' is the enhanced representation of the neck network feature; the generated weight is used for one of the partial enhanced features, and the fusion weight of the other partial enhanced feature can be expressed as 1-W; therefore, the SDFM fusion process can be expressed as:

9. The road defect detection method based on DRR module and SDFM according to claim 6, characterized in that: The specific steps of step S3 are: by setting training hyperparameters, using the training set in step S1 to train the improved model in step S3, and obtaining a road defect detection model based on the DRR module and SDFM.

10. The road defect detection method based on DRR module and SDFM according to claim 6, characterized in that: The specific steps of step S4 are: setting verification parameters to verify the road defect detection model based on the DRR module and SDFM trained in step S3 on a verification set, taking mAP@0.5, Precision, Recall, Parameters, and Gflops as evaluation indicators, wherein mAP@0.5 is the average precision of each type of detection result, Precision is the proportion of correct targets among the targets detected by the model, Recall is the proportion of targets correctly identified by the model among the total number of all real targets, Parameters is used to measure the size and complexity of the model; Gflops is used to measure the execution time of the model; finally, the verified road defect detection model based on the DRR module and SDFM is deployed to a visual interface to realize real-time detection of road defects.

Images