Pavement distress detection method and apparatus, device and medium

Abstract

The present invention relates to the technical field of pavement engineering. Disclosed are a pavement distress detection method and apparatus, a device and a medium. The method comprises: carrying out distress detection on a pavement radar image respectively on the basis of a well-trained large-object detection model and a well-trained small-object detection model, to correspondingly obtain a large-object detection result and a small-object detection result, respectively; and fusing the large-object detection result and the small-object detection result to determine a distress detection result of the pavement radar image, wherein the large-object detection model comprises a deformable convolution layer, a dual-path detection head, and a soft non-maximum suppression operation layer, and the small-object detection model comprises a multi-scale convolutional layer, a deformable ROI pooling layer, a dual-path detection head, a balanced line loss function, and a soft non-maximum suppression operation layer. The structure and corresponding function of a dual-path object detection model are controlled in a targeted manner, and detection results of the two paths are fused, such that the reliability of a final distress detection result is guaranteed.

Description

Road defect detection methods, devices, equipment and media Technical Field

[0001] This invention relates to the field of road engineering technology, and in particular to a method, apparatus, equipment and medium for detecting road defects. Background Technology

[0002] In the field of computer vision, early research in deep learning primarily focused on convolutional neural networks (CNNs). With technological advancements and increasing task demands, researchers have developed object detection algorithms based on CNNs to identify and locate specific objects in images. Each image may contain objects of different types and sizes to be detected, and the goal of object detection algorithms is to output the type and precise location information of all detected objects as accurately as possible. In computer vision, research tasks such as image segmentation and pose detection are extensions and expansions of object detection research.

[0003] In the research on intelligent localization algorithms for hidden defects in asphalt pavements, a target detection model based on a regional convolutional neural network can effectively locate hidden defects in road radar images. However, there is still room for improvement in the localization accuracy of this model. Furthermore, experiments optimizing the basic target detection model revealed difficulties in locating small-target defects, which severely limits the improvement of the model's localization performance.

[0004] Therefore, existing technologies suffer from low reliability in detecting road defects.

[0005] Summary of the Invention

[0006] In view of this, it is necessary to provide a method, device, equipment and medium for detecting road defects, so as to solve the problem of low reliability of defect detection results in the process of detecting road defects in the prior art.

[0007] To address the above problems, this invention provides a method for detecting road defects, comprising:

[0008] Based on the well-trained large target detection model and small target detection model, road radar images are used to detect defects, and the corresponding large target detection results and small target detection results are obtained respectively.

[0009] The detection results of large targets and small targets are fused to determine the defect detection results of road radar images;

[0010] The large target detection model includes deformable convolutional layers, dual-path detection heads, and smoothing non-maximum suppression operation layers; the small target detection model includes multi-scale convolutional layers, deformable ROI pooling layers, dual-path detection heads, balanced linear loss functions, and smoothing non-maximum suppression operation layers.

[0011] In one possible implementation, before performing defect detection on road radar images based on a well-trained large target detection model to obtain the large target detection results, the following steps are also included:

[0012] Obtain a large target defect detection data sample set, which includes large target road radar image samples and their corresponding large target defect results;

[0013] An initial large target detection model is constructed by inputting large target road radar image samples into the initial large target detection model, taking the large target defects as the output, and calculating the sample IOU value. Based on the sample IOU value, the weights of the large target defect detection data sample set are redistributed, and the large target defect detection data sample set with adjusted weights is iteratively trained until a fully trained large target detection model is obtained.

[0014] In one possible implementation, a large-target disease detection model is obtained by iteratively training on a large-target disease detection data sample set until a fully trained large-target detection model is obtained, including:

[0015] Large target road radar image samples are input into the initial large target detection model. The large target road radar image samples are divided into targets according to the dual-path detection head to obtain the region of interest image of the large target.

[0016] The large target radar feature map is obtained by performing irregular convolution processing on the region of interest image of the large target using deformable convolutional layers;

[0017] The detection boxes for large target defects are obtained by filtering the radar feature map of large targets using the smoothing nonmaximum suppression operation layer.

[0018] In one possible implementation, before performing defect detection on road radar images based on a well-trained small target detection model to obtain the small target detection results, the following steps are also included:

[0019] Obtain a small target defect detection data sample set, which includes small target road radar image samples and their corresponding small target defect results;

[0020] An initial small target detection model is constructed by inputting small target road radar image samples into the initial small target detection model, taking the small target defects as the output, and calculating the sample loss value. The model is iteratively trained on a small target defect detection data sample set with sample loss values ​​less than a preset loss threshold until a fully trained small target detection model is obtained.

[0021] In one possible implementation, a small target detection model is obtained by iteratively training a sample set of small target disease detection data until a fully trained small target detection model is obtained, including:

[0022] Small target road radar image samples are input into the initial small target detection model. Multi-scale convolution is performed on the small target road radar image samples according to the multi-scale convolution layer to obtain multi-scale feature images.

[0023] The irregular region image is divided into multiple scale feature images based on the deformable ROI pooling layer to obtain an irregular deformable region image.

[0024] The dual-path detection head is used to segment the image of the irregularly deformed region to obtain the region of interest image of the small target.

[0025] The detection boxes for small target diseases are obtained by filtering the region of interest image of small target based on the smoothing non-maximum suppression operation layer;

[0026] The loss value of the small target disease detection box and the small target disease result is evaluated based on the balanced line loss function, and the initial small target detection model with the smallest loss value is determined as the fully trained small target detection model.

[0027] In one possible implementation, obtaining a large target disease detection data sample set includes:

[0028] Obtain the initial disease detection dataset, which includes road radar image data and its corresponding disease annotation information;

[0029] Road radar image data containing subsidence-type diseases and hidden crack-type diseases were selected as the first-largest target road radar image sample, and the disease annotation information corresponding to the first-largest target road radar image sample was used as the first-largest target disease result.

[0030] The road radar image data containing loose defects was used as the second largest target road radar image sample, and the defect annotation information corresponding to the second largest target road radar image sample was determined to be empty.

[0031] In one possible implementation, obtaining a sample set of small-target disease detection data includes:

[0032] Obtain the initial disease detection dataset, which includes road radar image data and its corresponding disease annotation information;

[0033] Road radar image data containing loose defects were selected as the first small target road radar image sample, and the defect annotation information corresponding to the first small target road radar image sample was used as the first small target defect result.

[0034] Road radar image data containing subsidence-type defects and hidden crack-type defects are used as second small target road radar image samples, and the defect labeling information corresponding to the second small target road radar image samples is determined to be empty.

[0035] To address the above problems, the present invention also provides a road defect detection device, comprising:

[0036] The defect detection module is used to detect defects in road radar images based on well-trained large target detection models and small target detection models, respectively, and obtain large target detection results and small target detection results respectively.

[0037] The defect detection result determination module is used to fuse the detection results of large targets and small targets to determine the defect detection results of road radar images;

[0038] The large target detection model includes deformable convolutional layers, dual-path detection heads, and smoothing non-maximum suppression operation layers; the small target detection model includes multi-scale convolutional layers, deformable ROI pooling layers, dual-path detection heads, balanced linear loss functions, and smoothing non-maximum suppression operation layers.

[0039] To address the aforementioned problems, the present invention also provides an electronic device, including a memory and a processor, wherein,

[0040] Memory, used to store programs;

[0041] The processor, coupled to the memory, executes the program stored in the memory to perform the steps in the road defect detection method described above.

[0042] To address the aforementioned problems, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the road defect detection method described above.

[0043] The beneficial effects of the above embodiments are as follows: This invention provides a road defect detection method, which realizes dual-path target detection by setting up two models, a large target detection model and a small target detection model, to detect defects in road radar images; by specifically controlling the structure and corresponding functions of the models and fusing the detection results of the two models, the final detection accuracy is improved; by setting up structures such as multi-scale convolutional layers in the small target detection model, the detection capability of small target defects can be greatly improved, thereby ensuring the reliability of the final defect detection results. Attached Figure Description

[0044] Figure 1 is a flowchart illustrating an embodiment of the road defect detection method provided by the present invention;

[0045] Figure 2 is a flowchart illustrating an embodiment of the large target detection model provided by the present invention.

[0046] Figure 3 is a flowchart illustrating an embodiment of the present invention for obtaining a large target disease detection data sample set;

[0047] Figure 4 is a flowchart illustrating an embodiment of the training large target detection model provided by the present invention.

[0048] Figure 5 is a schematic diagram of the results of an embodiment of generating deformable convolution offsets provided by the present invention;

[0049] Figure 6 is a schematic diagram of the results of an embodiment of the sampling variation of deformable convolution provided by the present invention;

[0050] Figure 7 is a flowchart illustrating an embodiment of the small target detection model provided by the present invention;

[0051] Figure 8 is a flowchart illustrating an embodiment of the training small target detection model provided by the present invention;

[0052] Figure 9 is a schematic diagram of the structure of an embodiment of multi-scale convolution provided by the present invention;

[0053] Figure 10 is a schematic diagram of the results of an embodiment of the main computation process of deformable ROI pooling provided by the present invention;

[0054] Figure 11 is a structural block diagram of an embodiment of the road defect detection device provided by the present invention;

[0055] Figure 12 is a structural block diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0056] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which constitute a part of the present invention and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0057] ROI (Region of Interest) is a commonly used concept in image processing and object detection. It refers to a specific region in an image that is of interest to the user, and this region usually contains the target or object of interest.

[0058] IOU (Intersection over Union) is a metric used to evaluate the performance of object detection algorithms. It measures the degree of overlap between predicted bounding boxes (i.e., candidate boxes generated by the algorithm) and actual bounding boxes (i.e., the ground truth labeled boxes).

[0059] RCNN (Region-based Convolutional Neural Networks) is a deep learning model for object detection that combines region proposal and convolutional neural networks to achieve object localization and classification.

[0060] To address the aforementioned problems, this invention provides a method, apparatus, equipment, and medium for detecting road defects, which will be described in detail below.

[0061] Figure 1 is a flowchart illustrating an embodiment of the road defect detection method provided by the present invention. As shown in Figure 1, the road defect detection method includes:

[0062] S101: Based on the well-trained large target detection model and small target detection model, road radar images are used to detect defects, and the corresponding large target detection results and small target detection results are obtained respectively;

[0063] S102: The detection results of large targets and small targets are fused to determine the defect detection results of the road radar image;

[0064] The large target detection model includes deformable convolutional layers, dual-path detection heads, and smoothing non-maximum suppression operation layers; the small target detection model includes multi-scale convolutional layers, deformable ROI pooling layers, dual-path detection heads, balanced linear loss functions, and smoothing non-maximum suppression operation layers.

[0065] It should be noted that the large target model refers to the model used to detect diseases that are larger in size—the large target disease model, while the small target model refers to the model used to detect diseases that are smaller in size—the small target disease model.

[0066] Among them, road defects mainly include subsidence defects, hidden crack defects, and loosening defects. Subsidence defects and hidden crack defects are generally larger in size, while loosening defects are generally smaller in size. Therefore, subsidence defects and hidden crack defects are defined as large-target defects, and loosening defects are defined as small-target defects.

[0067] It should also be noted that the large target detection model and the small target detection model are parallel, and their detection order can be adjusted as needed, without any restrictions here.

[0068] In this embodiment, firstly, two models, a large target detection model and a small target detection model, are set up to detect defects in road radar images, thus achieving dual-path target detection. By specifically controlling the structure and corresponding functions of the models and fusing the detection results of the two models, the final detection accuracy is improved. By setting up structures such as multi-scale convolutional layers in the small target detection model, the detection capability of small target defects can be greatly improved, thereby ensuring the reliability of the final defect detection results.

[0069] In a preferred embodiment, in S101, before performing defect detection on the road radar image based on the well-trained large target detection model and small target detection model, it is necessary to obtain the large target detection model and small target detection model respectively.

[0070] As shown in Figure 2, Figure 2 is a flowchart illustrating an embodiment of the large target detection model provided by the present invention, including:

[0071] S201: Obtain a large target defect detection data sample set, which includes large target road radar image samples and their corresponding large target defect results;

[0072] S202: Construct an initial large target detection model. Input large target road radar image samples into the initial large target detection model, output the large target defects, and calculate the sample IOU value. Based on the sample IOU value, redistribute the weights of the large target defect detection data sample set, and iteratively train the large target defect detection data sample set with adjusted weights until a fully trained large target detection model is obtained.

[0073] It should be noted that by setting up an RCNN module that shares parameters with the initial large object detection model to calculate the sample IOU value, this RCNN module only needs to perform forward computation to obtain the sample IOU value.

[0074] In one specific embodiment, the training samples are uniformly sampled based on the IOU value, that is, the weights corresponding to the IOU values ​​of the samples are uniformly determined.

[0075] In this embodiment, the training samples are reassigned by grouping them according to their IOU value. Since the IOU value can reflect the difficulty of the training samples to a certain extent, this invention can treat samples with high difficulty and samples with low difficulty equally, thereby improving the generalization ability of the model after training.

[0076] Specifically, statistical analysis revealed that a large proportion of training samples had IOU values ​​between 0 and 0.05, while only a small portion were distributed in other IOU ranges. This uneven distribution causes the model to miss many difficult negative samples outside the 0-0.05 IOU range during random sampling. The balanced sampling method based on IOU values ​​divides the sampling intervals according to IOU values ​​and performs uniform sampling within each interval. This makes the sampling distribution closer to the distribution of difficult samples, allowing the large object detection model to collect more difficult samples during training than with random sampling, thus improving the model's training effectiveness and generalization performance.

[0077] Specifically, in S201, in order to obtain a large target disease detection data sample set, as shown in Figure 3, Figure 3 is a schematic flowchart of an embodiment of obtaining a large target disease detection data sample set provided by the present invention, including:

[0078] S301: Obtain the initial disease detection dataset, which includes road radar image data and its corresponding disease annotation information;

[0079] S302: Select road radar image data containing subsidence-type diseases and hidden crack-type diseases as the first major target road radar image sample, and use the disease annotation information corresponding to the first major target road radar image sample as the first major target disease result.

[0080] S303: Use road radar image data containing loose defects as the second largest target road radar image sample, and determine that the defect annotation information corresponding to the second largest target road radar image sample is empty.

[0081] In this embodiment, by adaptively deleting loose disease information in the initial disease detection dataset, information about small target diseases can be removed, thereby obtaining pure large target disease sample information, which can improve the training efficiency and targeting of the large target detection model in the subsequent training process.

[0082] In S202, in order to iteratively train the large target disease detection data sample set until a fully trained large target detection model is obtained, as shown in Figure 4, Figure 4 is a flowchart illustrating an embodiment of the training of the large target detection model provided by the present invention, including:

[0083] S401: Input the large target road radar image sample into the initial large target detection model, and divide the large target road radar image sample into targets according to the dual-path detection head to obtain the large target region of interest image;

[0084] S402: The large target region of interest image is processed by irregular convolution based on deformable convolutional layer to obtain large target radar feature map;

[0085] S403: Based on the smoothing nonmaximum suppression operation layer, the detection box of the radar feature map of the large target is filtered to obtain the detection box of the large target defect.

[0086] It should be noted that the dual-path detection head comprises convolutional layers and fully connected (FC) layers operating in parallel. Generally, R-CNN structures use two FC layers for classification and regression tasks. However, comparing two-stage object detection algorithms reveals that convolutional layers are more suitable for regression tasks, while FC layers are better suited for classification. Because FC layers have higher spatial sensitivity, they can better distinguish between global and local targets in classification tasks. However, FC layers are less robust in regression tasks, while convolutional layers achieve better results in regression. Therefore, this invention improves the dual-path detection head, effectively enhancing the model's detection accuracy.

[0087] Deformable convolutional layers refer to layers that replace the traditional convolution operation in the original backbone of the base model with deformable convolution. To explain the difference between deformable convolutional layers and traditional convolution, we first define the traditional convolution operation as follows:

[0088] In the formula, M in For the original image input to a traditional convolutional layer, M out Let l be the feature map output after traditional convolution operation, l be the position parameter in the feature map, C be the convolution matrix, c be the position parameter in the convolution matrix, and w be the sampling weight.

[0089] By adding an offset to the traditional convolution kernel, uniform convolution is transformed into irregular convolution, resulting in deformable convolution:

[0090] In the formula, Q in Given the original image as input for deformable convolution, Q out Let l be the feature map output after deformable convolution, l be the position parameter in the feature map, C be the convolution matrix, c be the position parameter in the convolution matrix, w be the sampling weight, and Δc be the offset added to the traditional convolution kernel by deformable convolution.

[0091] By applying the offset to the traditional convolution kernel, the original uniform convolution becomes an irregular convolution with a variable shape. The convolutional network that generates the offset also participates in backpropagation optimization during model training. After training, the generated offset will better match the characteristics of the lesions in the image. Figure 5 shows a schematic diagram of the result of an embodiment of generating deformable convolution offsets provided by the present invention, and Figure 6 shows a schematic diagram of the result of an embodiment of sampling variation of deformable convolution provided by the present invention. In Figure 6, the light gray dots represent the position before the offset, the dark black dots represent the position after the offset, and the arrows represent the offset vector.

[0092] The Smoothing Non-Maximum Suppression (SMS) layer replaces the traditional Non-Maximum Suppression (NMS) operation with Smoothing Non-Maximum Suppression (SMS). First, when predicting object detection boxes, the number of predicted boxes far exceeds the number of objects. However, each object should only have one true detection box. Therefore, it is necessary to remove the redundant predicted boxes and retain the prediction box that best fits the object as the final object detection box. The effect of the traditional NMS operation is as follows:

[0093] NMS removes redundant detection boxes by retaining the detection box with the highest prediction score and filtering out those with an overlap exceeding the IOU threshold.

[0094] However, while NMS is simple and effective, it has the following problems in complex object detection scenarios: when the objects in the detection scenario are densely distributed or two objects are close to each other, NMS may cause the detection box of one object to suppress the unique detection box of its neighboring object, resulting in missed detection of objects and reducing the final detection accuracy of the model. Therefore, Soft-NMS is used to avoid this situation.

[0095] Specifically, when removing redundant detection boxes, Soft-NMS no longer directly removes detection boxes that exceed the IOU threshold. Instead, it uses a Gaussian function, as shown below, to reduce their prediction scores.

[0096] In the formula: s i Let M be the predicted score of the detection box, and b be the detection box with the highest predicted score. i Let be the detection bounding box to be filtered, D be the final result after Soft-NMS, and G(x) be the Gaussian function.

[0097] It is obvious that the closer the detection box is to M, the more its score will decrease, while the farther the detection box is from M, the less its score will decrease. Therefore, by using Soft-NMS, many detection boxes that would otherwise be mistakenly deleted by NMS can be retained, thereby reducing the chance of missed detections and improving the model's detection performance.

[0098] In this embodiment, a dual-path detection head is used to segment large-target road radar image samples, which not only ensures the accuracy of the classification results but also improves the regression stability. A deformable convolutional layer is used to perform irregular convolution processing on the large target region of interest image, which better adapts to the needs of image defect detection. A smoothing non-maximum suppression operation layer is used to filter detection boxes on the large target radar feature map, which better ensures the reliability of the final target detection boxes. In summary, by adaptively adjusting some structures in the initial large target detection model, it can better meet the needs of large target defect detection and improve detection accuracy.

[0099] Further, as shown in Figure 7, which is a flowchart illustrating an embodiment of the small target detection model provided by the present invention, including:

[0100] S701: Obtain a small target defect detection data sample set, which includes small target road radar image samples and their corresponding small target defect results;

[0101] S702: Construct an initial small target detection model. Input small target road radar image samples into the initial small target detection model, output the small target defects, and calculate the sample loss value. Iterate the training on the small target defect detection data sample set with sample loss values ​​less than the preset loss threshold until a fully trained small target detection model is obtained.

[0102] In one specific embodiment, training samples can be divided into hard samples and easy samples based on detection difficulty. During training, making the model focus more on hard samples helps improve the model's generalization performance. The original R-CNN structure of the base model uses a random sampling method, which cannot guarantee the number of hard samples after sampling.

[0103] Therefore, to increase the number of hard samples collected by RCNN, this invention replaces random sampling with Online Hard Example Mining (OHEM). OHEM considers samples with large loss values ​​as hard samples, sorts the samples according to their loss values, and selects the hard samples with the highest loss values, thereby ensuring the number of hard samples after sampling.

[0104] In one specific embodiment, the OHEM sampling method uses an RCNN module that shares parameters with the original RCNN module to calculate the sample loss. This RCNN module only needs to perform forward computation to obtain the loss value, without needing to perform backpropagation. After calculating the loss value, the training samples are sorted in ascending order of loss value. After NMS, the 256 samples with the largest loss values ​​are selected for training the original RCNN module. After OHEM sampling, the small object detection model can obtain more training times for difficult samples, which is beneficial to improving the generalization performance of the small object detection model.

[0105] In this embodiment, by using the sample loss value as the grouping criterion to redistribute the training samples, it is possible to effectively combine the characteristics of the small target itself to select more comprehensive samples, thereby improving the model training effect and ensuring the reliability of the small target detection model after training.

[0106] In a preferred embodiment, in S701, similar to the process of obtaining a large target disease detection data sample set, in order to obtain a small target disease detection data sample set, firstly, an initial disease detection dataset is obtained, which includes road radar image data and its corresponding disease annotation information; then, road radar image data containing loose diseases are selected as the first small target road radar image sample, and the disease annotation information corresponding to the first small target road radar image sample is used as the first small target disease result; finally, road radar image data containing subsidence diseases and hidden crack diseases are selected as the second small target road radar image sample, and the disease annotation information corresponding to the second small target road radar image sample is determined to be empty.

[0107] In this embodiment, by adaptively deleting the annotation information of subsidence-type diseases and hidden crack-type diseases in the initial disease detection dataset, information about large target diseases can be removed, thereby obtaining only small target disease sample information, which can improve the training efficiency and targeting of the small target detection model in the subsequent training process.

[0108] In a preferred embodiment, in S702, in order to iteratively train the small target disease detection data sample set until a fully trained small target detection model is obtained, as shown in Figure 8, Figure 8 is a flowchart illustrating an embodiment of the training of the small target detection model provided by the present invention, including:

[0109] S801: Input the small target road radar image samples into the initial small target detection model, and perform multi-scale convolution on the small target road radar image samples according to the multi-scale convolution layer to obtain the multi-scale feature image;

[0110] S802: Irregular regions are divided into multiple scale feature images based on deformable ROI pooling layers to obtain irregular deformable region images;

[0111] S803: Based on the dual-path detection head, the image of the irregularly deformed region is divided into targets to obtain the region of interest image of the small target;

[0112] S804: Based on the smoothing non-maximum suppression operation layer, the detection box of the small target region of interest image is filtered to obtain the small target disease detection box;

[0113] S805: Evaluate the loss value of the small target defect detection box and the small target defect result based on the balance line loss function, and determine the initial small target detection model with the smallest loss value as the fully trained small target detection model.

[0114] Multi-scale convolutional layers refer to transforming the intermediate convolutional layers of the residual structure into multi-scale convolutions to improve the network's multi-scale representation capability. Specifically, by replacing the ResNet-50 (intermediate convolutional layer) used in the original Backbone model with Res2Net (multi-scale convolution), it is beneficial to improve the feature extraction capability of small object detection models for small objects. The basic structure of Res2Net is shown in Figure 9. Figure 9 is a schematic diagram of the structure of an embodiment of multi-scale convolution provided by the present invention. Res2Net's fusion of multi-scale convolution can enable the output features to have different receptive fields and contain feature information of the target at different scales.

[0115] The deformable ROI pooling layer refers to replacing the original ROI Align operation in the ROI alignment module of the basic model with deformable ROI pooling. Deformable ROI pooling can introduce the correlation information of other regions into the aligned ROI features, which is beneficial for small target detection models to detect small-sized disease targets.

[0116] Standard ROI pooling divides the ROI region into a 7x7 grid and then uses max pooling on each of the resulting sub-blocks to output aligned ROI features. Deformable ROI pooling, however, adds an offset to each sub-block within the 7x7 grid, transforming the uniformly divided region into an irregular, deformable one. The deformable ROI pooling operation can be defined as follows:

[0117] In the formula, f R For the feature image matrix of the deformable ROI pooling layer as input, R out This is the ROI feature matrix output by the deformable ROI pooling layer, where i and j represent the horizontal and vertical coordinates in the matrix, BIN is the matrix of uniformly divided region sub-blocks, b is the element value of the region sub-block matrix, and Δb i,j Offset added to the sub-block of the region for deformable ROI pooling.

[0118] As shown in Figure 10, Figure 10 is a schematic diagram of the results of an embodiment of the main operation process of deformable ROI pooling provided by the present invention. In this figure, the offset of the region sub-block is generated by adding an FC layer after the ROI features output by the ordinary ROI pooling layer, and the weight parameters of the FC layer also participate in the backpropagation optimization process during model training.

[0119] Small target detection models are primarily used to detect small-sized disease targets. Their detection accuracy is significantly affected by the target's location, thus requiring high accuracy in target location regression. On the other hand, since small target detection models only detect loosely distributed diseases, their classification requirements are relatively low. Based on the above analysis, it is necessary to increase the weight of the regression loss in the overall loss calculation of small target detection models, i.e., increase the weight coefficient before the regression loss. However, directly increasing the weight of the regression loss can lead to gradient imbalance between easy and difficult samples.

[0120] Therefore, in the original RCNN structure of the basic model, this application replaces the SmoothL1Loss function with the BalancedL1Loss function (balanced line loss function) for calculating the regression loss.

[0121] The gradient change during backpropagation can be obtained by taking the derivative of the SmoothL1Loss function:

[0122] In the formula, SmoothL1 is the SmoothL1 loss function, and x represents the regression error value.

[0123] Training samples with regression error values ​​x greater than 1 can be considered hard samples, and their propagation gradient value is constant at 1. Training samples with regression error values ​​x less than 1 can be considered easy samples, and their propagation gradient is related to x. Increasing the weight coefficient λ of the regression loss results in a much larger gradient increase for hard samples compared to easy samples, leading to a gradient imbalance between hard and easy samples, which is detrimental to improving model training performance. Furthermore, small object detection models often detect small, hard objects, which exacerbates this gradient imbalance.

[0124] The BalancedL1Loss function is:

[0125] In the formula, α is 0.5, γ is 1.5, and b is used to ensure the continuity of the function. The value of b needs to satisfy the equation αln(b+1)=γ.

[0126] Taking the derivative of the BalancedL1Loss function, we can obtain the gradient changes during backpropagation as follows:

[0127] By replacing the original SmoothL1 loss function with the BalancedL1 loss function, the gradient rise of simple samples will become faster, and the gradient contribution of easy and difficult samples will become more balanced after the regression loss weight coefficient λ is increased. This improves the aforementioned gradient imbalance problem and enhances the training effect of the small object detection model.

[0128] In this embodiment, multi-scale convolution is performed on small target road radar image samples using multi-scale convolutional layers, which increases the dimensionality of feature information. Deformable ROI pooling layers are used to divide irregular regions in the multi-scale feature images, achieving adaptive adjustment of region division to better meet practical needs. Dual-path detection heads are used to segment targets in irregularly deformed regions, ensuring classification accuracy while improving regression stability. A smoothing non-maximum suppression operation layer is used to filter detection boxes in the small target region of interest images, better ensuring the reliability of the final target detection boxes. A balanced line loss function is used to evaluate the loss value of small target hazard detection boxes and results, improving the aforementioned gradient imbalance problem and enhancing the training effect of the small target detection model. In summary, by adaptively adjusting some structures in the initial small target detection model, it better meets the needs of small target hazard detection and improves detection accuracy.

[0129] By using the above method, two models, a large target detection model and a small target detection model, are set up to detect defects in road radar images, achieving dual-path target detection. By specifically controlling the structure and corresponding functions of the models and fusing the detection results of the two models, the final detection accuracy is improved. By setting up structures such as multi-scale convolutional layers in the small target detection model, the detection capability of small target defects can be greatly improved, thereby ensuring the reliability of the final defect detection results.

[0130] To address the aforementioned problems, the present invention also provides a road defect detection device, as shown in FIG11. FIG11 is a structural block diagram of an embodiment of the road defect detection device provided by the present invention. The road defect detection device 1100 includes:

[0131] The defect detection module 1101 is used to perform defect detection on road radar images based on a well-trained large target detection model and a small target detection model, respectively, and obtain large target detection results and small target detection results respectively.

[0132] The defect detection result determination module 1102 is used to fuse the large target detection results and small target detection results to determine the defect detection results of the road radar image;

[0133] The large target detection model includes deformable convolutional layers, dual-path detection heads, and smoothing non-maximum suppression operation layers; the small target detection model includes multi-scale convolutional layers, deformable ROI pooling layers, dual-path detection heads, balanced linear loss functions, and smoothing non-maximum suppression operation layers.

[0134] As shown in Figure 12, the present invention also provides an electronic device 1200. The electronic device 1200 includes a processor 1201, a memory 1202, and a display 1203. Figure 12 only shows some components of the electronic device 1200; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively.

[0135] In some embodiments, processor 1201 may be a central processing unit (CPU), microprocessor or other data processing chip, used to run program code stored in memory 1202 or process data, such as the road defect detection method of the present invention.

[0136] In some embodiments, processor 1201 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 1201 may be local or remote. In some embodiments, processor 1201 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, internal cloud, multi-cloud, etc., or any combination thereof.

[0137] In some embodiments, memory 1202 may be an internal storage unit of electronic device 1200, such as a hard disk or memory of electronic device 1200. In other embodiments, memory 1202 may also be an external storage device of electronic device 1200, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 1200.

[0138] Furthermore, the memory 1202 may include both internal storage units of the electronic device 1200 and external storage devices. The memory 1202 is used to store application software and various types of data installed on the electronic device 1200.

[0139] In some embodiments, display 1203 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 1203 is used to display information from electronic device 1200 and to display a visual user interface. Components 1201-1203 of electronic device 1200 communicate with each other via a system bus.

[0140] In one embodiment, when the processor 1201 executes the road defect detection program in the memory 1202, the following steps can be implemented:

[0141] Based on the well-trained large target detection model and small target detection model, road radar images are used to detect defects, and the corresponding large target detection results and small target detection results are obtained respectively.

[0142] The detection results of large targets and small targets are fused to determine the defect detection results of road radar images;

[0143] The large target detection model includes deformable convolutional layers, dual-path detection heads, and smoothing non-maximum suppression operation layers; the small target detection model includes multi-scale convolutional layers, deformable ROI pooling layers, dual-path detection heads, balanced linear loss functions, and smoothing non-maximum suppression operation layers.

[0144] It should be understood that when the processor 1201 executes the road defect detection program in the memory 1202, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0145] Furthermore, the embodiments of the present invention do not specifically limit the type of the electronic device 1200 mentioned. The electronic device 1200 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1200 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0146] Accordingly, embodiments of the present invention also provide a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the road defect detection methods provided in the above-described method embodiments.

[0147] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0148] The road defect detection method, apparatus, equipment, and medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting road defects, characterized in that, include: Based on the well-trained large target detection model and small target detection model, road radar images are used to detect defects, and the corresponding large target detection results and small target detection results are obtained respectively. The detection results of the large target and the detection results of the small target are fused to determine the defect detection results of the road radar image; The large target detection model includes a deformable convolutional layer, a dual-path detection head, and a smooth non-maximum suppression operation layer; the small target detection model includes a multi-scale convolutional layer, a deformable ROI pooling layer, the dual-path detection head, a balanced linear loss function, and the smooth non-maximum suppression operation layer.

2. The road defect detection method according to claim 1, characterized in that, Before obtaining the large target detection results by performing defect detection on road radar images based on a well-trained large target detection model, the following steps are also included: Obtain a large target defect detection data sample set, which includes large target road radar image samples and their corresponding large target defect results; An initial large target detection model is constructed by inputting the large target road radar image samples into the initial large target detection model, outputting the large target defects, and calculating the sample IOU value. Based on the sample IOU value, the weights of the large target defect detection data sample set are redistributed, and the large target defect detection data sample set with adjusted weights is iteratively trained until a fully trained large target detection model is obtained.

3. The road defect detection method according to claim 2, characterized in that, By iteratively training the large target disease detection data sample set until a fully trained large target detection model is obtained, including: The large target road radar image sample is input into the initial large target detection model, and the large target road radar image sample is divided into targets according to the dual-path detection head to obtain the region of interest image of the large target. The large target region of interest image is subjected to irregular convolution processing based on the deformable convolutional layer to obtain a radar feature map of the large target. The detection boxes for large target defects are obtained by filtering the radar feature map of the large target based on the smoothing nonmaximum suppression operation layer.

4. The road defect detection method according to claim 1, characterized in that, Before obtaining the small target detection results by performing defect detection on road radar images based on a well-trained small target detection model, the following steps are also included: Obtain a small target defect detection data sample set, which includes small target road radar image samples and their corresponding small target defect results; An initial small target detection model is constructed by inputting the small target road radar image samples into the initial small target detection model, taking the small target defects as the output, and calculating the sample loss value. The small target defect detection data sample set with sample loss values ​​less than a preset loss threshold is iteratively trained until a fully trained small target detection model is obtained.

5. The road defect detection method according to claim 4, characterized in that, By iteratively training the small target disease detection data sample set until a fully trained small target detection model is obtained, including: The small target road radar image sample is input into the initial small target detection model, and the small target road radar image sample is convolved at multiple scales according to the multi-scale convolutional layer to obtain a multi-scale feature image. The multi-scale feature image is divided into irregular regions based on the deformable ROI pooling layer to obtain an irregular deformable region image. The dual-path detection head is used to segment the image of the irregularly deformed region to obtain an image of the region of interest for a small target. The image of the region of interest of the small target is filtered by the smoothing non-maximum suppression operation layer to obtain the small target disease detection box; Based on the equilibrium line loss function, the small target disease detection box and the small target disease results are analyzed. The initial small object detection model with the smallest loss value is determined to be a fully trained small object detection model.

6. The road defect detection method according to claim 2, characterized in that, The acquisition of a large target disease detection data sample set includes: Obtain an initial disease detection dataset, which includes road radar image data and its corresponding disease annotation information; The road radar image data containing subsidence-type defects and hidden crack-type defects are selected as the first major target road radar image sample, and the defect annotation information corresponding to the first major target road radar image sample is taken as the first major target defect result. The road radar image data containing loose-type defects is used as the second largest target road radar image sample, and the defect annotation information corresponding to the second largest target road radar image sample is determined to be empty.

7. The road defect detection method according to claim 4, characterized in that, The acquisition of the small target disease detection data sample set includes: Obtain an initial disease detection dataset, which includes road radar image data and its corresponding disease annotation information; The road radar image data containing loose diseases are selected as the first small target road radar image sample, and the disease annotation information corresponding to the first small target road radar image sample is used as the first small target disease result; The road radar image data containing subsidence-type defects and hidden crack-type defects are used as the second small target road radar image samples, and the defect labeling information corresponding to the second small target road radar image samples is determined to be empty.

8. A road defect detection device, characterized in that, include: The defect detection module is used to detect defects in road radar images based on well-trained large target detection models and small target detection models, respectively, and obtain large target detection results and small target detection results respectively. The disease detection result determination module is used to determine the detection results of the large target and the small target. The results are then fused to determine the damage detection results of the road radar images; The large target detection model includes a deformable convolutional layer, a dual-path detection head, and a smooth non-maximum suppression operation layer; the small target detection model includes a multi-scale convolutional layer, a deformable ROI pooling layer, the dual-path detection head, a balanced linear loss function, and the smooth non-maximum suppression operation layer.

9. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps in the road defect detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can perform the steps in the road defect detection method according to any one of claims 1 to 7.

Images