A road pavement disease detection method, medium and equipment based on multi-model integration

Abstract

The present invention provides a road pavement disease detection method, medium, and equipment based on multi-model integration, including: constructing a diverse, high-quality pavement disease image dataset; building a detection model based on the RT-DETR network and modifying the loss function to increase the model's convergence speed; separately constructing a single-category pothole disease dataset for use as a pre-trained model in training the detection model; using the isolation forest algorithm during the detection model training process to reduce the impact of extreme data on the model; constructing a ResNet-based classification model to improve the detection effect of pothole categories; and integrating the detection model with the classification model to build a complete pavement disease detection model. The present invention not only effectively alleviates the problem of decreased detection accuracy caused by extremely unbalanced datasets, but also specifically optimizes categories prone to false positives, such as potholes, significantly reducing the false positive rate.

Description

Technical Field

[0001] The present invention belongs to the field of road disease detection and deep learning technology, and specifically relates to a road pavement disease detection method, medium and equipment based on multi-model integration. Background Art

[0002] Over time, road surfaces can become damaged and age. If repairs aren't made promptly, the constant pressure of pedestrians and vehicles, combined with prolonged exposure to the sun and the erosion of rain and snow, will exacerbate the damage, further exacerbating the severity of the damage. This can not only create safety hazards for vehicles and pedestrians but also negatively impact the environment and cityscape. Therefore, regular road inspections to promptly detect and repair damage are essential for maintaining road safety and the city's appearance.

[0003] In the current field of road maintenance and management, deep learning technology is gradually becoming a mainstream means of detecting road defects. This technology has revolutionized the traditional manual inspection model. By training complex neural network models, it can automatically learn and identify various road defects such as cracks, potholes, and damage from massive amounts of image or video data. The introduction of deep learning methods has not only greatly improved the efficiency and accuracy of defect detection, but also reduced labor costs, making road maintenance more intelligent and refined.

[0004] Although deep learning-based pavement defect detection has made significant progress both domestically and internationally, it still faces numerous challenges in practical application. Two key challenges are of particular concern: First, the significant imbalance in the dataset. Since cracks, a common early form of pavement defect, far outnumber other types of defects, such as potholes and damage, this imbalance significantly impacts model training and overall detection accuracy. Second, the complex and ever-changing road environment, including obstructions from obstacles (such as railings, vehicles, and pedestrians), as well as interference from natural factors like lighting changes and rain, leads to frequent false positives during detection, primarily in non-defective areas with similar characteristics to those of the defect, further reducing the reliability and practicality of the system. Summary of the Invention

[0005] The present invention addresses the dataset imbalance and false positives problems in complex environments that exist in existing pavement defect detection tasks. In this dataset imbalance, cracks, as the primary manifestation of early-stage pavement defects, dominate the dataset, while other types of defects, such as potholes and damage, are relatively scarce. This imbalance poses a significant challenge to the training of deep learning models, significantly reducing the recognition accuracy of minority classes and even leading to a large number of missed detections. Regarding the false positives problem in complex environments, the actual road surface conditions are complex, with dense traffic markings creating a complex background. Furthermore, obstructions such as vehicles, pedestrians, and railings significantly increase the difficulty of detection. Furthermore, factors such as uneven pavement and post-rainwater accumulation can also lead to inaccurate detection. To address these issues, the present invention provides a method, medium, and device for road pavement defect detection based on multi-model integration. These methods not only effectively alleviate the problem of decreased detection accuracy caused by extreme dataset imbalance, but also specifically optimize categories prone to false positives, such as potholes, significantly reducing the false positive rate.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a road pavement disease detection method based on multi-model integration, characterized by comprising:

[0008] Step 1: Collect pavement disease sample images to form a pavement disease dataset, which includes various pavement disease types including potholes;

[0009] Step 2: Build a detection model based on RT-DETR network to detect road pavement defects;

[0010] Step 3: Obtain a pothole dataset and iteratively pre-train the detection model using the pothole dataset.

[0011] Step 4: Use the pre-trained weights and the pavement disease dataset to further train the detection model;

[0012] Step 5: Embed the detection model trained in step 4 into the vehicle system and run it online. Collect the image data of the running detection and classify the detection results into positive and negative samples. Positive samples and negative samples represent samples with correct detection results and false positives, respectively.

[0013] Step 6: Use the isolation forest algorithm to identify outliers in the positive samples and use it to update the pavement disease dataset. Return to step 4 to train the detection model for the next round until the optimal model is obtained.

[0014] Step 7: Build a ResNet-based binary classification model and train it based on negative samples. The binary classification model is used to distinguish the false positive categories of potholes in the detection results.

[0015] Step 8: Integrate the optimal model obtained in step 6 and the binary classification model trained in step 7, and use the integrated model to detect road pavement defects.

[0016] Optionally, in step 1, the method of collecting pavement disease sample images includes manual shooting, frame extraction from vehicle-mounted camera video, and collection of public data; the multiple pavement disease types include transverse cracks, longitudinal cracks, block cracks, potholes, subway exploration grooves, damage, and pitting.

[0017] Optionally, in step 2, the detection model uses ResNet101 as the backbone network and uses XIoU as the loss function.

[0018] Optionally, in step 3, the detection model is iteratively pre-trained using the pit dataset, specifically:

[0019] The acquired pothole dataset is trained on the detection model to obtain a preliminary detection model;

[0020] Collect actual road video data and extract frames at intervals, and adjust the frame size;

[0021] Use the preliminary detection model to detect the frame data, save the detection results as an XML file, use the labelImg annotation tool to process the detection results, adjust the detection box used to locate the pits in the detection results, and filter out data with false positives or missed detections as training data for the next round of training of the preliminary detection model.

[0022] Optionally, in step 4, the pavement disease dataset is enhanced using the Albumentations data enhancement library before training.

[0023] Optionally, in step 6, for positive samples with correct detection results, the isolation forest algorithm is used to divide the positive samples that are different from other positive samples and are smaller in number than other positive samples as extreme samples; the saved extreme sample data are detected using the detection model. If the detection result is correct, it is included in the pavement disease dataset for the next round of training of the detection model; otherwise, it is saved in the extreme dataset according to the pavement disease type. When the extreme dataset reaches the set data number, all of it is put into the pavement disease dataset for the next round of training of the detection model.

[0024] Optionally, in step 7, the pothole false alarm categories include accumulated water and reflection.

[0025] Optionally, in step 7, the training data set of the binary classification model is a pit type false alarm data set collected from negative samples.

[0026] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program enables a computer to execute the road pavement defect detection method based on multi-model integration as described in the first aspect.

[0027] In a third aspect, the present invention provides an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements the road pavement defect detection method based on multi-model integration as described in the first aspect.

[0028] The present invention is applicable to detecting a variety of road surface damage types in complex scenarios, improving model training and overall detection accuracy even when the data samples are extremely unbalanced. Furthermore, integrating a binary classification model into the overall model further enhances detection of a single category (potholes), further strengthening model training and overall detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of the road pavement disease detection method based on multi-model integration in the present invention.

[0030] Figure 2 This figure compares the performance of the preliminary and final models during online testing. (a) and (b) show the preliminary model's misidentification of reflections and accumulated water, respectively; (c) and (d) show the final model's detection performance, which eliminates reflections and accumulated water as potholes. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0032] In one embodiment, if Figure 1 As shown, the present invention constructs a road pavement defect detection method based on multi-model integration. First, a pre-trained model is designed to enhance the model's feature extraction capabilities for potholes, thereby improving the model's overall detection accuracy even when pothole data is minimal. Second, a binary classification model is constructed and integrated to reduce the false positive rate for pothole types. Furthermore, based on multi-model integration, the present invention further optimizes the algorithm, improving detection accuracy and robustness in complex environments.

[0033] The road pavement disease detection method based on multi-model integration of this embodiment specifically includes the following steps:

[0034] Step 1: Build a diverse and high-quality pavement disease dataset.

[0035] This example captures a large number of diverse, high-quality pavement defect sample images through manual photography, frame extraction from vehicle-mounted camera recordings, and the collection of selected public datasets. These images include various types of pavement defects, including transverse cracks, longitudinal cracks, block cracks, potholes, subway exploration trenches, damage, joint damage, and pitting. The images are manually annotated using the labelImg annotation tool, ultimately generating XML tag files in VOC format to form a standard detection dataset.

[0036] Step 2: Build a detection model based on RT-DETR network.

[0037] Based on the target detection algorithm RT-DETR, this embodiment builds a model network framework for road disease detection. The framework mainly consists of four parts: a backbone feature extraction network, a hybrid encoder, a Transformer decoder module, and a detection head. Given that the model needs to be embedded in the vehicle system, there are strict requirements on the model size and computational complexity. Therefore, ResNet101 with moderate model size and computational complexity is selected as the backbone network, based on its superior feature extraction capabilities while maintaining the practicality of the model. In order to further improve network performance, accelerate convergence speed and enhance robustness, this embodiment uses XIoU to replace the original loss function. In addition, the other structural configurations of the original RT-DETR remain unchanged to ensure the overall performance and stability of the model.

[0038] Step 3: Manually obtain and annotate a dataset of potholes with obvious features, and train a pre-trained model on the detection model in step 2.

[0039] A dedicated pothole dataset is created by carefully selecting and annotating pothole samples with distinct features. This dataset is then used to pre-train the detection model from step 2. After training the preliminary detection model, actual road video data is collected from a vehicle camera. The video data is then resized to 640×640 pixels using the PIL resize method. The preliminary detection model is then used to infer the resized frames. Finally, the inference results are saved as XML files and manually processed using the labelImg annotation tool to adjust the detection bounding boxes, generating high-quality training data. To further improve model performance, multiple iterations of training are performed. In each iteration, the optimized training data is used for secondary training, while samples with excessively high confidence levels are excluded to prevent overfitting. Data with false positives or missed detections is used as training data for secondary training. This process is repeated multiple times to obtain a model with good performance, which serves as the final pre-trained model. However, overfitting can occur during multiple iterations of training. To address this, high-confidence supplementary image data is filtered out during the iterations of adding new samples.

[0040] Step 4: Use the pre-trained weights and the pavement damage detection dataset to train a detection model, where the image is preprocessed before training begins.

[0041] Using the pre-trained model weights obtained in Step 3, the constructed pavement damage dataset was fed into the pavement damage detection network. The images were sized at 640×640 pixels. The original image inputs were augmented using the Albumentations data augmentation library, which included random translation, scaling, and rotation, as well as random image changes. After all training rounds were completed, the model weight file with the best performance was selected and saved.

[0042] Step 5: Embed the detection model in step 4 into the vehicle system and run it online, while continuously updating the sample library.

[0043] Running the model online is the best way to identify issues with the test results and is also a crucial step in enriching the training sample. After the model converges, it is embedded in the vehicle system and put into operation. Daily image data from the test runs is collected and manually screened. The test results are classified into positive and negative samples. The processed positive samples are added to the sample library, and issues are identified from the negative samples and addressed one by one during subsequent iterative training.

[0044] Step 6: During the training process of step 5, the isolation forest algorithm is used to solve the problem of low detection accuracy caused by some extreme samples.

[0045] Extreme samples are samples that can be manually distinguished as positive samples, but whose characteristics differ significantly from other positive samples and whose number is extremely low. The Isolation Forest algorithm is a fast anomaly detection method based on Ensemble. It detects outliers in a dataset by constructing multiple isolated trees. Since the presence of extreme samples can lead to low detection accuracy, the Isolation Forest algorithm is used to separate extreme data and operate on them separately. During daily operation, the code automatically saves extreme data. After training, the saved extreme data is inferred. If the inference result is correct, it is included in the training sample. Otherwise, it is classified according to the disease and saved in the extreme database. When the number of category data reaches a certain level, it is all added to the training sample for training. The above process is repeated, and all steps are completed automatically.

[0046] In step 7, a ResNet-based binary classification model is constructed to distinguish and classify the false positive categories of potholes that appear in the detection results of step 6: accumulated water and reflection.

[0047] Since the final model requires integrating two models and embedding them into the vehicle system, the less complex ResNet18 was selected as a binary classification model to classify the accumulated water and reflections in the detection results as false positives for potholes. The false positive samples of potholes in the detection results of step 6 are used to distinguish between correct samples and false positive samples (accumulated water, reflections). That is, the binary classification model is used to detect false positives and divide them into three categories, thereby improving the overall model's detection effect on potholes. In the detection results, if the Softmax function return value of a certain category is greater than 0.7, the data is labeled as this category, which is a false positive. If the Softmax function return values of the two categories are similar, both around 0.5, the data is labeled as the pothole category, which is a positive positive.

[0048] In step 8, the model obtained in step 7 is integrated with the model obtained in step 6 to finally form a road surface detection model. Figure 2 This is a comparison of the online testing results of the preliminary model and the final model. (a) and (b) show the preliminary model's misidentification of reflections and accumulated water, respectively. (c) and (d) show the detection results of the final model, which does not misidentify accumulated water and reflections as potholes.

[0049] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, which enables a computer to execute the road pavement defect detection method based on multi-model integration as described in the first embodiment.

[0050] In another embodiment, the present invention provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the road pavement defect detection method based on multi-model integration as described in Example 1 is implemented.

[0051] In the embodiments disclosed herein, computer storage media can be tangible media that can contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. Computer storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0052] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0053] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A road pavement disease detection method based on multi-model integration, characterized in that: include: Step 1: Collect pavement disease sample images to form a pavement disease dataset, which includes various pavement disease types including potholes; Step 2: Build a detection model based on RT-DETR network to detect road pavement defects; Step 3: Obtain a pothole dataset and iteratively pre-train the detection model using the pothole dataset. Specifically: The acquired pothole dataset is trained on the detection model to obtain a preliminary detection model; Collect actual road video data and extract frames at intervals, and adjust the frame size; Use the preliminary detection model to detect the frame data and save the detection results as an XML file. Use the labelImg annotation tool to process the detection results, adjust the detection boxes used to locate the pits in the detection results, and filter out data with false positives or missed detections as training data for the next round of training of the preliminary detection model. Step 4: Use the pre-trained weights and the pavement disease dataset to further train the detection model; Step 5: Embed the detection model trained in step 4 into the vehicle system and run it online. Collect the image data of the running detection and classify the detection results into positive and negative samples. Positive samples and negative samples represent samples with correct detection results and false positives, respectively. Step 6: Use the isolation forest algorithm to classify outliers in the positive samples and use them to update the pavement disease dataset. Return to step 4 to conduct the next round of training for the detection model until the optimal model is obtained. In step 6, for positive samples with correct detection results, use the isolation forest algorithm to classify positive samples that are different from other positive samples and have a smaller number than other positive samples as extreme samples. Use the detection model to test the saved extreme sample data. If the test result is correct, it is included in the pavement disease dataset for the next round of training of the detection model. Otherwise, it is saved in the extreme dataset according to the pavement disease type. When the extreme dataset reaches the set number of data, all of them are put into the pavement disease dataset for the next round of training of the detection model. Step 7: Build a ResNet-based binary classification model and train it based on negative samples. The binary classification model is used to distinguish the false positive categories of potholes in the detection results. Step 8: Integrate the optimal model obtained in step 6 and the binary classification model trained in step 7, and use the integrated model to detect road pavement defects.

2. The road pavement defect detection method based on multi-model integration according to claim 1, characterized in that: In step 1, the methods for collecting sample images of road surface defects include manual photography, frame extraction from vehicle-mounted camera recordings, and collection of public data; the various types of road surface defects include transverse cracks, longitudinal cracks, block cracks, potholes, subway exploration grooves, damage, and pitting.

3. The road pavement defect detection method based on multi-model integration according to claim 1, characterized in that: In step 2, the detection model uses ResNet101 as the backbone network and XIoU as the loss function.

4. The road pavement defect detection method based on multi-model integration according to claim 1, characterized in that: In step 4, the pavement disease dataset is enhanced using the Albumentations data enhancement library before training.

5. The road pavement defect detection method based on multi-model integration according to claim 1, characterized in that: In step 7, the pothole false alarm categories include water accumulation and reflection.

6. The road pavement defect detection method based on multi-model integration according to claim 1, characterized in that: In step 7, the training data set of the binary classification model is a pit type false alarm data set collected from negative samples.

7. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the road pavement defect detection method based on multi-model integration as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for detecting road pavement defects based on multi-model integration as described in any one of claims 1 to 6 is implemented.

Images