Pavement damage condition detection method and device based on YOLO-V5

Abstract

The invention relates to a road surface damage condition detection method and device based on YOLO-V5. The method comprises the following steps: collecting real-time road surface image data; inputting the real-time pavement image data into a pavement damage condition detection model, detecting whether pavement damage exists or not, and if yes, outputting a damage type and coordinate information; wherein the pavement damage condition detection model is constructed by combining a YOLO-V5 model with a CBAM attention mechanism and a detection layer, and is obtained by training based on model training data, and the model training data comprises an image containing a pavement damage information mark. According to the method, road surface damage features of different scales can be captured through the improved YOLO-V5 model, so that the detection precision of small target and dense region damage is improved, attention mechanisms such as a convolutional block attention module (CBAM) and the like are introduced, the feature extraction capability of the model on a damaged region can be enhanced, and the detection precision is further improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing detection technology, and in particular to a road damage condition detection method and device based on YOLO-V5. Background Art

[0002] In recent years, natural factors such as wind and sun and human behavior have had a serious impact on road safety, so a lot of money has been invested in the maintenance and upkeep of road facilities. Using more scientific and accurate methods can help reduce the capital expenditure on road maintenance.

[0003] The most widely used method at present is the videography method. The videography method is a detection technology based on image processing. It installs a special fast or high-speed camera on the measuring car, shoots the road surface at a certain speed and camera angle, and records various diseases on the road surface (such as cracks, potholes, etc.) into the camera tape. Subsequently, it is quickly processed into data on site or indoors through image analysis algorithms to evaluate the damage of the road surface. Although the videography method can realize automatic shooting and preliminary processing of images, manual intervention or the use of complex algorithms is often required in the subsequent image analysis and disease identification stages. This limits the degree of automation of the method to a certain extent and increases labor and time costs. Especially when processing large-scale image data, the workload of manual interpretation and annotation is huge, and problems such as fatigue and misjudgment are prone to occur.

[0004] With the development of digital image processing technology, the era when maintenance personnel used manual inspection methods to detect road damage has become a thing of the past. At present, both at home and abroad, cars equipped with road detection cameras have begun to collect data on a large scale. Compared with the method of manual inspection, the use of highway monitoring probes not only reduces time costs, but also saves labor costs. Therefore, using a more accurate data collection method is crucial to improving the efficiency of road damage detection. Although YOLO-V5 has improved in small target detection, its sensitivity to small-sized damage is still limited. Since small targets have less feature information in the image, the model may find it difficult to accurately capture and identify these small damages; in areas with dense damage, the occlusion and overlap between targets are more serious, which increases the difficulty of detection; when YOLO-V5 processes complex scenes, overlapping frames, missed detections, or false detections may occur. The shape and texture of damage are often complex and changeable, which places high demands on the generalization ability of the model; when YOLO-V5 processes damage with large changes in shape and texture, inaccurate detection may occur. Summary of the invention

[0005] The purpose of the present invention is to provide a road damage condition detection method and device based on YOLO-V5, improve the detection capability of small targets, improve the detection effect in dense scenes and enhance the robustness and generalization ability of the model, thereby reducing missed detection and false detection.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] On the one hand, a road damage condition detection method based on YOLO-V5 is provided, including:

[0008] Collect real-time road image data;

[0009] The real-time road surface image data is input into a road surface damage condition detection model to detect whether there is road surface damage. If there is road surface damage, the damage type and coordinate information are output; wherein the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is obtained based on model training data training, and the model training data includes images marked with road surface damage information.

[0010] Optionally, obtaining the model training data includes:

[0011] Collecting road surface image data, screening the road surface image data, and obtaining image data containing road surface damage information;

[0012] The image data containing the road damage information is framed, labeled, and named to obtain the model training data.

[0013] Optionally, after acquiring the image data containing the road surface damage information, the method includes: performing data enhancement and image enhancement on the image data containing the road surface damage information;

[0014] The data enhancement includes: flipping, translating, rotating, scaling and adding noise to the image data containing the road damage information;

[0015] The image enhancement includes: adding noise, changing brightness, cropping, translation, rotation, mirroring, and cutout.

[0016] Optionally, the road damage condition detection model includes:

[0017] Replace the original convolutional layer or module of the YOLO-V5 model with a convolutional layer or module containing the CBAM attention mechanism;

[0018] A detection layer is added to the output layer of the YOLO-V5 model.

[0019] Optionally, replacing the original convolutional layer or module of the YOLO-V5 model with a convolutional layer or module including a CBAM attention mechanism includes: adopting a CBAM attention mechanism module in the last layer C3 module of the backbone part of the YOLO-V5 model.

[0020] Optionally, the real-time road surface image data is input into a road surface damage condition detection model, and the damage type and coordinate information are output, including:

[0021] The real-time road surface image data is input into Backbone for preprocessing and downsampling, and feature maps of different scales are input;

[0022] The feature maps of different scales are input into Neck for feature fusion, the fused feature maps are input into the detection layer, and the damage type and coordinate information are output.

[0023] Optionally, outputting the damage type and coordinate information includes: feeding back the detected road damage status information to a monitoring center or a relevant maintenance department in real time.

[0024] On the other hand, a road damage condition detection device based on YOLO-V5 is also provided, comprising: a monitoring probe and a detection module;

[0025] The monitoring probe is used to collect real-time road surface image data;

[0026] The detection module is used to input the real-time road surface image data into a road surface damage condition detection model to detect whether there is road surface damage, and if there is road surface damage, output the damage type and coordinate information; wherein the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is obtained based on model training data training, and the model training data includes images marked with road surface damage information.

[0027] The beneficial effects of the present invention are:

[0028] (1) The present invention adopts the CBAM attention mechanism module in the last layer C3 module of the backbone part, combined with the attention module (CAM) and the spatial attention module (SAM), to improve the extraction and expression capabilities of the model in complex environments. In the NECK part, the 80 feature map is sampled into 160 and added to the 160 feature map in the backbone, thereby improving the detection capability of small targets, improving the detection effect in dense scenes, and improving the robustness and generalization ability of the model, thereby reducing missed detection and false detection.

[0029] (2) By conducting inspections of highway pavement conditions through automation and remote monitoring, the efficiency of inspections can be effectively improved. Road surfaces that need to be inspected can be automatically identified and given priority, thereby improving the inspection efficiency of the road operation and maintenance department. This not only saves human resources, but also allows intervention at the early stage of problem discovery, greatly reducing the probability of traffic accidents.

[0030] (3) The YOLO-V5 model has an extremely high detection speed, which can reach 140FPS (frames per second). It can process a large amount of image data in a very short time and realize real-time detection of road conditions. This is crucial for road condition monitoring and management that requires instant feedback. The improved YOLO-V5 model can capture road damage features of different scales, thereby improving the detection accuracy of small targets and dense area damage. The introduced attention mechanism such as the convolutional block attention module (CBAM) can enhance the model's feature extraction ability for damaged areas and further improve detection accuracy. The YOLO-V5 model itself has a fast detection speed, and the improved model can further improve real-time performance while maintaining high accuracy by optimizing the network structure and calculation process. This enables the model to meet the needs of real-time road damage detection. In busy traffic scenes such as highways, real-time detection of road conditions (such as cracks, potholes, obstacles, etc.) can ensure traffic safety and smoothness. The high-speed real-time detection capability of YOLO-V5 provides a strong guarantee for this demand.

[0031] (4) Compared with other complex detection models, YOLO-V5 has a smaller model and fewer parameters, which makes it easier to deploy on mobile and embedded devices. This advantage is particularly important in scenarios with limited hardware resources such as high-speed probes. In addition, YOLO-V5 supports multiple programming languages ​​and platforms, such as Python, PyTorch, etc. This allows users to quickly deploy and customize the detection system according to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 This is a flow chart of a road damage condition detection method based on YOLO-V5 according to an embodiment of the present invention;

[0034] Figure 2 This is a diagram of the improved and optimized YOLO-V5 structure of an embodiment of the present invention;

[0035] Figure 3 After training, use tensorboard to view the training visualization graph of the embodiment of the present invention;

[0036] Figure 4 The scatter plot and histogram generated after the improved and optimized YOLO-V5 training in the embodiment of the present invention;

[0037] Figure 5 This is a result diagram of the visualization training according to an embodiment of the present invention;

[0038] Figure 6 This is a result diagram of the damage type and coordinate information output by the improved and optimized YOLO-V5 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] This embodiment provides a road damage condition detection method based on YOLO-V5, including:

[0042] Collect real-time road image data;

[0043] The real-time road surface image data is input into the road surface damage condition detection model to detect whether there is road surface damage. If there is road surface damage, the damage type and coordinate information are output; the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is trained based on the model training data, which includes images marked with road surface damage information.

[0044] In this embodiment, monitoring equipment is used to collect road surface image data on the expressway.

[0045] Furthermore, obtaining model training data includes:

[0046] Collect road image data, filter the road image data, and obtain image data containing road damage information; use labelimg to select, label, and name the image data containing road damage information to obtain model training data.

[0047] Specifically, the collected image data is screened, irrelevant images are eliminated, and images containing road damage information are retained. This article uses the labelimg open source image labeling tool to select, label, and name the images in the dataset. First, use the labelimg tool to open the prepared dataset, and specify that the txt file generated after labeling is consistent with the dataset storage location. Then use Create RectBox to select the minimum rectangle of the road damage to be identified, and generate a txt file after saving. The content stored in the file is the two-dimensional coordinate position of the selected damage in the image. Finally, the image data is divided into training set, validation set, and test set in a ratio of 8:1:1; the improved and optimized YOLO-V5 structure diagram is shown below. Figure 2 As shown in the figure, the improved YOLO-V5 model is trained using the preprocessed training set data to obtain a trained road damage detection model. The trained model is tested using the validation set, and the model parameters are adjusted according to the test results until the model performance reaches the optimal level.

[0048] Furthermore, after obtaining the image data containing the road damage information, it includes: performing data enhancement and image enhancement on the image data containing the road damage information; data enhancement includes: flipping, translating, rotating, scaling and adding noise to the image data containing the road damage information; image enhancement includes: adding noise, changing brightness, cropping, translating, rotating, mirroring, and cutout.

[0049] Specifically, the image enhancement code in this embodiment includes methods such as adding noise, changing brightness, cropping, translating, rotating, mirroring, and cutout. You can select a specific method and modify the parameter settings in the code. Data enhancement methods include: flipping, translating, rotating, scaling, adding noise, and other transformations to the image data to expand the scale of the data set and improve the generalization ability of the model.

[0050] Furthermore, the road damage condition detection model includes: replacing the original convolutional layer or module of the YOLO-V5 model with a convolutional layer or module including the CBAM attention mechanism; and adding a detection layer to the output layer of the YOLO-V5 model.

[0051] Furthermore, the original convolutional layer or module of the YOLO-V5 model is replaced with a convolutional layer or module containing the CBAM attention mechanism, including: the last layer C3 module of the backbone part of the YOLO-V5 model adopts the CBAM attention mechanism module.

[0052] Furthermore, the real-time road surface image data is input into the road surface damage condition detection model, and the damage type and coordinate information output include:

[0053] The real-time road surface image data is input into Backbone for preprocessing and downsampling, and feature maps of different scales are input;

[0054] The feature maps of different scales are input into Neck for feature fusion, the fused feature maps are input into the detection layer, and the damage type and coordinate information are output.

[0055] Specifically, in this embodiment, improvements are made on the basis of the YOLO-V5 model, using the CBAM attention mechanism, combining channel attention (Channel Attention Module, CAM) and spatial attention (Spatial Attention Module, SAM), while paying attention to the channel and spatial information of the feature map. Add the class definition of the attention mechanism in the models.py file, including the initialization method (__init__) and the forward propagation method (forward). Then define the channel attention module (CAM) and the spatial attention module (SAM) respectively, and combine these two modules in the CBAM class. In the models / yolov5s.py file, find the place where the model network structure is built, replace the original convolutional layer or module with the convolutional layer or module containing the attention mechanism, and replace the convolutional layer, so that the existing convolutional layer has stronger adaptability and higher extraction ability for the trained pictures. This operation of replacing the convolutional layer does not change the overall architecture of the model, but on the premise of keeping the basic functions of the model unchanged, fine-tune the internal implementation to achieve better performance.

[0056] The original image data is input into Backbone. The image data is preprocessed by scaling, normalization, etc. to meet the input requirements of the model. The image data passes through a series of convolutional layers, using different convolution kernels to extract features from the image. As the convolutional layer goes deeper, the extracted features gradually transition from low-level edges, textures, etc. to high-level semantic information. Between the convolutional layers, pooling layers are interspersed to perform downsampling operations, while increasing the number of channels to make the feature map more abstract and compact. After multiple convolution and downsampling operations, Backbone outputs feature maps of different scales. The Neck part receives feature maps of different scales from Backbone and fuses these feature maps together through feature fusion strategies.

[0057] Modify yolov5s.yaml, add a small target detection layer, and send the fused feature map to the detection layer to output the damage type and coordinate type. After adding the detection layer to the network output layer, the detection ability of small targets is improved, the detection effect in dense scenes is improved, and the robustness and generalization ability of the model are improved.

[0058] In this embodiment, the prediction head is optimized, the size and proportion of the anchor frame are adjusted, so as to more accurately capture the object to be identified, optimize the structure of the classifier, and further improve the accuracy of identifying the object. At the same time, a more effective feature fusion strategy is introduced to reduce the interference of background information, thereby greatly improving the speed and accuracy of the prediction head. The prediction head contains multiple detection layers, each of which is responsible for detecting objects of different scales or types.

[0059] Among them, the feature fusion strategy is: adopting the feature pyramid strategy, by fusing feature maps of different scales, constructing a feature pyramid to adapt to objects of different sizes. The pyramid fuses high-level features with low-level features in a bottom-up and top-down manner, and ensures the complementarity of features through lateral connections. Horizontal connections are used to fuse upsampled high-level features with low-level features to enhance the semantics and spatial resolution of features.

[0060] Furthermore, after outputting the damage type and coordinate information, the method includes: feeding back the detected road damage status information to a monitoring center or a relevant maintenance department in real time.

[0061] The present invention will be further described below in conjunction with the accompanying drawings:

[0062] like Figure 1 As shown, the method of this embodiment specifically includes the following steps:

[0063] S1. Use Anaconda software to create a virtual environment, and enter "condaactivate the name of the virtual environment you created" on the AnacondaPrompt page to activate it. After the environment is successfully activated, add the "pip install" command to install its corresponding package (pytorch, labellmg, etc.), and then enter "conda env list" to view the installed libraries;

[0064] S2. Preprocess the image, find the folder where you downloaded yolov5 and create a new folder VOCDate. Create two files in the VOCDate folder, named "Annotations" and "images". Use the labelimg tool to open the prepared dataset, select the smallest rectangle of the damaged area to be identified, save the dataset to the "images" file, and then save the xml file generated by the labeled dataset to the "Annotations" file.

[0065] S3. Create and run the program "split_train_val.py" in the VOCData directory. After running, the "ImagesSets\Main" folder will be generated, and the collected data will be preprocessed and divided into training set, validation set, and test set; then create and run the program "text_to_yolo.py" in the VOCData directory. After running, two folders "labels" and "dataSet_path" will be generated and viewed. The xml file generated by the data set has been converted into a yolo_txt file. The content stored in the file is the two-dimensional coordinate position of the selected road damage in the image; create a new myvoc.yaml file in the data folder under the yolov5 directory, open it with Notepad, modify the paths of the training set and validation set (train.txt and val.txt) as well as the number and category names of the target, and then open the train.py file in python to randomly scale, flip and overlay the data. When processing different images, the position coordinates of the target in the image will also change. Therefore, the position coordinates of the target should be processed while processing the image. Enter the corresponding code and run it. After the operation, the transformed "images", "bboxes" and "y_trues" will be returned. The returned "y_true" is (3, 80, 80, 6), (3, 40, 40, 6), (3, 20, 20, 6). The three array sizes here correspond to the three output feature layers of the backbone network, 80X80, 40X40, and 20X20. 3 means that each point in each feature layer has 3 candidate anchors of different sizes and shapes. 6 means 5+1, where 1 means a total of 1 class is trained, 5=4+1, where 1 means there is a target (with a target or only background);

[0066] S4. Train the data set. Apply the deep learning image processing algorithm to the recognition of highway pavement images. Input the collected image information into a deep learning detection network model. Through machine learning and training, add the training command "python train.py --weights weights / yolov5s.pt --cfg models / yolov5s.yaml --data data / myvoc.yaml --epoch 200 --batch-size 8 --img 640 --devicecpu" to realize the recognition of highway pavement damage and intelligent status analysis.

[0067] S5. After training, use tensorboard to view the training visualization, such as Figure 3As shown, the loss and accuracy of the training process are monitored. The confusion_matrix.png file is stored in the runs folder. The file name is confusion matrix, which can show the accuracy of label prediction. The columns in the two-dimensional chart represent predictions, and the rows represent the names of the labels previously in labelimg. True means that the prediction result is correct. The final test shows that the accuracy probability of "meiyoukeng" is 0.80, the accuracy probability of "youkeng" is 0.73, and the accuracy probability of "background" is 0.8;

[0068] The scatter plot and histogram generated after yolov5 training are as follows: Figure 4 As shown. The histogram shows that the number of instances of the "meiyoukeng" category is about 250, and the number of instances of the "youkeng" category is about 230. It can be concluded that the number of instances recognized by the two is similar. It can be determined that each recognized instance image basically contains the two categories of "meiyoukeng" and "youkeng". The scatter plot represents the approximate location of the x and y coordinates of the pits in each recognized image. It can be concluded that the locations of most pits in the instance images are around x: 0.4-0.6, y: 0.2-0.4. The scatter plot shows the approximate width and length of the pits in each recognized image. Therefore, it can be concluded that the length of the pit is about 0.1m to 0.3m, and the width is about 0.1m to 0.3m;

[0069] The results of the visualization training are presented as follows Figure 5 As shown, the two figures in the first column represent the mean of the loss function during training and verification. The loss function represents the difference between the part predicted by the model and the part selected by the real box. It is a non-negative function, and the smaller the value, the better. The two figures in the second column represent the mean loss during target detection, and the smaller the better. Although the second figure in the second column shows an upward trend when adding instance pictures for recognition, the final curve value is very small and has little impact on the final result. The mean classification loss of the two figures in the third column is close to 0, and no negative values ​​are shown. The last figure in the first row observes the fluctuation of the complete probability of the recognized category at a macro level. In this figure, it shows a steady increase and little fluctuation and tends to be stable, indicating that the training effect is good;

[0070] The results of this training are in line with the experimental expectations. In the comparison between the predicted box selection and the formal box selection, the loss function tends to 0 with the training, reflecting that the machine prediction effect is good. And the accuracy rate is basically maintained at around 0.80. The experimental results are accurate and desirable;

[0071] S6. Model testing and optimization: Use the validation set to test the trained model and adjust the model parameters according to the test results until the model performance reaches the optimal level.

[0072] S7, Damage Detection: Input the test set and the real-time collected road image data into the trained road damage detection model, automatically identify and locate the road damage condition, and output the damage type and coordinate information, such as Figure 6 The output image file is analyzed and the recognition result shows the monitoring effect of road potholes. The average credibility of road pothole recognition is 0.80, which can complete the recognition and positioning task well.

[0073] This embodiment also provides a road damage condition detection device based on YOLO-V5, including: a monitoring probe and a detection module;

[0074] Monitoring probe, used to collect real-time road image data;

[0075] The detection module is used to input real-time road surface image data into the road surface damage condition detection model to detect whether there is road surface damage. If there is road surface damage, the damage type and coordinate information are output; wherein, the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is trained based on the model training data, and the model training data includes images marked with road surface damage information.

[0076] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A road damage detection method based on YOLO-V5, characterized in that: include: Collect real-time road image data; The real-time road surface image data is input into a road surface damage condition detection model to detect whether there is road surface damage. If there is road surface damage, the damage type and coordinate information are output; wherein the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is obtained based on model training data training, and the model training data includes images marked with road surface damage information.

2. The road damage detection method based on YOLO-V5 according to claim 1, characterized in that: Acquiring the model training data includes: Collecting road surface image data, screening the road surface image data, and obtaining image data containing road surface damage information; The image data containing the road damage information is framed, labeled, and named to obtain the model training data.

3. The road damage detection method based on YOLO-V5 according to claim 2, characterized in that: After acquiring the image data containing the road surface damage information, the method includes: performing data enhancement and image enhancement on the image data containing the road surface damage information; The data enhancement includes: flipping, translating, rotating, scaling and adding noise to the image data containing the road damage information; The image enhancement includes: adding noise, changing brightness, cropping, translation, rotation, mirroring, and cutout.

4. The road damage detection method based on YOLO-V5 according to claim 1, characterized in that: The road damage condition detection model includes: Replace the original convolutional layer or module of the YOLO-V5 model with a convolutional layer or module containing the CBAM attention mechanism; A detection layer is added to the output layer of the YOLO-V5 model.

5. The road damage detection method based on YOLO-V5 according to claim 4 is characterized in that: Replacing the original convolutional layer or module of the YOLO-V5 model with a convolutional layer or module including a CBAM attention mechanism includes: adopting a CBAM attention mechanism module in the last layer C3 module of the backbone part of the YOLO-V5 model.

6. The road damage detection method based on YOLO-V5 according to claim 4, characterized in that: The real-time road surface image data is input into the road surface damage condition detection model, and the damage type and coordinate information are outputted, including: The real-time road surface image data is input into Backbone for preprocessing and downsampling, and feature maps of different scales are input; The feature maps of different scales are input into Neck for feature fusion, the fused feature maps are input into the detection layer, and the damage type and coordinate information are output.

7. The road damage detection method based on YOLO-V5 according to claim 1, characterized in that: Outputting the damage type and coordinate information includes: feeding back the detected road damage status information to the monitoring center or relevant maintenance department in real time.

8. A road damage detection device based on YOLO-V5, characterized in that: include: Monitoring probes and detection modules; The monitoring probe is used to collect real-time road surface image data; The detection module is used to input the real-time road surface image data into a road surface damage condition detection model to detect whether there is road surface damage, and if there is road surface damage, output the damage type and coordinate information; wherein the road surface damage condition detection model is constructed by combining the YOLO-V5 model with the CBAM attention mechanism and the detection layer, and is obtained based on model training data training, and the model training data includes images marked with road surface damage information.

Images