A lightweight road surface defect detection method based on improved YOLOv7

Abstract

The present invention discloses a lightweight road surface defect detection method based on improved YOLOv7, which specifically includes the following steps: (1) establishing a road surface defect image data set; (2) adding annotation information to the images in the data set; (3) constructing a lightweight road surface defect detection model based on improved YOLOv7; (4) improving the loss function; (5) using a training set and a validation set to train the model and save the trained model; (6) using a test set to test the model, the accuracy of the model meets the generalization requirement, that is, the final lightweight road surface defect detection model is obtained. Compared with the prior art, the lightweight road surface defect detection method based on improved YOLOv7 disclosed by the present invention can effectively improve the accuracy of road surface defect detection, and at the same time, the model has better lightweight characteristics, which is convenient for deployment on resource-constrained mobile hardware platforms.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a lightweight road surface defect detection method based on improved YOLOv7. Background Art

[0002] As an important transportation infrastructure, highways are closely related to economic development and improvement of people's livelihood. During the operation of highways, various defects such as cracks and potholes may occur on the road surface due to factors such as changes in ambient temperature, traffic load, and rain erosion. If these defects cannot be detected and repaired in time, the road surface condition will further deteriorate, which will not only shorten the service life of the highway, but also bring serious hidden dangers to driving safety. Therefore, accurate and timely identification and repair of road surface defects are crucial to maintaining the service life of the highway and ensuring driving safety.

[0003] At present, highway pavement defect detection still relies on manual inspections, which is inefficient and easily interfered by subjective factors, resulting in an increased risk of false detection and missed detection. With the rapid development and widespread application of deep learning technology, highway pavement defect detection methods are moving towards automation. At the same time, the driving recorders equipped on modern vehicles generate a large amount of data, which provides a data basis for deep learning methods to detect pavement defects. Pavement defect detection based on deep learning models can accurately identify and locate pavement defects in images, improving detection accuracy and speed while saving manpower and material resources.

[0004] Object detection algorithms based on deep learning are mainly divided into two categories: single-stage and two-stage. The two-stage object detection algorithm first identifies the candidate area containing the object, and then classifies and locates the object. The main algorithms include R-CNN, Fast-R-CNN, etc. Although the two-stage detection algorithm performs well in detection accuracy, its speed is slow and it is difficult to meet the needs of real-time detection. The single-stage object detection method directly generates the classification and positioning coordinates of the object. The main algorithms include YOLO, SSD, etc. Although the single-stage object detection method has a higher detection speed, it usually sacrifices a certain degree of detection accuracy. In addition, since deep learning models generally have a large number of parameters and high computational complexity, their real-time performance is poor and they are difficult to deploy on hardware platforms with limited resources. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a lightweight pavement defect detection method based on improved YOLOv7. By improving the YOLOv7 target detection model, the problem of false detection and missed detection in pavement defect detection caused by factors such as low clarity of the pavement defect data set and long target distance is solved. At the same time, the volume and parameter amount of the pavement defect detection model are reduced, so that it can be deployed in actual application scenarios with limited resources.

[0006] The technical solution provided by the present invention comprises the following steps:

[0007] Step 1: Acquire road surface defect images to form a first data set;

[0008] Step 2: adding annotation information to the images in the first data set to form a second data set, and dividing the second data set into a training set, a validation set, and a test set;

[0009] Step 3: Build a lightweight road surface defect detection model based on improved YOLOv7;

[0010] Step 4: Use a new loss function;

[0011] Step 5: Using the training set and the validation set to train the lightweight pavement defect detection model based on the improved YOLOv7;

[0012] Step 6: Use the pavement defect test set to test the optimal model obtained through training and obtain the final pavement defect detection model.

[0013] Furthermore, in step 1, the road surface image in the first data set may be captured by a digital camera, or may be obtained from a driving recorder or a surveillance video;

[0014] Preferably, in step 2, the LabelImg annotation tool can be used to add annotation information to the road surface defects; the training set, the validation set and the test set can be divided according to a ratio of 8:1:1.

[0015] Furthermore, the step 3 further includes steps 3.1 to 3.3:

[0016] Step 3.1: Construct a feature extraction Backbone network, which includes a CBS layer, a MobileOne module, an MP-1 module, and an ELAN module; the input image first passes through four CBS convolutional layers, and then passes through the MobileOne module, the MP-1 module, and the ELAN module for feature extraction;

[0017] Step 3.2: Construct a lightweight feature fusion Neck network, which consists of a CBS layer, an SPPCSPC module, a UPSmaple upsampling module, a C3 module, and an MP-2 module; the features extracted in step 3.1 are processed by the SPPCSPC module to adjust the number of channels, and then processed by four C3 module feature fusion processes to obtain features of three sizes: large, medium, and small;

[0018] Step 3.3: The features fused in step 3.2 are sent to the Head network, and the detection results are obtained after being processed by the REP layer and the CBM layer.

[0019] Furthermore, in step 4, a new loss function L is used WIoUv3 , and its calculation formula is:

[0020] (1)

[0021] In formula (1), r is the gradient gain, R WIoU is the distance attention, L IoU Bounding box loss function; where r is calculated as:

[0022] (2)

[0023] In formula (2), β represents the outlier degree, which is used to describe the quality of the anchor box. α and δ are both adjustable parameters. α is used to control the mapping smoothness of β and is set to 1.9. δ is used to control the amplitude adjustment of the gradient gain r and is set to 3.0. The calculation formula of β is:

[0024] (3)

[0025] In formula (3), is the gradient gain of the monotonic focusing coefficient, It is a sliding average, which dynamically updates the highest gradient gain according to the training process to keep the overall β at a high level;

[0026] In formula (1), R WIoU The calculation formula is:

[0027] (4)

[0028] In formula (4), the superscript * indicates that W g With H g Separate from the calculation graph, x, y represents the center coordinates of the prediction box; x gt ,y gt Indicates the center coordinates of the real frame; W g ,H g Indicates the width and height of the smallest outer box that contains both the predicted box and the true box;

[0029] In formula (1), L IoU The calculation formula is:

[0030] (5)

[0031] Among them, IoU is a constant that measures the overlap between the predicted box and the true box, W t and H t Represents the width and height of the intersection of the predicted box and the true box; W g ,Hg Indicates the width and height of the minimum outer box that contains both the predicted box and the true box.

[0032] Furthermore, the step 5 specifically includes steps 5.1 to 5.4:

[0033] Step 5.1: Setting the training parameters of the lightweight road surface defect detection model based on improved YOLOv7, the model training parameters include: learning rate, momentum, weight decay, optimizer, number of iterations, and batch size;

[0034] Step 5.2: Input the training set and validation set images and corresponding labels into the improved YOLOv7 lightweight road defect detection model, use the back propagation algorithm to calculate the gradient of the loss function to the model parameters, and update the parameters of the model according to the gradient to gradually reduce the loss function;

[0035] Step 5.3: Monitor the loss function value and performance indicators during the training process. When the loss function of the training set and the validation set no longer decreases, and the evaluation indicators mean average precision mAP, recall rate R, and accuracy P no longer improve, stop training to avoid overfitting of the model;

[0036] Step 5.4: After the training is completed, save the trained models and select the best model.

[0037] Furthermore, the step 6 specifically includes steps 6.1 to 6.3:

[0038] Step 6.1: Input the test set into the optimal model selected in step 5 for testing;

[0039] Step 6.2: Calculate the model performance indicators: accuracy P, recall R, average precision mAP, number of parameters, computational complexity (GFLOPs), model size. The specific formula is as follows:

[0040] (6)

[0041] (7)

[0042] (8)

[0043] (9)

[0044] In formula (6) to formula (9), P is the precision, R is the recall, mAP is the average precision of all categories, AP is the average precision, TP is the number of positive samples correctly identified as positive samples, FP is the number of negative samples incorrectly identified as positive samples, FN is the number of positive samples incorrectly identified as negative samples, m is the total number of road defect ...i is the AP of the i-th type of pavement defect, P(R) is the function of P with respect to the variation of R;

[0045] Step 6.3: Evaluate the performance indicators of the test set. If the accuracy of the test set is similar to that of the training set, it means that the model meets the generalization requirements, and the final lightweight pavement defect detection model based on the improved YOLOv7 is obtained.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention discloses a lightweight road defect detection method based on improved YOLOv7, which improves the road defect detection accuracy while reducing the model size by introducing the MobileOne module in the Backbone network; designs a lightweight Neck structure to further reduce the calculation amount and parameter amount of the model; adopts a new WIoU loss function to make the model pay more attention to samples of normal quality, thereby improving the overall performance of the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the lightweight road surface defect detection method based on improved YOLOv7 of the present invention;

[0049] Figure 2 This is a schematic diagram of the structure of a lightweight road surface defect detection model based on improved YOLOv7 in the present invention;

[0050] Figure 3 This is a schematic diagram of the MobileOne structure;

[0051] Figure 4 It is a schematic diagram of the C3 structure;

[0052] Figure 5 Schematic diagram of WIoU loss function parameters; DETAILED DESCRIPTION

[0053] In order to make the technical solution, structural features, achieved purposes and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and accompanying drawings. It should be noted that the specific embodiments described herein are only used to more clearly explain the present invention and are not used to limit the present invention.

[0054] Figure 1 This is a flowchart of a lightweight road surface defect detection method based on improved YOLOv7 disclosed in the present invention, and its implementation process is as follows:

[0055] Step 1: Acquire road surface defect images to form a first data set; the road surface images in the first data set can be taken by a digital camera, or acquired from a driving recorder or surveillance video.

[0056] In this embodiment, in order to better evaluate the detection effect of a lightweight pavement defect detection method based on improved YOLOv7 disclosed in the present invention, the Janpan data in the public GRDDC2020 data set was used for pavement defect detection, and four types of pavement defects, namely transverse cracks, longitudinal cracks, network cracks and potholes, were selected for detection, thus forming the first data set.

[0057] Step 2: Add annotation information to the images in the first data set to form a second data set, and divide the second data set into a training set, a validation set, and a test set.

[0058] Preferably, label information is added to the images in the first data set using the LabelImg labeling tool. Since the public data set GRDDC2020 used in this embodiment already has label information, this step is skipped.

[0059] The label file in the xml format in the first data set described in this embodiment is converted into a label file in the txt format required by YOLOv7 to form the second data.

[0060] Preferably, the second data set is divided into a training set, a validation set and a test set according to a ratio of 8:1:1; the divided training set includes 6320 pictures, the test set includes 790 pictures, and the validation set includes 790 pictures. In this embodiment, the image size of the second data set is 600*600.

[0061] Step 3: Build a lightweight road defect detection model based on improved YOLOv7. The improved YOLOv7 model structure is as follows: Figure 2 As shown, the model building process specifically includes steps 3.1 to 3.3:

[0062] Step 3.1: Construct a feature extraction Backbone network, replace the first and last ELAN modules in the original YOLOv7 model Backbone network with MobileOne modules, and the Backbone network includes a CBS layer, a MobileOne module, an MP-1 module, and an ELAN module; the input image first passes through 4 CBS convolutional layers, and then passes through the MobileOne module, the MP-1 module, and the ELAN module for feature extraction.

[0063] Furthermore, the structure of MobileOne is as follows Figure 3 As shown; it has two states, training state and inference state, Figure 3 The left side shows the state during training, and the right side shows the state during inference.

[0064] In the training state, the deep convolution module has three branches, namely 1×1 deep convolution branch, 3×3 deep convolution branch and a BN layer branch. The point convolution module has two branches, namely 1×1 convolution branch and a BN layer branch.

[0065] In the inference state, the depth convolution module and the point convolution module have only one 3×3 depth convolution and a 1×1 point convolution, with no other additional branches, and the convolution and BN can be further integrated.

[0066] Step 3.2: Construct a lightweight feature fusion Neck network, replace all ELAN-W modules in the original YOLOv7 model Neck network with lightweight C3 modules, and the Neck network consists of a CBS layer, an SPPCSPC module, a UPSmaple upsampling module, a C3 module, and an MP-2 module; the features extracted in step 3.1 are adjusted to have the number of channels by the SPPCSPC module, and then are fused by four C3 modules to obtain features of three sizes: large, medium, and small.

[0067] Furthermore, the structure of the C3 module is as follows Figure 4 As shown in Figure 1, it is an important feature extraction module in YOLOv5, which consists of three standard convolutions and multiple bottleneck layers, where the bottleneck layer also includes a simple residual connection.

[0068] Step 3.3: The features fused in step 3.2 are sent to the Head network, and the detection results are obtained after being processed by the REP layer and the CBM layer.

[0069] Step 4: Use a new loss function L WIoUv3 ,The new loss function can speed up the convergence of the model and further ensure the detection performance of the detection model.

[0070] Furthermore, L WIoUv3 The calculation of each parameter is as follows Figure 5 As shown, the calculation formula is:

[0071] (1)

[0072] In formula (1), r is the gradient gain, R WIoU is the distance attention, L IoU Bounding box loss function; the calculation formula of r is:

[0073] (2)

[0074] In formula (2), β represents the outlier degree, which is used to describe the quality of the anchor box. α and δ are both adjustable parameters. α is used to control the mapping smoothness of β and is set to 1.9. δ is used to control the amplitude adjustment of the gradient gain r and is set to 3.0. The calculation formula of β is:

[0075] (3)

[0076] In formula (3), is the gradient gain of the monotonic focusing coefficient, It is a sliding average, which dynamically updates the highest gradient gain according to the training process to keep the overall β at a high level;

[0077] In formula (1), R WIoU The calculation formula is:

[0078] (4)

[0079] In formula (4), the superscript * indicates that W g With H g Separate from the calculation graph, x, y represents the center coordinates of the prediction box; x gt ,y gt Indicates the center coordinates of the real frame; W g ,H g Indicates the width and height of the smallest outer box that contains both the predicted box and the true box;

[0080] In formula (1), L IoU The calculation formula is:

[0081] (5)

[0082] Among them, IoU is a constant that measures the overlap between the predicted box and the true box, W t and H t Represents the width and height of the intersection of the predicted box and the true box; W g ,H g Indicates the width and height of the minimum outer box that contains both the predicted box and the true box.

[0083] Step 5: Input the training set and the validation set into the lightweight pavement defect detection model based on the improved YOLOv7 described in step 3 for training, specifically including steps 5.1 to 5.4:

[0084] Step 5.1: Set the training parameters of the lightweight road defect detection model based on the improved YOLOv7. The model training parameters include: learning rate, momentum, weight decay, optimizer, number of iterations, and batch size.

[0085] In this embodiment, the optimizer is Adam, the initial learning rate 1r0 is 0.01, the momentum is 0.937, the weight decay weight_decay is 0.0005, the batch size batchsize is 8, and the iteration round epoch is 200.

[0086] Step 5.2: Input the training set and validation set images and corresponding labels into the improved YOLOv7 lightweight road defect detection model, and use the back propagation algorithm to calculate the gradient of the loss function to the model parameters.

[0087] Specifically, in each training iteration, first, the error between the model's predicted output and the true label is calculated through forward propagation, and then the error is used to calculate the loss function.

[0088] Second, the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters.

[0089] Again, an optimization algorithm such as gradient descent is used to update the parameters of the model.

[0090] By repeatedly iterating this process, the parameters of the model will gradually be adjusted to the position where the loss function is minimized, allowing the model to detect road defects more accurately.

[0091] Step 5.3: Monitor the loss function value and performance indicators during the training process. The performance indicators include mean average precision (mAP), recall (R), and accuracy (P). When the loss function of the training set and the validation set no longer decreases, and the evaluation indicators mAP, recall R, and accuracy P no longer improve, stop training to avoid overfitting the model to the training data and ensure the generalization ability of the model on the test data.

[0092] Step 5.4: After the training is completed, save the trained model to ensure that the model can be used for road defect detection tasks at any time and can also be easily compared and evaluated with other models.

[0093] Step 6: Use the test set to test the optimal model described in step 5, evaluate the test set test results, and meet the accuracy requirements, that is, obtain the final lightweight pavement defect detection model based on improved YOLOv7. Specifically, step 6 further includes steps 6.1 to 6.3:

[0094] Step 6.1: Input the test set into the optimal model selected in step 5 for testing.

[0095] Step 6.2: Calculate model performance indicators: The performance indicators include precision P, recall R, mAP, number of parameters, computational complexity GFLOPs, and model size. The specific calculation formula is as follows:

[0096] (6)

[0097] (7)

[0098] (8)

[0099] (9)

[0100] In formula (6) to formula (9), P is the precision, R is the recall, mAP is the average precision of all categories, AP is the average precision, TP is the number of positive samples correctly identified as positive samples, FP is the number of negative samples incorrectly identified as positive samples, FN is the number of positive samples incorrectly identified as negative samples, m is the total number of road defect ... i is the AP of the i-th type of pavement defect, and P(R) is the function of P with respect to R.

[0101] Step 6.3: Evaluate the performance indicators of the test set. If the accuracy of the test set is similar to that of the training set, it means that the model meets the generalization requirements, and the final lightweight pavement defect detection model based on the improved YOLOv7 is obtained.

[0102] In this embodiment, in order to verify the effect of the improved model disclosed in the present invention, the test set obtained by dividing the Japan data in the GRDDC2020 dataset in a ratio of 8:1:1 is input into the lightweight pavement defect detection model based on the improved YOLOv7, the YOLOv7 model and other models for testing. The evaluation results are shown in Table 1.

[0103] Table 1 Comparative experimental results

[0104] ;

[0105] According to the data in Table 1, it can be seen that the road surface defect detection model disclosed by the present invention has improved in terms of accuracy P, mAP@0.5, FPS and other indicators, indicating that the model has higher detection speed and accuracy. At the same time, compared with the original YOLOv7 model, this model has lower performance indicators such as parameter quantity, model size, and computational complexity GFLOPs. This improvement makes the model have better lightweight characteristics and is easier to deploy on mobile devices.

[0106] The above description is only one embodiment of the present invention and does not limit the patent scope of the present invention. For those skilled in the art, the present invention may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight road surface defect detection method based on improved YOLOv7, characterized in that: The specific steps include: Step 1: Obtain road surface defect images to form a first data set; in the first data set, the road surface defect images can be photographed by a digital camera, obtained from a driving recorder or a surveillance video; Step 2: adding annotation information to the images in the first data set to form a second data set, and dividing the second data set into a training set, a validation set, and a test set in a ratio of 8:1:1; Step 3: Construct a lightweight pavement defect detection model based on improved YOLOv7. The construction of the model further includes steps 3.1 to 3.3: Step 3.1: Construct a feature extraction Backbone network, which includes a CBS layer, a MobileOne module, an MP-1 module, and an ELAN module; the input image first passes through four CBS convolutional layers, and then passes through the MobileOne module, the MP-1 module, and the ELAN module for feature extraction; Step 3.2: Construct a lightweight feature fusion Neck network, which consists of a CBS layer, an SPPCSPC module, a UPSmaple upsampling module, a C3 module, and an MP-2 module; the features extracted in step 3.1 are processed by the SPPCSPC module to adjust the number of channels, and then processed by four C3 module feature fusion processes to obtain features of three sizes: large, medium, and small; Step 3.3: The features fused in step 3.2 are sent to the Head network, and the detection results are obtained after being processed by the REP layer and the CBM layer; Step 4: Use a new loss function L WIoUv3 , and its calculation formula is: (1) In formula (1), r is the gradient gain, R WIoU is the distance attention, L IoU Bounding box loss function; where r is calculated as: (2) In formula (2), β represents the outlier degree, which is used to describe the quality of the anchor box. α and δ are both adjustable parameters. α is used to control the mapping smoothness of β and is set to 1.

9. δ is used to control the amplitude adjustment of the gradient gain r and is set to 3.

0. The calculation formula of β is: (3) In formula (3), is the gradient gain of the monotonic focusing coefficient, It is a sliding average, which dynamically updates the highest gradient gain according to the training process to keep the overall β at a high level; In formula (1), R WIoU The calculation formula is: (4) In formula (4), the superscript * indicates that W g With H g Separate from the calculation graph, x, y represents the center coordinates of the prediction box; x gt ,y gt Indicates the center coordinates of the real frame; W g ,H g Indicates the width and height of the smallest outer box that contains both the predicted box and the true box; In formula (1), L IoU The calculation formula is: (5) Among them, IoU is a constant that measures the overlap between the predicted box and the true box, W t and H t Represents the width and height of the intersection of the predicted box and the true box; W g ,H g Indicates the width and height of the smallest outer box that contains both the predicted box and the true box; Step 5: Use the training set and the validation set to train the lightweight pavement defect detection model based on the improved YOLOv7 described in step 3, and save the trained model. Step 5 further includes steps 5.1 to 5.4: Step 5.1: Setting the training parameters of the lightweight road surface defect detection model based on improved YOLOv7, the model training parameters include: learning rate, momentum, weight decay, optimizer, number of iterations, and batch size; Step 5.2: Input the training set and validation set images and corresponding labels into the improved YOLOv7 lightweight road defect detection model, use the back propagation algorithm to calculate the gradient of the loss function to the model parameters, and update the parameters of the model according to the gradient to gradually reduce the loss function; Step 5.3: Monitor the loss function value and performance indicators during the training process. When the loss function of the training set and the validation set no longer decreases, and the evaluation indicators mean average precision mAP, recall rate R, and accuracy P no longer improve, stop training to avoid overfitting of the model; Step 5.4: After the training is completed, save the trained models and select the best model; Step 6: Use the test set to test the optimal model selected in step 5.4, evaluate the test set test results, and the accuracy of the model meets the generalization requirements, that is, the final lightweight pavement defect detection model based on improved YOLOv7 is obtained. Step 6 further includes steps 6.1 to 6.3: Step 6.1: Input the test set into the optimal model selected in step 5 for testing; Step 6.2: Calculate the model performance indicators: accuracy P, recall R, average precision mAP, number of parameters, computational complexity GFLOPs, model size, and frames per second FPS. The specific calculation formula for the model performance indicators is as follows: (6) (7) (8) (9) In formulas (6) to (9), P is the precision, R is the recall, mAP is the average precision of all categories, AP is the average precision, TP is the number of positive samples correctly identified as positive samples, FP is the number of negative samples incorrectly identified as positive samples, FN is the number of positive samples incorrectly identified as negative samples, m is the total number of road defect categories, and AP is i is the AP of the i-th type of pavement defect, P(R) is the function of P with respect to the variation of R; Step 6.3: Evaluate the performance indicators of the test set. If the accuracy of the test set is similar to that of the training set, it means that the model meets the generalization requirements, and the final lightweight pavement defect detection model based on the improved YOLOv7 is obtained.

Images