Road surface damage detection methods and systems

Abstract

This invention provides a method and system for road damage detection, relating to the field of target detection technology. The method includes: constructing a road damage detection model based on YOLOv5; training the road damage detection model using a road damage loss function; the road damage loss function includes: coordinate localization loss, confidence loss, and classification loss based on multiple distance constraints; the multiple distance constraints include: overlap area, center point distance, aspect ratio of the ground truth bounding box to the predicted bounding box, aspect ratio of the intersecting bounding box to the predicted bounding box, distance between the top left corner and the top right corner; and using the trained road damage detection model to detect road damage on real image data. This invention improves the accuracy of road damage detection by modifying the CIoU loss function.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a road surface damage detection method and system based on a coordinate positioning loss function with multiple distance constraints. Background Technology

[0002] With the significant increase in the number of vehicles and roads, road maintenance and management have become a challenge. It is well known that due to settlement, most road surfaces frequently experience various types of damage during use. This damage not only affects the structural performance and load-bearing capacity of the road surface but also its service life. To provide timely, comfortable, safe, and efficient high-level road services, and to enhance road safety and improve traffic efficiency, accurate detection of road surface damage is necessary.

[0003] Road surface damage detection can be viewed as target detection, and can be performed using target detection algorithms. Existing target detection algorithms, such as YOLOv5, can detect targets such as road surface damage.

[0004] However, the coordinate localization loss function used in this target detection model has low accuracy in bounding box position when detecting road surface loss, resulting in incomplete detection of the detected object in the detection box, and both accuracy and recall need to be improved. Summary of the Invention

[0005] In view of this, the present invention provides a road surface damage detection method based on a coordinate positioning loss function with multiple distance constraints, so as to improve the accuracy of road surface damage detection.

[0006] Therefore, the present invention provides the following technical solution:

[0007] This invention provides a road surface damage detection method based on a coordinate positioning loss function with multiple distance constraints, comprising:

[0008] Construct a road surface damage detection model based on YOLOv5;

[0009] The road damage detection model is trained using a road damage loss function, which includes: coordinate localization loss, confidence loss, and classification loss based on multiple distance constraints; the multiple distance constraints include: overlapping area, center point distance, aspect ratio of ground truth bounding box to predicted bounding box, aspect ratio of intersecting bounding box to predicted bounding box, top left corner distance, and top right corner distance.

[0010] A trained road damage detection model is used to detect road damage on real image data.

[0011] Furthermore, it also includes: data augmentation of the real image data to be detected.

[0012] Furthermore, the data enhancement includes: augmenting the data using a brightness contrast adjustment algorithm, the calculation formula of which is as follows:

[0013] Z1 = αP1 + β;

[0014] Where P1 is the input, Z1 is the output, parameter α is the gain, used to adjust the contrast of the road surface image; β is the deviation parameter, used to adjust the brightness of the road surface image.

[0015] Furthermore, the road surface damage detection model includes:

[0016] The backbone network uses repVGG as the feature extraction network.

[0017] The neck network employs a feature fusion network that combines a recursive feature pyramid with a path aggregation network.

[0018] The head network employs a nonmaximum suppression algorithm.

[0019] Furthermore, the coordinate positioning loss based on multiple distance constraints is:

[0020]

[0021] Where d is the distance between the center points of the ground truth box and the predicted box, K represents the Euclidean distance between the top left corner of the predicted box and the ground truth box, B is the Euclidean distance between the top right corner of the predicted box and the ground truth box, c is the diagonal length of the minimum bounding box between the ground truth box and the predicted box, α·v is the penalty term for the aspect ratio between the ground truth box and the predicted box, α1v1 is the penalty direction for the aspect ratio between the intersecting box and the predicted box, and λ is the weight optimization loss.

[0022] The calculation expressions for αv and α1v1 are as follows:

[0023]

[0024]

[0025]

[0026]

[0027] Among them, W gt With H gt Let W and H represent the width and height of the ground truth bounding box, respectively, and let W and H represent the width and height of the predicted bounding box, respectively. in With H in These represent the width and height of the intersecting boxes, respectively. v and v1 are used to measure the consistency of the aspect ratio, and α and α1 are weighting coefficients.

[0028] Furthermore, the confidence loss adopts the cross-entropy loss function.

[0029] Furthermore, the classification loss employs a binary cross-entropy function.

[0030] This invention also provides a road surface damage detection system based on a coordinate positioning loss function with multiple distance constraints, characterized in that it includes:

[0031] The model building module is used to build a road surface damage detection model based on YOLOv5;

[0032] The model training module is used to train the road damage detection model using the road damage loss function. The road damage loss function includes: coordinate localization loss, confidence loss, and classification loss based on multiple distance constraints. The multiple distance constraints include: overlapping area, center point distance, aspect ratio of the ground truth box to the predicted box, aspect ratio of the intersecting box to the predicted box, top left corner distance, and top right corner distance.

[0033] The target detection module uses a trained road damage detection model to detect road damage in real image data.

[0034] Advantages and positive effects of the present invention: The present invention proposes a road damage detection method based on a coordinate positioning loss function with multiple distance constraints. By improving the coordinate positioning loss function, the commonly used CIoU-Loss experimental results show that the accuracy of the bounding box position is low, resulting in the detection box not fully containing the detected object. However, the coordinate positioning loss function WLCIoU-Loss with multiple distance constraints proposed in this invention can accurately detect road damage while reducing the false negative rate. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the road surface damage detection method in an embodiment of the present invention;

[0037] Figure 2 This is a diagram of the backbone network and neck network structure in the road surface damage detection model in this embodiment of the invention;

[0038] Figure 3 This is a schematic diagram illustrating the case where the aspect ratios of the intersecting box and the predicted box are different.

[0039] Figure 4This diagram illustrates various scenarios where IoU becomes ineffective.

[0040] Figure 5 This is a diagram illustrating the situation where the distance to the top left corner is no longer effective.

[0041] Figure 6 The figure shows the experimental results of various IoU comparisons in the embodiments of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] This invention provides a road damage detection model based on YOLOv5, consisting of four parts. The first part is the backbone network, which uses repVGG as the feature extraction network. The second part is the neck network, which uses FPN+PAN as the feature fusion network. The third part, the head, uses the NMS non-maximum suppression algorithm. The fourth part is the loss function, where the coordinate localization loss function uses WLCIoU-Loss (a coordinate localization loss function with multiple distance constraints) to train the model. Weights are then applied for detection.

[0045] like Figure 1As shown, the above-mentioned road damage detection model can be used for road damage detection, including: labeling real images to be detected using labels, generating corresponding labels, dividing the dataset into training and testing sets, training with an improved YOLOv5, testing with the generated weight file, and selecting the model with the highest F1-score to detect images. By improving the bounding box regression loss function, the accuracy of road damage detection is improved. The road damage detection method provided in this embodiment of the invention specifically includes the following steps:

[0046] S1. Obtain real image data to be detected and perform data augmentation.

[0047] The data is augmented using a brightness and contrast adjustment algorithm. The calculation formula is as follows:

[0048] Z1 = αP1 + β;

[0049] In the above formula, P1 is the input, Z1 is the output, parameter α is the gain, which is used to adjust the contrast of the road surface image; β is the deviation parameter, which is used to adjust the brightness of the road surface image.

[0050] S2. Feature extraction is performed using the backbone network in the road damage detection model;

[0051] The backbone uses the reparameterized repVGG network as the feature extraction network. Since repVGG has the characteristic of being comparable to the most advanced feature extraction networks, the reparameterization idea transforms the multi-path structure of the training network into the single-path structure of the inference network. During inference, the multi-branch model is converted into a single-path model.

[0052] S3. Feature fusion is performed on the extracted features through the neck network in the road damage detection model;

[0053] The neck network presents a feature fusion network that combines a recursive feature pyramid with a path aggregation network. After the FPN layer, a bottom-up path aggregation network is added. This operation complements the FPN and passes the strong localization features from the lower layers to the upper layers.

[0054] Backbone network and neck network structure diagram as follows Figure 2 As shown.

[0055] S4. The candidate bounding boxes obtained after feature fusion are filtered using the non-maximum suppression algorithm to obtain the road surface damage bounding boxes.

[0056] Input a set of candidate bounding boxes (each with a confidence score), an IoU threshold, and an initial empty set D. Sort the candidate bounding boxes in descending order of their confidence scores. Select the first candidate box (with the highest confidence score) from the set, add it to set D, and remove it from the set. Iterate through each candidate box in the set and calculate its IoU value with the corresponding candidate box in set D. If the IoU value is greater than the IoU threshold, remove it from the set. Repeat this process until the set is empty, and finally output D.

[0057] S5. Train the road damage detection model using the loss function.

[0058] The loss function of YOLOv5 for object detection consists of three parts: coordinate localization loss, confidence loss, and classification loss, all of which are applied to the object detection algorithm.

[0059] Bounding box regression, a fundamental task in object detection, is widely used in computer vision. It locates objects in an image by regressing the predicted bounding boxes. The calculation method involves determining the distance deviation between the predicted bounding box and the ground truth bounding box, calculating the loss value using a specific function, and then iteratively updating the weight parameters through backpropagation until the predicted bounding box approximates the true object.

[0060] To date, bounding box regression methods are based on Intersection over Union (IoU-Loss), Generalized Intersection over Union (GIoU-Loss), Distance Intersection over Union (DIoU-Loss), and Complete Intersection over Union (CIoU-Loss) and their variants.

[0061] IoU-Loss uses the intersection-union ratio (IoU) to measure the distance between the ground truth bounding box and the target bounding box; its calculation method is simple. Let D represent the predicted bounding box, D... gt Representing the ground truth bounding box, IoU is the intersection-over-union ratio between the predicted bounding box and the ground truth bounding box. The calculation formula is as follows:

[0062]

[0063] GIoU-Loss addresses the issue of failing to update weight parameters when the predicted and ground truth bounding boxes do not intersect. C represents the smallest bounding rectangle between the predicted and ground truth bounding boxes. When the difference between the predicted and ground truth bounding boxes is too large, C increases, resulting in a smaller GIoU value. It can measure the loss when the predicted and ground truth bounding boxes are mutually inclusive, and it can also calculate the distance when the two boxes are not inclusive. The calculation formula is as follows:

[0064]

[0065] DIoU-Loss introduces the distance between the center points of the predicted bounding box and the ground truth bounding box into the IoU and GIoU loss functions, avoiding the divergence problem during training. Let d represent the Euclidean distance between the center points of the predicted and ground truth bounding boxes, and c represent the diagonal distance enclosed by the smallest outer rectangles of the two boxes. The calculation formula is as follows:

[0066]

[0067] CIoU-Loss designs its loss function from three perspectives: intersection-to-union ratio (CRO), center point distance, and aspect ratio information. Let W... gt With H gt Let W and H represent the width and height of the object detection box, respectively, and let α represent the width and height of the predicted box, respectively. Let α represent the weighting coefficient, v represent a function that measures aspect ratio consistency, and αv represent a penalty term that measures aspect ratio. The calculation formula is as follows:

[0068]

[0069]

[0070]

[0071] CIoU-Loss has become the mainstream loss function for coordinate localization in object detection. CIoU-Loss considers both the intersection-union ratio (IU) between the predicted and ground truth boxes and uses center distance and aspect ratio penalties, thus making bounding box regression more accurate in most cases. However, it also has problems, specifically as follows:

[0072] (1) As Figure 3 As shown in the figure, the green box represents the ground truth box and the red box represents the predicted box. When the aspect ratios of the ground truth box and the predicted box remain unchanged, the aspect ratio penalty term has no effect, and the model convergence speed will be slower.

[0073] (2) Figure 4 As shown, when the predicted bounding box and the ground truth bounding box are contained within each other, the values ​​of IoU-Loss, GIoU-Loss, DIoU-Loss, and CIoU-Loss are all the same because the IoU size, aspect ratio, and center point distance are all the same. Therefore, the algorithm cannot provide a clear gradient direction for optimization, and all the IoU values ​​become ineffective.

[0074] (3) Figure 5 As shown, when the predicted bounding box and the ground truth bounding box are contained within each other, the box will also lose its function because the IoU size, aspect ratio, center point distance, and top left corner distance are all the same.

[0075] To address the aforementioned issues, this invention further improves the coordinate positioning loss function CIoU-Loss, proposing a coordinate positioning loss function based on multiple distance constraints that offers higher accuracy and lower false negative rates. By improving the coordinate positioning loss function, road surface damage can be detected more accurately while reducing the false negative rate.

[0076] (1) Coordinate positioning loss function

[0077] In this embodiment of the invention, based on the currently mainstream coordinate positioning loss functions CIoU and WLCIoU, the specific formulas are as follows:

[0078]

[0079] WLCIoU is composed of six geometric factors: overlap area, center-to-center distance, aspect ratio of the ground truth bounding box and the predicted bounding box, aspect ratio of the intersecting bounding box and the predicted bounding box, top-left corner distance, and top-right corner distance. Let d be the center-to-center distance between the ground truth bounding box and the predicted bounding box, K be the Euclidean distance of the top-left corner coordinates between the predicted and ground truth bounding boxes, B be the Euclidean distance of the top-right corner coordinates between the predicted and ground truth bounding boxes, c be the diagonal length of the minimum bounding box between the ground truth and the predicted bounding box, α·v be the penalty term for the aspect ratio of the ground truth bounding box and the predicted bounding box, and α1v1 be the penalty direction for the aspect ratio of the intersecting bounding box and the predicted bounding box. Considering that cases where the predicted and target bounding boxes contain each other and the center-to-center distance, aspect ratio, and top-left corner distance cannot provide optimization directions are relatively rare, λ is assigned as a weighted optimization loss. Let W... gt With H gt Let W and H represent the width and height of the ground truth bounding box, respectively, and let W and H represent the width and height of the predicted bounding box, respectively. in With H in Let v and v1 represent the width and height of the intersecting boxes, respectively. v and v1 are used to measure the consistency of the aspect ratio. α and α1 are weighting coefficients. The calculation expressions for αv and α1v1 are as follows:

[0080]

[0081]

[0082]

[0083]

[0084] (2) Confidence loss

[0085] Cross-entropy loss function is used as the confidence loss (loss2). Let N represent the number of grid cells in the image, and D represent the number of rectangular bounding boxes. This indicates whether the target exists in the j1-th rectangular bounding box of the i1-th grid. If it exists, then... If it does not exist, Conversely, when the j1st rectangular edge in the i1th grid

[0086] When the target is not in the bounding box, Otherwise, it is 0. This indicates the confidence level of the predicted bounding box. The label represents the sample; if the sample is a positive sample, then... If the sample is a negative sample, then λ noobj The weighting coefficient represents the weight when the object does not exist. The confidence loss2 is defined as follows:

[0087]

[0088] (3) Classification loss

[0089] By employing the binary cross-entropy function as the classification loss, the use of the Softmax function is avoided, thus reducing computational complexity. The label represents the sample information of the j1-th rectangular bounding box in the i1-th grid. If the sample is a positive sample, then... If the sample is a negative sample, then Let $\frac{j}{i}$ be the class probability of the $j1$-th predicted bounding box in the $i1$-th grid. The classification loss $Loss3$ is defined as follows:

[0090]

[0091] The loss function formula for road damage detection, consisting of coordinate positioning loss, confidence loss, and classification loss, is as follows:

[0092] Loss = Loss1 + Loss2 + Loss3.

[0093] The training parameters of the road surface damage detection model are shown in Table 1.

[0094] Table 1

[0095]

[0096] The results of the comparison experiment of multiple IoU are shown in the figure below. Figure 6 As shown, WLCIoU in this invention can accurately and completely detect road surface damage, while CIoU-Loss experimental results show that the accuracy of the bounding box position is low, resulting in the detection box not fully containing the detected object. The existing GIoU and DIoU also have varying degrees of target missing and position inaccuracy.

[0097] Corresponding to the road surface damage detection method in the above embodiments, this embodiment of the invention also provides a road surface damage detection system, including:

[0098] The model building module is used to build a road surface damage detection model based on YOLOv5;

[0099] The model training module is used to train the road damage detection model using the road damage loss function. The road damage loss function includes: coordinate localization loss, confidence loss, and classification loss based on multiple distance constraints. The multiple distance constraints include: overlapping area, center point distance, aspect ratio of the ground truth box to the predicted box, aspect ratio of the intersecting box to the predicted box, top left corner distance, and top right corner distance.

[0100] The target detection module uses a trained road damage detection model to detect road damage in real image data.

[0101] The road damage detection system of this invention corresponds to the road damage detection method in the above embodiments, so the description is relatively simple. For related similarities, please refer to the description of the road damage detection method in the above embodiments, which will not be described in detail here.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1.A road damage detection method based on a coordinate positioning loss function of multiple distance constraints, characterized by, The application relates to a pavement damage detection model based on YOLOv5. The pavement damage detection model is trained by using a pavement damage loss function. The pavement damage loss function comprises a coordinate positioning loss based on multiple distance constraints, a confidence loss and a classification loss; the multiple distance constraints comprise an overlapping area, a center point distance, a real frame and a predicted frame width-height ratio, an intersecting frame and a predicted frame width-height ratio, a left upper corner distance and a right upper corner distance. The pavement damage detection model comprises a backbone network, a neck network and a head network; the backbone network adopts repVGG as a feature extraction network; the neck network adopts a feature fusion network combining a recursive feature pyramid and a path aggregation network; and the head network adopts a non-maximum suppression algorithm. The coordinate positioning loss based on the multiple distance constraints is as follows: The confidence loss adopts a cross-entropy loss function. ； wherein, d is the distance between the center points of the real and predicted boxes, K represents the Euclidean distance between the top-left corner coordinates of the predicted and real boxes, B is the Euclidean distance between the top-right corner coordinates of the predicted and real boxes, c is the diagonal length of the minimum enclosing box of the real and predicted boxes, is the penalty term for the aspect ratio of the real and predicted boxes, is the penalty term for the aspect ratio of the intersection and predicted boxes, is the weighted optimization loss; and The calculation expression of is as follows: ； ； ； ； wherein, denote the width and height of the ground truth box, denote the width and height of the predicted box, and denote the width and height of the intersection box, denote the width and height of the ground truth box, denote the width and height of the intersection box, and are used to measure the consistency of the aspect ratio, and are weight coefficients; The classification loss adopts a binary cross-entropy function. Real image data to be detected is subjected to pavement damage detection by using the trained pavement damage detection model. The application further relates to a pavement damage detection method. 2.The road damage detection method based on the coordinate positioning loss function of multiple distance constraints according to claim 1, wherein, The real image data to be detected is subjected to data enhancement. The data enhancement comprises data expansion by using a brightness-contrast adjustment algorithm, and a calculation formula of the brightness-contrast adjustment algorithm is as follows: 3.The road damage detection method based on the coordinate positioning loss function of multiple distance constraints according to claim 2, wherein, The application relates to a pavement damage detection model based on YOLOv5. ； wherein, is an input, is an output, parameter is a gain, used to adjust the contrast of the road surface image; is a bias parameter, used to adjust the brightness of the road surface image. 4.A road damage detection system based on a coordinate positioning loss function of multiple distance constraints, characterized by, The pavement damage detection model is trained by using a pavement damage loss function. The pavement damage loss function comprises a coordinate positioning loss based on multiple distance constraints, a confidence loss and a classification loss; the multiple distance constraints comprise an overlapping area, a center point distance, a real frame and a predicted frame width-height ratio, an intersecting frame and a predicted frame width-height ratio, a left upper corner distance and a right upper corner distance. The coordinate positioning loss based on the multiple distance constraints is as follows: The confidence loss adopts a cross-entropy loss function. The classification loss adopts a binary cross-entropy function. ； wherein, d is the distance between the center points of the real and predicted boxes, K represents the Euclidean distance between the top-left corner coordinates of the predicted and real boxes, B is the Euclidean distance between the top-right corner coordinates of the predicted and real boxes, c is the diagonal length of the minimum enclosing box of the real and predicted boxes, is the penalty term for the aspect ratio of the real and predicted boxes, is the penalty term for the aspect ratio of the intersection and predicted boxes, is the weighted optimization loss; and The calculation expression of is as follows: ； ； ； ； wherein, denote the width and height of the real box, respectively, denote the width and height of the predicted box, respectively, and denote the width and height of the intersection box, respectively, and denote the width and height of the intersection box, respectively, and are used to measure the consistency of the aspect ratio, and are weight coefficients; Real image data to be detected is subjected to pavement damage detection by using the trained pavement damage detection model. ​ ​

Images