Pavement disease detection method based on improved YOLOv8

Abstract

The invention relates to a pavement disease detection method based on improved YOLOv8. The method comprises the following steps: collecting to-be-detected pavement video data; inputting the to-be-detected pavement video data into a preset pavement disease detection model, and obtaining a video frame image set after disease classification; wherein the pavement disease detection model is constructed based on an improved YOLOv8 model, and is obtained by training a pre-constructed pavement disease image data set; and determining the disease category of the same position of a single-frame image in the video frame image set, and performing de-weighting and counting processing on the diseases of the same position in continuous multiple-frame images to obtain a pavement disease detection result. According to the method, the dependence on manual intervention is reduced, and the accuracy and efficiency of pavement disease detection are remarkably improved. Through the automatic and intelligent detection process, the labor cost is reduced, and the phenomena of misjudgment and missed judgment caused by human factors are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of road inspection, and in particular to a road surface disease detection method based on improved YOLOv8. Background Art

[0002] Road inspection is an important part of ensuring traffic safety. It can timely discover and repair road problems, reduce traffic accidents and road damage. With the expansion of the transportation network and the increase in transportation tools, the importance of road inspection has become increasingly prominent. At the same time, the cost of road maintenance is usually 2-3 times the construction cost. At present, my country's road inspection mainly adopts the method of manual inspection, which usually requires a lot of manpower and material resources, but problems such as missed inspections, false inspections, low efficiency, and low accuracy are common. In addition, inspectors need to stop and take photos and measure at any time, which not only affects traffic, but also lacks safety guarantees. Since road inspections are usually carried out outdoors, they are affected by environmental factors such as weather and traffic conditions. When bad weather occurs, the equipment may not work properly, and even additional safety measures may be required to protect the inspectors. These factors may reduce the efficiency of road inspections. In addition, for management departments, the lack of effective structured data support has brought great difficulties to road maintenance decisions, and it is often difficult to achieve cost reduction and efficiency improvement. Therefore, how to improve the efficiency and accuracy of road inspections has become a major problem before us.

[0003] Road inspection requires real-time or near-real-time data processing and feedback to timely discover and deal with road problems. In the case of a changing inspection environment, how to transmit the acquired data in real time and reduce delays, improve the accuracy of data processing, and make AI models and edge computing algorithms have strong environmental adaptability, security and robustness are key technological innovations that need to be broken through at present. The pavement disease detection algorithm based on deep learning realizes end-to-end learning from raw images to disease recognition through automatic feature extraction, and does not require complex preprocessing work like previous methods, which simplifies the detection process. In addition, the generalization ability of the deep learning model enables it to adapt to disease detection tasks under different lighting, weather and road conditions. Through transfer learning and incremental learning, the training cost is reduced, the comprehensiveness of the algorithm is improved, and the dynamic needs of road inspection are met. The algorithm combines with the attention mechanism to enable the model to focus on identifying key disease areas in the image, integrate multi-source sensor data, and improve the accuracy and robustness of detection. The characteristics of model compression and optimization technology also enable the deep learning model to adapt to the resource limitations of the on-board computing platform, maintain efficient disease detection performance, and greatly improve detection efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a road surface disease detection method based on improved YOLOv8, which is used to improve the accuracy of road surface disease detection, improve the efficiency of road intelligent inspection and ensure traffic safety.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A road surface disease detection method based on improved YOLOv8, comprising:

[0007] Collecting road surface video data to be detected;

[0008] Input the road surface video data to be detected into a preset road surface disease detection model to obtain a set of video frame images after disease classification; wherein the road surface disease detection model is constructed based on an improved YOLOv8 model and is trained using a pre-constructed road surface disease image dataset;

[0009] The defect category at the same position of a single frame image in the video frame image set is determined, and duplicate removal and counting processing are performed on the defects at the same position in multiple consecutive frames of images to obtain a road surface defect detection result.

[0010] Optionally, pre-constructing the pavement damage image dataset includes:

[0011] Collecting road surface video frame image data, the road surface video frame image data includes: damage image data of several types of damage under different road types, different lighting and different weather conditions, the damage types include: longitudinal cracks, longitudinal cracks, cracks, potholes, transverse intersection lines, longitudinal intersection lines, asphalt repairs, rutting, lane line damage, and sidewalk damage;

[0012] The target position is annotated on the road surface video frame image data, and the annotated image data is preprocessed to obtain the data set.

[0013] Optionally, preprocessing the annotated image data includes:

[0014] The adversarial network with quadratic loss function is used to enhance the labeled image data.

[0015] The enhanced image data is fitted using an improved cubic curve active learning method, and the preset image data is selected by analyzing the curvature of the curve;

[0016] The improved cubic curve active learning method is:

[0017] AL (D augmented ,M0)→(D train ,D val ,D test )

[0018] Among them, AL represents the active learning process, M0 is the initial model, and D train , D val ,D test They represent the training set, validation set, and test set respectively.

[0019] Optionally, improving the YOLOv8 model includes:

[0020] A CNX2f feature extraction module is introduced into the backbone network of the YOLOv8 model to replace the C2f feature extraction module, and a RepConv reparameterization module is introduced to replace the ordinary convolution module in the original network, and a DBB module is introduced in the neck network and the Head. Both modules use a multi-branch structure in the training stage and are converted into a single-branch structure through reparameterization in the inference stage. At the same time, a shared parameter structure is adopted for the Head of the network, and a new reparameterization module RBB is introduced; and the SPPF module in the YOLOv8 network is improved to construct a dual-branch structure SPPFAvg module.

[0021] Optionally, in the improved YOLOv8 model, the convolutional layer and the BN layer are automatically fused during the inference phase;

[0022] The convolutional layer is:

[0023]

[0024] Among them, x is the input value, W is the weight of the convolution module, b is the bias, and avg is the input mean;

[0025] The BN layer is:

[0026]

[0027] Among them, σ is the input variance, γ is the BN weight, and β is the weight;

[0028] The fusion process of the convolutional layer and the BN layer is:

[0029]

[0030] Among them, F fuse are the fused weights and bias values.

[0031] Optionally, improving the YOLOv8 model further includes: introducing a multi-scale feature fusion module to fuse multi-scale features from different sensors during the training phase;

[0032] The multi-scale feature fusion module is:

[0033]

[0034] Among them, F multi-scale represents the result of multi-scale feature fusion, σ is the activation function, W scale,i and b i is the convolution kernel weight and bias of the i-th scale, and N is the number of multi-scale features.

[0035] Optionally, use a pre-built pavement damage image dataset for training including:

[0036] Input the pavement disease image dataset into the improved YOLOv8 model for recognition, obtain the detection frame and category information, input the detection frame, category information and the annotation information of the pavement video frame image data into a preset optimization loss function, and calculate the loss value; wherein the loss function includes: border regression loss CIoU Loss, DFL loss, and category classification cross entropy loss;

[0037] A gradient descent algorithm is set to optimize the model parameters of the improved YOLOv8 model, and training rounds are set to obtain the trained improved YOLOv8 target detection model as the pavement disease detection model.

[0038] Optionally, determining the category of the damage at the same position of a single frame image in the video frame image set, performing deduplication and counting processing on the damage at the same position in multiple consecutive frames of images, and obtaining the road surface damage detection result includes:

[0039] If the same position of a single frame image in the video frame image set is marked by several detection frames, it is determined whether the categories of the detection frames are the same: if the categories of the detection frames are different, no processing is performed, and the category of the detection frame is the category of the corresponding disease; if the categories of the detection frames are the same, the pavement disease detection results in the video frame image set are deduplicated;

[0040] Based on the adjacent multi-frame multi-target defect tracking algorithm, the video frame image set after deduplication is counted to obtain the road surface defect detection result.

[0041] Optionally, deduplication of the road surface disease detection results in the video frame image set includes:

[0042] Determine whether the two rectangles intersect in space. If so, merge the two rectangles.

[0043] Assume that the coordinates of the upper left corner and lower right corner of rectangular boxes A and B are (x01, y01), (x02, y02), (x11, y11), (x12, y12)

[0044] Assume that the distance Dx between the two center points of rectangular frame A and rectangular frame B in the X-axis direction is:

[0045] Dx=abs((x01+x02) / 2-(x11+x12) / 2)

[0046] Assume that the distance Dy between the two center points of rectangular frame A and rectangular frame B in the Y-axis direction is:

[0047] Dy=abs((y01+y02) / 2-(y11+y12) / 2)

[0048] Assume that the lengths of rectangular frames A and B in the X-axis direction are Dax and Dbx:

[0049] Dax=abs(x01-x02),Dbx=abs(x11-x12)

[0050] Assume that the lengths of rectangles A and B in the Y-axis direction are Day and Dby:

[0051] Day=abs(y01-y02),Dby=abs(y11-y12)

[0052] If rectangles A and B intersect,

[0053] Dx<=(Dax+Dbx) / 2&&Dy<=(Day+Dby) / 2

[0054] Suppose the coordinates of the upper left corner and lower right corner of the merged rectangle are (x1, y1) and (x2, y2):

[0055] x1=min(x01,x11),y1=min(y01,y11)

[0056] x2=max(x02,x12), y2=max(y02,y12).

[0057] Optionally, based on the adjacent multi-frame multi-target disease tracking algorithm, counting the deduplicated video frame image set includes:

[0058] Assigning a tracker to each target disease in the deduplicated video frame image set to predict the position of the target disease in the next frame;

[0059] The predicted position is corrected according to the detection result of the pavement defect position in the next frame to obtain the optimal position estimate of the target defect tracked in this frame;

[0060] Compare the disease at the optimal position in the current frame with the tracked disease in the previous frame: if the two are the same disease, then the target disease is counted in the previous and next frames to remove duplicates; if the two are not the same disease, then count the tracked disease to obtain the real-time tracking count and detection results of the same disease between consecutive frames;

[0061] Assign a global counter to each different category of pavement disease; initialize a tracking number list to save the numbers of pavement diseases that have been counted; if it is confirmed to be the same disease, check whether the tracking number of the pavement disease already exists in the tracking number list: if it does, cancel the repeated counting of the target disease; if it does not exist, add one to the global counter of the corresponding category to update the statistical information of the pavement disease.

[0062] The beneficial effects of the present invention are:

[0063] The present invention is based on an improved YOLOv8 road surface disease detection method, and aims to accurately meet the disease detection requirements in scenes with complex road surface backgrounds and different scales of disease targets. A complete disease detection and analysis statistical system is designed, and a deep learning algorithm is used to build a road surface disease detection model framework. Based on the detection results of road surface diseases in the video, the accurate statistics of road surface diseases in the video are completed from the spatial position of the single-frame disease and the time relationship between adjacent multiple frames. Finally, the present invention is applied to the inspection and analysis of actual road surface diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0065] Figure 1 A schematic flow chart of a road surface disease detection method based on improved YOLOv8 according to an embodiment of the present invention;

[0066] Figure 2 This is a diagram of the improved YOLOv8 network structure of an embodiment of the present invention;

[0067] Figure 3 It is a flowchart of a single-frame target disease deduplication algorithm according to an embodiment of the present invention;

[0068] Figure 4 The present invention is a flowchart of a method for implementing road disease statistics according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] The deep learning-based pavement disease detection algorithm achieves end-to-end learning from raw images to disease recognition through automated feature extraction. It does not require complex preprocessing like previous methods, simplifying the detection process. In addition, the generalization ability of the deep learning model enables it to adapt to disease detection tasks under different lighting, weather and road conditions. It reduces training costs through transfer learning and incremental learning, improves the comprehensiveness of the algorithm, and meets the dynamic needs of road inspections. The algorithm is combined with the attention mechanism to enable the model to focus on identifying key disease areas in the image, integrate multi-source sensor data, and improve the accuracy and robustness of detection. The characteristics of model compression and optimization technology also enable the deep learning model to adapt to the resource limitations of the on-board computing platform, maintain efficient disease detection performance, and greatly improve detection efficiency.

[0072] like Figure 1 As shown, this embodiment proposes a road surface disease detection method based on improved YOLOv8, including:

[0073] Collecting road surface video data to be detected;

[0074] The road surface video data to be detected is input into a preset road surface disease detection model to obtain a set of video frame images after disease classification; wherein the road surface disease detection model is constructed based on an improved YOLOv8 model and is trained using a pre-constructed road surface disease image dataset;

[0075] Determine the category of the damage at the same position in a single frame image in the video frame image set, remove duplicates and count the damage at the same position in multiple consecutive frames of images, and obtain the pavement damage detection results.

[0076] Specifically, this embodiment uses the on-board camera of the inspection vehicle to collect road video frame images and construct a training sample set; construct an improved YOLOv8 model that introduces feature extraction and re-parameterization; then train the improved YOLOv8 model based on the training set to obtain the trained improved YOLOv8 model as a road disease detection model; input the training sample set into the model, use the gradient descent method to iteratively train, and obtain the optimized model; classify the road diseases in the inspection video; determine the disease category at the same position of a single frame image in the video frame image set; remove and count the diseases at the same position in multiple consecutive frames, and finally obtain accurate road disease detection results. The present invention reduces the dependence on manual intervention and significantly improves the accuracy and efficiency of road disease detection. Through the automated and intelligent detection process, not only the labor cost is reduced, but also the misjudgment and missed judgment caused by human factors are reduced. This efficient and accurate disease detection method can timely discover and locate road problems, facilitate rapid response and repair by relevant departments, thereby extending the service life of the road and ensuring traffic safety. In addition, the present invention exhibits good adaptability in different environments and conditions, and is suitable for the inspection and maintenance of various road surfaces, thus promoting the modernization and intelligent development of road maintenance work.

[0077] Furthermore, the pre-built pavement damage image dataset includes:

[0078] Collect road surface video frame image data, including: disease image data of several types of diseases under different road types, different lighting and different weather conditions, including: longitudinal cracks, longitudinal cracks, cracks, potholes, transverse intersections, longitudinal intersections, asphalt repairs, rutting, lane line damage, and sidewalk damage;

[0079] The target position is annotated on the road surface video frame image data, and the annotated image data is preprocessed to obtain a data set.

[0080] Furthermore, preprocessing the annotated image data includes:

[0081] The adversarial network with quadratic loss function is used to enhance the labeled image data.

[0082] The enhanced image data is fitted using an improved cubic curve active learning method, and the preset image data is selected by analyzing the curvature of the curve;

[0083] The improved cubic curve active learning method is:

[0084] AL (D augmented ,M0)→(D train ,D val ,D test )

[0085] Among them, AL represents the active learning process, M0 is the initial model, and D train , D val ,D test They represent the training set, validation set, and test set respectively.

[0086] Specifically, in this embodiment, road surface video frame images are captured by a vehicle-mounted camera, and the established road surface mainly comes from disease images containing ten types of diseases under various road types, various lighting conditions, and various weather conditions. The acquired images are annotated with target positions, and then data set enhancement is performed to reduce the uneven distribution of the number of category disease labels. Finally, the test set, training set, and validation set are divided and selected.

[0087] Get the test set, training set, and validation set. The specific steps include:

[0088] S11. Use the on-board camera of the inspection vehicle to shoot road video frame images, and extract video frame images of disease images from various road types, various lighting and various weather conditions. The disease images include longitudinal cracks, longitudinal cracks, cracks, potholes, transverse intersections, longitudinal intersections, asphalt repairs, rutting, lane line pollution, and sidewalk pollution. The data is represented as follows:

[0089] D raw ={(I i ,P i )|i=1,2,…,N}

[0090] Among them, I i represents the i-th road picture, P i Represents the corresponding location information, and N represents the total number of collections.

[0091] S12. Using the image annotation tool LabelImg, label each image in the dataset according to the format requirements of the PASCAL VOC dataset to obtain a label set containing disease information:

[0092] D labeled ={(I i ,P i ,L i )|i=1,2,…,M}

[0093] Among them, L i It indicates the disease type mark in i pictures, and M indicates the total number of marks.

[0094] S13. In the improved quadratic generative adversarial network, a quadratic loss function is introduced. Let the generator be G, the discriminator be D, and the real data distribution be p data , the generated data distribution pg , the loss function of the generator is:

[0095]

[0096] Where z is the noise vector, p z (z) is the noise distribution and G(z) is the generated diseased image.

[0097] The discriminator loss function is:

[0098]

[0099] Among them, x is the real image, D(x) is the discrimination result of the real image, and D(G(z)) is the discrimination result of the generated image.

[0100] New disease images are generated through GAN and added to the original dataset to reduce the uneven distribution of the number of disease labels.

[0101] The formula means:

[0102] D augmented =D labeled ∪{G(z)|z~p z (z)}

[0103] S14. Combine the label sets of different disease information with their corresponding original video frame images to form a sample set. In the active learning process, use the cubic curve to fit the characteristics of the data samples. By analyzing the curvature of the curve, dynamically select high-information samples. Select representative and high-information samples according to the curvature value. Randomly select no more than one-tenth of the samples as the test set, and divide the remaining part into a training set and a validation set at a ratio of 9:1: use a part of the labeled data to train the initial model; select new samples for labeling according to the uncertainty or information content of the model; and update the model with the newly labeled samples.

[0104] AL (D augmented ,M0)→(D train ,D val ,D test )

[0105] Among them, AL represents the active learning process, M0 is the initial model, and D train , D val ,D test They represent the training set, validation set, and test set respectively.

[0106] Further improvements to the YOLOv8 model include:

[0107] The CNX2f feature extraction module is introduced into the backbone network of the YOLOv8 model to replace the C2f feature extraction module, and the RepConv reparameterization module is introduced to replace the ordinary convolution module in the original network, and the DBB module is introduced in the neck network and the Head. Both modules use a multi-branch structure in the training phase and are converted into a single-branch structure through reparameterization in the inference phase. At the same time, a shared parameter structure is adopted for the Head of the network, and a new reparameterization module RBB is introduced; the SPPF module in the YOLOv8 network is improved, and a dual-branch structure SPPFAvg module is constructed.

[0108] Specifically, in this embodiment, first, the CNX2f module is introduced into the backbone network of the model. The introduction of the 7x7 depth-separable convolution expands the receptive field of feature extraction, improves the correlation between features, and better distinguishes the pavement disease features from the background environment features. Secondly, the re-parameterization module RepConv is introduced into the backbone network, and the DBB module is introduced into the neck network and the Head. Both modules use a multi-branch structure in the training stage to improve the training effect of the model. In the inference stage, the re-parameterization is converted into a single-branch structure to improve the inference speed, and solve the problem of false detection caused by the network's insufficient ability to extract and fuse pavement disease features. At the same time, a shared parameter structure is used for the Head of the network, and a new re-parameterization module RBB is introduced to solve the problem of redundant head parameters and improve the head network feature extraction capability. Finally, the SPPF module in the YOLOv8 network is improved, and a dual-branch structure SPPFAvg module is constructed to prevent the loss of pavement disease features and solve the problem of insufficient multi-scale feature extraction of pavement diseases. The improved YOLOv8n network structure is shown in the figure. Figure 2 shown.

[0109] Considering the similarity between pavement disease features and some background environment features, using a larger convolution kernel to extract features can cover a larger receptive field, thereby capturing more contextual information, making the features more distinguishable and interpretable, and effectively solving the problem of target features and background features being easily confused. This study constructed a CNX2f module and introduced a YOLOv8 network to replace the C2f feature extraction module. Experimental verification found that introducing the CNX2f module only into the backbone network has the greatest improvement on the model. The specific location is as follows Figure 2 shown.

[0110] The CNX2f module adopts a spilt structure, using two CNX modules to replace the two ordinary convolutions C in C2f. The CNX module reduces the number of channels through 1x1 ordinary convolution, reduces the number of model parameters and calculations, and the 7x7 depth-separable convolution expands the receptive field of the model and improves the detection effect of the model.

[0111] The multi-branch structure can better extract multi-scale features, and its detection effect is better than the single-branch structure. However, due to the use of a multi-branch parallel structure, the amount of calculation and the number of parameters are large. The re-parameterization module integrates the multi-branch structure and the single-branch structure. In the model training stage, the multi-branch parallel structure is used to better realize feature extraction and obtain better weight parameters. In the inference stage, the multi-branch structure is converted into a single-branch structure to complete the detection task more efficiently.

[0112] In the task of pavement disease detection, the characteristics of longitudinal cracks, transverse cracks and mesh cracks are similar, and false detection is prone to occur. The use of a heavy parameter multi-branch structure to realize multi-scale feature extraction can effectively solve the problem of false detection of crack types, and at the same time, the re-parameterization will not affect the detection efficiency of the model. This study introduces the RepConv module in the backbone network of the model to replace the ordinary convolution module in the original network, improves the multi-scale feature extraction ability of the backbone network, and better integrates multi-scale features to obtain model parameters. The "DBB" module is introduced at the end of the neck network to improve the multi-scale feature expression ability of the feature map, effectively alleviating the multi-scale information loss problem caused by sampling on the neck network.

[0113] The RepConv (Re-param Conv) module is the basic module of the RepVGG network. It adopts the convolutional reparameterization method. In the Train stage, it uses the parallel structure of BatchNorm, 1x1 convolution and 3x3 convolution to enrich multi-scale information. In the Val stage, the convolutional reparameterization method is used to merge multiple branches into a single branch structure. The reparameterization module achieves efficient and high-precision detection through a 3x3 convolution.

[0114] In the YOLOv8 algorithm, the convolution layer and the BN layer are automatically fused during the inference phase to accelerate inference. The convolution layer formula is shown in formula (1), the BN layer formula is shown in formula (2), and the fusion process is shown in formula (3):

[0115]

[0116]

[0117]

[0118] Among them, x is the input value, W is the convolution module weight, b is the bias, avg is the input mean, σ is the input variance, γ is the BN weight, β is the weight, and fuse is the weight and bias value after fusion. During the model conversion process, the weight size of each branch needs to be converted to 3×3 size, among which the 1×1 convolution is converted by pad operation, and finally the weights and biases of each branch are summed up as the weights and biases of a single branch convolution.

[0119] RepConv is introduced into the model backbone network to achieve better feature extraction, and the number of parameters and calculations of the model in the training and inference stages. Although the introduction of RepConv increases the number of parameters and calculations in the training stage, the number of parameters and calculations in the inference stage is smaller than that of the YOLOv8n network, and the average detection accuracy is improved by 1 percentage point. In summary, the introduction of RepConv improves the detection accuracy without increasing the number of parameters and calculations in the model inference stage.

[0120] Furthermore, the present embodiment constructs an improved YOLOv8 model. First, a dual-channel feature extraction module is constructed, in which one channel is an improved CNX2f feature extraction module and the other channel is a reparameterization module. A fusion network is introduced to extract pavement damage features through a larger convolution kernel to solve the problem that pavement damage features and background environment features are easily confused. Secondly, a multi-scale feature fusion module is introduced, and multi-scale features from different sensors are fused in the training stage to solve the problem that the size of pavement damage varies greatly. Reparameterization is used in the inference stage to reduce the number of model parameters and improve the detection speed, thereby constructing an improved YOLOv8 model that introduces feature extraction, multi-scale fusion and reparameterization.

[0121] The first channel uses an improved CNX2f feature extraction module, which focuses on extracting two-dimensional image features and uses larger convolution kernels to capture more contextual information, thereby improving the ability to extract pavement disease features. The RepConv module uses a convolution reparameterization method. In the training phase, it uses a parallel structure of BatchNorm, 1x1 convolution, and 3x3 convolution to enrich multi-scale information. In the verification phase, the convolution reparameterization method is used to fuse multiple branches into a single branch structure. The reparameterization module uses a 3x3 convolution to achieve efficient and high-precision detection. The convolution layer and the BN layer are fused to achieve the purpose of accelerating reasoning. The convolution layer formula is shown as follows:

[0122]

[0123] The BN layer is shown as follows:

[0124]

[0125] The fusion process is shown in the formula:

[0126]

[0127] Among them, x is the input value, W is the convolution module weight, b is the bias, avg is the input mean, σ is the input variance, γ is the BN weight, β is the weight, F fuseare the fused weights and bias values. In the model conversion process, the weights of each branch need to be converted to 3×3 size, where the 1×1 convolution is converted by padding operation, and finally the weights and biases of each branch are summed up as the weights and biases of a single branch convolution.

[0128] The second channel uses the DBB reparameterization module with a separate branch structure. In the training phase, the separate branch structure is used to extract multi-scale features, obtain feature information of different sizes and complexities, and improve feature expression. In the verification phase, separate branches of different sizes and complexities are combined to increase the feature space, improve the expressive power of a single convolution, and achieve fast model reasoning. DBB fuses 1×1 convolution and k×k convolution on the same branch. Fusion of 1×1 convolution and average pooling on the same branch can be seen as a fusion of 1×1 convolution and k×k convolution, where each weight of the k×k convolution is 1 / (k×k). The fusion process is shown in the formula:

[0129]

[0130] Among them, k and b represent the fused weight and bias, k1 and b1 represent the weight and bias of 1×1 convolution, k k and b k Represents the weight and bias of the k×k convolution. First, the modules of the same branch are fused, and then the weight parameters of the modules of different branches are summed to realize the conversion of the module from multi-branch to single-branch.

[0131] This embodiment also introduces a multi-scale feature fusion module to fuse multi-scale features from different sensors during the training phase to solve the problem of large differences in the size of road surface diseases. The specific implementation is as follows:

[0132]

[0133] Among them, F multi-scale represents the result of multi-scale feature fusion, σ is the activation function, W scale,i and b i is the convolution kernel weight and bias of the i-th scale, and N is the number of multi-scale features.

[0134] Furthermore, in the training phase, the present embodiment first initializes the model parameters of the improved YOLOv8 target detection model;

[0135] The pavement defect images in the pavement disease image dataset are input into the improved YOLOv8 target detection model for recognition to obtain the detection frame and category information; the detection frame and category information and the annotation information of the pavement defect image are input into the preset optimization loss function to calculate the loss value. The loss function is the bounding box regression loss CIoU Loss and DFL loss, and the category classification cross entropy loss;

[0136] The gradient descent algorithm is set to SGD, the model parameters of the improved YOLOv8 target detection model are optimized, and the training round is set to 650 epochs to obtain the trained improved YOLOv8 target detection model as the pavement disease detection model.

[0137] This embodiment proposes a mixed precision training technology, which combines half-precision and single-precision floating-point calculations to reduce memory usage and improve calculation speed; applies transfer learning and multi-task learning methods, uses pre-trained models and shared features to improve the generalization ability and multi-task processing ability of the model; in the training process, adopts dynamic learning rate adjustment and gradient accumulation strategies to ensure stable training of the model on large-scale data sets; optimizes the key hyperparameters of the model through automatic hyperparameter tuning tools, and obtains an optimized improved YOLOv8 model as a road surface disease detection model. Specifically, it includes:

[0138] Mixed precision training combines half-precision (FP16) and single-precision (FP32) floating point calculations to reduce memory usage and increase computing speed.

[0139]

[0140] in, represents the loss calculated in single precision, Represents the loss computed at half precision.

[0141] Transfer learning uses the weights of the pre-trained model as the initial value to improve the generalization ability of the model. Multi-task learning processes multiple related tasks by sharing features to improve multi-task processing capabilities.

[0142] θ new =θ pre-trained +Δθ

[0143] Among them, θ new are the parameters of the new model, θ pre-trained are the parameters of the pre-trained model, and Δθ are the parameter adjustments on the new task.

[0144]

[0145] in, is the loss of the ith task, w i is the weight of the task, and M is the number of tasks.

[0146] During the training process, dynamic learning rate adjustment and gradient accumulation strategies are used to ensure stable training of the model on large-scale datasets.

[0147]

[0148] Among them, ηt is the learning rate of the tth iteration, η0 is the initial learning rate, T is the total number of iterations, and p is the exponential decay parameter.

[0149]

[0150] Among them, Δθ is the accumulated gradient, K is the number of steps of gradient accumulation, is the gradient at the kth iteration.

[0151] Use the automatic hyperparameter tuning tool to optimize the key hyperparameters of the model and obtain an optimized improved YOLOv8 model.

[0152]

[0153] Among them, θ optimal is the optimized hyperparameter, is the loss function on the validation set.

[0154] The classification of road surface diseases in road inspection videos mainly includes the following steps:

[0155] S41, using the on-board camera of the inspection vehicle to shoot the road surface video to be inspected as the input of the road surface disease detection model, using frame-by-frame and pixel-by-pixel comparison, on each frame of the image, the model identifies and marks the location and category of the road surface disease, and gives the corresponding confidence information;

[0156] S42, inputting the road surface video into the road surface disease detection model, and performing frame-by-frame and pixel-by-pixel detection and comparison on the video. The compared video includes each frame image marked with the road surface disease category and confidence information.

[0157] S43, divide the video into frames to obtain a video frame image set V = {I1, I2, ..., I T}, where I t represents the t-th frame of video image, each image contains the detected road disease information P t ={p t1 ,p t2 ,…,p tk},C t ={c t1 ,c t2 ,…,c tk}, respectively representing the location and category of the disease in the tth frame.

[0158] S44. Modeling the dynamic changes of pavement damage using time series models t =f(H t-1 ,P t ,C t ), where H tIt represents the time series information of time t. During the modeling process, factors such as the duration and diffusion rate of pavement diseases are taken into account.

[0159] That is, in this embodiment, a video of the road surface to be detected is taken by the on-board camera of the inspection vehicle as the input of the road surface disease detection model, and a frame-by-frame and pixel-by-pixel comparison is adopted to obtain a video containing road surface disease category and confidence information marked thereon; the video is divided into frames to obtain a video frame image set.

[0160] Furthermore, the category of the damage at the same position of a single frame image in the video frame image set is determined, and the damage at the same position in the continuous multiple frames of images is deduplicated and counted to obtain the road surface damage detection results including:

[0161] If the same defect in a single frame image in the video frame image set is marked by several detection frames, it is determined whether the categories of the detection frames are the same: if the categories of the detection frames are different, no processing is performed, and the category of the detection frame is the category of the corresponding defect; if the categories of the detection frames are the same, the pavement defect detection results in the video frame image set are deduplicated;

[0162] Based on the adjacent multi-frame multi-target defect tracking algorithm, the deduplicated video frame image set is counted to obtain the pavement defect detection result.

[0163] Specifically, in this embodiment. Figure 3 As shown, deduplication of the road surface disease detection results in the video frame image set includes:

[0164] If the defect at the same position in a single frame image in the video frame image set C is marked by multiple detection frames, it is determined whether the categories of the detection frames are the same: if the categories of the detection frames are different, no processing is performed, and the category of the detection frame is the category of the defect; if the categories of the detection frames are the same, the single-frame target defect deduplication algorithm is executed to complete the deduplication of the road surface defect detection results. The algorithm is as follows:

[0165] Determine whether the two rectangles intersect in space. If so, merge the two rectangles. Assume that the coordinates of the upper left corner and lower right corner of rectangles A and B are (x01, y01), (x02, y02), (x11, y11), (x12, y12)

[0166] Assume that the distance Dx between the two center points of rectangular frame A and rectangular frame B in the X-axis direction is:

[0167] Dx=abs((x01+x02) / 2-(x11+x12) / 2)

[0168] Assume that the distance Dy between the two center points of rectangular frame A and rectangular frame B in the Y-axis direction is:

[0169] Dy=abs((y01+y02) / 2-(y11+y12) / 2)

[0170] Assume that the lengths of rectangular frames A and B in the X-axis direction are Dax and Dbx:

[0171] Dax=abs(x01-x02),Dbx=abs(x11-x12)

[0172] Assume that the lengths of rectangles A and B in the Y-axis direction are Day and Dby:

[0173] Day=abs(y01-y02),Dby=abs(y11-y12)

[0174] If rectangles A and B intersect,

[0175] Dx<=(Dax+Dbx) / 2&&Dy<=(Day+Dby) / 2

[0176] Suppose the coordinates of the upper left corner and lower right corner of the merged rectangle are (x1, y1) and (x2, y2):

[0177] x1=min(x01,x11),y1=min(y01,y11)

[0178] x2=max(x02,x12), y2=max(y02,y12).

[0179] Furthermore, based on the adjacent multi-frame multi-target disease tracking algorithm, counting processing is performed on the deduplicated video frame image set, including:

[0180] Assign a tracker to each target disease in the deduplicated video frame image set to predict the location of the target disease in the next frame;

[0181] The predicted position is corrected according to the detection result of the pavement defect position in the next frame to obtain the optimal position estimate of the target defect tracked in this frame;

[0182] Compare the disease at the optimal position in the current frame with the tracked disease in the previous frame: if the two are the same disease, then the target disease is counted in the previous and next frames to remove duplicates; if the two are not the same disease, then count the tracked disease to obtain the real-time tracking count and detection results of the same disease between consecutive frames;

[0183] Assign a global counter to each different category of pavement disease; initialize a tracking number list to save the numbers of the pavement diseases that have been counted; if it is confirmed that it is the same disease, check whether the tracking number of the pavement disease already exists in the tracking number list: if it exists, cancel the repeated counting of the target disease; if it does not exist, add one to the global counter of the corresponding category to update the statistical information of the pavement disease, such as Figure 4 shown.

[0184] Specifically, this embodiment uses the multi-target tracking SORT in the Kalman filter algorithm to create unique identifiers for different categories of targets entering the video frame image, that is, assigned trackers. Each tracker corresponds to a disease tracking number. The algorithm is as follows:

[0185] The state of the road surface disease relative to the vehicle in the current frame, that is, the position and velocity, satisfies the Gaussian distribution and is expressed as:

[0186]

[0187] Let the acceleration of the disease relative to the shooting car be u t , and the state relationship of this disease in the previous frame can be expressed as:

[0188]

[0189] v t =v t-1 +u t ×Δt

[0190] Converted to matrix form:

[0191]

[0192] The state prediction equation 1 is obtained:

[0193]

[0194] Introducing the uncertainty and noise of the system, the state prediction equation 2 is obtained:

[0195]

[0196] Introducing the observation uncertainty matrix R, the observation matrix is ​​H. The relationship between the observed value and the true value is:

[0197]

[0198] The observed value and the true value are used to correct the state prediction equation, and the following correction equation is obtained:

[0199]

[0200] K t =N tk H T (HN tk H T +R) -1

[0201] N t =(1-K t H)N tk

[0202] State prediction equations 1 and 2 are used to predict the state of pavement defects in a certain frame in the next frame. Combined with the correction equation, the optimal estimate of the target pavement defect can be obtained.

[0203] Compare the optimally located defect in the current frame with the tracked defect in the previous frame:

[0204] If the two are the same disease, the disease is counted in the two frames before and after, and the implementation is as follows:

[0205] Assign a global counter to each different type of pavement disease; initialize a tracking number list to save the numbers of the pavement diseases that have been counted; and determine whether the tracking number of the pavement disease exists in the tracking number list, provided that it is the same disease:

[0206] If it exists, the repeated counting of the tracked disease will be cancelled during the counting;

[0207] If it does not exist, the global counter of this category is increased by 1 to obtain the updated road surface disease statistics. If the two are not the same disease, the tracked disease is counted to obtain the real-time tracking count and detection results of the same disease between consecutive frames.

[0208] The above embodiments are only descriptions of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A road surface disease detection method based on improved YOLOv8, characterized in that: include: Collecting road surface video data to be detected; Input the road surface video data to be detected into a preset road surface disease detection model to obtain a set of video frame images after disease classification; wherein the road surface disease detection model is constructed based on an improved YOLOv8 model and is trained using a pre-constructed road surface disease image dataset; The defect category at the same position of a single frame image in the video frame image set is determined, and duplicate removal and counting processing are performed on the defects at the same position in multiple consecutive frames of images to obtain a road surface defect detection result.

2. The pavement disease detection method based on improved YOLOv8 according to claim 1, characterized in that: The pre-built pavement damage image dataset includes: Collecting road surface video frame image data, wherein the road surface video frame image data includes: disease image data of several disease types under different road types, different lighting conditions and different weather conditions; The target position is annotated on the road surface video frame image data, and the annotated image data is preprocessed to obtain the data set.

3. The pavement disease detection method based on improved YOLOv8 according to claim 2 is characterized in that: Preprocessing of the annotated image data includes: Use adversarial networks to enhance the labeled image data; The enhanced image data is fitted using an improved cubic curve active learning method, and the preset image data is selected by analyzing the curvature of the curve; The improved cubic curve active learning method is: AL(D augmented ,M0)→(D train ,D val ,D test ) Among them, AL represents the active learning process, M0 is the initial model, and D train , D val ,D test They represent the training set, validation set, and test set respectively.

4. The pavement disease detection method based on improved YOLOv8 according to claim 1, characterized in that: Improvements to the YOLOv8 model include: A CNX2f feature extraction module is introduced into the backbone network of the YOLOv8 model to replace the C2f feature extraction module, and a RepConv reparameterization module is introduced to replace the ordinary convolution module in the original network, and a DBB module is introduced in the neck network and the Head. The RepConv reparameterization module and the DBB module both use a multi-branch structure in the training stage, and are converted into a single-branch structure through reparameterization in the reasoning stage. At the same time, a shared parameter structure is adopted for the Head of the network, and a new reparameterization module RBB is introduced; and the SPPF module in the YOLOv8 network is improved to construct a dual-branch structure SPPFAvg module.

5. The pavement damage detection method based on improved YOLOv8 according to claim 4 is characterized in that: In the improved YOLOv8 model, the convolutional layer and the BN layer are automatically fused during the inference phase; The convolutional layer is: Among them, x is the input value, W is the convolution module weight, b is the bias, and avg is the input mean; The BN layer is: Among them, σ is the input variance, γ is the BN weight, and β is the weight; The fusion process of the convolutional layer and the BN layer is: Among them, F fuse are the fused weights and bias values.

6. The pavement disease detection method based on improved YOLOv8 according to claim 4, characterized in that: Improving the YOLOv8 model also includes: introducing a multi-scale feature fusion module to fuse multi-scale features from different sensors during the training phase; The multi-scale feature fusion module is: Among them, F multi-scale represents the result of multi-scale feature fusion, σ is the activation function, W scale,i and b i is the convolution kernel weight and bias of the i-th scale, and N is the number of multi-scale features.

7. The pavement disease detection method based on improved YOLOv8 according to claim 2, characterized in that: And use the pre-built pavement disease image dataset for training including: Input the pavement disease image dataset into the improved YOLOv8 model for recognition, obtain the detection frame and category information, input the detection frame, category information and the annotation information of the pavement video frame image data into a preset optimization loss function, and calculate the loss value; wherein the loss function includes: border regression loss CIoU Loss, DFL loss, and category classification cross entropy loss; A gradient descent algorithm is set to optimize the model parameters of the improved YOLOv8 model, and training rounds are set to obtain the trained improved YOLOv8 target detection model as the pavement disease detection model.

8. The pavement disease detection method based on improved YOLOv8 according to claim 1, characterized in that: Determining the category of the damage at the same position of a single frame image in the video frame image set, performing deduplication and counting processing on the damage at the same position in multiple consecutive frames of images, and obtaining the road surface damage detection result includes: If the same position of a single frame image in the video frame image set is marked by several detection frames, it is determined whether the categories of the detection frames are the same: if the categories of the detection frames are different, no processing is performed, and the category of the detection frame is the category of the corresponding disease; if the categories of the detection frames are the same, the pavement disease detection results in the video frame image set are deduplicated; Based on the adjacent multi-frame multi-target defect tracking algorithm, the video frame image set after deduplication is counted to obtain the road surface defect detection result.

9. The pavement disease detection method based on improved YOLOv8 according to claim 8, characterized in that: Deduplication of the road surface disease detection results in the video frame image set includes: Based on the detection frames of the same category, determine whether the two rectangles intersect in space. If so, merge the two rectangles.

10. The pavement damage detection method based on improved YOLOv8 according to claim 8, characterized in that: Based on the adjacent multi-frame multi-target disease tracking algorithm, counting processing is performed on the deduplicated video frame image set, including: Assigning a tracker to each target disease in the deduplicated video frame image set to predict the position of the target disease in the next frame; The predicted position is corrected according to the detection result of the pavement defect position in the next frame to obtain the optimal position estimate of the target defect tracked in this frame; Compare the disease at the optimal position in the current frame with the tracked disease in the previous frame: if the two are the same disease, then the target disease is counted in the previous and next frames to remove duplicates; if the two are not the same disease, then count the tracked disease to obtain the real-time tracking count and detection results of the same disease between consecutive frames; Assign a global counter to each different category of pavement disease; initialize a tracking number list to save the numbers of pavement diseases that have been counted; if it is confirmed to be the same disease, check whether the tracking number of the pavement disease already exists in the tracking number list: if it does exist, cancel the repeated counting of the target disease; if it does not exist, add one to the global counter of the corresponding category and update the statistical information of the pavement disease.

Images