A Deep Learning-Based Automated Detection Method and System for Road Surface Defects

Abstract

This invention belongs to the field of image recognition and relates to an automated method and system for detecting pavement defects based on deep learning. The method includes: anomaly detection of image data, filtering out abnormal images to obtain normal image data; image registration to obtain image point cloud data; pavement identification of the normal image data to obtain pavement type; defect detection of the normal image data using a defect detection algorithm to obtain pavement defect targets; segmentation of standard defect targets to obtain standard instance defect segmentation results; target tracking and tracking ID allocation of the detected pavement defect targets to obtain pavement defect tracking results; and defect index calculation for each pavement defect target to obtain defect indices. This provides a pavement disaster identification solution that meets engineering requirements, enabling rapid, accurate, and scale-consistent detection of various types of pavement defects.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and specifically discloses an automated method and system for detecting road surface defects based on deep learning. Background Technology

[0002] In the fields of pavement inspection and highway maintenance, pavement defect detection is a core component. Traditional pavement defect inspection typically relies on manual patrols, a method that is inefficient, labor-intensive, subjective, and lacks real-time accuracy. In recent years, with the rise of artificial intelligence, image recognition technology has been gradually applied to road inspection. However, in the process of developing towards fully automated detection, some pavement defects are difficult to classify and standardize due to varying causes, acquired effects, and a lack of relevant standards. How to ensure detection accuracy, unify classification scales, and simultaneously consider practical application efficiency remains a key research focus. Existing technologies require manual verification of the detection results after real-time data detection, followed by deduplication, size calculation, and evaluation of duplicated targets. However, the shortcomings of existing technologies are mainly reflected in the following aspects: First, traditional methods rely on manual patrols, requiring a large number of patrol personnel for long-term deployment, especially in large urban areas with dense roads. When encountering road surface defects, it is necessary to get out of the vehicle to observe and record, resulting in low efficiency. Most road surface inspections use a vertical downward scanning and photographing method, but this method can only inspect one lane at a time. If a basic inspection unit (road segment) contains multiple lanes, it is necessary to perform back-and-forth inspections, which is inefficient. Secondly, due to the characteristics of target detection, some methods may detect the same target multiple times, leading to duplicate road segment statistics. Then, some methods may miss detections due to numerous algorithm modules or the discarding of some raw data. In addition, some methods rely solely on the statistical data of a single inspection, which inevitably leads to subjective misjudgments by different road maintenance personnel. Finally, some methods, due to numerous algorithm modules or the inability to achieve real-time detection, have delayed information feedback, making it difficult to achieve rapid synchronization and decision-making.

[0003] In view of this, and to address the shortcomings of existing technologies, this invention provides an automated method and system for detecting pavement defects based on deep learning. This method aims to achieve rapid, accurate, and complete detection of pavement defects, improve the efficiency and quality of road inspection, and provide strong technical support for the construction of digital highways and highway patrol and maintenance. Summary of the Invention

[0004] The purpose of this invention is to provide an automated method and system for detecting pavement defects based on deep learning, addressing the problem of how to achieve rapid, accurate, and comprehensive detection of pavement defects; the specific solution is as follows:

[0005] An automated pavement distress detection method based on deep learning includes: Step 1, acquiring image data, LiDAR point cloud data, and integrated navigation data; Step 2, performing anomaly detection on the image data, filtering out abnormal images, and obtaining normal image data; abnormal images refer to image anomalies caused by camera malfunctions; Step 3, performing image registration on the normal image data and LiDAR point cloud data to obtain image point cloud data; Step 4, performing pavement identification on the normal image data using a target classification algorithm to obtain pavement type; Step 5, based on the pavement type, performing distress detection on the normal image data using a distress detection algorithm to obtain pavement distress targets; pavement distress targets include standard distress targets and non-standard distress targets; Step 6, performing segmentation on standard distress targets using a distress segmentation algorithm to obtain standard instance distress segmentation results; Step 7, performing target tracking and tracking ID allocation on the detected pavement distress targets using a multi-target tracking algorithm to obtain pavement distress tracking results; Step 8, calculating distress indicators for each pavement distress target based on the pavement distress tracking results and image point cloud data to obtain distress indicators.

[0006] Further, step 5 includes: Step 5.1, preprocessing the input normal image data through the input layer to obtain preprocessed image data; the preprocessed image data refers to the road surface image adapted to the network structure requirements of YOLOv11; Step 5.2, performing multi-level feature processing on the preprocessed image data through the backbone network to obtain multi-level feature maps; the multi-level feature maps include at least high-level feature maps, mid-level feature maps, and low-level feature maps; Step 5.3, processing the multi-level feature maps through the detection head to obtain multi-scale feature maps; the multi-scale feature maps are used to adapt to road surface defects of different sizes; Step 5.4, processing the multi-scale feature maps through the output layer to obtain the defect bounding box, defect confidence, and defect type of the road surface defect target.

[0007] Further, step 6 includes: inputting the multi-scale feature map into the instance segmentation module and outputting a segmentation mask of the standard disease target; the instance segmentation module includes a convolution module, which processes the input multi-scale feature map to generate a prototype mask and mask weight coefficients corresponding to each disease target box; and linearly combining the mask weight coefficients and the prototype mask to generate the final segmentation mask.

[0008] Further, step 7 includes: Step 7.1, dividing the target bounding boxes of road defects into high-scoring boxes and low-scoring boxes according to the defect confidence level; high-scoring boxes refer to target bounding boxes of road defects with a defect confidence level greater than or equal to the confidence threshold; low-scoring boxes refer to target bounding boxes of road defects with a defect confidence level less than the confidence threshold; Step 7.2, performing one-to-one matching between high-scoring boxes and historical defect activity trajectories using IoU distance calculation and the Hungarian algorithm to obtain the identification results of historical road defect targets in the current frame, and constructing the first current defect activity trajectory based on the identification results of the current frame; historical defect activity trajectory refers to the tracking results of historical road defect targets before the current frame; historical road defect targets refer to road defect targets identified before the current frame; the current defect activity trajectory is used to represent the tracking results of historical road defect targets in the current frame and before; Step 7.3, determining whether all historical defect activity trajectories match the high-scoring boxes of the current frame, if so, then... The high-resolution bounding box that matches the historical pavement activity trajectory is used as the new pavement activity trajectory, and a tracking ID is assigned for matching pavement activity trajectories in subsequent frames. The first current pavement activity trajectory is used as the pavement pavement tracking result of the current frame. Step 7.4: For historical pavement activity trajectories that do not match the high-resolution bounding box of the current frame, the low-resolution bounding box is matched one-to-one with the historical pavement activity trajectory through IoU distance calculation and Hungarian algorithm to obtain the identification result of the historical pavement pavement target in the current frame, and the second current pavement activity trajectory is constructed based on the identification result of the current frame. Step 7.5: Determine whether all historical pavement activity trajectories match the pavement target bounding box of the current frame. If so, the matching ends. Step 7.6: The historical pavement activity trajectories that do not match the pavement target bounding box of the current frame are temporarily stored. If the temporarily stored historical pavement activity trajectory still does not match the pavement target bounding box within a preset number of consecutive frames, the historical pavement activity trajectory is cleared.

[0009] Further, step 8 includes: step 8.1, determining the target of the defect to be judged based on the standard instance defect segmentation results and road defect tracking results; step 8.2, when the target of the defect to be judged is within the vehicle statistics range, calculating the defect target index and defect coordinates; the defect target index includes the area, depth, length and width of the defect.

[0010] Furthermore, calculating the area of ​​the target disease to be identified includes: extracting the image point cloud data based on the segmentation mask of the target disease to obtain the planar coordinates of the target disease; constructing the convex hull or ordered polygon vertex set of the target disease based on the planar coordinates; and calculating the planar coordinates of the convex hull or ordered polygon vertex set to obtain the two-dimensional planar area of ​​the target disease.

[0011] ;

[0012] Where A is the two-dimensional planar area of ​​the target disease to be judged; n is the number of polygon vertices; and i is the vertex index (subscript variable). and Let be the two-dimensional coordinates of the i-th vertex; and Let be the two-dimensional coordinates of the (i+1)th vertex.

[0013] Furthermore, the depth of the target disease to be judged is calculated, including: filtering the image point cloud data based on the segmentation mask of the target disease to be judged to obtain the point cloud data of the target disease; extracting the coordinate values ​​of the point cloud data of the target disease in the vertical axis direction; and taking the difference between the maximum and minimum values ​​of the coordinate values ​​in the vertical axis direction as the depth of the target disease to be judged.

[0014] Furthermore, the length and width of the target disease to be judged are calculated as follows: Based on the segmentation mask of the target disease, the image point cloud data is filtered to obtain the target disease point cloud data; the target disease point cloud data is centered, and the covariance matrix of the centered target disease point cloud data is calculated; the centering process refers to subtracting the mean coordinate of the target disease point cloud data from the three-dimensional coordinates of each point, so that the center of the processed data distribution is located at the origin of the coordinate system; the covariance matrix is ​​decomposed into eigenvalues ​​to obtain multiple principal component directions and their variance intensities; the eigenvalue of the principal component direction with the highest variance intensity is obtained, and the length of the target disease to be judged is determined based on the multiple of this eigenvalue; the eigenvalue of the principal component direction with the second highest variance intensity is obtained, and the width of the target disease to be judged is determined based on the multiple of this eigenvalue.

[0015] Furthermore, the latitude and longitude of the target to be identified as having defects are calculated as follows: The current vehicle position information is obtained from the integrated navigation data; the current vehicle position information includes latitude and longitude, elevation, and vehicle attitude; the vehicle attitude includes heading angle; the current vehicle position information is used as the origin of the ENU coordinate system, and a local ENU coordinate system is established; the relative three-dimensional position of the target to be identified as having defects in the lidar coordinate system is extracted; based on the vehicle attitude, the relative three-dimensional position is transformed to the local ENU coordinate system to obtain the defect ENU coordinates of the target to be identified; the defect ENU coordinates are offset from the origin of the ENU coordinate system to obtain the absolute ENU coordinates of the target to be identified relative to the origin of the ENU coordinate system; the absolute ENU coordinates are converted to coordinates in the WGS84 coordinate system using a geographic projection transformation algorithm to obtain the latitude and longitude of the target to be identified as having defects.

[0016] This invention also provides a deep learning-based automated pavement distress detection system according to the aforementioned deep learning-based automated pavement distress detection method, comprising a data acquisition module, an image filtering module, a registration module, a recognition module, a detection module, a segmentation module, a tracking module, and an index calculation module: the data acquisition module is used to acquire image data, LiDAR point cloud data, and integrated navigation data; the image filtering module is used to perform anomaly detection on the image data, filter out abnormal images, and obtain normal image data; abnormal images refer to image anomalies caused by camera malfunctions; the registration module is used to perform image registration on the normal image data and LiDAR point cloud data to obtain image point cloud data; the recognition module is used to identify the target... The classification algorithm identifies road surface types from normal image data; the detection module detects road surface defects based on these types using a defect detection algorithm, identifying road defect targets; these targets include standard and non-standard defects; the segmentation module segments standard defects using a defect segmentation algorithm, obtaining standard instance defect segmentation results; the tracking module tracks detected road defect targets and assigns tracking IDs using a multi-target tracking algorithm, obtaining road defect tracking results; and the index calculation module calculates defect indices for each road defect target based on the road defect tracking results and image point cloud data, obtaining defect indices.

[0017] The present invention has the following advantages and beneficial effects:

[0018] This invention, through the application of algorithms, can achieve rapid, accurate, and comprehensive detection of various types of pavement defects, avoiding the problem of missed detection caused by too many sub-tasks and subjective inconsistencies among different personnel. It achieves the engineering goal of real-time detection and real-time display of results, and has the advantages of full automation and intelligence, providing strong technical support for the construction of highway digitalization and highway patrol and maintenance. Attached Figure Description

[0019] Figure 1 This is a comparison diagram of automated and non-automated development in this invention;

[0020] Figure 2 An exemplary flowchart of an automated pavement distress detection method based on deep learning provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] This invention employs a pipeline framework to achieve a fully automated process; it uses the ROS 2 (Robot Operating System 2) framework for data access; it uses the YOLOv11 algorithm for target detection and instance segmentation; it uses the ByteTrack algorithm for multi-target tracking; and it uses a MySQL database for data storage. Figure 2 As shown, the complete process of the automation method is as follows:

[0023] Step 1: Acquire image data, LiDAR point cloud data, and integrated navigation data.

[0024] For example, an onboard camera continuously captures images of the road ahead at 30 frames per second with a resolution of 1920×1080, using the resulting RGB color images as image data. A lidar system emits high-frequency short pulses from a laser; the beam reflects off the target object, and the receiver captures the reflected signal. By precisely measuring the time difference between the laser's emission and return, and combining this with the speed of light, the target distance is calculated. Each laser beam corresponds to a ranging point, and the resulting massive point cloud of the current road segment is used as lidar point cloud data. Accurate, continuous, and real-time attitude, velocity, and position information is obtained through the automotive-grade IMU platform and the full-system, full-frequency dual-antenna RTK system within the multi-sensor integrated navigation device, and this acquired attitude, velocity, and position information is used as integrated navigation data.

[0025] After the edge computing device is powered on, it waits for time synchronization and automatically starts the camera, LiDAR and integrated navigation equipment. After starting, it travels at a constant speed in the basic detection road segment unit and obtains image data, LiDAR point cloud data and integrated navigation data by subscribing to image topics, LiDAR point cloud topics and integrated navigation data topics from the topics published by ROS 2.

[0026] The system compares the timestamps of LiDAR point cloud data, image data, and integrated navigation data, selecting a set of data that meets the synchronization requirements and inputting it into a callback function to obtain image data, LiDAR point cloud data, and integrated navigation data from the same time for subsequent processing. For example, in time synchronization, it first waits for system time correction. Considering that LiDAR and camera do not support external triggering, this design adopts a software synchronization scheme, aligning data based on the millisecond-level timestamp in the message header of the ROS2 topic message. During initial startup, the camera driver will continuously check the millisecond timestamp of the currently captured image until it reaches a whole 100 milliseconds before officially starting data publishing. The LiDAR, by setting angle triggering, enabling and setting the synchronization angle function, and setting the scan frame rate to 20Hz, can collect data every 50 milliseconds, with the initial timestamp having a millisecond portion of 000 (e.g., 000ms, 050ms, 100ms…950ms). The integrated navigation related data update frequency is 100Hz, and data with timestamps having a millisecond portion of a whole 10 millisecond can be published through the ROS2 driver. Using the method described above, three-party data sources can achieve millisecond-level time synchronization.

[0027] Step 2: Perform anomaly detection on the image data, filter out abnormal images, and obtain normal image data; abnormal images refer to image anomalies caused by camera malfunctions.

[0028] After obtaining data from various sources by subscribing to different topics using ROS2, it is necessary to perform anomaly detection and processing on the data. Anomaly detection and processing includes anomaly image detection. First, the image data obtained in real time in the callback function is subjected to anomaly detection, and abnormal images caused by accidental camera malfunctions are filtered out. For example, image data with automatic white balance algorithm failure (such as a scene dominated by a single color) and image tearing or disorder are filtered out as abnormal image data.

[0029] After acquiring the normal image data, LiDAR point cloud data, and integrated navigation data, the data is saved. For example, the raw data can be saved as files or database tables. Alternatively, the LiDAR point cloud data can be compressed before saving.

[0030] Step 3: Perform image registration on the normal image data and the lidar point cloud data to obtain image point cloud data.

[0031] For example, by using the extrinsic parameters obtained from the joint calibration of the lidar and the camera, the lidar point cloud data is rotated and translated in the coordinate system to obtain the registered image data and lidar point cloud data. Then, the lidar point cloud data in the area of ​​the normal image data is selected to obtain the image point cloud data.

[0032] Step 4: Use a target classification algorithm to identify the road surface in the normal image data to obtain the road surface type.

[0033] For example, the YOLOv11 algorithm can be used to identify the road surface type in the current image, which facilitates the classification of subsequent damage types. Road surface types include asphalt pavement and cement concrete pavement, among others.

[0034] Step 5: Based on the road surface type, perform disease detection on normal image data using a disease detection algorithm to obtain road surface disease targets; road surface disease targets include standard disease targets and non-standard disease targets.

[0035] For example, a pavement defect detection algorithm can be implemented using the YOLOv11 target detection algorithm. Normal image data is input into the YOLOv11 algorithm, which then performs target detection on all defects, identifying all pavement defect targets in the current frame and outputting defect location, classification, and confidence information. Standard defect targets refer to pavement defect types classified according to industry standards. Non-standard defect targets refer to pavement defect types that cannot be classified according to industry standards. For example, some defects cannot be accurately classified into a specific category in the industry standards due to ambiguity or comprehensive characteristics. These defects will be uniformly classified as "defects," and also fall under the category where different road maintenance personnel have differing classifications.

[0036] The YOLOv11 network structure consists of five parts: input, backbone, neck, head, and output.

[0037] Step 5.1: Preprocess the input normal image data through the input layer to obtain preprocessed image data; the preprocessed image data refers to the road surface image that is adapted to the network structure requirements of YOLOv11.

[0038] Preprocessing includes image scaling, image normalization, and data augmentation. For example, image scaling uniformly adjusts normal image data to a fixed size of 640×640. Normalization normalizes the pixel values ​​of the image from [0,255] to the [0,1] range to accelerate model convergence. Data augmentation can include random cropping, flipping, color dithering, and mosaic enhancement to improve the model's generalization ability.

[0039] Step 5.2: Perform multi-level feature processing on the preprocessed image data through a backbone network to obtain multi-layer feature maps. The multi-layer feature maps include at least high-level feature maps, mid-level feature maps, and low-level feature maps.

[0040] The backbone network extracts feature maps from multi-level features of the input image. Its lightweight structure includes a downsampling module, a C3k2 module, a C2PSA module, and an SPPF module. The downsampling module downsamples the preprocessed image data or the first feature map to obtain a downsampled feature map. The C3k2 module performs feature extraction and channel interaction processing on the downsampled or convolutional feature maps to obtain the first feature map. The C2PSA module enhances the first feature map with an attention mechanism to obtain the second feature map. The SPPF module expands the receptive field of the first feature map to obtain the third feature map. Multiple downsampling operations (convolutions with stride=2) progressively transform image features from low-level edge information to high-level semantic information. The backbone network incorporates the efficient C3k2 module for feature extraction and channel interaction, and the SPPF module expands the receptive field to better capture multi-scale contextual information. Furthermore, the C2PSA attention mechanism enhances the model's ability to focus on key regions, enabling the network to achieve stronger feature representation capabilities without significantly increasing computational cost. The size of high-level feature maps is smaller than that of mid-level feature maps, and the size of mid-level feature maps is smaller than that of low-level feature maps. For example, the size of a high-level feature map can be 20x20, the size of a mid-level feature map can be 40x40, and the size of a low-level feature map can be 80x80.

[0041] Step 5.3 involves processing the multi-layer feature map using the detection head to obtain a multi-scale feature map. This multi-scale feature map is used to adapt to pavement defects of different sizes.

[0042] The Head (detection head) employs a feature pyramid structure for multi-scale feature fusion, combining classic FPN (top-down) and PAN (bottom-up) approaches. It includes an upsampling module and a concatenation module, fusing high-level semantic features with low-level detail features to generate feature maps for small, medium, and large target detection. For example, the top-down multi-scale feature fusion process involves: multiple upsampling operations on the high-level feature map to obtain upsampled mid-level and low-level feature maps; fusing the mid-level feature map output from the backbone network with the upsampled mid-level feature map to obtain a concatenated mid-level feature map; and fusing the low-level feature map output from the backbone network with the upsampled low-level feature map to obtain a concatenated low-level feature map. The backbone network then performs bottom-up multi-scale feature fusion, sequentially obtaining low-level, mid-level, and high-level feature maps. After each fusion layer, the C3k2 module further processes the data to improve feature quality. For example, by using the C3k2 module to enhance the fused mid-level feature map and the fused low-level feature map separately using an attention mechanism, a 40x40 mid-scale feature map and an 80x80 large-scale feature map are obtained; and the high-level feature map is used as a small-scale feature map. This design can effectively handle the problem of large differences in target size in the image, thereby improving the overall detection accuracy.

[0043] Step 5.4: The multi-scale feature map is processed through the output layer to obtain the target bounding box, confidence level, and type of pavement defects.

[0044] The output layer (Detect) is the module that outputs the final prediction result. It receives feature maps from three scales: P3 (small-scale feature map), P4 (medium-scale feature map), and P5 (large-scale feature map), and outputs the bounding box coordinates (x, y, w, h), objectness, and class probability for each anchor point. YOLOv11's Detect layer supports end-to-end training and incorporates efficient non-maximum suppression (e.g., NMS) for post-processing, ultimately providing the object detection result in the image. This module has a compact design and fast inference speed, making it very suitable for real-time applications.

[0045] Step 6: Segment the standard disease target using the disease segmentation algorithm to obtain the standard instance disease segmentation result.

[0046] For example, the disease segmentation algorithm can be implemented using YOLOv11. The instance segmentation algorithm segments instance disease targets classified according to relevant industry standards. The disease segmentation algorithm can include an input layer, Backbone, Neck, Head, and an instance segmentation module. For details on the input layer, Backbone, Neck, and Head, please refer to step 5 and its related description. The input to the instance segmentation module is a multi-scale feature map, and the output is a segmentation mask for the standard disease target. The multi-scale feature map is input to the convolution module to generate a prototype mask and mask weight coefficients corresponding to the detection boxes of each standard disease target. Then, the mask weight coefficients are linearly combined with the prototype mask to generate the final target segmentation mask. The instance mask for each standard disease target has the same size as the entire image, with pixel values ​​of 1 for instance regions and 0 for other regions.

[0047] Step 7: Using a multi-target tracking algorithm, target tracking and tracking ID assignment are performed on the detected pavement defects to obtain the pavement defect tracking results.

[0048] The ByteTrack algorithm is used to track all targets and assign tracking IDs, ensuring that each disease is evaluated only once, including:

[0049] Step 7.1: Divide the disease target boxes into high-scoring boxes and low-scoring boxes based on disease confidence level. High-scoring boxes are disease target boxes with a disease confidence level greater than or equal to the confidence threshold; low-scoring boxes are disease target boxes with a disease confidence level less than the confidence threshold. For example, input the disease target boxes, disease confidence levels, and disease classification results from Step 5 into the multi-target tracking algorithm ByteTrack and assign a unique target ID. ByteTrack divides all detection boxes output by the detection algorithm into high-scoring boxes (≥ threshold 0.5) and low-scoring boxes (< threshold 0.5) based on their confidence level.

[0050] Step 7.2: High-resolution bounding boxes are matched one-to-one with historical pavement defect activity tracks using IoU distance calculation and the Hungarian algorithm to obtain the identification results of historical pavement defect targets in the current frame. Based on the identification results of the current frame, a first current defect activity track is constructed. Historical defect activity tracks refer to the tracking results of historical pavement defect targets before the current frame. Historical pavement defect targets refer to pavement defect targets identified before the current frame. The current defect activity track represents the tracking results of historical pavement defect targets in the current frame and before. For example, the first round of matching is only performed between high-resolution bounding boxes and existing active tracks, using IoU distance calculation and the Hungarian algorithm for one-to-one matching, where the IoU threshold is 0.85.

[0051] Step 7.3: Determine whether all historical defect activity trajectories match the high-resolution bounding box of the current frame. If so, take the high-resolution bounding box that does not match the historical defect activity trajectory as the new defect activity trajectory and assign a tracking ID to it for defect activity trajectory matching in subsequent frames. Take the first current defect activity trajectory as the road defect tracking result of the current frame.

[0052] Step 7.4: For historical pavement activity trajectories with high-scoring bounding boxes that do not match the current frame, low-scoring bounding boxes are matched one-to-one with the historical pavement activity trajectories using IoU distance calculation and the Hungarian algorithm to obtain the identification results of the historical pavement defects in the current frame. A second current pavement activity trajectory is then constructed based on the identification results of the current frame. For example, if some trajectories fail to match successfully in the first round, ByteTrack will perform a second round of matching: these unmatched trajectories will be matched again with low-scoring bounding boxes using IoU to salvage targets that may have lower scores due to occlusion, blurring, or other reasons but still exist.

[0053] Step 7.5: Determine whether all historical disease activity trajectories match the disease target box in the current frame. If so, end the matching process.

[0054] Step 7.6: Temporarily store historical disease activity trajectories that do not match the target bounding box in the current frame. If a stored historical disease activity trajectory still does not match the target bounding box within a preset number of consecutive frames, then the historical disease activity trajectory is cleared. For example, successfully matched trajectories are retained, while unmatched trajectories enter a "temporary storage" state. If they still cannot match within 30 consecutive frames, they are cleared. Ultimately, only trajectories matching high-resolution bounding boxes are output for display or evaluation; low-resolution bounding boxes only participate in trajectory maintenance and do not participate in result output.

[0055] Step 8: Based on the standard instance disease segmentation results and image point cloud data, calculate the disease index for each pavement disease target to obtain the disease index.

[0056] According to relevant industry standards, the relevant indicators of the disease are calculated using instance segmentation results and point cloud data. These indicators include length, width, area, and depth.

[0057] Step 8.1: Based on the standard instance defect segmentation results and pavement defect tracking results, determine the defect targets to be judged. For example, the standard defect target corresponding to each tracking ID in the pavement defect tracking results can be used as the defect target to be judged for the defect indicator.

[0058] Step 8.2: When the target of the defect to be judged is within the vehicle statistics range, calculate the defect target index and defect coordinates; the defect target index includes the area, depth, length and width of the defect. Determine whether the defect target is within the vehicle statistics range. If the following conditions are met, calculate the target index and coordinates: (1) The target is in the center of the screen and within 9 meters of the front of the vehicle. (2) The target is in the adjacent lane of the current vehicle and within 13 meters of the front of the vehicle.

[0059] Calculating the area of ​​the target disease to be identified includes: extracting the point cloud data from the image based on the segmentation mask of the target disease to obtain its planar coordinates; ignoring the remaining coordinate axes, constructing the convex hull or ordered polygon vertex set of the target disease based on the planar coordinates; and calculating the planar coordinates of the convex hull or ordered polygon vertex set using the Shoelace formula to obtain the two-dimensional planar area of ​​the target disease.

[0060] ;

[0061] Where A is the two-dimensional planar area of ​​the target disease to be judged; n is the number of polygon vertices; and i is the vertex index (subscript variable). and Let be the two-dimensional coordinates of the i-th vertex; and Let be the two-dimensional coordinates of the (i+1)th vertex.

[0062] Calculating the depth of the target disease to be identified includes: filtering the image point cloud data based on the segmentation mask of the target disease to be identified to obtain the point cloud data of the target disease; extracting the coordinate values ​​of the point cloud data of the target disease in the vertical axis direction; and taking the difference between the maximum and minimum values ​​of the coordinate values ​​in the vertical axis direction as the depth or height difference of the target disease to be identified.

[0063] The target length and width are estimated using a principal component analysis (PCA)-based method. The approximate length and width of the target object on a two-dimensional plane are estimated through eigenvalue decomposition of the covariance matrix of the point cloud data. Specifically, the length and width of the target to be identified are calculated as follows: Based on the segmentation mask of the target, the image point cloud data is filtered to obtain the target point cloud data; the target point cloud data is centered, and the covariance matrix of the centered target point cloud data is calculated; centering refers to subtracting the mean coordinates of the target point cloud data from the three-dimensional coordinates of each point, so that the center of the processed data distribution is located at the origin of the coordinate system; eigenvalue decomposition is performed on the covariance matrix to obtain multiple principal component directions and their variance strengths; the eigenvalue of the principal component direction with the highest variance strength is obtained, and the length of the target to be identified is determined based on a multiple of this eigenvalue; the eigenvalue of the principal component direction with the second highest variance strength is obtained, and the width of the target to be identified is determined based on a multiple of this eigenvalue. For example, the square root of the eigenvalues ​​of the first two principal directions is multiplied by 2 to obtain the length and width estimates.

[0064] The calculation of the latitude and longitude of the target to be identified as having defects is as follows: Obtain the vehicle's current position information from the integrated navigation data; the vehicle's current position information includes latitude and longitude, elevation, and vehicle attitude; the vehicle attitude includes heading angle; use the vehicle's current position information as the origin of the ENU (East-North-Up) coordinate system and establish a local ENU coordinate system; extract the relative three-dimensional position (X, Y, Z) of the target to be identified in the lidar coordinate system; based on the vehicle attitude (e.g., heading angle), transform the relative three-dimensional position to the local ENU coordinate system to obtain the defect ENU coordinates of the target to be identified; calculate the position offset between the defect ENU coordinates and the origin of the ENU coordinate system to obtain the absolute ENU coordinates of the target to be identified relative to the origin of the ENU coordinate system; convert the absolute ENU coordinates to coordinates in the WGS84 coordinate system using a geographic projection transformation algorithm to obtain the latitude and longitude (longitude, latitude, and altitude) of the target to be identified as having defects.

[0065] Step 9: Store the image data, lidar point cloud data, integrated navigation data, and disease indicators.

[0066] For example, the data for disease identification calculations is stored in a database, using MySQL tables to store disease target indicators, latitude and longitude information, etc. Images of detected targets and their related data are saved. For "disease" types whose indicators have not been calculated, image data, tracking ID, classification, and latitude and longitude data are saved; for other types, relevant indicator data and radar point cloud data are additionally saved. Images and point clouds are saved as files, while other data, along with image and point cloud paths, are stored in the MySQL database.

[0067] Step 10: The backend Java service will continuously transmit the saved data to the server, enabling the client to call the relevant data for real-time detection results.

[0068] This invention employs a pipelined framework to perform task partitioning, parallel scheduling, and resource optimization for all modules, maximizing hardware utilization while reducing blocking and coupling between modules to meet the requirements of real-time detection in engineering. Examples of comparisons between pipelined and non-pipelined frameworks are provided. Figure 1 As shown in the example above, there are five modules: object detection, instance segmentation, multi-object tracking, indicator and coordinate calculation, and data storage. Each module takes 10 milliseconds. In a non-pipelined, serial development approach, all steps are written in a large function or process, and the modules lack independent structures. Data 2 can only be input into the object detection module to begin its series of tasks after data 1 has completed all module tasks, resulting in an output time interval of 50 milliseconds between adjacent input data. However, in a pipelined development framework, each module is independent, and it can begin working on the next set of data as soon as it completes the task for the previous set. This reduces the output time interval between adjacent input data to only 10 milliseconds, significantly improving data processing speed and ensuring real-time inference requirements.

[0069] This invention also provides an automated pavement distress detection system based on deep learning, comprising a data acquisition module, an image filtering module, a registration module, a recognition module, a detection module, a segmentation module, a tracking module, and an index calculation module: the data acquisition module is used to acquire image data, LiDAR point cloud data, and integrated navigation data; the image filtering module is used to perform anomaly detection on the image data, filter out abnormal images, and obtain normal image data; abnormal images refer to image anomalies caused by camera malfunctions; the registration module is used to perform image registration on the normal image data and LiDAR point cloud data to obtain image point cloud data; the recognition module is used to perform pavement distress detection on the normal image data using a target classification algorithm. The system consists of several modules: a surface recognition module to identify the road surface type; a detection module to detect road surface defects in normal image data based on the road surface type using a defect detection algorithm; road surface defect targets include standard defect targets and non-standard defect targets; a segmentation module to segment standard defect targets using a defect segmentation algorithm to obtain standard instance defect segmentation results; a tracking module to track detected road surface defect targets and assign tracking IDs using a multi-target tracking algorithm to obtain road surface defect tracking results; and an index calculation module to calculate defect indices for each road surface defect target based on the standard instance defect segmentation results and image point cloud data.

[0070] This invention solves the problems of low efficiency in road surface defect inspection and inconsistent manual classification standards in existing technologies. By applying this method, various types of road surface defects can be detected quickly and accurately, improving the efficiency and quality of road inspections and providing strong technical support for the development of digital highway systems and highway maintenance.

[0071] Example 1

[0072] Example of Crack-Type Defect Detection and Location. This example illustrates the method of the present invention using the automatic detection, tracking, and location of crack-type road defects as an example. In this example, a Jetson AGX Orin is selected as the edge computing device to perform real-time inference calculations for road defect detection, segmentation, and tracking; a ZED2i binocular camera is selected as the image acquisition device, with an image resolution of 1920×1080 and an acquisition frequency of 30 frames / second; a Hesai Technology AT128 hybrid solid-state LiDAR is selected as the point cloud acquisition device; and a Yuanji Technology FSS-NAV680D multi-sensor integrated navigation device is selected to acquire high-precision vehicle pose information. The processing flow for crack-type defect detection includes the following steps:

[0073] S1, Data Acquisition Steps: Road image data is acquired using the ZED2i binocular camera, and road 3D point cloud data at the corresponding time is acquired using the AT128 LiDAR. The FSS-NAV680D integrated navigation device outputs the vehicle's latitude and longitude, attitude angle, and unified timestamp information to achieve time alignment of multi-sensor data.

[0074] S2, Crack Target Detection Step: Detect crack-type defects in the acquired road images. By setting the target confidence threshold conf_threshold = 0.25 and the intersection-union ratio threshold iou_threshold = 0.45, the detection results are filtered and suppressed, and the target bounding boxes of crack defects and their corresponding category information are output.

[0075] S3, Crack Target Segmentation Step: Based on the target detection results, pixel-level segmentation processing is performed on the crack disease target to extract the target segmentation boundary of the crack disease for subsequent geometric feature calculation.

[0076] S4, Crack Target Tracking Steps: Based on a multi-target tracking algorithm, crack targets in consecutive frames are associated. By setting the target tracking confidence threshold track_thresh to 0.5, the target matching similarity threshold match_thresh to 0.8, and the target trajectory buffer length track_buffer to 30 frames, and combining the target aspect ratio threshold aspect_ratio_thresh to 1.6 and the minimum target box area threshold min_box_area to 10 pixels, abnormal targets are filtered out. A unique target tracking ID is assigned to each consecutive crack, achieving stable tracking across frames.

[0077] S5, Crack Location and Index Calculation Steps: Combining the segmentation boundary of the crack target in the image, the spatial information of the laser point cloud, and the vehicle pose data output by the integrated navigation equipment, calculate the latitude and longitude coordinates of the crack, and further calculate the length, width, area, and other indicators of the crack.

[0078] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automated detection of pavement defects based on deep learning, characterized in that, include: Step 1: Acquire image data, LiDAR point cloud data, and integrated navigation data; Integrated navigation data includes vehicle attitude information, speed information, and position information; Step 2: Perform anomaly detection on the image data, filter out abnormal images, and obtain normal image data; abnormal images refer to image anomalies caused by camera malfunctions. Step 3: Perform image registration on the normal image data and the lidar point cloud data to obtain image point cloud data; Step 4: Use a target classification algorithm to identify the road surface in the normal image data to obtain the road surface type; the target classification algorithm is the YOLOv11 algorithm. Step 5: Based on the road surface type, perform disease detection on normal image data using a disease detection algorithm to obtain road surface disease targets; road surface disease targets include standard disease targets and non-standard disease targets; Step 6: Segment the standard disease targets using the disease segmentation algorithm to obtain the standard instance disease segmentation results; Step 7: Using a multi-target tracking algorithm, target tracking and tracking ID assignment are performed on the detected pavement defects to obtain the pavement defect tracking results; Step 8: Based on the pavement distress tracking results and image point cloud data, calculate the distress index for each pavement distress target to obtain the distress index.

2. The automated pavement distress detection method based on deep learning according to claim 1, characterized in that, Step 5 includes: Step 5.1: The input normal image data is preprocessed through the input layer to obtain preprocessed image data; the preprocessed image data refers to the road surface image that is adapted to the network structure requirements of YOLOv11. Step 5.2: Perform multi-level feature processing on the preprocessed image data through the backbone network to obtain multi-level feature maps; the multi-level feature maps include at least high-level feature maps, mid-level feature maps, and low-level feature maps; Step 5.3: The multi-layer feature map is processed by the detection head to obtain a multi-scale feature map; the multi-scale feature map is used to adapt to pavement distresses of different sizes. Step 5.4: The multi-scale feature map is processed through the output layer to obtain the target bounding box, confidence level and type of pavement distress.

3. The automated pavement distress detection method based on deep learning according to claim 2, characterized in that, Step 6 includes: The multi-scale feature map is input into the instance segmentation module, which outputs a segmentation mask for the standard disease target. The instance segmentation module includes a convolution module, which processes the input multi-scale feature map to generate a prototype mask and mask weight coefficients corresponding to each disease target box. The final segmentation mask is generated by linearly combining the mask weight coefficients and the prototype mask.

4. The automated pavement distress detection method based on deep learning according to claim 1, characterized in that, Step 7 includes: Step 7.1: Divide the disease target boxes into high-scoring boxes and low-scoring boxes according to the disease confidence level; high-scoring boxes are disease target boxes with a disease confidence level greater than or equal to the confidence threshold; low-scoring boxes are disease target boxes with a disease confidence level less than the confidence threshold. Step 7.2: Using IoU distance calculation and the Hungarian algorithm, the high-resolution bounding box is matched one-to-one with the historical pavement defect activity trajectory to obtain the identification result of the historical pavement defect target in the current frame, and the first current defect activity trajectory is constructed based on the identification result of the current frame; the historical defect activity trajectory refers to the tracking result of the historical pavement defect target before the current frame; the historical pavement defect target refers to the pavement defect target identified before the current frame; the current defect activity trajectory is used to represent the tracking result of the historical pavement defect target in the current frame and before. Step 7.3: Determine whether all historical disease activity trajectories match the high-resolution bounding box of the current frame. If so, take the high-resolution bounding box that does not match the historical disease activity trajectory as the new disease activity trajectory and assign a tracking ID to it for disease activity trajectory matching in subsequent frames. Take the first current disease activity trajectory as the road surface disease tracking result of the current frame. Step 7.4: For historical pavement activity trajectories with high-resolution bounding boxes that are not matched in the current frame, low-resolution bounding boxes are matched one-to-one with historical pavement activity trajectories using IoU distance calculation and Hungarian algorithm to obtain the identification results of historical pavement pavement targets in the current frame, and a second current pavement activity trajectory is constructed based on the identification results of the current frame. Step 7.5: Determine whether all historical disease activity trajectories match the disease target box in the current frame. If so, end the matching process. Step 7.6: Temporarily store the historical disease activity trajectory that has not matched the disease target box in the current frame. If the temporarily stored historical disease activity trajectory still does not match the disease target box within a preset number of consecutive frames, then clear the historical disease activity trajectory.

5. The automated pavement distress detection method based on deep learning according to claim 1, characterized in that, Step 8 includes: Step 8.1: Based on the standard example of disease segmentation results and pavement disease tracking results, determine the target diseases to be judged; Step 8.2: When the target of the defect to be determined is within the vehicle statistics range, calculate the defect target index and the defect coordinates; the defect target index includes the area, depth, length and width of the defect.

6. The automated pavement distress detection method based on deep learning according to claim 5, characterized in that, Calculate the area of ​​the target area to be diagnosed with the disease, including: Based on the segmentation mask of the target disease to be identified, the point cloud data of the image is extracted to obtain the planar coordinates of the target disease to be identified; Construct a set of convex hull or ordered polygon vertices of the target disease to be judged based on planar coordinates; The two-dimensional planar area of ​​the target defect is obtained by calculating the planar coordinates of the convex hull or ordered polygon vertex set: ； Where A is the two-dimensional planar area of ​​the target disease to be judged; n is the number of polygon vertices; and i is the vertex index. and Let be the two-dimensional coordinates of the i-th vertex; and Let be the two-dimensional coordinates of the (i+1)th vertex.

7. The automated pavement distress detection method based on deep learning according to claim 5, characterized in that, Calculate the depth of the target disease to be identified, including: Based on the segmentation mask of the target disease to be identified, the image point cloud data is filtered to obtain the point cloud data of the target disease. Extract the coordinate values ​​of the diseased target point cloud data along the vertical axis; The difference between the maximum and minimum values ​​of the coordinates along the vertical axis is taken as the depth of the target disease to be judged.

8. The automated pavement distress detection method based on deep learning according to claim 5, characterized in that, The length and width of the target defect to be identified are calculated as follows: Based on the segmentation mask of the target disease to be identified, the image point cloud data is filtered to obtain the point cloud data of the target disease. The point cloud data of disease targets is centered, and the covariance matrix of the centered point cloud data of disease targets is calculated. The centering process refers to subtracting the mean coordinate of the point cloud data of disease targets from the three-dimensional coordinates of each point so that the distribution center of the processed data is located at the origin of the coordinate system. Eigenvalue decomposition of the covariance matrix yields multiple principal component directions and their variance strengths; Obtain the eigenvalues ​​of the principal component directions with the highest variance intensity, and determine the length of the target disease to be judged based on the multiple of these eigenvalues; The eigenvalues ​​of the principal component directions with the second highest variance intensity are obtained, and the width of the target disease to be judged is determined based on the multiple of these eigenvalues.

9. The automated pavement distress detection method based on deep learning according to claim 5, characterized in that, The latitude and longitude of the target area to be diagnosed with the disease are calculated as follows: Obtain the vehicle's current location information from the integrated navigation data; the vehicle's current location information includes latitude and longitude, elevation, and vehicle attitude; the vehicle attitude includes heading angle. Use the vehicle's current location information as the origin of the ENU coordinate system, and establish a local ENU coordinate system; Extract the relative three-dimensional position of the target defect in the lidar coordinate system; Based on the vehicle's attitude, the relative three-dimensional position is transformed into a local ENU coordinate system to obtain the ENU coordinates of the target defect to be identified. The positional offset between the ENU coordinates of the disease and the origin of the ENU coordinate system is calculated to obtain the absolute ENU coordinates of the target disease relative to the origin of the ENU coordinate system. The absolute ENU coordinates are converted to coordinates in the WGS84 coordinate system using a geographic projection conversion algorithm to obtain the latitude and longitude of the target to be diagnosed.

10. A deep learning-based automated pavement distress detection system according to any one of claims 1-9, characterized in that, It includes a data acquisition module, an image filtering module, a registration module, a recognition module, a detection module, a segmentation module, a tracking module, and an index calculation module. The data acquisition module is used to acquire image data, LiDAR point cloud data, and integrated navigation data; the integrated navigation data includes vehicle attitude information, speed information, and position information. The image filtering module is used to identify anomalies in image data, filter out abnormal images, and obtain normal image data; abnormal images refer to image anomalies caused by camera malfunctions. The registration module is used to register normal image data and LiDAR point cloud data to obtain image point cloud data; The recognition module is used to identify road surfaces from normal image data using a target classification algorithm to determine the road surface type; the target classification algorithm is the YOLOv11 algorithm. The detection module is used to detect road surface defects in normal image data based on road surface type and through defect detection algorithms to obtain road surface defect targets; road surface defect targets include standard defect targets and non-standard defect targets; The segmentation module is used to segment standard disease targets using a disease segmentation algorithm to obtain standard instance disease segmentation results; The tracking module is used to track and assign tracking IDs to detected pavement defects using a multi-target tracking algorithm, and obtain pavement defect tracking results. The index calculation module is used to calculate the disease index for each pavement disease target based on the pavement disease tracking results and image point cloud data, and obtain the disease index.

Images