Road surface defect detection method and system based on multi-sensor fusion

Abstract

The invention provides a road surface defect detection method and system based on multi-sensor fusion, and the method comprises the steps: 1, capturing a road surface image through an RGB camera, and obtaining the three-dimensional point cloud data of a road surface through LiDAR; 2, performing feature extraction on the road surface image captured by the RGB camera by using a YOLOv5 network; step 3, processing the road surface three-dimensional point cloud data acquired by the LiDAR by using a Point Net + + network; step 4, carrying out deep fusion on the image features extracted by the YOLOv5 and the point cloud features extracted by the Point Net + +; and 5, detecting and classifying the road surface defects by adopting a quantitative and qualitative combined method. According to the method, an RGB camera and LiDAR (light detection and distance measurement) are adopted, a YOLOv5 network is combined with PointNet + +, high-precision and real-time detection and classification of road surface defects are achieved, and the defects of a traditional method are overcome.

Description

Technical Field

[0001] The present invention relates to the field of road surface defect recognition, and in particular, to a road surface defect detection method and system based on multi-sensor fusion. Background Art

[0002] With the development of intelligent transportation systems and autonomous driving technologies, the demand for real-time monitoring of road surface conditions is increasing. Traditional methods for detecting road defects rely primarily on manual inspections, which are inefficient, costly, and pose safety risks.

[0003] As deep learning-based road surface defect detection methods gradually become mainstream, they have achieved remarkable results in this field. For example, real-time object detection networks based on the YOLO (You Only Look Once) series and other convolutional neural network (CNN) technologies not only offer high detection accuracy and speed, but also possess a relatively lightweight network structure.

[0004] For example, Li Jintao et al. proposed a road surface defect detection method based on TAS-YOLO, in which a TAS-YOLO network model was established for road surface defect data. The model optimized the network loss function based on YOLOv5s, added a feature extraction layer, and used a task-specific context decoupling head.

[0005] For example, Yin Jing et al. proposed a road surface defect detection method based on an improved YOLOv8. To address the redundant computation issues inherent in YOLOv8 and enhance the model's feature processing capabilities, the method replaced the C2f model in the backbone network with a lightweight and highly accurate FasterNext model. Secondly, to capture global features, a spatial attention mechanism was employed, enabling the model to adaptively learn attention weights for different regions. This allows the model to focus more on important areas while ignoring less important ones, improving accuracy.

[0006] For example, Chinese invention patent application CN116402750A discloses an automatic road crack detection method based on morphological operations and YOLOv5, including the following steps: 1) extracting crack features through morphological operations; 2) selecting the Yolov5 network for model training; and 3) detecting road cracks using the Yolov5 model.

[0007] For example, Chinese invention patent application publication CN117437201A discloses a road crack detection method based on an improved YOLOv7, including the following steps: Step 1: Acquire a road crack dataset; Step 2: Preprocess the images in the dataset and divide the processed dataset into a training set, a validation set, and a test set; Step 3: Construct an improved YOLOv7 model; Step 4: Use the training set and the validation set to train the improved YOLOv7 model; Step 5: Use the test set to evaluate the performance of the improved YOLOv7 model. Once the set performance indicators are met, it is determined to be a crack detection model; Step 6: Input the road crack image to be detected into the crack detection model for detection.

[0008] For example, China's invention patent application CN117576073A discloses a road defect detection method based on an improved YOLOv8 model, including: collecting road defect images and dividing them into training set, validation set and test set; building a road defect detection model based on the improved YOLOv8 model, iteratively training the road defect detection model using the training set, verifying the model performance through the validation set, and obtaining the model with the best performance; collecting real road defect images, inputting them into the trained road defect detection model, and outputting the road defect type.

[0009] While all of the aforementioned methods utilize the YOLO family of algorithms, they primarily rely on RGB image data, which is significantly affected by ambient lighting. This makes detection accuracy difficult to guarantee in complex environments. Furthermore, a single sensor cannot effectively identify non-surface or subsurface defects, presenting limitations.

[0010] Compared to existing technologies, this invention offers significant innovation and technical advantages: by fusing dual-modal data from RGB cameras and LiDAR, combining the image feature extraction capabilities of YOLOv5 with the point cloud geometric feature analysis capabilities of PointNet++, it achieves high-precision, robust detection of road surface defects across all scenarios. Existing technologies (such as CN116402750A, CN117437201A, and CN117576073A) rely on single RGB image data. While these technologies improve detection in some scenarios through morphological preprocessing, attention mechanisms, or network structure optimization, they are still limited by their sensitivity to illumination and dependence on surface texture, making it difficult to reliably identify targets in non-surface defects (such as loose roadbed) or complex environments (such as occluded by water or fallen leaves).

[0011] This invention uses a groundbreaking multi-sensor deep fusion architecture: first, sensor calibration is used to achieve spatial alignment between the RGB image and the LiDAR point cloud, and the geometric features of the point cloud, such as normal and curvature, are projected onto the image plane; then, a bidirectional cross-attention module is designed to dynamically aggregate the deep mutation information of the point cloud (such as crack edges) using image features as queries, while simultaneously enhancing abnormal texture areas in the image (such as potholes and broken zones) using point cloud features as queries, thereby achieving deep complementarity between geometric and visual features. This method not only significantly improves detection accuracy, but also breakthroughs in covering all scales of defect types, from microscopic cracks to loose roadbeds, and maintains stable performance in complex environments such as strong light, rain, and fog. In addition, through multi-scale feature fusion and end-to-end progressive training strategies, this invention balances real-time performance with model generalization capabilities, providing reliable technical support for intelligent road inspections. Summary of the Invention

[0012] To address the above issues, the present invention provides a road surface defect detection method based on multi-sensor fusion. This method uses an RGB camera and LiDAR (Light Detection and Ranging) and combines the YOLOv5 network with PointNet++ to achieve high-precision, real-time detection and classification of road surface defects, thus overcoming the shortcomings of traditional methods.

[0013] The present invention provides a road surface defect detection method based on multi-sensor fusion, comprising the following steps: Step 1: Capture the road surface image through the RGB camera and obtain the 3D point cloud data of the road surface through LiDAR; Step 2: Use the YOLOv5 network to extract features from the road surface image captured by the RGB camera. The RGB image is input into the YOLOv5 network, and the backbone network extracts multi-scale feature maps. The neck network performs bidirectional fusion of the different scale features output by the backbone network. The measurement head uses the fused feature maps to predict targets at three scales. Step 3: Use the PointNet++ network to process the 3D point cloud data of the road surface acquired by LiDAR to extract the 3D geometric features of the road surface. The processing includes sampling grouping, local feature extraction, multi-scale and recursive abstraction, and feature propagation and refinement. Step 4: Deeply fuse the image features extracted by YOLOv5 with the point cloud features extracted by PointNet++. The fusion includes feature alignment, point cloud projection to the image plane, bidirectional cross attention, and fused feature output. Step 5: Use a combination of quantitative and qualitative methods to detect and classify road surface defects.

[0014] Preferably, the distance between the RGB camera and the LiDAR in step 1 is 0.5 meters, and the field of view overlap rate is ≥80%.

[0015] Preferably, the RGB camera intrinsic parameters are calibrated using the calibrateCamera function of OpenCV. 20 sets of images are collected using a 9×6 corner point checkerboard, and the intrinsic parameter matrix K is calculated to ensure that the reprojection error is ≤1.5 pixels. The LiDAR camera extrinsic parameters are based on the pointToPlaneICP algorithm of PCL to align the point cloud and image, solve the rotation matrix R and translation vector t, and establish a 3D→2D projection relationship, as shown in formula (1), to achieve multimodal sensor spatial alignment.

[0016]

[0017] Preferably, step 1 further includes a data preprocessing step, Data preprocessing for road surface images involves first resizing the input image to 640×640 to meet the YOLOv5 model input requirements; then performing channel-by-channel normalization based on the statistical characteristics of the dataset; and finally, implementing random horizontal flipping, brightness adjustment, and 3×3 Gaussian blur data augmentation to effectively improve the model's robustness to scale changes and lighting interference. Preprocessing of 3D point cloud data involves first downsampling the data to a 0.02m resolution voxel grid to reduce data density and filtering out non-ground point clouds through RANSAC plane fitting (with a height threshold of 0.1m). A 3D ROI is then generated based on a 10cm expansion of the bounding box of the image annotation. 1024 points / blocks are extracted from this 3D ROI, and the coordinates (x, y, z), reflectivity, normal vector (x, y, z), and curvature of each point are concatenated into a 7-dimensional feature vector. Finally, standardized point cloud data blocks suitable for deep learning model input are constructed.

[0018] Preferably, in step 2, the backbone network performs hierarchical feature extraction with the help of 5-layer convolution and 4-layer C3 modules. The shallow output retains the crack texture details, and the deep output focuses on the semantics of potholes. The neck network (PANet) uses upsampling to introduce deep semantic features and downsampling to introduce shallow position features to achieve bidirectional cross-scale fusion and construct a multi-scale feature expression that takes into account both semantics and accuracy.

[0019] Preferably, in step three, PointNet++ uses a two-level Set Abstraction module to extract point cloud features, where the SA1 layer searches for 32 neighboring points with a radius of 0.05m to extract local geometric features, and the SA2 layer searches for 64 neighboring points with a radius of 0.1m, and combines the multi-scale grouping strategy to enhance the multi-scale expression capability, and finally outputs a 1024-dimensional feature vector containing normal, curvature and semantic information.

[0020] Preferably, in step three, sampling and grouping include: at each level, using farthest point sampling to select representative "seed points", and searching their spatial local neighborhood point sets through radius grouping or K-nearest neighbor strategy; local feature extraction includes: inputting a small shared multi-layer perceptron into each neighborhood point set, improving the feature dimension point by point, using maximum pooling to aggregate all point features in the neighborhood, and generating a robust local feature vector representing the area; multi-scale and recursive abstraction includes: combining a multi-scale grouping strategy, extracting features from neighborhoods of different radius / scales at the same time and splicing them, to enhance the scale adaptability from micro cracks to macro potholes. The above process is recursive layer by layer, with the neighborhood radius increasing and the number of points decreasing, gradually constructing a multi-level feature pyramid from details to semantics. Feature propagation and refinement include: to restore spatial resolution and point-by-point details, high-level semantic features are propagated back to low-level dense points through inverse distance weighted interpolation, and then spliced ​​and fused with the original low-level high-resolution features. High-resolution point-level feature descriptors are then generated through MLP refinement. The final output is a fused feature vector containing spatial position, surface normal, local curvature, and contextual semantics for subsequent defect determination.

[0021] Preferably, step 4 specifically includes: projecting the point cloud geometric features extracted by PointNet++ onto the image plane using camera and LiDAR calibration parameters, generating a multi-channel geometric feature map that is spatially aligned with the YOLOv5 feature map, and strictly matching the multi-scale resolution through interpolation. Subsequently, a bidirectional cross-attention module is embedded in each scale feature map of YOLOv5: using image features as queries to dynamically weight and aggregate the related point cloud geometric context, while also using point cloud features as queries to reversely enhance the texture areas in the image corresponding to geometric anomalies, thereby enhancing the network's detection capabilities for defects of different sizes and types.

[0022] Preferably, the bidirectional cross attention module in step 4 executes the following steps: the bidirectional cross attention module operates in two stages. First, the image features are used as query and the point cloud features are used as key / value to aggregate geometric contexts such as crack depth mutations; then, the point cloud features are used as query and the image features are used as key / value to enhance texture abnormality areas such as potholes and broken zones; and the point cloud feature channels are dynamically adjusted to 64 / 128 / 256 with the help of 1×1 convolution, matching the P5 / P4 / P3 layers, and output after 3×3 convolution compression.

[0023] Preferably, step five includes performing non-maximum suppression on the output results to generate the final road defect detection results, wherein the non-maximum suppression adopts an IoU threshold of 0.5, and the final output supports 2D and 3D visualization, wherein the 2D results are superimposed with a depth pseudo-color map, and the 3D results are annotated with defect areas by back-projection of the point cloud.

[0024] The present invention also provides a road surface defect detection system based on multi-sensor fusion, which is characterized by comprising corresponding modules for executing each step of the road surface defect detection method as described above.

[0025] The present invention has the following beneficial effects: 1) By fusing the texture and color information of RGB images with the precise three-dimensional geometric information (depth, normal, and curvature) of LiDAR point clouds, this invention comprehensively utilizes the visual appearance and intrinsic structural characteristics of defects, significantly improving the detection and classification accuracy of various road surface defects (such as cracks and potholes), overcoming the deficiency of insufficient information from a single sensor.

[0026] 2) Thanks to the PointNet++ network's multi-scale grouping (MSG) recursive abstraction mechanism and YOLOv5's multi-scale feature fusion mechanism (PANet), this method can effectively capture road defect features of different scales, types, and complexities, from microscopic crack textures to macroscopic pothole shapes, and has strong adaptability to both environmental and defect types.

[0027] 3) It adopts spatial alignment and interpolation operations based on sensor calibration to ensure accurate matching between images and point cloud feature maps, and innovatively embeds a bidirectional cross-attention module to achieve deep, dynamic interaction and complementarity between image features and point cloud geometric features (such as image-guided aggregation of key geometric contexts, and geometric features-guided focusing on key texture areas), greatly enhancing the network's ability to capture subtle changes (such as sudden depth changes at crack edges) and complex scenes (such as broken textures around potholes). BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a road surface defect detection method based on multi-sensor fusion of the present invention; Figure 2 1 is a schematic diagram of the network structure of the YOLO v5 algorithm in the road surface defect detection method based on multi-sensor fusion of the present invention; Figure 3 This is a PANet network structure diagram in a road surface defect detection method based on multi-sensor fusion according to the present invention; Figure 4 This is a diagram of the PointNet++ network structure in a road surface defect detection method based on multi-sensor fusion according to the present invention; Figure 5 This is a schematic diagram of the improved algorithm network structure in the road surface defect detection method based on multi-sensor fusion of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only illustrative and not restrictive.

[0030] Figure 1 This is a flow chart of a road surface defect detection method based on multi-sensor fusion according to the present invention. Figure 1 As shown, the present invention provides a road surface defect detection method based on multi-sensor fusion, comprising the following steps: Step 1: Capture the road surface image through the RGB camera and obtain the 3D point cloud data of the road surface through LiDAR; Step 2: Use the YOLOv5 network to extract features from the road surface image captured by the RGB camera. The RGB image is input into the YOLOv5 network, and the backbone network extracts multi-scale feature maps. The neck network performs bidirectional fusion of the different scale features output by the backbone network. The measurement head uses the fused feature maps to predict targets at three scales. Step 3: Use the PointNet++ network to process the 3D point cloud data of the road surface acquired by LiDAR to extract the 3D geometric features of the road surface. The processing includes sampling grouping, local feature extraction, multi-scale and recursive abstraction, and feature propagation and refinement. Step 4: Deeply fuse the image features extracted by YOLOv5 with the point cloud features extracted by PointNet++. The fusion includes feature alignment, point cloud projection to the image plane, bidirectional cross attention, and fused feature output. Step 5: Use a combination of quantitative and qualitative methods to detect and classify road surface defects.

[0031] Optionally, the distance between the RGB camera and the LiDAR in step 1 is 0.5 meters, and the field of view overlap rate is ≥80%.

[0032] A Basler acA2040-180km RGB camera and a Velodyne VLP-16 LiDAR were mounted on top of the test vehicle, 0.5 meters apart. Through hardware collaboration, the camera captured road surface texture details and the LiDAR captured 3D geometric information simultaneously. A field of view overlap of ≥80% ensured spatial alignment of multimodal data.

[0033] Optionally, the RGB camera intrinsic parameters are calibrated using the calibrateCamera function of OpenCV. 20 sets of images are collected using a 9×6 corner point checkerboard, and the intrinsic parameter matrix K is calculated to ensure that the reprojection error is ≤1.5 pixels. The LiDAR camera extrinsic parameters are based on the pointToPlaneICP algorithm of PCL to align the point cloud and image, solve the rotation matrix R and translation vector t, and establish a 3D→2D projection relationship, as shown in formula (1), to achieve multimodal sensor spatial alignment.

[0034]

[0035] This enables multi-stage calibration.

[0036] Optionally, step 1 further includes a data preprocessing step. Data preprocessing for road surface images involves first resizing the input image to 640×640 to meet the YOLOv5 model input requirements; then performing channel-by-channel normalization based on the statistical characteristics of the dataset; and finally, implementing random horizontal flipping, brightness adjustment, and 3×3 Gaussian blur data augmentation to effectively improve the model's robustness to scale changes and lighting interference. Preprocessing of 3D point cloud data involves first downsampling the data to a 0.02m resolution voxel grid to reduce data density and filtering out non-ground point clouds through RANSAC plane fitting (with a height threshold of 0.1m). A 3D ROI is then generated based on a 10cm expansion of the bounding box of the image annotation. 1024 points / blocks are extracted from this 3D ROI, and the coordinates (x, y, z), reflectivity, normal vector (x, y, z), and curvature of each point are concatenated into a 7-dimensional feature vector. Finally, standardized point cloud data blocks suitable for deep learning model input are constructed.

[0037] Optionally, in step 2, the backbone network performs hierarchical feature extraction with the help of 5-layer convolution and 4-layer C3 modules. The shallow output retains the crack texture details, and the deep output focuses on the semantics of potholes. The neck network (PANet) uses upsampling to introduce deep semantic features and downsampling to introduce shallow position features to achieve bidirectional cross-scale fusion and construct a multi-scale feature expression that takes into account both semantics and accuracy.

[0038] Step 2 specifically involves inputting the RGB image into the YOLOv5 backbone network. Convolutional layers (downsampling + channel multiplication) and the C3 module extract features layer by layer: shallow layers focus on details (crack texture) and deep layers on semantics (pothole shape). The neck network (PANet structure) performs bidirectional fusion of the different-scale features output by the backbone network: upsampling incorporates deep semantics, while downsampling introduces shallow positional details, generating a multi-scale feature map that balances semantics and accuracy. The detection head uses this fused feature map to predict objects at three scales. For each location, it outputs the defect category, confidence score, and bounding box coordinates, enabling preliminary detection of defect areas on the road surface.

[0039] Optionally, in step three, PointNet++ uses a two-level Set Abstraction module for point cloud feature extraction, where the SA1 layer searches for 32 nearest neighbors with a radius of 0.05m to extract local geometric features, and the SA2 layer searches for 64 nearest neighbors with a radius of 0.1m. It also combines a multi-scale grouping strategy to enhance multi-scale expression capabilities, and finally outputs a 1024-dimensional feature vector containing normal, curvature and semantic information.

[0040] Optionally, in step three, the sampling grouping includes: at each level, selecting a representative "seed point" using farthest point sampling, and searching its spatial local neighborhood point set through radius grouping or K-nearest neighbor strategy.

[0041] Local feature extraction involves feeding each neighborhood point set into a small shared multi-layer perceptron, which increases the feature dimension (including position, reflectivity, etc.) point by point. Max pooling (a symmetric function) is then used to aggregate all point features within the neighborhood to generate a robust local feature vector representing the region.

[0042] Multi-scale and recursive abstraction involves combining the Multi-Scale Grouping (MSG) strategy to simultaneously extract and stitch features from neighborhoods of different radii and sizes, enhancing adaptability to scales from microscopic cracks to macroscopic potholes. This process is recursive, increasing the neighborhood radius while decreasing the number of points, gradually constructing a multi-level feature pyramid from details to semantics.

[0043] Feature propagation and refinement involves propagating high-level semantic features back to low-level dense points via inverse distance weighted interpolation to restore spatial resolution and point-by-point detail. These features are then concatenated and fused with the original low-level high-resolution features. These features are then refined via MLP to generate high-resolution point-level feature descriptors. The final output is a fused feature vector containing spatial position, surface normal, local curvature, and contextual semantics, which is used for subsequent defect detection.

[0044] Optionally, step four specifically includes: projecting the point cloud geometric features (such as normals, curvature, and semantics) extracted by PointNet++ onto the image plane using camera and LiDAR calibration parameters, generating a multi-channel geometric feature map that is spatially aligned with the YOLOv5 feature map, and strictly matching the multi-scale resolution through interpolation. Subsequently, a bidirectional cross-attention module is embedded in each scale feature map (P3 / P4 / P5) of YOLOv5: using image features as queries to dynamically weight and aggregate the related point cloud geometric context (such as the depth mutation information of crack edges), while also using point cloud features as queries to reversely enhance texture areas in the image corresponding to geometric anomalies (such as the broken texture around potholes), thereby enhancing the network's detection capabilities for defects of different sizes and types.

[0045] Optionally, the bidirectional cross-attention module in step 4 executes the following steps: the bidirectional cross-attention module operates in two stages. First, the image features are used as queries and the point cloud features are used as keys / values ​​to aggregate geometric contexts such as crack depth mutations. Then, the point cloud features are used as queries and the image features are used as keys / values ​​to enhance texture abnormality areas such as potholes and broken zones. The point cloud feature channels are dynamically adjusted to 64 / 128 / 256 with the help of 1×1 convolution, matching the P5 / P4 / P3 layers, and output after 3×3 convolution compression.

[0046] In step 4, the projection alignment process includes: First, the 3D coordinates of the point cloud are converted to the image coordinate system. Then, the normal (x / y components) and curvature are spliced ​​to generate a 3-channel geometric feature map. Finally, through bilinear interpolation, the geometric feature map is made to match the P3 / P4 / P5 layer resolution of YOLOv5 (80×80, 40×40, and 20×20, respectively), achieving multi-scale adaptation of point cloud geometric features and image features.

[0047] Figure 2 This is a schematic diagram of the network structure of the YOLOv5 algorithm in the road surface defect detection method based on multi-sensor fusion of the present invention. Figure 3 This is a PANet network structure diagram in a road surface defect detection method based on multi-sensor fusion according to the present invention. Figure 2 As shown in , the network adopts a multi-level feature extraction and fusion architecture: the backbone network consists of 5 layers of convolution and 4 layers of C3 modules for efficient gradient propagation and feature abstraction; Figure 3 The neck network shown uses an improved PANet structure to enhance multi-scale information interaction. The detection head outputs feature maps at three scales: 80×80, 40×40, and 20×20, achieving high-precision positioning and classification for small, medium, and large road defects, respectively, forming an end-to-end multi-scale detection capability.

[0048] During the training phase, the AdamW optimizer (beta_1=0.9, beta_2=0.999) was used. The initial learning rate was set to 0.001 and decayed by 0.1 times every 20 epochs, for a total of 120 epochs. Efficiency was improved through mixed precision training (batch_size=16) combined with TensorRT acceleration. In terms of loss function design, Focal Loss (gamma=2) was used to optimize classification to address the problem of class imbalance, DIoU loss (alpha=0.5) was used to improve the accuracy of bounding box regression, and binary cross entropy (BCE) was used to constrain target confidence to achieve multi-task collaborative optimization.

[0049] Figure 4 This is a diagram of the PointNet++ network structure used in the multi-sensor fusion-based road surface defect detection method of the present invention. As shown, the network employs a hierarchical point cloud feature extraction architecture: First, downsampling and local feature aggregation are performed through a two-layer SetAbstraction module. The SA1 layer selects seed points based on the farthest point sampling from the FPS, searching for 32 nearest neighbors within a 0.05m radius to extract local geometric features. The SA2 layer expands the receptive field to a 0.1m radius and employs 64 nearest neighbors plus multi-scale grouping to enhance multi-scale feature expression. Finally, the feature propagation layer uses inverse distance weighted interpolation to restore the resolution to 1024 points, outputting a 1024-dimensional global feature vector that fuses geometric attributes with high-level semantics, achieving point cloud feature encoding from local to global scale.

[0050] During the pre-training phase, a multi-task joint learning strategy is adopted: normal prediction, curvature regression, and semantic classification are used to simultaneously optimize the geometric and semantic feature expressions of the point cloud. Training is performed with batch_size=8, epochs=100, and a cosine annealing schedule with an initial learning rate of 0.0005. The pre-trained weights of ModelNet40 are loaded for transfer learning initialization, significantly improving the model's geometric sensitivity to road defects and its semantic discrimination capabilities.

[0051] The feature fusion and end-to-end training in the road surface defect detection method based on multi-sensor fusion of the present invention include: 1) The spatial alignment module achieves cross-modal information interaction through multi-scale geometric-image feature fusion: First, the 3D coordinates of the point cloud are projected onto the 80×80 / 40×40 / 20×20 feature maps of YOLOv5 based on calibration parameters, and the resolution is matched through bilinear interpolation. At the same time, the normal (x / y components, 2 channels) and curvature (1 channel) of the point cloud are spliced ​​into a 3-channel geometric feature map, which is spatially aligned with the 64-channel features of the image P5 layer. This constructs a fusion representation that combines geometric constraints and semantic information, enhancing the robustness of multimodal detection.

[0052] 2) This module uses a bidirectional cross-attention mechanism to achieve deep interaction between image and point cloud features: In stage 1, image features are used as queries and point cloud features as keys / values ​​(four-head attention), and deep mutation areas such as crack edges are enhanced through geometric context aggregation; in stage 2, point cloud features are used as queries and image features as keys / values, focusing on abnormal patterns such as potholes and broken textures; finally, 1×1 convolution is used to dynamically adjust the point cloud feature channels to 64 / 128 / 256 (matching the P5 / P4 / P3 layers), and after 3×3 convolution compression and splicing, multimodal fusion features are output to achieve complementary enhancement of geometric and texture information.

[0053] 3) A two-stage progressive training strategy is adopted: in stage 1, the pre-training layer parameters are frozen and only the fusion module is optimized (50 epochs, lr=0.0001) to ensure the stability of multimodal feature alignment; in stage 2, all network layers are unfrozen and end-to-end joint training is performed (30 epochs, lr=0.00005). Through the YOLO detection loss + 0.1× attention L2 regularization term, the detection accuracy and model generalization are balanced, ultimately achieving efficient convergence and performance improvement of the fusion model.

[0054] In a road surface defect detection method based on multi-sensor fusion of the present invention, the testing and performance evaluation includes: 1) During the testing phase, a 640×640 image and a raw point cloud are input simultaneously. YOLOv5 is used to extract multi-scale image features. PointNet++ is then used to generate 7-dimensional point cloud features including coordinates, reflectivity, normal, and curvature, and complete projection alignment. A bidirectional attention module is then used to achieve cross-modal feature fusion. The detection head outputs bounding boxes, categories, and confidence scores. Finally, NMS post-processing (intersection-over-union threshold = 0.5) is used to filter out redundant predictions, forming an end-to-end multimodal defect detection process.

[0055] 2) The evaluation uses a combination of quantitative and qualitative methods: the detection accuracy of cracks and potholes is quantified using mAP@0.5 and mAP@0.5:0.95. The detection performance of fine cracks (≥2 mm) and shallow potholes (≥8 mm) is analyzed based on 2D / 3D visualization (such as depth pseudo-color overlays and point cloud back-projection annotations). The focus is on evaluating the edge missed detection rate and small target false detection rate, comprehensively verifying the robustness of the model in real-world scenarios.

[0056] The detection head makes predictions at three scales: an 80×80 layer for detecting small cracks with a width ≥ 2mm, a 40×40 layer for detecting medium potholes with a depth ≥ 8mm, and a 20×20 layer for detecting large repair areas. The output parameters for each position include defect category, center offset, bounding box (width / height / angle), and confidence level.

[0057] The data collection of this invention covers urban main roads and rural roads, including various scenes such as dryness, waterlogging, and fallen leaves, and a total of 8,000 sets of image-point cloud alignment data are obtained; in terms of annotation, LabelImg is used to annotate the image bounding boxes for 5,000 cracks and 3,000 potholes, and CloudCompare is used to annotate the point cloud defect areas, and geometric features such as normals and curvatures are calculated simultaneously to construct a multimodal defect dataset.

[0058] Figure 5 This is a schematic diagram of the improved algorithm network structure in the road surface defect detection method based on multi-sensor fusion of the present invention.

[0059] Optionally, step five includes performing non-maximum suppression on the output structure to generate the final road defect detection result, where the non-maximum suppression uses an IoU threshold of 0.5, and the final output supports 2D and 3D visualization, where the 2D result is superimposed with a depth pseudo-color map, and the 3D result is annotated with defect areas through point cloud back projection.

[0060] The present invention also provides a road surface defect detection system based on multi-sensor fusion, which is characterized by comprising corresponding modules for executing each step of the road surface defect detection method as described above.

[0061] In summary, this invention offers significant innovation and technical advantages over existing technologies. By fusing dual-modal data from RGB cameras and LiDAR, combined with the image feature extraction capabilities of YOLOv5 and the point cloud geometric feature analysis capabilities of PointNet++, it achieves high-precision, robust detection of road surface defects across all scenarios. Existing technologies (such as CN116402750A, CN117437201A, and CN117576073A) rely on single RGB image data. While these technologies improve detection in some scenarios through morphological preprocessing, attention mechanisms, or network structure optimization, they are still limited by their sensitivity to illumination and dependence on surface texture, making it difficult to reliably identify targets in non-surface defects (such as loose roadbed) or complex environments (such as stagnant water or fallen leaves).

[0062] This invention uses a groundbreaking multi-sensor deep fusion architecture: first, sensor calibration is used to achieve spatial alignment between the RGB image and the LiDAR point cloud, and the geometric features of the point cloud, such as normal and curvature, are projected onto the image plane; then, a bidirectional cross-attention module is designed to dynamically aggregate the deep mutation information of the point cloud (such as crack edges) using image features as queries, while simultaneously enhancing abnormal texture areas in the image (such as potholes and broken zones) using point cloud features as queries, thereby achieving deep complementarity between geometric and visual features. This method not only significantly improves detection accuracy, but also breakthroughs in covering all scales of defect types, from microscopic cracks to loose roadbeds, and maintains stable performance in complex environments such as strong light, rain, and fog. In addition, through multi-scale feature fusion and end-to-end progressive training strategies, this invention balances real-time performance with model generalization capabilities, providing reliable technical support for intelligent road inspections.

[0063] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A road surface defect detection method based on multi-sensor fusion, characterized in that: The following steps are involved: Step 1: Capture the road surface image through the RGB camera and obtain the 3D point cloud data of the road surface through LiDAR; Step 2: Use the YOLOv5 network to extract features from the road surface image captured by the RGB camera. The RGB image is input into the YOLOv5 network, and the backbone network extracts multi-scale feature maps. The neck network performs bidirectional fusion of the different scale features output by the backbone network. The measurement head uses the fused feature maps to predict targets at three scales. Step 3: Use the PointNet++ network to process the 3D point cloud data of the road surface acquired by LiDAR to extract the 3D geometric features of the road surface. The processing includes sampling grouping, local feature extraction, multi-scale and recursive abstraction, and feature propagation and refinement. Step 4: Deeply fuse the image features extracted by YOLOv5 with the point cloud features extracted by PointNet++. The fusion includes feature alignment, point cloud projection to the image plane, bidirectional cross attention, and fused feature output. Step 5: Use a combination of quantitative and qualitative methods to detect and classify road surface defects.

2. The road surface defect detection method according to claim 1, characterized in that: The distance between the RGB camera and LiDAR in step 1 is 0.5 meters, and the field of view overlap rate is ≥80%.

3. The road surface defect detection method according to claim 2, characterized in that: The RGB camera intrinsic parameters are calibrated using the calibrateCamera function of OpenCV. 20 sets of images are collected using a 9×6 corner point checkerboard. The intrinsic parameter matrix K is calculated to ensure that the reprojection error is ≤1.5 pixels. The LiDAR camera extrinsic parameters are aligned with the point cloud and image based on the pointToPlaneICP algorithm of PCL. The rotation matrix R and translation vector t are solved and the 3D→2D projection relationship is established as shown in formula (1), achieving multimodal sensor spatial alignment. 。 4. The road surface defect detection method according to claim 1, characterized in that: After step one, there is also a data preprocessing step. Data preprocessing for road surface images involves first resizing the input image to 640×640 to meet the YOLOv5 model input requirements; then performing channel-by-channel normalization based on the statistical characteristics of the dataset; and finally, implementing random horizontal flipping, brightness adjustment, and 3×3 Gaussian blur data augmentation to effectively improve the model's robustness to scale changes and lighting interference. Preprocessing of 3D point cloud data involves first downsampling the data using a 0.02m resolution voxel grid to reduce data density and filtering out non-ground point clouds through RANSAC plane fitting (height threshold 0.1m). A 3D ROI is then generated based on a 10cm expansion of the bounding box of the image annotation. 1024 points / blocks are extracted from this 3D ROI, and the coordinates (x, y, z), reflectivity, normal vector (x, y, z), and curvature of each point are concatenated into a 7-dimensional feature vector. Finally, standardized point cloud data blocks suitable for deep learning model input are constructed.

5. The road surface defect detection method according to claim 1, characterized in that: In step 2, the backbone network uses 5-layer convolution and 4-layer C3 modules to perform hierarchical feature extraction. The shallow output retains the texture details of the cracks, and the deep output focuses on the semantics of the potholes. The neck network (PANet) uses upsampling to introduce deep semantic features and downsampling to introduce shallow position features to achieve bidirectional cross-scale fusion and construct a multi-scale feature expression that takes into account both semantics and accuracy.

6. The road surface defect detection method according to claim 1, characterized in that: In step three, PointNet++ uses a two-level Set Abstraction module to extract point cloud features. The SA1 layer searches for 32 nearest neighbors with a radius of 0.05m to extract local geometric features, and the SA2 layer searches for 64 nearest neighbors with a radius of 0.1m. It also combines a multi-scale grouping strategy to enhance multi-scale expression capabilities, and finally outputs a 1024-dimensional feature vector containing normal, curvature, and semantic information.

7. The road surface defect detection method according to claim 6, characterized in that: In step three, sampling and grouping involve selecting representative "seed points" at each level using farthest point sampling and searching for their local spatial neighborhood points using radius grouping or a K-nearest neighbor strategy. Local feature extraction involves feeding each neighborhood point set into a small shared multi-layer perceptron, increasing the feature dimension point by point, and aggregating the features of all points in the neighborhood using max pooling to generate a robust local feature vector representing the region. Multiscale and recursive abstraction involves combining a multiscale grouping strategy to simultaneously extract and concatenate features from neighborhoods of different radii / scales, enhancing adaptability to scales from microscopic cracks to macroscopic potholes. This process is repeated layer by layer, increasing the neighborhood radius while decreasing the number of points, gradually constructing a multi-level feature pyramid from detail to semantics. Feature propagation and refinement involves propagating high-level semantic features back to the dense lower-level points via inverse distance weighted interpolation to restore spatial resolution and point-by-point detail. These features are then concatenated and fused with the original high-resolution features from the lower levels. A high-resolution point-level feature descriptor is then generated through MLP refinement. The final output is a fused feature vector containing spatial position, surface normal, local curvature, and contextual semantics for subsequent defect assessment.

8. The road surface defect detection method according to claim 1, characterized in that: Step 4 specifically includes: projecting the point cloud geometric features extracted by PointNet++ onto the image plane using camera and LiDAR calibration parameters, generating a multi-channel geometric feature map that is spatially aligned with the YOLOv5 feature map and strictly matching the multi-scale resolution through interpolation; then, embedding a bidirectional cross-attention module on each scale feature map of YOLOv5, using image features as queries to dynamically weight and aggregate the associated point cloud geometric context. At the same time, using point cloud features as queries to reversely enhance the texture areas in the image corresponding to geometric anomalies, thereby enhancing the network's detection capabilities for defects of different sizes and types; The execution steps of the bidirectional cross attention module include: the bidirectional cross attention module operates in two stages. First, the image features are used as queries and the point cloud features are used as keys / values ​​to aggregate geometric contexts such as crack depth mutations; then, the point cloud features are used as queries and the image features are used as keys / values ​​to enhance texture abnormal areas such as potholes and broken zones; and the point cloud feature channels are dynamically adjusted to 64 / 128 / 256 with the help of 1×1 convolution, matching the P5 / P4 / P3 layers, and output after 3×3 convolution compression.

9. The road surface defect detection method according to claim 1, characterized in that: Step five involves performing non-maximum suppression on the output structure to generate the final road defect detection results. Non-maximum suppression uses an IoU threshold of 0.

5. The final output supports 2D and 3D visualization. The 2D result is superimposed with a depth pseudo-color map, and the 3D result is annotated with defect areas through point cloud back-projection.

10. A road surface defect detection system based on multi-sensor fusion, characterized by: The method comprises corresponding modules for executing each step of the road surface defect detection method according to any one of claims 1 to 9.

Images