A Multimodal Road Defect Detection Method Based on Vehicle-Mounted Measurement

By combining a vehicle-mounted mobile measurement system with point cloud data and image verification in a multimodal detection method, the problem of low accuracy in road defect detection has been solved, achieving efficient and accurate defect identification and location.

CN122715094BActive Publication Date: 2026-11-10CHONGQING INST OF SURVEYING & MAPPING SCI & TECH (CHONGQING MAP COMPILATION CENT)
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611207301.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-11
Publication Date
2026-11-10
Estimated Expiration
2046-08-11

AI Technical Summary

Technical Problem

Existing technologies have low accuracy in detecting road defects. Traditional manual inspections are inefficient, costly, and pose high safety risks. Furthermore, intelligent detection methods are not robust enough in complex road conditions and are difficult to adapt to the needs of efficient, accurate, and safe maintenance of large-scale road networks.

Method used

A multimodal detection method based on vehicle-mounted mobile measurement is adopted, which combines point cloud data and orthophoto verification. The method performs two verifications using point cloud influence factors and image correction coefficients to identify road defects and improve detection accuracy.

Benefits of technology

It reduces the false detection rate, improves the accuracy of road defect detection, and accurately locates the defect location through image verification, which facilitates maintenance and management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122715094B_ABST
    Figure CN122715094B_ABST
Patent Text Reader

Abstract

This invention provides a multimodal road defect detection method based on vehicle-mounted mobile measurement, comprising: acquiring road surface point cloud data and road centerline data, and calculating the relative elevation of all point clouds in the road surface point cloud data; regularizing the road surface point cloud into a grid, and then dividing all grids into multiple defect grid classes based on the relative elevation of the point clouds within each grid, and then generating corresponding defect regions according to each defect grid class and calculating the corresponding point cloud influence factor; projecting all point clouds and grids onto an orthophoto image, performing image verification on each defect region, and then calculating the confidence level corresponding to each verified defect region based on the point cloud influence factor; locating each defect region with a confidence level greater than a preset value according to the road centerline data, thereby obtaining a road defect location map. This invention solves the problem of low road defect detection accuracy in existing technologies.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of intelligent road maintenance technology, and in particular to a multimodal road defect detection method based on vehicle-mounted mobile measurement. Background Technology

[0002] With the continuous expansion of my country's highway network, by the end of 2025, the total length of highways nationwide had exceeded 5.5 million kilometers, of which expressways accounted for over 190,000 kilometers. This massive scale of road infrastructure presents challenges to daily maintenance, defect detection, and rapid repair. Traditional manual inspection methods rely primarily on maintenance personnel driving at low speeds or walking, conducting inspections through visual inspection, measuring with tape measures, and paper records. This approach is not only inefficient, costly, and risky, but also results in subjective and inaccurate findings, making it unsuitable for the efficient, precise, and safe maintenance needs of a large-scale road network. Against this backdrop, the Mobile Mapping System (MMS), integrating vehicle-mounted LiDAR, high-definition cameras, and GNSS / IMU, has emerged as a core technological means to achieve automated and intelligent detection of road defects.

[0003] Existing intelligent inspection solutions include two-dimensional image-based methods, which are currently the most widely used technical approach. The core of this approach is to continuously acquire RGB images or video streams of the road surface using a vehicle-mounted high-resolution camera, and then use computer vision algorithms to automatically identify, locate, and classify defects. Early research mainly relied on traditional digital image processing techniques, using algorithms such as edge detection (Canny, Sobel), threshold segmentation (Otsu), morphological operations (dilation, erosion), and region growing to extract the geometric and textural features of defects such as cracks and depressions. In recent years, breakthroughs in deep learning technology have completely revolutionized this field. Deep learning-based methods (U-Net, YOLO, Mask R-CNN) have become mainstream due to their powerful feature self-learning capabilities. However, such methods can only utilize shallow image features and are very sensitive to lighting, shadows, and stains, resulting in poor robustness and poor adaptability to complex road conditions. Two-dimensional images only record pixel grayscale and color information, without containing real spatial scale and depth information, making it impossible to obtain key maintenance indicators such as pothole depth, subsidence height difference, and damage area. Damage size can only be estimated through pixel ratio, and the accuracy is greatly affected by shooting distance, angle, and road surface undulation. Generalization ability is limited, relying on large-scale labeled data. The road surface material, color, and wear degree vary significantly in different regions. After the model is trained on a certain road section, directly transferring it to other road conditions often results in a significant drop in accuracy. It requires the collection of massive, diverse, and manually labeled datasets, which is costly and time-consuming. The model is difficult to converge in small sample scenarios.

[0004] Existing road defect detection methods based on 3D laser point clouds rely on high-speed scanning of the road surface using vehicle-mounted LiDAR to acquire high-density 3D point cloud data (x, y, z) at a rate of millions per kilometer. This data is then analyzed from a purely geometric perspective, examining road elevation anomalies, surface curvature changes, and plane fitting residuals to detect damage. However, the point cloud data obtained by this method only contains coordinates and reflection intensity, lacking color and texture details. This method is prone to misjudging normal structures such as expansion joints, manhole covers, road markings, and construction joints. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a multimodal road defect detection method based on vehicle-mounted mobile measurement, which solves the problem of low accuracy in road defect detection in existing technologies.

[0006] According to an embodiment of the present invention, a multimodal road distress detection method based on vehicle-mounted motion measurement includes: Acquire road surface point cloud data and road centerline data, and calculate the relative elevation of all point clouds in the road surface point cloud data; The road surface point cloud is regularized into a grid. Then, based on the relative elevation of the point cloud in each grid, all grids are divided into multiple defect grid classes. Then, the corresponding defect area is generated according to each defect grid class and the corresponding point cloud influence factor is calculated. All point clouds and grids are projected onto the orthophoto, and each defect region is image-verified. Then, the confidence level of each defect region that passes the verification is calculated based on the point cloud influence factor. Based on the road centerline data, each defect area with a confidence level greater than a preset value is located to obtain a road defect location map.

[0007] Preferably, the method for dividing all grids into multiple defect grid classes based on the relative elevation of the point cloud within each grid includes: For each grid, the average relative elevation of all point clouds within the grid is taken as the elevation value of that grid. Then, the gradient magnitude of each grid is calculated to generate a point cloud elevation gradient magnitude map. The point cloud elevation gradient magnitude map is binarized, and the probabilistic Hough transform is used to extract straight line segments. The grid where all straight line segments are located is divided into linear crack grids. Divide each point cloud into a neighborhood, and map the local RMSE values ​​of all point clouds within the neighborhood to the grid to generate a residual feature map. The residual feature map is binarized and then morphological closing operation is performed to obtain multiple connected regions. A direction histogram is statistically analyzed for each connected region, and the entropy H of the direction histogram is calculated. The grid of connected regions with entropy H greater than a preset value is divided into a mesh-like crack grid. For each point cloud, calculate the standard deviation of the relative elevation and the local RMSE value of all point clouds in the corresponding neighborhood, and then calculate the roughness skewness of each point cloud based on the standard deviation. Point clouds with standard deviation and local RMSE values ​​both greater than the preset value and roughness skewness are classified as road depression seed points, and point clouds with standard deviation and local RMSE values ​​both greater than the preset value and roughness skewness are classified as road protrusion seed points. DBSCAN clustering is performed on the road depression seed points and road convex seed points respectively to obtain the corresponding depression cluster regions and convex cluster regions. Then, the grid where the depression cluster region is located is used as the road depression grid, and the grid where the convex cluster region is located is used as the road convex grid.

[0008] Preferably, all straight line segments are clustered and merged to obtain multiple linear cracks. Then, the ratio of the average gradient magnitude of the linear crack grid to a preset gradient threshold is used as the point cloud influence factor of the linear crack. The connected regions corresponding to the mesh of mesh cracks are taken as mesh cracks, and the ratio of the average local RMSE value of the mesh crack mesh to the preset RMSE threshold is taken as the point cloud influence factor of the mesh crack. The concave and convex cluster regions are used as road surface concave patches and road surface convex patches, respectively. Then, the ratio of the absolute value of the average relative elevation of all point clouds in the road surface concave patches and road surface convex patches to the preset elevation threshold is used as the point cloud influence factor of the corresponding road surface concave patches and road surface convex patches.

[0009] Preferably, the method for image verification of each defective region includes: The linear crack is converted to grayscale to obtain the corresponding first grayscale image. Then, samples are taken along the linear crack at equal pixel intervals to obtain multiple sampling points. At each sampling point, pixels in the first grayscale image within a preset range on both sides of its normal direction are taken to form a local image strip. For each local image strip, the Canny operator is used to extract the strip edge and calculate the edge density of the strip edge. If the edge density is greater than the preset value, the linear crack is retained; otherwise, it is deleted. The mesh cracks are converted to grayscale to obtain the corresponding second grayscale image. For each pixel in the second grayscale image, the neighboring pixels of the pixel are found at a preset direction and step size. Then, the pixel value of the pixel and the corresponding neighboring pixel are combined to form a grayscale pair. After normalizing the occurrence frequency of all grayscale pairs, a GLCM matrix is ​​generated. Then, the entropy of the GLCM matrix is ​​calculated. If the entropy is greater than a preset value, the mesh crack is retained; otherwise, it is deleted. Convert road surface depressions or bumps to the HSV color space to obtain the corresponding first color value map, extract the luminance channel of the first color value map, and calculate the luminance variance based on the luminance channel. The surrounding road area is divided around the depression or protrusion of the road surface. The surrounding road area is converted to the HSV color space to obtain a second color value map, and the luminance channel of the second color value map is extracted. The luminance ratio of the first color value map and the second color value map is calculated based on the corresponding luminance channel. If the luminance variance is greater than the preset value and the luminance ratio meets the preset conditions, the road surface depression patch or road surface protrusion patch is retained; otherwise, it is deleted.

[0010] Preferably, the method for calculating the confidence level corresponding to each verified defect region based on the point cloud impact factor includes: For each defective area that passes verification, the defective area is converted into a grayscale image, and the image confidence level is calculated based on the grayscale image. PCA principal component analysis is performed on the horizontal coordinates of all point clouds in the defect area to obtain the corresponding principal axis direction. All point clouds are projected onto the principal axis direction, and multiple cross-sectional lines are generated by sampling at equal intervals along the principal axis direction. Extract the brightness sequence and high-order sequence of all cross-sectional lines, calculate the rank correlation coefficient between the high-order sequence and brightness sequence of each cross-sectional line, and then calculate the mean of all rank correlation coefficients to obtain the multimodal similarity. Image correction coefficients are calculated based on the entropy / brightness ratio of the edge density / GLCM matrix and multimodal similarity. Then, confidence is calculated based on the image correction coefficients and point cloud influence factors.

[0011] Preferably, the method for calculating image confidence based on grayscale images includes: For each defect area that passes verification, calculate the corresponding illumination uniformity and exposure coefficient; The Sobel operator is used to calculate the gray-level gradient magnitude of each pixel in the gray-level image corresponding to the defect area. Then, the gradient operator is obtained by normalizing the mean of all gray-level gradient magnitudes and calculating the gradient factor based on the gradient operator. Image confidence is calculated based on illumination uniformity, exposure factor, and gradient factor.

[0012] Preferably, the formula for calculating the grayscale gradient magnitude is as follows:

[0013]

[0014] , in, and All are Sobel operators. Let G be the pixel matrix of the grayscale image, and G be the grayscale gradient magnitude. This represents the gradient in the x-direction obtained after the convolution operation. This represents the gradient in the y-direction obtained after the convolution operation.

[0015] Preferably, the method for extracting the brightness sequence and high-order sequence of all cross-sectional lines includes: Set a strip half-width on the cross-section line, and project the point cloud in an area with a width of 2 times the strip half-width onto the cross-section line, with the cross-section line as the center, to obtain multiple projected point clouds. The projected point cloud is resampled at a fixed sampling interval to obtain multiple first sampling points. The relative elevations corresponding to all the first sampling points are sorted from smallest to largest and combined into an elevation sequence. The cross-sectional line is projected onto the orthophoto to obtain the projected image, and the projected image is converted to the HSV color space. Multiple second sampling points are generated at a fixed sampling interval. All second sampling points are sorted in ascending order of brightness channel in the HSV color space and combined into a brightness sequence.

[0016] Preferably, the formula for calculating the image correction coefficient is as follows:

[0017] in, The preset correction strength coefficient, For image trust, denoted as edge density or the entropy or brightness ratio of the GLCM matrix, and S is the multimodal similarity.

[0018] Preferably, the method for locating each defect area with a confidence level greater than a preset value based on road centerline data includes: Establish a Frenet coordinate system with the road centerline, and project the point cloud of all defect areas onto the Frenet coordinate system to obtain the rectangular range of each defect area; Transform the rectangular area to the global coordinate system to obtain the start and end positions and coverage of each defect region.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention detects defect areas based on point cloud data, then projects the point cloud onto an orthophoto image for image verification, and calculates the image correction coefficient of the defect area on the orthophoto image as the basis for evaluating the image recognition result, and calculates the confidence of the defect recognition. In this way, the detection of the defect area is determined through two verifications, thereby reducing the false detection rate and improving the accuracy.

[0020] (2) Accurately locate the defect area based on the road center data to facilitate road maintenance management personnel in finding it. Attached Figure Description

[0021] Figure 1 This diagram illustrates a multimodal road defect detection method based on vehicle-mounted mobile measurement, according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] like Figure 1 As shown in the figure, this invention proposes a multimodal road distress detection method based on vehicle-mounted mobile measurement, including: Acquire road surface point cloud data and road centerline data, and calculate the relative elevation of all point clouds in the road surface point cloud data; The three-dimensional laser point cloud of the road to be inspected was synchronously acquired using a vehicle-mounted mobile measurement system. The three-dimensional laser point cloud was downsampled with a voxel grid of 0.01m, and then noise was removed by statistical filtering. After that, the road surface point cloud was extracted from the original three-dimensional laser point cloud containing non-road surface points such as shoulders, guardrails, vehicles, and vegetation using the M-EstimatorSampleConsensus method.

[0024] A sliding window (with a selectable window size of 0.5m × 0.5m and a step size of 0.25m) is set on the road surface point cloud. For each sliding window point cloud, the MSAC algorithm is used to fit a local reference plane, and the distance Z_x (positive for upward normal and negative for downward normal) from each point cloud within the sliding window to the local reference plane is calculated. This Z_x value is used as the relative elevation of the point cloud. For cases where points fall in overlapping areas of multiple windows, this invention employs an inverse distance weighting method, using the point cloud... Planar distance to the center point of each sliding window The reciprocal of the square is used as a weighting coefficient. The core idea is that the closer a point cloud is to the center of a sliding window, the greater the influence (higher weight) of the reference plane fitted by that window on the point cloud.

[0025] For point clouds Its coordinates are The relative elevation corresponding to the i-th sliding window is It can calculate the distance from the point cloud P to the center point of each sliding window. Planar distance:

[0026] Then calculate the weight coefficients for each window. :

[0027] Calculate the final Z_x value at this point: = As the relative elevation of that point, if point cloud P happens to fall exactly at the center of a sliding window, the distance is... If the value is 0, then the weight is 1, meaning the weight is calculated using this sliding window. The value is used as the relative elevation of that point. This allows the extracted road surface point clouds and the relative elevation of each point cloud to be combined into road surface point cloud data.

[0028] Read the road centerline of the road segment to be detected (text format, format is vertex name, X(m), Y(m), Z(m), mileage(m)). If the vertex spacing is too large (>1m), perform cubic spline interpolation to encrypt the data, generate integer mileage points with a spacing of 1m, and calculate the tangent direction vector t, normal direction vector n (perpendicular to the tangent in the horizontal plane, pointing to the right) and curvature of each point in the direction from smallest to largest mileage.

[0029] The road surface point cloud is regularized into a grid. Then, based on the relative elevation of the point cloud in each grid, all grids are divided into multiple defect grid classes. Then, the corresponding defect area is generated according to each defect grid class and the corresponding point cloud influence factor is calculated. Road defects mainly include four types: linear cracks, network cracks, road surface depressions, and road surface bumps. The detection methods for these four types of defects in this invention are as follows: (1) Detection of linear cracks in roads The road surface point cloud is regularized into a 0.01m grid. The average relative elevation of all point clouds within each grid is taken as the elevation value of that grid, resulting in an elevation map Z(u,v) composed of all grids and their corresponding elevation values. For each grid, the elevation gradient magnitude map G is generated by calculating the elevation change rate, i.e., the gradient magnitude, between adjacent grids.

[0030]

[0031]

[0032] Where u and v are the indices of the elevation map in the x and y directions, respectively, after the point cloud has been regularized into a grid. This represents the grid spacing of a single grid cell in the X direction. This represents the grid spacing of a single grid in the Y direction, as shown in the example. and The values ​​are all 0.01m. This represents the gradient magnitude of the elevation map in the x-direction (horizontal). This represents the gradient magnitude of the elevation map in the y-direction (vertical).

[0033] The point cloud elevation gradient magnitude map is binarized (threshold is 0.02m), and the probabilistic Hough transform is used to extract straight line segments. The minimum line segment length is set to 0.5m, the maximum gap is set to 0.1m, and the grid containing the straight line segments is divided into linear crack grids.

[0034] Since actual road cracks are often curved or broken lines, a single long straight line cannot fully describe them. This invention clusters and merges all extracted straight line segments: The Euclidean distance and the included angle between the endpoints of any two straight line segments are calculated. If the Euclidean distance between the endpoints is less than 0.2m and the included angle is less than 30°, these two straight line segments are connected as components of the same crack. This process is repeated until no new connections are made, thus piecing together discrete short straight line segments into multiple complete linear cracks. For each linear crack, the ratio of the average gradient magnitude to the gradient threshold of all linear crack meshes within its coverage area is calculated and used as the point cloud influence factor of the linear crack.

[0035] Where G_mean is the average gradient magnitude of all linear crack meshes corresponding to the linear crack, and t_g is the gradient threshold, which is 0.02.

[0036] (2) Detection of network cracks in roads After removing the point cloud containing the linear crack, a neighborhood is divided for each point cloud with a search radius of 0.15m. For each point cloud P, all point clouds within the corresponding neighborhood (search radius 0.15m) are searched. The least squares method is used to fit a plane to these point clouds to obtain the equation of the fitted plane. Then, the distance d_i from each point cloud within the neighborhood to the fitted plane is calculated, and the local RMSE value (which reflects the roughness of the local area) is calculated. , where n is the number of point clouds in the neighborhood.

[0037] Map the local RMSE values ​​of all point clouds onto the mesh to generate residual feature maps, and binarize them. Perform morphological closing operations (dilation followed by erosion) on the binarized residual feature maps, connect neighboring regions to obtain multiple connected regions, and calculate the orientation histogram for each connected region (divided into 8 directions, counting the number of points at each direction angle). Calculate the entropy H of the orientation histogram.

[0038] Where p_i represents the probability in the i-th direction, the magnitude of entropy H is used to determine whether it is a network crack. If the entropy H is greater than the threshold, it is judged as a network crack (the threshold of H is 2.5). For network cracks, the ratio of the mean of the local RMSE values ​​of all point clouds in the connected region to the RMSE threshold is calculated as the point cloud influence factor of the network crack.

[0039]

[0040] Wherein, RMSE_mean is the mean of the local RMSE values ​​of all point clouds within the connected regions of the mesh crack, and t_rmse is the RMSE threshold, which is 0.005.

[0041] (3) Road depression detection After removing point clouds containing linear and network cracks, each point cloud is divided into neighborhoods with a search radius of 0.15m. The standard deviation of the relative elevation (roughness, also representing the roughness of that point cloud) and the local RMSE value of all point clouds in the neighborhood of each point cloud are calculated. Then, the skewness of the roughness of each point cloud in the mesh is calculated.

[0042] in σ_rough is the mean roughness of all point clouds within the mesh, and σ_rough is the standard deviation of the roughness.

[0043] Points with a roughness greater than 0.005m, a local RMSE value greater than 0.005, and a skewness less than -0.5 are used as seed points for road depressions. DBSCAN clustering is performed on these seed points (neighborhood radius set to 0.15m, minimum sample size 30), resulting in multiple depression cluster regions. Within each cluster region, all scattered seed points are grouped into connected regions to obtain complete road depression patches. The grid covered by these patches is defined as the road depression grid. For each road depression patch, the ratio of the average depth (absolute value of average relative elevation) of all point clouds within the corresponding road depression grid to a depth threshold is calculated as the point cloud influence factor for the road depression patch.

[0044] Where deep_mean is the absolute value of the average relative elevation of all point clouds within the road depression grid, and t_deep is the depth threshold, which is 0.015m.

[0045] (4) Road bump detection After removing point clouds from areas containing linear cracks, network cracks, and road depressions, the method is the same as for road depression detection. Points with a standard deviation of relative elevation greater than 0.005m, RMSE greater than 0.005, and skewness greater than 0.5 are used as seed points for road protrusions. DBSCAN clustering is performed on these seed points to obtain multiple protrusion cluster regions. Within each cluster region, all scattered seed points are grouped into connected regions to obtain complete road protrusion patches. The grid covered by these patches is defined as the road protrusion grid. For each protrusion patch, the ratio of the average height (absolute value of average relative elevation) of all point clouds within the corresponding grid to a height threshold is calculated as the point cloud influence factor for the protrusion patch.

[0046] Where height_mean is the absolute value of the average relative elevation of all points within the road raised grid, and t_height is the height threshold, which is 0.015m.

[0047] All point clouds and grids are projected onto the orthophoto, and each defect region is image-verified. Then, the confidence level of each defect region that passes the verification is calculated based on the point cloud influence factor. After all point clouds and meshes are projected onto the orthophoto (orthophoto resolution is 0.01m), the coordinates of the point clouds and meshes on the orthophoto will not change. However, the orthophoto can be used to verify four types of road defects and determine whether the defect detection is correct.

[0048] (1) Image verification of linear cracks in roads The corresponding linear crack is converted to grayscale to obtain the first grayscale image. Sampling is performed along the linear crack at equal pixel intervals (2 pixels). At each sampling point, a local image strip is formed by taking a range of 5 pixels in the normal direction of the first grayscale image. For each local image strip, the edge is extracted using the Canny operator, with a low threshold of 50 and a high threshold of 150. The edge density edge_density (number of edge pixels / total number of pixels in the strip) is calculated. At the same time, the median of the edge densities of all strips is taken as the image influence factor of the linear crack.

[0049] If the edge density edge_density is greater than the preset threshold, it indicates that the area where the linear crack is located has obvious linear texture, the verification is passed, and the linear crack is retained; otherwise, it is deleted. The preset threshold t_edge is set to 0.2.

[0050] (2) Image verification of road network cracks The corresponding mesh cracks are grayscaled to 16 levels to obtain a second grayscale image. For each pixel (x, y) in the second grayscale image, its neighboring pixels at a distance of 1 and directions of 0°, 45°, 90°, and 135° are found. Grayscale pairs (i, j) are formed based on the grayscale values ​​of the pixel and its neighbors. The occurrence frequency of each grayscale pair (i, j) is counted and normalized to construct a GLCM matrix. Then, the entropy of the GLCM matrix is ​​calculated.

[0051] Where p(i,j) is the probability of gray-level pair (i,j). If the entropy is greater than the preset threshold (usually 2.8), it indicates that the texture is complex (irregular) and meets the characteristics of a mesh crack, thus passing the verification and retaining the mesh crack; otherwise, it is deleted.

[0052] (3) Image verification of road depressions The corresponding road surface depression patches are converted to the HSV color space to obtain the first color value map. The luminance channel V of the first color value map is extracted, and the luminance variance var_V is calculated (in the depression area, under sunlight or ambient light, the side wall will produce shadows, and the bottom of the pit may darken due to water accumulation or exposed materials, which will lead to uneven luminance distribution in the image and a large luminance variance). Then, the annular area 0.2m outside the road surface depression patch is taken as the surrounding road surface area, and the surrounding road surface area is converted to the HSV color space to obtain the second color value map, and the luminance channel of the second color value map is extracted.

[0053] Calculate the ratio of the mean brightness of the brightness channel of the road surface depression to the mean brightness of the brightness channel of the surrounding road surface area (due to geometric depressions, water and dust easily accumulate at the bottom of the depression, or shadows are generated on the side walls, resulting in a significantly lower image brightness than the surrounding normal road surface, so the brightness ratio < 1, and the further it deviates from 1, the more realistic the depression). If the brightness variance var_V is greater than the threshold and the ratio < 0.85, the verification is passed and the road surface depression is retained; otherwise, it is deleted. The threshold is set to 0.02.

[0054] (4) Image verification of road protrusions The verification steps are the same as those for road depression images, but the judgment method is different. When verifying images of road protrusions, if the brightness variance var_V is greater than the threshold and ratio>1.15, the verification is passed and the road protrusion patch is retained; otherwise, it is deleted. The threshold t_v is set to 0.02.

[0055] For each defect area that passes verification, calculate its image confidence level in the orthophoto. First, calculate the uniformity of illumination. (Normalized entropy of brightness histogram)

[0056] Under uniform illumination, the brightness histogram is dispersed, and the entropy value is high. Approaching 1; Under strong shadows or excessively strong light, the brightness histogram is concentrated. Smaller, of which Let k be the probability of brightness level k.

[0057] Calculate the exposure ratio :

[0058] in, This indicates the number of overexposed pixels (pixel value greater than 240). This indicates the number of underexposed pixels (pixel value less than 15). This represents the total number of pixels.

[0059] Calculate the average gradient magnitude First, the image of the defect area under orthophoto is converted into a grayscale image. Then, the Sobel operator is used to calculate the grayscale gradient magnitude of each pixel in the grayscale image. The Sobel operator includes... , For grayscale image I, calculate

[0060]

[0061] in, Let G be the pixel matrix of the grayscale image, and G be the grayscale gradient magnitude. This represents the horizontal gradient obtained after the convolution operation, specifically the magnitude of the pixel grayscale gradient in the horizontal direction. This represents the y-direction (vertical) gradient obtained after the convolution operation, that is, the pixel grayscale gradient magnitude in the vertical direction.

[0062] The gradient factor is calculated as follows:

[0063] in, The result is obtained by averaging the grayscale gradient magnitude G of all pixels and then normalizing it. Take 0.1.

[0064] The grayscale gradient magnitude reflects the image's sharpness and texture richness; a blurry image has a low gradient value, while a sharp image has a high gradient value.

[0065] Ultimately, the image trust level is obtained. The calculation is as follows:

[0066] For the point cloud data of each defect region, perform PCA principal component analysis on the horizontal coordinates of all point clouds in the defect region. Input the horizontal coordinates (X,Y) of all point clouds in the defect region and calculate the mean of the horizontal coordinates of all point clouds in the region. The mean value is subtracted from the coordinates of each point cloud to shift the data center to the origin (i.e., decentralization). Then, a 2×2 covariance matrix is ​​constructed and singular value decomposition (SVD) is performed to obtain multiple unit eigenvectors and their corresponding eigenvalues. All eigenvalues ​​are sorted from largest to smallest, and the unit eigenvector corresponding to the largest eigenvalue is taken as the principal axis direction of the defect region.

[0067] Project all point clouds onto the principal axis direction, and sample at equal intervals along the principal axis direction to generate cross-sectional lines. The maximum and minimum values ​​in the principal axis direction can be determined based on the coordinates of the point clouds. Based on the maximum and minimum values ​​in the principal axis direction, the number of cross-sectional lines is usually determined to be 5-8 in the principal axis direction.

[0068] On each cross-section line, a strip with a half-width of 0.03m is set. Centered on the cross-section line, all points within a 0.06m wide interval (twice the half-width of the strip) are projected onto the cross-section line, resulting in multiple projected point clouds. The projected coordinates of each point cloud are recorded. The corresponding relative elevation H is then resampled at a sampling interval of 0.01m (if there are no point cloud points at the sampling point, they are filled by linear interpolation), finally yielding the relative elevation sequence. , will relatively high program column The point cloud is arranged in ascending order of relative elevation. The position (rank) of each element after sorting is recorded to obtain the ordered column. .

[0069] For the same cross-section, the cross-section line is projected onto the orthophoto image, and sampling points are generated at a fixed sampling interval of 1 pixel (the sampling interval is consistent with the point cloud resampling interval). At each sampling point, the luminance channel V value in the HSV color space is taken, and finally, a luminance sequence is obtained. The number of elements in the brightness sequence is consistent with the number of elements in the relative height sequence of the point cloud. The brightness sequence... Luminance channel in Arrange the elements in ascending order, record the rank of each element after sorting, and obtain the ordered column. .

[0070] For each cross section, calculate the Spearman rank correlation coefficient between the point cloud height sequence and the image brightness sequence. :

[0071] in , They are respectively ordered columns order sequence The mean. The value range is [0,1] (if it is negative, it is assigned the value 0), multimodal similarity The rank correlation coefficient of all cross-sections within the defect region. The mean.

[0072] In addition, under special circumstances: ① If the number of sampling points sampled on a certain cross-section is less than 5, then the cross-section is discarded.

[0073] ② If the projected length or width of the defect area along the principal axis is less than 0.15m, effective cross-sectional sampling cannot be performed. In this case, centroid radial sampling is used instead: with the centroid of the area as the center, four rays (0°, 45°, 90°, 135°) are taken, and each ray extends outward to the boundary. The relative elevation and brightness sequences on the rays are extracted, and the rank correlation coefficient is calculated and averaged.

[0074] The image correction factor is then calculated using the following formula:

[0075] in A value of 0.8 is used; this coefficient controls the influence of the imagery on the overall result. This indicates the image impact factor.

[0076] Final confidence score calculation and labeling: The final confidence score is calculated using the following formula:

[0077] Point_factor represents the point cloud influence factor corresponding to the defect area. If the confidence level is greater than or equal to 0.65, proceed to the next step. If the confidence level is less than 0.65, the location is temporarily fixed and marked as "requires manual review". After manual review, continue to the next step.

[0078] Based on the road centerline data, each defect area with a confidence level greater than a preset value is located to obtain a road defect location map.

[0079] Based on the given road centerline, establish a Frenet coordinate system (s, d), where s is the mileage along the centerline and d is the signed lateral offset perpendicular to the road centerline (positive for the right and negative for the left). Iterate through each segment of the centerline (1m segments, with the starting and ending points of each segment serving as mileage points). For each point (X, Y) in the point cloud data, calculate its distance to each segment's centerline, recording the line segment corresponding to the shortest distance and its perpendicular point. s represents the starting mileage of the corresponding line segment and the distance from the perpendicular point to the starting point, while d represents the shortest distance (determined by the direction of the normal vector at the perpendicular point).

[0080] For all point clouds within the defect region, calculate their coordinates (s_i, d_i) in the Frenet coordinate system to obtain the sets s_i and d_i.

[0081] For a specific diseased area, calculate the maximum and minimum values ​​of s and d: max(s_i), max(d_i), min(s_i), min(d_i) to obtain the corner points of the bounding rectangle. Simultaneously, calculate the coordinates of the center point of the rectangle's range, the rectangle's width, and the rectangle's length based on these corner points. Center point coordinates:

[0082]

[0083] Rectangle width:

[0084] Rectangle length:

[0085] Therefore, the rectangular range of the diseased area in the Frenet coordinate system is determined based on the center point coordinates, rectangle width, and rectangle length. Then, the rectangular range is reprojected onto the global coordinate system.

[0086]

[0087] Where x_globle and y_globle are the coordinates in the global coordinate system, x_center and y_center are the global coordinates of the midpoint of the rectangle projected onto the center line of the road, t is the tangent vector of the global coordinate system (positive in the direction of mileage increase), and n is the normal vector of the global coordinate system (positive on the right).

[0088] This allows us to locate the start and end points (including the starting and ending mileage) and the corresponding coverage area of ​​the defect area in the global coordinate system. By combining all the defect areas, we can obtain a road defect location map with all defect areas on the road under test.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multimodal road distress detection method based on vehicle-mounted mobile measurement, characterized in that: include: Acquire road surface point cloud data and road centerline data, and calculate the relative elevation of all point clouds in the road surface point cloud data; The road surface point cloud is regularized into a grid, and then based on the relative elevation of the point cloud within each grid, including: For each grid, the average relative elevation of all point clouds within the grid is taken as the elevation value of that grid. Then, the gradient magnitude of each grid is calculated to generate a point cloud elevation gradient magnitude map. The point cloud elevation gradient magnitude map is binarized, and the probability Hough transform is used to extract line segments. The grid where all line segments are located is divided into linear crack grids. For each point cloud, a neighborhood is divided. The local RMSE values ​​of all point clouds within the neighborhood are mapped to the grid to generate a residual feature map. The residual feature map is binarized and then morphological closing operation is performed to obtain multiple connected regions. A direction histogram is statistically analyzed for each connected region, and the entropy H of the direction histogram is calculated. The grid of connected regions with entropy H greater than a preset value is divided into a mesh-like crack grid. For each point cloud, the point cloud is divided into road depression seed points or road convex seed points according to the relative elevation of all point clouds in the corresponding neighborhood. DBSCAN clustering is performed on the road depression seed points and road convex seed points respectively to obtain the corresponding depression cluster regions and convex cluster regions. Then, the grid where the depression cluster region is located is used as the road depression grid, and the grid where the convex cluster region is located is used as the road convex grid. All meshes are divided into multiple defect mesh classes. Then, for each defect mesh class, a corresponding defect region is generated, and the corresponding point cloud influence factor is calculated, including: All straight line segments are clustered and merged to obtain multiple linear cracks. Then, the ratio of the average gradient magnitude of the linear crack grid to the preset gradient threshold is used as the point cloud influence factor of the linear crack. The connected regions corresponding to the mesh of mesh cracks are taken as mesh cracks, and the ratio of the average local RMSE value of the mesh crack mesh to the preset RMSE threshold is taken as the point cloud influence factor of the mesh crack. The concave cluster region and the convex cluster region are respectively regarded as road surface concave patches and road surface convex patches. Then, the ratio of the absolute value of the average relative elevation of all point clouds in the road surface concave patches and road surface convex patches to the preset elevation threshold is used as the point cloud influence factor of the corresponding road surface concave patches and road surface convex patches. All point clouds and meshes are projected onto the orthophoto image, and image verification is performed for each defect region. Then, the confidence level corresponding to each verified defect region is calculated based on the point cloud influence factor, including: For each defective area that passes verification, the defective area is converted into a grayscale image, and the image confidence level is calculated based on the grayscale image. PCA principal component analysis is performed on the horizontal coordinates of all point clouds in the defect area to obtain the corresponding principal axis direction. All point clouds are projected onto the principal axis direction, and multiple cross-sectional lines are generated by sampling at equal intervals along the principal axis direction. Extract the brightness sequence and high-order sequence of all cross-sectional lines, calculate the rank correlation coefficient between the high-order sequence and brightness sequence of each cross-sectional line, and then calculate the mean of all rank correlation coefficients to obtain the multimodal similarity. Image correction coefficients are calculated based on the entropy / brightness ratio of the edge density / GLCM matrix and multimodal similarity, and then confidence is calculated based on the image correction coefficients and point cloud influence factors. Based on the road centerline data, each defect area with a confidence level greater than a preset value is located to obtain a road defect location map.

2. The multimodal road distress detection method based on vehicle-mounted mobile measurement as described in claim 1, characterized in that: Methods for image verification of each defect area include: The linear crack is converted to grayscale to obtain the corresponding first grayscale image. Then, samples are taken along the linear crack at equal pixel intervals to obtain multiple sampling points. At each sampling point, pixels in the first grayscale image within a preset range on both sides of its normal direction are taken to form a local image strip. For each local image strip, the Canny operator is used to extract the strip edge and calculate the edge density of the strip edge. If the edge density is greater than the preset value, the linear crack is retained; otherwise, it is deleted. The mesh cracks are converted to grayscale to obtain the corresponding second grayscale image. For each pixel in the second grayscale image, the neighboring pixels of the pixel are found at a preset direction and step size. Then, the pixel value of the pixel and the corresponding neighboring pixel are combined to form a grayscale pair. After normalizing the occurrence frequency of all grayscale pairs, a GLCM matrix is ​​generated. Then, the entropy of the GLCM matrix is ​​calculated. If the entropy is greater than a preset value, the mesh crack is retained; otherwise, it is deleted. Convert road surface depressions or bumps to the HSV color space to obtain the corresponding first color value map, extract the luminance channel of the first color value map, and calculate the luminance variance based on the luminance channel. The surrounding road area is divided around the depression or protrusion of the road surface. The surrounding road area is converted to the HSV color space to obtain a second color value map, and the luminance channel of the second color value map is extracted. The luminance ratio of the first color value map and the second color value map is calculated based on the corresponding luminance channel. If the luminance variance is greater than the preset value and the luminance ratio meets the preset conditions, the road surface depression patch or road surface protrusion patch is retained; otherwise, it is deleted.

3. The multimodal road defect detection method based on vehicle-mounted mobile measurement as described in claim 1, characterized in that: Methods for calculating image confidence based on grayscale images include: For each defect area that passes verification, calculate the corresponding illumination uniformity and exposure coefficient; The Sobel operator is used to calculate the gray-level gradient magnitude of each pixel in the gray-level image corresponding to the defect area. Then, the gradient operator is obtained by normalizing the mean of all gray-level gradient magnitudes and calculating the gradient factor based on the gradient operator. Image confidence is calculated based on illumination uniformity, exposure factor, and gradient factor.

4. The multimodal road defect detection method based on vehicle-mounted mobile measurement as described in claim 3, characterized in that: The formula for calculating the grayscale gradient magnitude is as follows: , in, and All are Sobel operators. Let G be the pixel matrix of the grayscale image, and G be the grayscale gradient magnitude. This represents the gradient in the x-direction obtained after the convolution operation. This represents the gradient in the y-direction obtained after the convolution operation.

5. The multimodal road defect detection method based on vehicle-mounted mobile measurement as described in claim 1, characterized in that: Methods for extracting the brightness sequence and high program sequence of all cross-sectional lines include: Set a strip half-width on the cross-section line, and project the point cloud in an area with a width of 2 times the strip half-width onto the cross-section line, with the cross-section line as the center, to obtain multiple projected point clouds. The projected point cloud is resampled at a fixed sampling interval to obtain multiple first sampling points. The relative elevations corresponding to all the first sampling points are sorted from smallest to largest and combined into an elevation sequence. The cross-sectional line is projected onto the orthophoto to obtain the projected image, and the projected image is converted to the HSV color space. Multiple second sampling points are generated at a fixed sampling interval. All second sampling points are sorted in ascending order of brightness channel in the HSV color space and combined into a brightness sequence.

6. The multimodal road defect detection method based on vehicle-mounted mobile measurement as described in claim 1, characterized in that: The methods for locating defect areas with a confidence level greater than a preset value based on road centerline data include: Establish a Frenet coordinate system with the road centerline, and project the point cloud of all defect areas onto the Frenet coordinate system to obtain the rectangular range of each defect area; Transform the rectangular area to the global coordinate system to obtain the start and end positions and coverage of each defect region.

Citation Information

Patent Citations

  • Intelligent inspection method and system for road infrastructure

    CN121392351A

  • Laser-vision progressive detection method for underwater concrete joints and cracks

    CN122244660A