Road pit detection method and system

Abstract

The invention relates to the technical field of road defect detection, in particular to a road pit detection method and system, and the method comprises the following steps: S1, obtaining an original image flow of a road surface through a vehicle-mounted multispectral camera, and carrying out the motion artifact elimination to generate a vibration compensation image; s2, generating a shadow suppression image through asymmetric gamma correction; s3, adopting a dual-threshold connected domain analysis method to extract candidate pothole regions; s4, when the contrast difference value exceeds a texture mutation threshold value, determining that the area is a surface damaged area; s5, extracting a continuous pixel cluster statistical area proportion, and when the proportion is greater than 60%, determining that the feature is an effective pothole feature; and S6, calculating the minimum enclosing ellipse eccentricity rate and the ellipse area ratio, and generating a final pothole detection report. According to the method, through a multi-source information fusion and multi-stage feature extraction method, high-precision identification and standardized output of the pothole area in a complex road environment are realized, and the detection accuracy and the application reliability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of road defect detection, and particularly to a road pothole detection method and system. Background Art

[0002] With the rapid development of urban roads and high-speed traffic infrastructure, the road surface quality is directly related to driving safety and road maintenance costs; especially under high-frequency heavy load traffic or harsh climate conditions, local damages such as potholes and cracks are likely to form on the road surface. If not discovered and repaired in time, it is easy to cause damage to the vehicle suspension system, an increase in traffic accidents, and even further deterioration of the road surface structure; to improve the efficiency of road inspection, automated detection technologies based on visual sensing and multi-source data fusion have been gradually developed, attempting to identify and locate road surface defects through in-vehicle image acquisition devices; however, due to multiple interference factors such as vehicle vibration, light change, shadow occlusion, and asphalt material differences in the road detection environment, the accuracy and robustness of traditional image processing methods are still greatly limited in practical applications.

[0003] When dealing with the problem of road pothole detection, the existing technologies generally have the following technical problems: First, there is a lack of an effective vibration artifact elimination mechanism, resulting in blurred images and blurred edges, affecting subsequent feature extraction; second, the adaptability to shadow and light interference is weak, prone to misjudgment; third, relying only on two-dimensional image information, lacking a three-dimensional perception of the road surface structure morphology, it is difficult to accurately judge the spatial continuity and true morphology of potholes; fourth, there is a lack of a unified morphological geometry evaluation mechanism, and it is impossible to standardize the expression of the detection results. Therefore, there is an urgent need for a road pothole detection method and system to solve the above problems. Summary of the Invention

[0004] Based on the above purpose, the present invention provides a road pothole detection method and system.

[0005] A road pothole detection method includes the following steps: S1: Obtain the original road surface image stream through an in-vehicle multi-spectral camera, synchronously collect the vibration data of a six-axis inertial measurement unit, and generate a vibration compensation image by eliminating motion artifacts based on the vibration data and a preset asphalt pavement material feature library; S2: Separate the visible light and near-infrared spectral channels of the vibration compensation image, dynamically construct a shadow probability map according to the spectral reflectance difference, and generate a shadow suppression image through asymmetric gamma correction; S3: Perform a directionally adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use a double-threshold connected component analysis method to extract candidate pothole regions; S4: Select a circular reference area around the candidate pothole area, calculate the contrast difference of the gray-level co-occurrence matrix, and determine it as the surface damage area when the contrast difference exceeds the texture mutation threshold; S5: Perform 3D point cloud reconstruction on the surface damage area and calculate the surface curvature gradient distribution, extract the statistical area ratio of continuous pixel clusters, and determine it as an effective pothole feature when the ratio is greater than 60%; S6: For the area corresponding to the effective pothole feature, calculate its minimum bounding ellipse eccentricity and ellipse area ratio. When the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85, mark the area as a complete pothole area with regular morphology, and incorporate its parameter information into the final pothole detection report.

[0006] Optionally, the specific steps of S1 include: S11: Obtain the original road surface image stream through a multi-spectral camera fixedly installed at the mid-axis position of the front bumper of the vehicle; S12: Synchronously collect the three-axis acceleration and three-axis angular velocity data of the vehicle at the image acquisition moment through a six-axis inertial measurement unit installed in the center of the vehicle chassis, with a data sampling frequency of 200 Hz; at the same time, linearly interpolate and register the time stamp of each frame of image with the three-axis acceleration and three-axis angular velocity data at the corresponding moment to ensure that the vibration data corresponds one-to-one with the image frames; S13: Input the three-axis acceleration and three-axis angular velocity data collected by the six-axis inertial measurement unit into an image transformation function based on forward motion compensation to compensate and transform the original image pixel positions. The expression of the image transformation function is: , where is the pixel gray value of the original image; is the pixel gray value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position; S14: Based on the image transformation function, perform pixel-level resampling and translation transformation on the original image stream frame by frame to generate a vibration compensation image that eliminates vehicle vibration and displacement artifacts.

[0007] Optionally, the specific steps of S2 include: S21: Split the vibration compensation image obtained in S1 by band channels, and extract the visible light channel image and the near-infrared channel image; S22: Calculate the spectral reflectance difference of each pixel point, and the formula is: , where is the normalized reflectance difference of the pixel point in the visible light and near-infrared channels; is the gray value of the pixel point in the visible light channel; is the gray value of the pixel in the near-infrared channel; is a constant to prevent the denominator from being zero, and its value is 0.01; S23: According to the normalized reflectance difference calculate the shadow probability , and its expression is: , when is greater than the set shadow determination threshold , then the corresponding pixel is regarded as having shadow occlusion; S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow suppression image.

[0008] Optionally, the specific steps of S3 are as follows: S31: Perform a directionally adjustable gradient operation on the shadow suppression image obtained in S2. Use Sobel convolution kernels with different directions to perform convolution on each pixel of the image, calculate the gradient components in each direction, and finally synthesize the gradient magnitude at each pixel position to obtain a complete gradient magnitude heat map; S32: Based on the gradient magnitude heat map, select two thresholds, a high threshold and a low threshold. Specifically, the high threshold is set to 70% of the maximum value of all pixel gradient magnitudes in the gradient magnitude heat map, and the low threshold is set to 30% of the maximum value of all pixel gradient magnitudes in the gradient magnitude heat map. And screen out the preliminary strong gradient pixel points through the high threshold; S33: Use the preliminary strong gradient pixel points as initial seed points, and perform iterative expansion on the pixels whose gradient magnitudes around the seed points exceed the low threshold based on the connected component growth algorithm. Successively include adjacent pixels into the current connected region until there is no pixel position in the region that can continue to expand, so as to obtain multiple independent candidate pothole connected regions; S34: Calculate the pixel area of each candidate pothole connected region, and remove the regions with an area less than 50 pixels in the connected region as noise regions, and retain the regions with an area greater than or equal to 50 pixels as the final candidate pothole regions.

[0009] Optionally, the specific steps of S31 are as follows: S311: Denote the shadow suppression image obtained in S2 as ; S312: For the image , respectively construct Sobel convolution kernel matrices with directions of 0°, 45°, 90°, and 135°. The corresponding gradient directions are the horizontal direction, the upper right direction, the vertical direction, and the upper left direction. Use the convolution kernels in each direction to perform two-dimensional convolution operations on the image to obtain the gradient response images in each direction; S313: Weightedly synthesize the gradient response images in multiple directions to obtain a direction-sensitive comprehensive gradient magnitude image for each pixel point, defined as the gradient magnitude heat map. .

[0010] Optionally, the specific steps of S33 are as follows: S331: Use the strong gradient pixel points screened out in S32 as initial seed points to establish a seed point set. S332: Take out the seed points from the seed point set one by one, calculate the gradient magnitude similarity between the current seed point and its surrounding 8-neighborhood pixels. When the gradient magnitude is not less than the preset low threshold, mark the corresponding adjacent pixels as belonging to the same connected domain as the current seed point, and add the adjacent pixels to the seed point set. S333: Repeat S332, continuously take out new seed points from the seed point set for expansion until the seed point set is empty. At this time, an independent and complete connected domain is obtained. S334: Traverse all the strong gradient pixel point sets obtained in S32, and repeat the iterative process of S331 to S333 for the unmarked strong gradient pixel points until all the strong gradient pixel points are processed. Finally, multiple independent candidate pothole connected regions are obtained.

[0011] Optionally, the specific steps of S4 are as follows: S41: For each candidate pothole region obtained in S3, expand it outward along the outer edge contour of the candidate region with a fixed width. The fixed width is set to 1 times the equivalent radius of the candidate region, and the region between the outer boundary and the candidate region contour is used as the annular reference region. S42: Calculate the gray-level co-occurrence matrices inside the candidate pothole region and inside the annular reference region respectively. S43: Calculate the texture contrast values for the two gray-level co-occurrence matrices respectively. The texture contrast inside the candidate pothole region is denoted as , and the texture contrast of the annular reference region is denoted as . S44: Calculate the texture contrast difference between the candidate pothole region and the annular reference region; when the difference exceeds the preset texture mutation threshold , determine that the candidate pothole region is a surface damage region.

[0012] Optionally, the specific steps of S5 are as follows: S51: Based on the surface damage area determined in S4, using multiple two-dimensional images continuously acquired by a camera during the vehicle's forward movement, combined with the pose information recorded during the vehicle's driving process, through a three-dimensional reconstruction algorithm based on the principle of structured light projection, high-density three-dimensional point cloud reconstruction of the surface damage area is achieved to obtain three-dimensional point cloud data; S52: Perform surface curvature calculation on the three-dimensional point cloud data. Using each point cloud data point and its neighboring points within a radius of 5 mm, a local quadratic surface is fitted by the least squares method, the Gaussian curvature value at each data point is calculated, and the spatial gradient of the obtained curvature value is obtained to generate a corresponding surface curvature gradient distribution map; S53: In the curvature gradient distribution map, perform binary processing with a preset curvature gradient threshold to obtain a binary image. The curvature gradient threshold is set to 0.05 mm⁻¹; and the eight-neighborhood connectivity analysis method is used for the binary image to extract multiple continuous pixel cluster regions; S54: Calculate the pixel area ratio for each of the extracted continuous pixel cluster regions; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damage area has effective pitting features.

[0013] Optionally, the specific steps of S6 are as follows: S61: For the region determined to have effective pitting features in S5, extract the set of all boundary pixel coordinates, and use the least squares ellipse fitting algorithm to perform the least enclosing ellipse fitting process on this pixel set to obtain the major axis length, minor axis length, center coordinates, and orientation angle of the fitting ellipse; S62: Calculate the eccentricity of the ellipse according to the fitting result. The calculation formula is: , where is the eccentricity of the fitting ellipse; is the major axis length of the fitting ellipse; is the minor axis length of the fitting ellipse; S63: Calculate the area ratio of the fitting ellipse to the actual pitting pixel area, which is defined as the ratio of the area of the effective pitting pixel area to the area of the fitting ellipse. The expression is: , where is the ellipse area ratio; is the actual pixel area within the effective pitting feature area; is the area of the fitting ellipse; S64: Judge the geometric characteristics of the fitting ellipse. If the eccentricity and the area ratio are satisfied, then mark the current region as a morphologically regular complete pitting region, write its spatial position, contour coordinates, major and minor axis dimensions, eccentricity, and area ratio into the standardized data structure, and finally generate a detection report containing the parameters of all effective pitting regions.

[0014] A road pothole detection system for implementing the above-mentioned road pothole detection method, comprising the following modules: Image acquisition module: It is used to obtain the multi-channel raw image stream of the road surface in real time through a multi-spectral camera installed in front of the vehicle, and synchronously start a six-axis inertial measurement unit to collect acceleration and angular velocity data during the vehicle operation; Artifact elimination module: Connected to the image acquisition module, based on the acceleration and angular velocity data, and combined with a preset asphalt pavement material feature library, it performs motion artifact elimination processing on the raw image stream and outputs a vibration compensation image; Spectral separation and shadow suppression module: Connected to the artifact elimination module, it is used to split the vibration compensation image into a visible light channel map and a near-infrared channel map, construct a shadow probability map according to the pixel reflectivity difference, and then generate a shadow suppression image through asymmetric gamma correction; Gradient heat map and candidate extraction module: Connected to the spectral separation and shadow suppression module, it is used to perform multi-directional gradient operations on the shadow suppression image to generate a gradient magnitude heat map, and use the double-threshold connected component growth algorithm to extract candidate pothole regions; Texture analysis module: Connected to the gradient heat map and candidate extraction module, it is used to generate an annular reference region outside the candidate pothole region, and calculate the contrast difference of the gray-level co-occurrence matrix inside and outside the region respectively. When the difference exceeds the set threshold, it is determined as a surface damage region; 3D reconstruction and structure determination module: Connected to the texture analysis module, it is used to perform image sequence point cloud reconstruction on the surface damage region, calculate the surface curvature gradient distribution, extract continuous high-curvature pixel clusters, calculate their area ratio, and determine whether it is an effective pothole feature; Morphological evaluation and result output module: Connected to the 3D reconstruction and structure determination module, it is used to perform minimum bounding ellipse fitting on the effective pothole region, calculate the eccentricity and area ratio of the ellipse, and output the final pothole detection report according to the set conditions.

[0015] Advantages of the present invention: In the present invention, by constructing an on-vehicle data acquisition system integrating multi-spectral imaging and six-axis inertial measurement, and combining a vibration artifact elimination algorithm based on material perception, the stability of the image quality is significantly improved, and the problems of image blurring and information deviation caused by vehicle movement during the traditional image detection process are solved. At the same time, by introducing the spectral reflectivity difference analysis and asymmetric gamma correction methods, the influence of shadow interference on the image brightness and contrast is effectively suppressed, and the adaptability to complex road surface environments is enhanced.

[0016] In the present invention, at the feature extraction and determination level, by integrating technical means such as multi-directional gradient operation, double-threshold connected component analysis, texture contrast judgment, point cloud curvature analysis, and ellipse shape fitting, a bottom-up multi-level screening mechanism is constructed, realizing multi-dimensional comprehensive recognition and judgment of pothole areas from two dimensions to three dimensions and from grayscale to shape, ensuring that the finally output pothole detection results have high integrity, accuracy, and standard expression ability, and meeting the requirements of high-precision real-time detection of the road intelligent inspection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only those of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 Schematic diagram of the road pothole detection method according to an embodiment of the present invention; Figure 2 Schematic diagram of the road pothole detection system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The present invention will be described in detail below with reference to the drawings and specific embodiments. At the same time, it should be noted here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments. For some well-known technologies, those skilled in the art can also adopt other alternative methods for implementation; and the drawings are only for more specifically describing the embodiments, and are not intended to specifically limit the present invention.

[0020] It should be pointed out that in the specification, when referring to "an embodiment", "embodiment", "exemplary embodiment", "some embodiments", etc., it indicates that the described embodiment may include specific features, structures, or characteristics, but not necessarily every embodiment includes such specific features, structures, or characteristics. In addition, when combining an embodiment to describe a specific feature, structure, or characteristic, implementing such a feature, structure, or characteristic in combination with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the relevant art.

[0021] Generally, the terms can be understood at least in part from their use in the context. For example, at least in part depending on the context, the term "one or more" used herein can be used to describe any feature, structure, or characteristic in a singular sense, or can be used to describe a combination of features, structures, or characteristics in a plural sense. In addition, the term "based on" can be understood as not necessarily intended to convey a set of exclusive factors, but rather, at least in part depending on the context, allowing the existence of other factors that may not be explicitly described.

[0022] As shown Figure 1 in the figure, a road pothole detection method includes the following steps: S1: Obtain the original road surface image stream through an in-vehicle multi-spectral camera, synchronously collect the vibration data of a six-axis inertial measurement unit, and perform motion artifact elimination based on the vibration data and a preset asphalt pavement material feature library to generate a vibration compensation image; S2: Separate the visible light and near-infrared spectral channels of the vibration compensation image, dynamically construct a shadow probability map according to the spectral reflectance difference, and generate a shadow suppression image through asymmetric gamma correction; S3: Perform a directionally adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use a double-threshold connected component analysis method to extract candidate pothole regions; S4: Select an annular reference region outside the candidate pothole region, calculate the contrast difference of the gray-level co-occurrence matrix, and determine it as a surface damage region when the contrast difference exceeds the texture mutation threshold; S5: Perform three-dimensional point cloud reconstruction on the surface damage region and calculate the surface curvature gradient distribution, extract the statistical area ratio of continuous pixel clusters, and determine it as an effective pothole feature when the ratio is greater than 60%; S6: For the region corresponding to the effective pothole feature, calculate its minimum bounding ellipse eccentricity and ellipse area ratio, and generate a final pothole detection report when the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85.

[0023] Specifically, S1 includes: S11: Obtain the original road surface image stream through a multi-spectral camera fixedly installed at the mid-axis position of the vehicle's front bumper. The multi-spectral camera has three channels of visible light, near-infrared, and short-wave infrared, with a frame rate of 60fps and an image resolution of not less than 1920×1080 pixels, and uses a hardware trigger method to synchronously start image acquisition; S12: Synchronously collect the three-axis acceleration and three-axis angular velocity data of the vehicle at the image acquisition moment through a six-axis inertial measurement unit (IMU) installed at the center of the vehicle chassis. The three-axis acceleration includes longitudinal acceleration, lateral acceleration, and vertical acceleration along the vehicle's forward direction; the three-axis angular velocity includes roll angular velocity rotating around the vehicle's forward direction axis, pitch angular velocity rotating around the transverse axis, and yaw angular velocity rotating around the vertical axis, and the data sampling frequency is 200Hz; at the same time, linearly interpolate and register the time stamp of each frame of image with the three-axis acceleration and three-axis angular velocity data at the corresponding moment to ensure that the vibration data corresponds one-to-one with the image frames; S13: Input the three-axis acceleration and three-axis angular velocity data collected by the six-axis inertial measurement unit into an image transformation function based on forward motion compensation to perform compensation transformation on the original image pixel positions. The expression of the image transformation function is: , where the pixel position offset is calculated by the formula: ; , where is the pixel gray value of the original image; is the pixel gray value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position; are the longitudinal and lateral accelerations of the vehicle at the image acquisition moment, with the unit of ; are the roll angular velocity and yaw angular velocity of the vehicle at the image acquisition moment, with the unit of ; is the acceleration compensation coefficient calibrated for different asphalt materials in the asphalt pavement material feature library, with the unit of pixel·s² / m; is the angular velocity compensation coefficient calibrated for different asphalt materials in the asphalt pavement material feature library, with the unit of pixel·s / rad; S14: Perform pixel-level resampling and translation transformation on the original image stream frame by frame based on the image transformation function to generate a vibration compensation image that eliminates vehicle vibration and displacement artifacts; Through the above steps, it is possible to ensure a high degree of consistency between image acquisition and vibration data in the time and space dimensions, and achieve targeted image correction under the influence of material differences, thereby improving the stability and clarity of the pothole area image, and contributing to the subsequent spectral analysis and geometric recognition accuracy.

[0024] S2 specifically includes: S21: Split the vibration compensation image obtained in S1 by band channels, extract the visible light channel image and the near-infrared channel image. The multispectral camera realizes spectral imaging of each band through the filter array. The channel wavelengths are: the visible light channel wavelength range is 400nm to 700nm, and the near-infrared channel wavelength range is 750nm to 950nm. The split images maintain the same resolution and pixel coordinates as the original image; S22: Calculate the spectral reflectance difference of each pixel point. The formula is: , where is the normalized reflectance difference of the pixel point in the visible light and near-infrared channels; is the gray value of the pixel point in the visible light channel; is the gray value of the pixel point in the near-infrared channel; is a constant to prevent the denominator from being zero, with a value of 0.01; S23: Calculate the shadow probability according to the normalized reflectance difference , and its expression is: , when Greater than the set shadow determination threshold , then the corresponding pixel is regarded as having shadow occlusion; S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow suppression image. The asymmetric gamma correction function is defined as: , where is the gray value of the corresponding pixel in the shadow suppression image; is the original pixel gray value in the vibration compensation image; is the enhanced gamma coefficient set for the shadow area, with a value range of 1.2 to 1.5; is the reduction gamma coefficient set for the non-shadow area, with a value range of 0.7 to 0.9; Through the above steps, channel separation and fusion processing of multi-spectral information can be achieved, the shadow area can be distinguished based on spectral reflection characteristics, and non-linear compensation can be performed using an adaptive gray mapping method, effectively suppressing the interference of shadows on subsequent gradient operations and edge extraction processes, and improving the accuracy of pothole feature extraction.

[0025] S3 specifically includes: S31: Perform directionally adjustable gradient operations on the shadow suppression image obtained in S2. Use Sobel convolution kernels with different directions to perform convolution on the image pixel by pixel, calculate the gradient components in each direction, and finally synthesize the gradient amplitude at each pixel position to obtain a complete gradient amplitude heat map; S32: Based on the gradient amplitude heat map, select two thresholds, a high threshold and a low threshold. Specifically, the high threshold is set to 70% of the maximum value of all pixel gradient amplitudes in the gradient amplitude heat map, and the low threshold is set to 30% of the maximum value of all pixel gradient amplitudes in the gradient amplitude heat map, and initially strong gradient pixel points are selected through the high threshold; S33: Use the initially strong gradient pixel points as initial seed points, and perform iterative expansion on the pixels whose gradient amplitudes around the seed points exceed the low threshold based on the connected component growth algorithm. Successively include adjacent pixels in the current connected region until there is no pixel position in the region that can continue to expand, thereby obtaining multiple independent candidate pothole connected regions; S34: Calculate the pixel area of each candidate pothole connected region, and remove the regions with an area less than 50 pixels in the connected region as noise regions, and retain the regions with an area greater than or equal to 50 pixels as the final candidate pothole regions; Through the above steps, comprehensive coverage of the gradient operation direction and accurate identification of the pothole region boundary can be achieved. At the same time, the connected component growth method with double thresholds effectively suppresses noise interference and ensures the reliability of subsequent pothole feature region determination.

[0026] S31 specifically includes: S311: Denote the shadow suppression image obtained in S2 as , where represents the position coordinates of pixels in the image; S312: For the image , construct Sobel convolution kernel matrices with directions of 0°, 45°, 90°, and 135° respectively. The corresponding gradient directions are the horizontal direction, the upper right direction, the vertical direction, and the upper left direction. Perform two-dimensional convolution operations on the image using the convolution kernels in each direction to obtain gradient response images in each direction, which are denoted as the horizontal direction gradient map , the upper right direction gradient map , the vertical direction gradient map ; the upper left direction gradient map ; S313: Weightedly synthesize the gradient response images in multiple directions to obtain a direction-sensitive comprehensive gradient magnitude image for each pixel point, defined as the gradient magnitude heat map , and its calculation formula is as follows: , where: is the finally obtained gradient magnitude heat map; are the weight coefficients in the corresponding directions respectively, and each weight coefficient satisfies . In this embodiment, the values of the four weight coefficients are uniformly taken as 0.25; through the above steps, the edge information in multiple directions can be integrated, the response ability to road pothole contours in different directions can be improved, and a stable and clear gradient magnitude heat map can be obtained, providing a high-precision gradient basis for subsequent connected component analysis.

[0027] S33 specifically includes: S331: Use the strong gradient pixel points screened out in S32 as initial seed points to establish a seed point set S332: Take out the seed points from the seed point set one by one, calculate the gradient magnitude similarity between the current seed point and its surrounding 8-neighborhood pixels. When the gradient magnitude is not less than the preset low threshold, mark the corresponding adjacent pixels as belonging to the same connected component as the current seed point, and add the adjacent pixels to the seed point set; The determination method of gradient magnitude similarity adopts the following formula: ; where, is the marking function for whether the current neighborhood pixel point is included in the connected component. The value of 1 means it is included in the current connected component, and the value of 0 means it is not included; is the preset low threshold; S333: Repeatedly execute S332, continuously extract new seed points from the seed point set for expansion until the seed point set is empty. At this time, an independent and complete connected domain is obtained; S334: Traverse all the strong gradient pixel point sets obtained in S32, and repeat the iterative process of S331 to S333 for the unmarked strong gradient pixel points until all strong gradient pixel points are processed. Finally, multiple independent candidate pothole connected regions are obtained. Through the above steps, the spatial continuity of the gradient amplitude can be fully utilized to accurately expand the connected domain, making the boundaries of the candidate pothole regions more accurate and effectively improving the accuracy of subsequent pothole feature determination.

[0028] S4 specifically includes: S41: For each candidate pothole region obtained in S3, expand it outward with a fixed width along the outer edge contour of the candidate region to obtain an outer boundary. The fixed width is set to 1 times the equivalent radius of the candidate region, and the region between the outer boundary and the candidate region contour is used as an annular reference region; S42: Calculate the gray-level co-occurrence matrices within the candidate pothole region and within the annular reference region respectively. The gray-level co-occurrence matrix is constructed based on the pixel gray levels of the gray-scale image. The gray levels are quantized into 256 levels, and the gray-level co-occurrence matrix is constructed in the horizontal direction with a pixel spacing of 1 pixel to obtain the co-occurrence matrix within the candidate region and the co-occurrence matrix within the annular reference region; S43: Calculate the texture contrast values of the two gray-level co-occurrence matrices respectively. Among them, the texture contrast within the candidate pothole region is denoted as , and the texture contrast within the annular reference region is denoted as ; The calculation formula for texture contrast is: , where is the texture contrast of the corresponding region; is the probability value of the co-occurrence of pixel pairs with gray levels of and in the gray-level co-occurrence matrix of the corresponding region; is the total number of gray levels. In this embodiment, ; S44: Calculate the texture contrast difference between the candidate pothole region and the annular reference region, and judge whether there is an obvious texture mutation in the region based on this difference . The calculation formula for the texture contrast difference is: ; When the difference exceeds the preset texture mutation threshold , it is determined that the candidate pothole region is a surface damage region, where the texture mutation threshold is the empirical threshold obtained according to the calibration of different road material surfaces; through the above steps, the texture difference between the pothole area and the surrounding normal road surface area can be accurately identified, and the false alarm area can be effectively excluded, thereby improving the reliability and accuracy of the identification of the damaged area on the pothole surface.

[0029] S5 specifically includes: S51: Based on the surface damaged area determined in S4, using multiple two-dimensional images continuously acquired by the camera during the vehicle's forward movement, combined with the pose information recorded during the vehicle's driving process, the high-density three-dimensional point cloud reconstruction of the surface damaged area is realized through a three-dimensional reconstruction algorithm based on the principle of structured light projection, and three-dimensional point cloud data is obtained; S52: Perform surface curvature calculation on the three-dimensional point cloud data. Using each point cloud data point and its neighboring points within a radius of 5 mm, the local quadratic surface is fitted by the least squares method, the Gaussian curvature value at each data point is calculated, and the spatial gradient of the obtained curvature value is obtained to generate the corresponding surface curvature gradient distribution map; the calculation formula for Gaussian curvature is: , where is the Gaussian curvature at the point cloud data point; are the principal curvatures obtained by fitting the local quadratic surface respectively, with the unit of ; S53: In the curvature gradient distribution map, perform binarization processing with a preset curvature gradient threshold to obtain a binary image. The curvature gradient threshold is set to 0.05 mm⁻¹; and the eight-neighborhood connectivity analysis method is used for the binary image to extract multiple continuous pixel cluster regions; S54: Calculate the pixel area ratio for each of the extracted continuous pixel cluster regions; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damaged area has effective pothole characteristics; the pixel area ratio is the ratio of the number of pixels in the current pixel cluster region to the total number of pixels in the entire surface damaged area. The specific calculation formula is: , where is the area ratio of the current pixel cluster region; is the number of pixels in the current pixel cluster region; is the total number of pixels in the entire surface damaged area; through the above steps, the three-dimensional structure and curvature characteristics of the surface damaged area can be identified and quantified with high precision, the misidentification of non-pothole characteristics can be effectively suppressed, and the reliability and accuracy of road pothole feature detection can be significantly improved.

[0030] S6 specifically includes: S61: For the area determined as an effective pothole feature in S5, extract the set of all boundary pixel coordinates, and use the least-squares ellipse fitting algorithm to perform the least enclosing ellipse fitting process on this pixel set, obtaining the major axis length, minor axis length, center coordinates, and orientation angle of the fitting ellipse; S62: Calculate the eccentricity of the ellipse according to the fitting result, and its calculation formula is: , where in the formula, is the eccentricity of the fitting ellipse; is the major axis length of the fitting ellipse; is the minor axis length of the fitting ellipse; the units are all pixels, and satisfy the condition ; S63: Calculate the area ratio of the fitting ellipse to the actual pothole pixel area, which is defined as the ratio of the area of the effective pothole pixel area to the area of the fitting ellipse, and the expression is: , where in the formula, is the ellipse area ratio; is the actual pixel area within the effective pothole feature area; is the area of the fitting ellipse; S64: Judge the geometric characteristics of the fitting ellipse. If the eccentricity and the area ratio are satisfied, then mark the current area as a morphologically regular complete pothole area, and write its spatial position, contour coordinates, major and minor axis dimensions, eccentricity, and area ratio into the standardized data structure, and finally generate a detection report containing the parameters of all effective pothole areas; Through the above steps, it is possible to achieve precise geometric evaluation of the pothole area based on effective geometric fitting combined with morphological feature indicators, thereby ensuring that the output detection results have high reliability and integrity.

[0031] As Figure 2 shown, a road pothole detection system for implementing the above-mentioned road pothole detection method includes the following modules: Image acquisition module: used to obtain the multi-channel raw image stream of the road surface in real time through a multi-spectral camera installed in front of the vehicle, and synchronously start the six-axis inertial measurement unit to collect the acceleration and angular velocity data during the vehicle operation; Artifact elimination module: connected to the image acquisition module, based on the acceleration and angular velocity data, and combined with the preset asphalt pavement material feature library, perform motion artifact elimination processing on the raw image stream, and output a vibration compensation image; Spectral separation and shadow suppression module: connected to the artifact elimination module, used to split the vibration compensation image into a visible light channel map and a near-infrared channel map, construct a shadow probability map according to the pixel reflectance difference, and then generate a shadow suppression image through asymmetric gamma correction; Gradient Heat Map and Candidate Extraction Module: Connected to the Spectral Separation and Shadow Suppression Module, it is used to perform multi-directional gradient operations on the shadow-suppressed image to generate a gradient magnitude heat map, and adopt a double-threshold connected component growth algorithm to extract candidate pothole regions; Texture Analysis Module: Connected to the Gradient Heat Map and Candidate Extraction Module, it is used to generate an annular reference region around the candidate pothole region, and calculate the difference in gray-level co-occurrence matrix contrast inside and outside the region respectively. When the difference exceeds the set threshold, it is determined as the surface damage region; 3D Reconstruction and Structure Judgment Module: Connected to the Texture Analysis Module, it is used to perform point cloud reconstruction of the image sequence for the surface damage region, calculate the surface curvature gradient distribution, extract continuous high-curvature pixel clusters, calculate their area ratio, and determine whether it is an effective pothole feature; Morphological Evaluation and Result Output Module: Connected to the 3D Reconstruction and Structure Judgment Module, it is used to perform minimum bounding ellipse fitting on the effective pothole region, calculate the eccentricity and area ratio of the ellipse, and output the final pothole detection report according to the set conditions.

[0032] This invention covers any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To enable the public to have a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments of this invention. However, those skilled in the art can fully understand this invention without the description of these details. Additionally, well-known methods, processes, procedures, components, and circuits are not described in detail to avoid unnecessary confusion to the essence of this invention.

[0033] The above are only the preferred embodiments of this invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of this invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this invention.

Claims

1. A method for detecting road potholes, characterized in that, It includes the following steps: S1: Obtain the original road surface image stream through an in-vehicle multi-spectral camera, synchronously collect the vibration data of a six-axis inertial measurement unit, and generate a vibration compensation image by eliminating motion artifacts based on the vibration data and a preset asphalt pavement material feature library; S2: Separate the visible light and near-infrared spectral channels of the vibration compensation image, dynamically construct a shadow probability map according to the spectral reflectance difference, and generate a shadow suppression image through asymmetric gamma correction; S3: Perform a directionally adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use the double-threshold connected component analysis method to extract candidate pothole regions; S4: Select an annular reference region outside the candidate pothole region, calculate the contrast difference of the gray-level co-occurrence matrix, and determine it as a surface damage region when the contrast difference exceeds the texture mutation threshold; S5: Perform three-dimensional point cloud reconstruction on the surface damage region and calculate the surface curvature gradient distribution, extract the statistical area ratio of continuous pixel clusters, and determine it as an effective pothole feature when the ratio is greater than 60%; S6: For the region corresponding to the effective pothole feature, calculate its minimum bounding ellipse eccentricity and ellipse area ratio. When the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85, mark the region as a complete pothole region with regular morphology, and incorporate its parameter information into the final pothole detection report.

2. The road pothole detection method according to claim 1, characterized in that The specific content of S1 includes: S11: Obtain the original road surface image stream through a multi-spectral camera fixedly installed at the mid-axis position of the front bumper of the vehicle; S12: Synchronously collect the three-axis acceleration and three-axis angular velocity data of the vehicle at the image acquisition moment through a six-axis inertial measurement unit installed at the center of the vehicle chassis, and the data sampling frequency is 200Hz; at the same time, linearly interpolate and register the time stamp of each frame of image with the three-axis acceleration and three-axis angular velocity data at the corresponding moment to ensure that the vibration data corresponds to each image frame one by one; S13: Input the three-axis acceleration and three-axis angular velocity data collected by the six-axis inertial measurement unit into the image transformation function based on forward motion compensation to perform compensation transformation on the original image pixel positions. The expression of the image transformation function is: , where is the pixel gray value of the original image; is the pixel gray value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position; S14: Perform pixel-level resampling and translation transformation on the original image stream frame by frame based on the image transformation function to generate a vibration compensation image that eliminates vehicle vibration and displacement artifacts.

3. The method for detecting road potholes according to claim 1, characterized in that, The specific content of S2 includes: S21: Split the vibration compensation image obtained in S1 according to the band channels, and extract the visible light channel image and the near-infrared channel image; S22: Calculate the spectral reflectance difference of each pixel point, with the formula: , where is the normalized reflectance difference of the pixel point in the visible light and near-infrared channels; is the gray value of the pixel point in the visible light channel; is the gray value of the pixel point in the near-infrared channel; is a constant to prevent the denominator from being zero, with a value of 0.01; S23: Calculate the shadow probability according to the normalized reflectance difference The expression is as follows: When is greater than the set shadow determination threshold , the corresponding pixel is regarded as having shadow occlusion;​ S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow suppression image.

4. A road pothole detection method according to claim 1, characterized in that, The specific content of S3 includes: S31: Perform a directionally adjustable gradient operation on the shadow suppression image obtained in S2. Use Sobel convolution kernels with different directions to convolve each pixel of the image, calculate the gradient components in each direction, and finally synthesize the gradient magnitude at each pixel position to obtain a complete gradient magnitude heat map; S32: Based on the gradient magnitude heat map, select two thresholds, a high threshold and a low threshold. Specifically, the high threshold is set to 70% of the maximum value of the gradient magnitudes of all pixels in the gradient magnitude heat map, and the low threshold is set to 30% of the maximum value of the gradient magnitudes of all pixels in the gradient magnitude heat map, and preliminarily screen out strong gradient pixel points through the high threshold; S33: Use the initially identified strong gradient pixels as initial seed points, and iteratively expand the pixels whose gradient magnitudes exceed the low threshold around the seed points based on the connected component growth algorithm. Sequentially incorporate adjacent pixels into the current connected region until there are no more pixels within the region that can be further expanded, thereby obtaining multiple independent candidate pothole connected regions; S34: Calculate the pixel area of each candidate pothole connected region. Discard the regions with an area less than 50 pixels as noise regions, and retain the regions with an area greater than or equal to 50 pixels as the final candidate pothole regions.

5. The method for detecting road potholes according to claim 4, wherein, The specific steps of S31 include: S311: Denote the shadow suppression image obtained in S2 as ; S312: For the image , construct Sobel convolution kernel matrices with directions of 0°, 45°, 90°, and 135° respectively. The corresponding gradient directions are the horizontal direction, the upper right direction, the vertical direction, and the upper left direction. Use the convolution kernels in each direction to perform two-dimensional convolution operations on the image to obtain gradient response images in each direction; S313: Weightedly synthesize the gradient response images in multiple directions to obtain the direction-sensitive comprehensive gradient magnitude image for each pixel point, which is defined as the gradient magnitude heat map .

6. The road pothole detection method according to claim 4, wherein, The specific steps of S33 include: S331: Use the strong gradient pixels selected in S32 as initial seed points and establish a seed point set; S332: Sequentially take out the seed points from the seed point set and calculate the gradient magnitude similarity between the current seed point and its 8-neighborhood pixels. When the gradient magnitude is not less than the preset low threshold, mark the corresponding adjacent pixels as belonging to the same connected domain as the current seed point, and add the adjacent pixels to the seed point set; S333: Repeat S332, continuously take out new seed points from the seed point set for expansion until the seed point set is empty. At this time, an independent and complete connected domain is obtained; S334: Traverse all the strong gradient pixel sets obtained in S32, and repeat the iterative process of S331 to S333 for the unmarked strong gradient pixels until all strong gradient pixels are processed. Finally, obtain multiple independent candidate pothole connected regions.

7. A road pothole detection method according to claim 1, characterized in that The specific steps of S4 include: S41: For each candidate pothole region obtained in S3, expand it outward along the outer edge contour of the candidate region with a fixed width to obtain an outer boundary. The fixed width is set to 1 times the equivalent radius of the candidate region, and use the region between the outer boundary and the candidate region contour as an annular reference region; S42: Calculate the gray-level co-occurrence matrices within the candidate pothole region and within the annular reference region respectively; S43: Calculate the texture contrast values of the two gray-level co-occurrence matrices respectively, where the texture contrast inside the candidate pothole region is denoted as , and the texture contrast of the annular reference region is denoted as ; S44: Calculate the difference in texture contrast between the candidate pitted area and the annular reference area ; When the difference exceeds a preset texture mutation threshold it is determined that the candidate pitted area is a surface damage area.

8. A road pothole detection method according to claim 1, characterized in that The specific steps of S5 include: S51: Based on the surface damage region determined in S4, use multiple two-dimensional images continuously acquired by a camera during the vehicle's forward movement, combined with the pose information recorded during the vehicle's driving process, and implement high-density three-dimensional point cloud reconstruction of the surface damage region through a three-dimensional reconstruction algorithm based on the structured light projection principle to obtain three-dimensional point cloud data; S52: Perform surface curvature calculation on the three-dimensional point cloud data. Use each point cloud data point and its neighborhood points within a radius of 5 mm, fit a local quadratic surface through the least squares method, calculate the Gaussian curvature value at each data point, and obtain the spatial gradient of the obtained curvature values to generate a corresponding surface curvature gradient distribution map; S53: In the curvature gradient distribution map, perform binary processing with a preset curvature gradient threshold to obtain a binary image. The curvature gradient threshold is set to 0.05 mm⁻¹; and use the eight-neighborhood connected analysis method for the binary image to extract multiple continuous pixel cluster regions; S54: Calculate the pixel area ratio for each of the extracted continuous pixel cluster regions; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damage region has effective pothole features.

9. The method for detecting road potholes according to claim 1, wherein The specific steps of S6 are as follows: S61: For the region determined to have effective pothole features in S5, extract the set of all boundary pixel coordinates, and use the least-squares ellipse fitting algorithm to perform the minimum bounding ellipse fitting process on this pixel set to obtain the major axis length, minor axis length, center coordinates, and orientation angle of the fitting ellipse. S62: Calculate the eccentricity of the ellipse according to the fitting result, and its calculation formula is: , where is the eccentricity of the fitted ellipse; is the major axis length of the fitted ellipse; is the minor axis length of the fitted ellipse; S63: Calculate the area ratio of the fitted ellipse to the actual pitted pixel area, which is defined as the ratio of the area of the effective pitted pixel area to the area of the fitted ellipse, and the expression is: , where is the ellipse area ratio; is the actual pixel area within the effective pitted feature area; is the area of the fitted ellipse; S64: Determine the geometric features of the fitted ellipse. If the eccentricity and the area ratio are satisfied, mark the current area as a complete pitted area with regular morphology, write its spatial position, contour coordinates, major and minor axis dimensions, eccentricity, and area ratio into the standardized data structure, and finally generate a detection report containing the parameters of all effective pitted areas.

10. A road pothole detection system for implementing a road pothole detection method according to any one of claims 1-9, characterized in that, It includes the following modules: Image acquisition module: It is used to obtain the multi-channel raw image stream of the road surface in real time through a multi-spectral camera installed in front of the vehicle, and synchronously start the six-axis inertial measurement unit to collect the acceleration and angular velocity data during the vehicle operation. Artifact elimination module: Connected to the image acquisition module, based on the acceleration and angular velocity data, and combined with the preset asphalt pavement material feature library, perform motion artifact elimination processing on the raw image stream and output the vibration compensation image. Spectral separation and shadow suppression module: Connected to the artifact elimination module, it is used to split the vibration compensation image into a visible light channel map and a near-infrared channel map, construct a shadow probability map according to the pixel reflectivity difference, and then generate a shadow suppression image through asymmetric gamma correction. Gradient heat map and candidate extraction module: Connected to the spectral separation and shadow suppression module, it is used to perform multi-directional gradient operations on the shadow suppression image to generate a gradient magnitude heat map, and use the double-threshold connected component growth algorithm to extract candidate pothole regions. Texture analysis module: Connected to the gradient heat map and candidate extraction module, it is used to generate an annular reference region outside the candidate pothole region, and calculate the difference in the gray-level co-occurrence matrix contrast inside and outside the region. When the difference exceeds the set threshold, it is determined as the surface damage region. 3D reconstruction and structure determination module: Connected to the texture analysis module, it is used to perform image sequence point cloud reconstruction on the surface damage region, calculate the surface curvature gradient distribution, extract continuous high-curvature pixel clusters, calculate their area ratios, and determine whether they are effective pothole features. Morphological evaluation and result output module: Connected to the 3D reconstruction and structure determination module, it is used to perform the minimum bounding ellipse fitting on the effective pothole region, calculate the eccentricity and area ratio of the ellipse, and output the final pothole detection report according to the set conditions.

Images