A method and system for detecting road potholes

Abstract

The present invention relates to the technical field of road defect detection, and specifically relates to a road pothole detection method and system, which includes the following steps: S1: Obtain the original image stream of the road surface through an in-vehicle multi-spectral camera, and perform motion artifact elimination to generate a vibration compensation image; S2: Generate a shadow suppression image through asymmetric gamma correction; S3: Use the double-threshold connected component analysis method to extract candidate pothole regions; S4: Determine it as a surface damage region when the contrast difference exceeds the texture mutation threshold; S5: Extract the statistical area ratio of continuous pixel clusters, and determine it as an effective pothole feature when the ratio is greater than 60%; S6: Calculate its minimum bounding ellipse eccentricity and ellipse area ratio, and generate a final pothole detection report. With the present invention, through the multi-source information fusion and multi-stage feature extraction method, high-precision identification and standardized output of pothole regions in complex road environments are achieved, improving the detection accuracy and application reliability.

Description

Technical Field

[0001] The present invention relates to the technical field of road defect detection, and in particular to a road pothole detection method and system. Background Art

[0002] With the rapid development of urban roads and high-speed transportation infrastructure, road surface quality is directly related to driving safety and road maintenance costs. Especially under high-frequency heavy traffic or severe weather conditions, the road surface is prone to local damage such as potholes and cracks. If not discovered and repaired in time, it is easy to cause damage to the vehicle suspension system, an increase in traffic accidents, and even further deterioration of the road surface structure. To improve the efficiency of road inspections, automated detection technologies based on visual sensing and multi-source data fusion are gradually being developed, attempting to identify and locate road surface defects through on-board image acquisition equipment. However, due to multiple interference factors in the road inspection environment, such as vehicle vibration, lighting changes, shadow occlusion, and differences in asphalt materials, the accuracy and robustness of traditional image processing methods are still greatly limited in practical applications.

[0003] Existing technologies for pothole detection face the following common technical challenges: First, they lack effective vibration artifact removal mechanisms, resulting in image blur and edge blurring, which hinders subsequent feature extraction; second, they are poorly adaptable to shadow and lighting interference, which can easily lead to misjudgments; third, they rely solely on two-dimensional image information and lack a three-dimensional understanding of the road surface structure, making it difficult to accurately determine the spatial continuity and true shape of potholes; and fourth, they lack a unified morphological and geometric evaluation mechanism, making it impossible to standardize the expression of detection results. Therefore, a pothole detection method and system are urgently needed to address these issues. Summary of the Invention

[0004] Based on the above objectives, the present invention provides a road pothole detection method and system.

[0005] A road pothole detection method comprises the following steps:

[0006] S1: The vehicle-mounted multispectral camera acquires the original image stream of the road surface, and simultaneously collects vibration data from the six-axis inertial measurement unit. Based on the vibration data and a preset asphalt pavement material feature library, motion artifacts are eliminated to generate a vibration-compensated image.

[0007] S2: Separate the visible and near-infrared spectral channels of the vibration-compensated image, dynamically construct a shadow probability map based on the spectral reflectance difference, and generate a shadow-suppressed image through asymmetric gamma correction;

[0008] S3: Perform a direction-adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use a dual-threshold connected domain analysis method to extract candidate pothole areas;

[0009] S4: Select an annular reference area around the candidate pothole area, calculate the contrast difference of the gray-level co-occurrence matrix, and determine it as the surface damage area when the contrast difference exceeds the texture mutation threshold;

[0010] S5: Perform 3D point cloud reconstruction on the surface damage area and calculate the surface curvature gradient distribution, extract the statistical area ratio of continuous pixel clusters, and determine it as an effective pothole feature when the ratio is greater than 60%;

[0011] S6: For the area corresponding to the effective pothole feature, calculate its minimum enclosing ellipse eccentricity and ellipse area ratio. When the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85, mark this area as a complete pothole area with regular morphology and include its parameter information in the final pothole detection report.

[0012] Optionally, the specific steps of S1 include:

[0013] S11: Obtain the original road surface image stream through a multispectral camera fixedly installed at the mid-axis position of the front bumper of the vehicle;

[0014] S12: Synchronously collect the three-axis acceleration and three-axis angular velocity data of the vehicle at the image acquisition moment through a six-axis inertial measurement unit installed at the center of the vehicle chassis, and the data sampling frequency is 200Hz; at the same time, linearly interpolate and register the time stamp of each frame of image with the three-axis acceleration and three-axis angular velocity data at the corresponding moment to ensure that the vibration data corresponds to each image frame one by one;

[0015] S13: Input the three-axis acceleration and three-axis angular velocity data collected by the six-axis inertial measurement unit into an image transformation function based on forward motion compensation to perform compensation transformation on the original image pixel positions. The expression of the image transformation function is: , where is the pixel gray value of the original image; is the pixel gray value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position;

[0016] S14: Perform pixel-level resampling and translation transformation on the original image stream frame by frame based on the image transformation function to generate a vibration compensation image that eliminates vehicle vibration and displacement artifacts.

[0017] Optionally, the specific steps of S2 include:

[0018] S21: Split the vibration compensation image obtained in S1 according to the band channels, and extract the visible light channel image and the near-infrared channel image;

[0019] S22: Calculate the spectral reflectance difference of each pixel point, and the formula is: , where is the normalized reflectance difference of the pixel in the visible light and near-infrared channels; is the gray value of the pixel in the visible light channel; is the gray value of the pixel in the near-infrared channel; is a constant to prevent the denominator from being zero, and its value is 0.01;

[0020] S23: Calculate the shadow probability according to the normalized reflectance difference [[ID=!16]]whose expression is: , when is greater than the set shadow determination threshold , the corresponding pixel is regarded as having shadow occlusion;

[0021] S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow suppression image.

[0022] Optionally, the specific steps of S3 are as follows:

[0023] S31: Perform a directionally adjustable gradient operation on the shadow suppression image obtained in S2. Use Sobel convolution kernels with different directions to perform convolution on each pixel of the image, calculate the gradient components in each direction, and finally synthesize the gradient magnitude at each pixel position to obtain a complete gradient magnitude heat map;

[0024] S32: Based on the gradient magnitude heat map, select two thresholds, a high threshold and a low threshold. Specifically, the high threshold is set to 70% of the maximum value of all pixel gradient magnitudes in the gradient magnitude heat map, and the low threshold is set to 30% of the maximum value of all pixel gradient magnitudes in the gradient magnitude heat map, and preliminarily strong gradient pixel points are selected through the high threshold;

[0025] S33: Use the preliminarily strong gradient pixel points as initial seed points, and perform iterative expansion on the pixels whose gradient magnitudes around the seed points exceed the low threshold based on the connected component growth algorithm. Successively include adjacent pixels in the current connected region until there is no pixel position in the region that can continue to expand, thereby obtaining multiple independent candidate pothole connected regions;

[0026] S34: Calculate the pixel area of each candidate pothole connected region, remove the regions with an area less than 50 pixels in the connected region as noise regions, and retain the regions with an area greater than or equal to 50 pixels as the final candidate pothole regions.

[0027] Optionally, the specific steps of S31 are as follows:

[0028] S311: Denote the shadow suppression image obtained in S2 as ;

[0029] S312: For the image , construct Sobel convolution kernel matrices with directions of 0°, 45°, 90°, and 135° respectively. The corresponding gradient directions are the horizontal direction, the upper right direction, the vertical direction, and the upper left direction. Use the convolution kernel in each direction to perform a two-dimensional convolution operation on the image to obtain the gradient response image in each direction;

[0030] S313: Weightedly synthesize the gradient response images in multiple directions to obtain the direction-sensitive comprehensive gradient magnitude image for each pixel point, which is defined as the gradient magnitude heat map .

[0031] Optionally, the specific steps of S33 are as follows:

[0032] S331: Use the strong gradient pixel points selected by S32 as the initial seed points to establish a seed point set;

[0033] S332: Take out the seed points from the seed point set one by one, calculate the gradient magnitude similarity between the current seed point and its surrounding 8-neighborhood pixels. When the gradient magnitude is not less than the preset low threshold, mark the corresponding adjacent pixels as belonging to the same connected domain as the current seed point, and add the adjacent pixels to the seed point set;

[0034] S333: Repeat S332, continuously take out new seed points from the seed point set for expansion until the seed point set is empty. At this time, an independent and complete connected domain is obtained;

[0035] S334: Traverse all the strong gradient pixel point sets obtained by S32, and repeat the iterative process of S331 to S333 for the unmarked strong gradient pixel points until all strong gradient pixel points are processed. Finally, multiple independent candidate pothole connected regions are obtained.

[0036] Optionally, the specific steps of S4 are as follows:

[0037] S41: For each candidate pothole region obtained by S3, expand it outward with a fixed width along the outer edge contour of the candidate region. The fixed width is set to 1 times the equivalent radius of the candidate region, and the region between the outer boundary and the candidate region contour is used as the annular reference region;

[0038] S42: Calculate the gray-level co-occurrence matrices inside the candidate pothole region and inside the annular reference region respectively;

[0039] S43: Calculate the texture contrast values of the two gray-level co-occurrence matrices respectively. The texture contrast inside the candidate pothole region is denoted as , and the texture contrast of the annular reference region is denoted as ;

[0040] S44: Calculate the texture contrast difference between the candidate pothole area and the annular reference area ; When the difference exceeds the preset texture mutation threshold it is determined that the candidate pothole area is a surface damage area.

[0041] Optionally, the specific steps of S5 are as follows:

[0042] S51: Based on the surface damage area determined in S4, use multiple two-dimensional images continuously acquired by the camera during the vehicle's forward movement, combined with the pose information recorded during the vehicle's driving process, and implement high-density three-dimensional point cloud reconstruction of the surface damage area through a three-dimensional reconstruction algorithm based on the principle of structured light projection to obtain three-dimensional point cloud data;

[0043] S52: Perform surface curvature calculation on the three-dimensional point cloud data. Use each point cloud data point and its neighborhood points within a radius of 5 mm, and fit a local quadratic surface through the least squares method to calculate the Gaussian curvature value at each data point, and obtain the spatial gradient of the obtained curvature value to generate a corresponding surface curvature gradient distribution map;

[0044] S53: In the curvature gradient distribution map, perform binarization processing with a preset curvature gradient threshold to obtain a binary image. The curvature gradient threshold is set to 0.05 mm⁻¹; and use the eight-neighborhood connectivity analysis method for the binary image to extract multiple continuous pixel cluster regions;

[0045] S54: Calculate the pixel area ratio for each of the extracted continuous pixel cluster regions; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damage area has effective pothole features.

[0046] Optionally, the specific steps of S6 are as follows:

[0047] S61: For the region determined to have effective pothole features in S5, extract the set of all boundary pixel coordinates, and use the least squares ellipse fitting algorithm to perform the least enclosing ellipse fitting process on this pixel set to obtain the major axis length, minor axis length, center coordinates, and orientation angle of the fitting ellipse;

[0048] S62: Calculate the eccentricity of the ellipse according to the fitting result. The calculation formula is: , where is the eccentricity of the fitting ellipse; is the major axis length of the fitting ellipse; is the minor axis length of the fitting ellipse;

[0049] S63: Calculate the area ratio between the fitted ellipse and the actual pothole pixel region, which is defined as the ratio of the area of the effective pothole pixel region to the area of the fitted ellipse, and the expression is: , where is the ellipse area ratio; is the actual pixel area within the effective pothole feature region; is the area of the fitted ellipse;

[0050] S64: Judge the geometric features of the fitted ellipse. If the eccentricity and the area ratio are satisfied, then mark the current region as a complete pothole region with regular morphology, and write its spatial position, contour coordinates, major and minor axis dimensions, eccentricity, and area ratio into the standardized data structure, and finally generate a detection report containing the parameters of all effective pothole regions.

[0051] A road pothole detection system for implementing the above-mentioned road pothole detection method includes the following modules:

[0052] Image acquisition module: used to continuously obtain the multi-channel raw image stream of the road surface through the multispectral camera installed in front of the vehicle, and simultaneously start the six-axis inertial measurement unit to collect the acceleration and angular velocity data during the vehicle operation;

[0053] Artifact elimination module: connected to the image acquisition module, based on the acceleration and angular velocity data, and combined with the preset asphalt pavement material feature library, perform motion artifact elimination processing on the raw image stream and output the vibration compensation image;

[0054] Spectral separation and shadow suppression module: connected to the artifact elimination module, used to split the vibration compensation image into visible light channel map and near-infrared channel map, construct the shadow probability map according to the pixel reflectance difference, and then generate the shadow suppression image through asymmetric gamma correction;

[0055] Gradient heat map and candidate extraction module: connected to the spectral separation and shadow suppression module, used to perform multi-directional gradient operation on the shadow suppression image to generate the gradient magnitude heat map, and use the double-threshold connected component growth algorithm to extract the candidate pothole regions;

[0056] Texture analysis module: connected to the gradient heat map and candidate extraction module, used to generate an annular reference region outside the candidate pothole region, and calculate the difference in the gray-level co-occurrence matrix contrast inside and outside the region respectively. When the difference exceeds the set threshold, it is determined as the surface damage region;

[0057] 3D reconstruction and structure determination module: connected to the texture analysis module, used to perform image sequence point cloud reconstruction on the surface damage region, calculate the surface curvature gradient distribution, extract the continuous high-curvature pixel clusters, calculate their area ratio, and determine whether it is an effective pothole feature;

[0058] Morphological evaluation and result output module: Connected to the three-dimensional reconstruction and structure determination module, it is used to perform minimum bounding ellipse fitting on the effective pothole area, calculate the eccentricity and area ratio of the ellipse, and output the final pothole detection report according to the set conditions.

[0059] Advantages of the present invention:

[0060] In the present invention, by constructing a vehicle-mounted data acquisition system integrating multi-spectral imaging and six-axis inertial measurement, combined with a vibration artifact elimination algorithm based on material perception, the stability of image quality is significantly improved, and the problems of image blurring and information offset caused by vehicle movement in the traditional image detection process are solved. At the same time, by introducing spectral reflectance difference analysis and asymmetric gamma correction methods, the influence of shadow interference on image brightness and contrast is effectively suppressed, and the adaptability to complex road environments is enhanced.

[0061] In the present invention, at the feature extraction and determination level, by integrating technical means such as multi-directional gradient operation, double-threshold connected component analysis, texture contrast judgment, point cloud curvature analysis, and ellipse morphology fitting, a bottom-up multi-level screening mechanism is constructed, realizing multi-dimensional comprehensive identification and judgment of pothole areas from two-dimensional to three-dimensional and from grayscale to morphology, ensuring that the final output pothole detection results have high integrity, accuracy, and standard expression ability, meeting the requirements of high-precision real-time detection of road intelligent inspection systems. Brief description of the drawings

[0062] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following drawings are only those of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0063] Figure 1 Schematic diagram of the road pothole detection method for the embodiment of the present invention;

[0064] Figure 2 Schematic diagram of the road pothole detection system for the embodiment of the present invention. Detailed implementation manners

[0065] The following will describe the present invention in detail in combination with the drawings and specific embodiments. At the same time, it should be noted here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments. For some well-known technologies, those skilled in the art can also adopt other alternative methods for implementation; moreover, the drawing part is only for more specifically describing the embodiments, and is not intended to specifically limit the present invention.

[0066] It should be noted that in the specification, references to "an embodiment", "embodiments", "exemplary embodiments", "some embodiments", etc. indicate that the described embodiments may include specific features, structures, or characteristics, but not necessarily every embodiment includes such specific features, structures, or characteristics. Additionally, when describing a specific feature, structure, or characteristic in connection with an embodiment, implementing such feature, structure, or characteristic in connection with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the relevant art.

[0067] Generally, terms can be understood, at least in part, from their use in context. For example, at least in part depending on the context, the term "one or more" used herein can be used to describe any feature, structure, or characteristic in a singular sense, or can be used to describe a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey a set of exclusive factors, but rather, at least in part depending on the context, can allow for the existence of other factors that may not be explicitly described.

[0068] As Figure 1 shown, a method for detecting road potholes includes the following steps:

[0069] S1: Obtain the original image stream of the road surface through an in-vehicle multispectral camera, synchronously collect the vibration data of a six-axis inertial measurement unit, and perform motion artifact elimination based on the vibration data and a preset asphalt pavement material feature library to generate a vibration compensation image;

[0070] S2: Separate the visible light and near-infrared spectral channels of the vibration compensation image, dynamically construct a shadow probability map according to the spectral reflectance difference, and generate a shadow suppression image through asymmetric gamma correction;

[0071] S3: Perform a directionally adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use a double-threshold connected component analysis method to extract candidate pothole regions;

[0072] S4: Select an annular reference region outside the candidate pothole region, calculate the contrast difference of the gray-level co-occurrence matrix, and determine it as a surface damage region when the contrast difference exceeds the texture mutation threshold;

[0073] S5: Perform three-dimensional point cloud reconstruction on the surface damage region and calculate the surface curvature gradient distribution, extract the continuous pixel cluster statistical area ratio, and determine it as an effective pothole feature when the ratio is greater than 60%;

[0074] S6: For the region corresponding to the effective pothole feature, calculate its minimum bounding ellipse eccentricity and ellipse area ratio, and generate a final pothole detection report when the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85.

[0075] S1 specifically includes:

[0076] S11: Obtain the original road surface image stream through a multi-spectral camera fixedly installed at the mid-axis position of the front bumper of the vehicle. The multi-spectral camera has three channels of visible light, near-infrared, and short-wave infrared, with a frame rate of 60fps, an image resolution of not less than 1920×1080 pixels, and uses the hardware trigger method to synchronously start image acquisition;

[0077] S12: Synchronously collect the triaxial acceleration and triaxial angular velocity data of the vehicle at the image acquisition moment through a six-axis inertial measurement unit (IMU) installed in the center of the vehicle chassis. The triaxial acceleration includes longitudinal acceleration, lateral acceleration, and vertical acceleration along the vehicle's forward direction; the triaxial angular velocity includes roll angular velocity rotating around the vehicle's forward direction axis, pitch angular velocity rotating around the transverse axis, and yaw angular velocity rotating around the vertical axis. The data sampling frequency is 200Hz; at the same time, linearly interpolate and register the time stamp of each frame of image with the triaxial acceleration and triaxial angular velocity data at the corresponding moment to ensure that the vibration data corresponds to each image frame one by one;

[0078] S13: Input the triaxial acceleration and triaxial angular velocity data collected by the six-axis inertial measurement unit into an image transformation function based on forward motion compensation to perform compensation transformation on the pixel positions of the original image. The expression of the image transformation function is: , where the pixel position offset The calculation formula is: ; , in the formula, is the pixel gray value of the original image; is the pixel gray value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position; are the longitudinal and lateral accelerations of the vehicle at the image acquisition moment, with the unit of ; are the roll angular velocity and yaw angular velocity of the vehicle at the image acquisition moment, with the unit of ; is the acceleration compensation coefficient calibrated for different asphalt materials in the asphalt road surface material feature library, with the unit of pixel·s² / m; is the angular velocity compensation coefficient calibrated for different asphalt materials in the asphalt road surface material feature library, with the unit of pixel·s / rad;

[0079] S14: Perform pixel-level resampling and translation transformation on the original image stream frame by frame based on the image transformation function to generate a vibration compensation image that eliminates vehicle vibration and displacement artifacts. Through the above steps, it is possible to ensure a high degree of consistency between image acquisition and vibration data in the time and space dimensions, and achieve targeted image correction under the influence of material differences, thereby improving the stability and clarity of the pothole area image, which helps to improve the accuracy of subsequent spectral analysis and geometric recognition.

[0080] S2 specifically includes:

[0081] S21: Split the vibration compensation image obtained in S1 by band channels, extract the visible light channel image and the near-infrared channel image. The multispectral camera realizes spectral imaging of each band through the filter array. The channel wavelengths are as follows: the visible light channel wavelength range is 400nm to 700nm, and the near-infrared channel wavelength range is 750nm to 950nm. The split images maintain the same resolution and pixel coordinates as the original image.

[0082] S22: Calculate the spectral reflectance difference of each pixel point. The formula is: , where is the normalized reflectance difference of the pixel point in the visible light and near-infrared channels; is the gray value of the pixel point in the visible light channel; is the gray value of the pixel point in the near-infrared channel; is a constant to prevent the denominator from being zero, with a value of 0.01;

[0083] S23: Calculate the shadow probability according to the normalized reflectance difference , and its expression is: , when is greater than the set shadow determination threshold , then the corresponding pixel point is considered to be shaded;

[0084] S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow suppression image. The asymmetric gamma correction function is defined as:

[0085] , where is the gray value of the corresponding pixel in the shadow suppression image; is the original pixel gray value in the vibration compensation image; is the enhanced gamma coefficient set for the shadow area, with a value range of 1.2 to 1.5; The reduction gamma coefficient set for the non-shadow area ranges from 0.7 to 0.9; through the above steps, channel separation and fusion processing of multi-spectral information can be achieved, the shadow area is distinguished based on spectral reflection characteristics, and non-linear compensation is performed using an adaptive gray mapping method, effectively suppressing the interference of shadows on the subsequent gradient operation and edge extraction processes, and improving the accuracy of pothole feature extraction.

[0086] S3 specifically includes:

[0087] S31: Perform a directionally adjustable gradient operation on the shadow-suppressed image obtained in S2. Use Sobel convolution kernels with different directions to perform convolution on the image pixel by pixel, calculate the gradient components in each direction, and finally synthesize the gradient magnitude at each pixel position to obtain a complete gradient magnitude heat map;

[0088] S32: Based on the gradient magnitude heat map, select two thresholds, a high threshold and a low threshold. Specifically, the high threshold is set to 70% of the maximum of the gradient magnitudes of all pixels in the gradient magnitude heat map, and the low threshold is set to 30% of the maximum of the gradient magnitudes of all pixels in the gradient magnitude heat map, and initially strong gradient pixel points are selected through the high threshold;

[0089] S33: Use the initially strong gradient pixel points as initial seed points, and based on the connected component growth algorithm, iteratively expand the pixels whose gradient magnitudes around the seed points exceed the low threshold, and sequentially include adjacent pixels into the current connected region until there is no pixel position in the region that can continue to expand, thus obtaining multiple independent candidate pothole connected regions;

[0090] S34: Calculate the pixel area of each candidate pothole connected region, and remove the regions with an area less than 50 pixels as noise regions, and retain the regions with an area greater than or equal to 50 pixels as the final candidate pothole regions; through the above steps, comprehensive coverage of the gradient operation direction and accurate identification of the boundaries of the pothole regions can be achieved, and at the same time, the double-threshold connected component growth method effectively suppresses noise interference, ensuring the reliability of the subsequent determination of the pothole feature region.

[0091] S31 specifically includes:

[0092] S311: Denote the shadow-suppressed image obtained in S2 as where represents the position coordinates of the pixels in the image;

[0093] S312: For the image , Sobel convolution kernel matrices with directions of 0°, 45°, 90°, and 135° are constructed respectively. The corresponding gradient directions are the horizontal direction, the upper right direction, the vertical direction, and the upper left direction. The two-dimensional convolution operation is performed on the image using the convolution kernel in each direction to obtain the gradient response images in each direction, which are respectively denoted as the horizontal direction gradient map , the upper right direction gradient map , the vertical direction gradient map ; the upper left direction gradient map ;

[0094] S313: The gradient response images in multiple directions are weighted and synthesized to obtain the direction-sensitive comprehensive gradient amplitude image of each pixel point, which is defined as the gradient amplitude heat map , and its calculation formula is as follows:

[0095] , where: is the finally obtained gradient amplitude heat map; are the weight coefficients in the corresponding directions respectively, and each weight coefficient satisfies . In this embodiment, the values of the four weight coefficients are uniformly taken as 0.25; through the above steps, the edge information in multiple directions can be integrated, the response ability to road pothole contours in different directions can be improved, and a stable and clear gradient amplitude heat map can be obtained, providing a high-precision gradient basis for subsequent connected component analysis.

[0096] S33 specifically includes:

[0097] S331: The strong gradient pixel points screened out in S32 are used as the initial seed points to establish a seed point set

[0098] S332: The seed points are taken out one by one from the seed point set, and the gradient amplitude similarity between the current seed point and its surrounding 8-neighborhood pixels is calculated. When the gradient amplitude is not less than the preset low threshold, the corresponding adjacent pixels are marked as belonging to the same connected component as the current seed point, and the adjacent pixels are added to the seed point set;

[0099] The determination method of gradient amplitude similarity adopts the following formula:

[0100] ;

[0101] where, is the marking function for whether the current neighborhood pixel point is included in the connected component. The value of 1 means it is included in the current connected component, and the value of 0 means it is not included; is the preset low threshold;

[0102] S333: Repeatedly execute S332, continuously extract new seed points from the seed point set for expansion until the seed point set is empty. At this time, an independent and complete connected domain is obtained;

[0103] S334: Traverse all the strong gradient pixel point sets obtained in S32, and repeat the iterative process of S331 to S333 for the unmarked strong gradient pixel points until all the strong gradient pixel points are processed. Finally, multiple independent candidate pit-connected regions are obtained; Through the above steps, the spatial continuity of the gradient amplitude can be fully utilized for accurate expansion of the connected domain, making the boundary of the candidate pit region more accurate and effectively improving the accuracy of subsequent pit feature determination.

[0104] S4 specifically includes:

[0105] S41: For each candidate pit region obtained in S3, expand it outward with a fixed width along the outer edge contour of the candidate region to obtain an outer boundary. The fixed width is set to 1 times the equivalent radius of the candidate region, and the region between the outer boundary and the candidate region contour is used as an annular reference region;

[0106] S42: Calculate the gray-level co-occurrence matrices inside the candidate pit region and in the annular reference region respectively. The gray-level co-occurrence matrix is constructed based on the pixel gray levels of the gray image. The gray levels are quantized into 256 levels, and the gray-level co-occurrence matrix is constructed in the horizontal direction with a pixel spacing of 1 pixel to obtain the co-occurrence matrix inside the candidate region and the co-occurrence matrix of the annular reference region;

[0107] S43: Calculate the texture contrast values of the two gray-level co-occurrence matrices respectively. Among them, the texture contrast inside the candidate pit region is denoted as and the texture contrast of the annular reference region is denoted as ;

[0108] The calculation formula for texture contrast is: , where in the formula, is the texture contrast of the corresponding region; is the probability value of the co-occurrence of pixel pairs with gray levels of and in the gray-level co-occurrence matrix of the corresponding region; is the total number of gray levels. In this embodiment, ;

[0109] S44: Calculate the texture contrast difference between the candidate pit region and the annular reference region, and judge whether there is an obvious texture mutation in the region based on this difference . The calculation formula for the texture contrast difference is: ; When the difference exceeds the preset texture mutation threshold When it is determined that the candidate pothole area is the surface damage area, where the texture mutation threshold is an empirical threshold obtained by calibrating the surfaces of different road materials; through the above steps, the texture difference between the pothole area and the surrounding normal road surface area can be accurately identified, false alarm areas can be effectively excluded, thereby improving the reliability and accuracy of the identification of the pothole surface damage area.

[0110] S5 specifically includes:

[0111] S51: Based on the surface damage area determined by S4, using multiple two-dimensional images continuously acquired by the camera during the vehicle's forward movement, combined with the pose information recorded during the vehicle's driving process, high-density three-dimensional point cloud reconstruction of the surface damage area is achieved through a three-dimensional reconstruction algorithm based on the principle of structured light projection, obtaining three-dimensional point cloud data;

[0112] S52: Perform surface curvature calculation on the three-dimensional point cloud data. Using each point cloud data point and its neighboring points within a radius of 5 mm, a local quadratic surface is fitted by the least squares method, the Gaussian curvature value at each data point is calculated, and the spatial gradient of the obtained curvature value is obtained to generate a corresponding surface curvature gradient distribution map; The calculation formula for Gaussian curvature is: , where is the Gaussian curvature at the point cloud data point; are the principal curvatures obtained by fitting the local quadratic surface respectively, with the unit of ;

[0113] S53: In the curvature gradient distribution map, perform binary processing with a preset curvature gradient threshold to obtain a binary image. The curvature gradient threshold is set to 0.05 mm⁻¹; and the eight-neighborhood connectivity analysis method is used for the binary image to extract multiple continuous pixel cluster regions;

[0114] S54: Calculate the pixel area ratio for each of the extracted continuous pixel cluster regions; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damage area has effective pothole characteristics; The pixel area ratio is the ratio of the number of pixels in the current pixel cluster region to the total number of pixels in the entire surface damage area. The specific calculation formula is: , where is the area ratio of the current pixel cluster region; is the number of pixels in the current pixel cluster region; is the total number of pixels in the entire surface damage area; through the above steps, the three-dimensional structure and curvature characteristics of the surface damage area can be identified and quantified with high precision, effectively suppressing the misidentification of non-pothole characteristics, and significantly improving the reliability and accuracy of road pothole feature detection.

[0115] S6 specifically includes:

[0116] S61: For the area determined as an effective pothole feature in S5, extract the set of all boundary pixel coordinates, and use the least squares ellipse fitting algorithm to perform the least enclosing ellipse fitting process on this pixel set, obtaining the major axis length, minor axis length, center coordinates, and orientation angle of the fitted ellipse;

[0117] S62: Calculate the eccentricity of the ellipse according to the fitting result, and its calculation formula is: , where is the eccentricity of the fitted ellipse; is the major axis length of the fitted ellipse; is the minor axis length of the fitted ellipse; the units are all pixels, and the condition is satisfied;

[0118] S63: Calculate the area ratio of the fitted ellipse to the actual pothole pixel area, which is defined as the ratio of the area of the effective pothole pixel area to the area of the fitted ellipse, and the expression is: , where is the ellipse area ratio; is the actual pixel area within the effective pothole feature area; is the area of the fitted ellipse;

[0119] S64: Judge the geometric characteristics of the fitted ellipse. If the eccentricity and the area ratio are satisfied, then mark the current area as a regularly shaped complete pothole area, and write its spatial position, contour coordinates, major and minor axis dimensions, eccentricity, and area ratio into the standardized data structure, and finally generate a detection report containing the parameters of all effective pothole areas; through the above steps, it is possible to achieve precise geometric evaluation of the pothole area based on effective geometric fitting combined with morphological feature indicators, thereby ensuring that the output detection results have high reliability and integrity.

[0120] As Figure 2 shown, a road pothole detection system for implementing the above-mentioned road pothole detection method includes the following modules:

[0121] Image acquisition module: used to continuously obtain the multi-channel original image stream of the road surface through the multispectral camera installed in front of the vehicle, and simultaneously start the six-axis inertial measurement unit to collect the acceleration and angular velocity data during the vehicle operation;

[0122] Artifact elimination module: connected to the image acquisition module, based on the acceleration and angular velocity data, and combined with the preset asphalt pavement material feature library, perform the motion artifact elimination process on the original image stream, and output the vibration compensation image;

[0123] Spectral separation and shadow suppression module: Connected to the artifact elimination module, it is used to split the vibration-compensated image into a visible light channel map and a near-infrared channel map, construct a shadow probability map based on the pixel reflectivity difference, and then generate a shadow suppression image through asymmetric gamma correction;

[0124] Gradient heat map and candidate extraction module: Connected to the spectral separation and shadow suppression module, it is used to perform multi-directional gradient operations on the shadow suppression image to generate a gradient magnitude heat map, and use the double-threshold connected component growth algorithm to extract candidate pitted areas;

[0125] Texture analysis module: Connected to the gradient heat map and candidate extraction module, it is used to generate an annular reference area around the candidate pitted area, and calculate the contrast difference of the gray-level co-occurrence matrix inside and outside the area respectively. When the difference exceeds the set threshold, it is determined as the surface damage area;

[0126] 3D reconstruction and structure determination module: Connected to the texture analysis module, it is used to perform image sequence point cloud reconstruction on the surface damage area, calculate the surface curvature gradient distribution, extract continuous high-curvature pixel clusters, calculate their area ratio, and determine whether it is an effective pitted feature;

[0127] Morphological evaluation and result output module: Connected to the 3D reconstruction and structure determination module, it is used to perform minimum bounding ellipse fitting on the effective pitted area, calculate the eccentricity and area ratio of the ellipse, and output the final pitted detection report according to the set conditions.

[0128] This invention covers any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To enable the public to have a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments of this invention. However, those skilled in the art can fully understand this invention without the description of these details. Additionally, to avoid unnecessary confusion to the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0129] The above are only the preferred embodiments of this invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of this invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this invention.

Claims

1. A road pothole detection method, characterized in that: The following steps are involved: S1: The vehicle-mounted multispectral camera acquires the original image stream of the road surface, and simultaneously collects vibration data from the six-axis inertial measurement unit. Based on the vibration data and a preset asphalt pavement material feature library, motion artifacts are eliminated to generate a vibration-compensated image. S11: Obtaining a raw image stream of the road surface through a multispectral camera fixedly installed at the center axis of the front bumper of the vehicle; S12: The six-axis inertial measurement unit (IMU) installed in the center of the vehicle chassis synchronously collects the vehicle's three-axis acceleration and three-axis angular velocity data at the time of image acquisition. The data sampling frequency is 200 Hz. At the same time, the timestamp of each image frame is linearly interpolated and aligned with the three-axis acceleration and three-axis angular velocity data at the corresponding moment to ensure a one-to-one correspondence between the vibration data and the image frame. S13: Input the three-axis acceleration and three-axis angular velocity data collected by the six-axis inertial measurement unit into the image transformation function based on forward motion compensation to perform compensation transformation on the original image pixel position. The image transformation function expression is: , where is the pixel gray value of the original image; is the pixel grayscale value of the image after vibration compensation; is the pixel position coordinate in the original image; is the compensation offset of the pixel position; S14: performing pixel-level resampling and translation transformation on the original image stream frame by frame based on the image transformation function to generate a vibration-compensated image that eliminates vehicle vibration and displacement artifacts; S2: Separate the visible and near-infrared spectral channels of the vibration-compensated image, dynamically construct a shadow probability map based on the spectral reflectance difference, and generate a shadow-suppressed image through asymmetric gamma correction; S3: Perform a direction-adjustable gradient operation on the shadow suppression image to generate a gradient magnitude heat map, and use a dual-threshold connected domain analysis method to extract candidate pothole areas; S4: Select an annular reference area outside the candidate pothole area, calculate the gray level co-occurrence matrix contrast difference, and determine it as a surface damage area when the contrast difference exceeds the texture mutation threshold; S5: Reconstruct the 3D point cloud of the surface damaged area and calculate the surface curvature gradient distribution. Extract the statistical area ratio of continuous pixel clusters. When the ratio is greater than 60%, it is determined to be a valid pothole feature. S6: For the area corresponding to the valid pothole feature, calculate its minimum enclosing ellipse eccentricity and ellipse area ratio. When the ellipse eccentricity is less than 0.7 and the ellipse area ratio is greater than 0.85, mark the area as a complete pothole area with regular morphology, and include its parameter information in the final pothole detection report.

2. The road pothole detection method according to claim 1, characterized in that: The S2 specifically includes: S21: Split the vibration-compensated image obtained in S1 according to the band channel, and extract the visible light channel image and the near-infrared channel image; S22: Calculate the spectral reflectance difference of each pixel point. The formula is: , where is the normalized reflectance difference of the pixel in the visible light and near-infrared channels; is the gray value of the pixel in the visible light channel; is the gray value of the pixel in the near-infrared channel; To prevent the constant from having a zero denominator, the value is taken as 0.01; S23: According to the normalized reflectivity difference Calculating shadow probability , whose expression is: ,when Greater than the set shadow judgment threshold When , the corresponding pixel is considered to be shadowed; S24: Perform asymmetric gamma correction on the image with shadow occlusion to generate a shadow-suppressed image.

3. The road pothole detection method according to claim 1, characterized in that: The S3 specifically includes: S31: Perform direction-adjustable gradient operation on the shadow suppression image obtained in S2, convolve the image pixel by pixel using Sobel convolution kernels with different directions, calculate the gradient components in each direction, and finally synthesize them into the gradient amplitude at each pixel position, thereby obtaining a complete gradient amplitude heat map; S32: Based on the gradient amplitude heat map, two thresholds, a high threshold and a low threshold, are selected. Specifically, the high threshold is set to 70% of the maximum gradient amplitude of all pixels in the gradient amplitude heat map, and the low threshold is set to 30% of the maximum gradient amplitude of all pixels in the gradient amplitude heat map. Preliminary strong gradient pixels are screened out using the high threshold. S33: Using the initial strong gradient pixel as the initial seed point, iteratively expand the pixels around the seed point whose gradient amplitude exceeds the lower threshold based on the connected domain growth algorithm, and sequentially assign adjacent pixels to the current connected domain until there are no more pixel positions in the domain that can be further expanded, thereby obtaining multiple independent candidate pothole connected regions; S34: Calculate the pixel area of each candidate pothole connected region, remove the connected region with an area less than 50 pixels as a noise region, and retain the region with an area greater than or equal to 50 pixels as the final candidate pothole region.

4. The road pothole detection method according to claim 3, characterized in that: The S31 specifically includes: S311: The shadow suppression image obtained in S2 is recorded as ; S312: For images , respectively construct Sobel convolution kernel matrices with directions of 0°, 45°, 90° and 135°, and the corresponding gradient directions are horizontal, upper right, vertical and upper left, respectively. Use the convolution kernel in each direction to perform a two-dimensional convolution operation on the image to obtain the gradient response image in each direction; S313: Perform weighted synthesis on the gradient response images in multiple directions to obtain the direction-sensitive integrated gradient amplitude image of each pixel, which is defined as the gradient amplitude heat map .

5. The road pothole detection method according to claim 3, characterized in that: The S33 specifically includes: S331: Using the strong gradient pixel points screened out in S32 as initial seed points to establish a seed point set; S332: Extract seed points one by one from the seed point set, calculate the gradient magnitude similarity between the current seed point and its eight neighboring pixels, and when the gradient magnitude is not less than a preset lower threshold, mark the corresponding neighboring pixels as belonging to the same connected domain as the current seed point, and add the neighboring pixels to the seed point set; S333: Repeat S332, continuously extracting new seed points from the seed point set for expansion, until the seed point set is empty, at which point an independent and complete connected domain is obtained; S334: Traverse all sets of strong gradient pixels obtained in S32, and repeat the iterative process from S331 to S333 for unlabeled strong gradient pixels until all strong gradient pixels are processed, and finally obtain multiple independent candidate pothole connected areas.

6. The road pothole detection method according to claim 1, characterized in that: The S4 specifically includes: S41: For each candidate pothole area obtained in S3, expand outward along the outer edge contour of the candidate area with a fixed width to obtain an outer boundary, where the fixed width is set to 1 times the equivalent radius of the candidate area, and the area between the outer boundary and the candidate area contour is used as an annular reference area; S42: Calculating the gray level co-occurrence matrix inside the candidate pothole area and within the annular reference area respectively; S43: Calculate the texture contrast values of the two gray-level co-occurrence matrices respectively, where the texture contrast inside the candidate pothole area is recorded as , the texture contrast of the annular reference area is recorded as ; S44: Calculate the texture contrast difference between the candidate pothole area and the annular reference area ; When the difference Exceeding the preset texture mutation threshold , the candidate pothole area is determined to be a surface damaged area.

7. The road pothole detection method according to claim 1, characterized in that: The S5 specifically includes: S51: Based on the surface damage area determined in S4, a high-density 3D point cloud of the surface damage area is reconstructed using a 3D reconstruction algorithm based on the principle of structured light projection, using multiple frames of 2D images continuously acquired by the camera during the vehicle's forward movement, combined with the posture information recorded during the vehicle's driving process, to obtain 3D point cloud data. S52: Perform surface curvature calculation on the three-dimensional point cloud data. Use each point cloud data point and its neighboring points within a radius of 5 mm to fit a local quadratic surface using the least squares method. Calculate the Gaussian curvature value at each data point, and calculate the spatial gradient of the obtained curvature value to generate a corresponding surface curvature gradient distribution map. S53: In the curvature gradient distribution map, a binary image is obtained by performing binarization processing with a preset curvature gradient threshold value, wherein the curvature gradient threshold value is set to ; And the eight-neighborhood connectivity analysis method is used to extract multiple continuous pixel cluster areas from the binary image; S54: Calculate the pixel area ratios for the extracted continuous pixel cluster regions respectively; when the area ratio of any continuous pixel cluster region is greater than 60%, it is determined that the surface damaged region has effective pothole features.

8. The road pothole detection method according to claim 1, characterized in that: The S6 specifically includes: S61: For the area determined as a valid pothole feature in S5, extract the coordinate set of all its boundary pixels, and use a least squares ellipse fitting algorithm to fit a minimum enclosing ellipse to the pixel set to obtain the major axis length, minor axis length, and center coordinates and orientation angle of the fitted ellipse; S62: Calculate the eccentricity of the ellipse according to the fitting result. The calculation formula is: , where is the eccentricity of the fitted ellipse; is the length of the major axis of the fitted ellipse; is the length of the minor axis of the fitted ellipse; S63: Calculate the area ratio of the fitted ellipse to the actual pothole pixel area, which is defined as the ratio of the effective pothole pixel area to the area of the fitted ellipse, and the expression is: , where is the ellipse area ratio; is the actual pixel area within the effective pothole feature area; is the area of the fitted ellipse; S64: Determine the geometric characteristics of the fitted ellipse. If the eccentricity is satisfied And the area ratio , the current area is marked as a complete pothole area with regular morphology, and its spatial position, contour coordinates, major and minor axis dimensions, eccentricity and area ratio are written into a standardized data structure, and finally a detection report containing all valid pothole area parameters is generated.

9. A road pothole detection system, used to implement the road pothole detection method according to any one of claims 1 to 8, characterized in that: Includes the following modules: Image acquisition module: This module is used to acquire a multi-channel raw image stream of the road surface in real time through a multispectral camera installed in front of the vehicle, and simultaneously activates the six-axis inertial measurement unit to collect acceleration and angular velocity data during vehicle operation; Artifact removal module: connected to the image acquisition module, based on acceleration and angular velocity data and combined with a preset asphalt pavement material feature library, it removes motion artifacts from the original image stream and outputs a vibration-compensated image; Spectral separation and shadow suppression module: This module is connected to the artifact removal module and is used to split the vibration-compensated image into a visible light channel map and a near-infrared channel map. It also constructs a shadow probability map based on the pixel reflectance difference and generates a shadow-suppressed image through asymmetric gamma correction. Gradient thermal and candidate extraction module: This module is connected to the spectral separation and shadow suppression module to perform multi-directional gradient operations on the shadow suppression image to generate a gradient amplitude thermal map and use a dual-threshold connected domain growing algorithm to extract candidate pothole areas. Texture analysis module: This module is connected to the gradient thermal and candidate extraction modules to generate a circular reference area around the candidate pothole area and calculate the gray-level co-occurrence matrix contrast difference inside and outside the area. When the difference exceeds the set threshold, it is determined to be a surface damage area. 3D reconstruction and structure determination module: This module is connected to the texture analysis module and is used to reconstruct image sequence point clouds of surface damage areas, calculate the surface curvature gradient distribution, extract continuous high-curvature pixel clusters, calculate their area ratio, and determine whether they are valid pothole features; Morphological assessment and result output module: connected to the three-dimensional reconstruction and structure judgment module, used to fit the minimum enclosing ellipse to the effective pothole area, calculate the ellipse eccentricity and area ratio, and output the final pothole detection report according to the set conditions.

Images