A method, device and storage medium for automatically detecting and estimating the depth of a road surface pit

Abstract

The application discloses a kind of road surface pit automatic detection and depth estimation method, equipment and storage medium, it is related to image recognition technical field.The method includes: obtaining road surface image and road surface laser point cloud;YOLOv5 deep learning model and DeepSORT target tracker are utilized to detect and track pit area in the road surface image, and SFM algorithm is utilized to reconstruct the road surface pit area image, obtain three-dimensional reconstruction point cloud;Point cloud post-processing is carried out to the three-dimensional reconstruction point cloud, and reconstruction point cloud calibration data is obtained;The coordinate system of the road surface laser point cloud and the reconstruction point cloud calibration data is converted into pixel coordinate system, and scale factor is calculated based on grid search algorithm, to determine road surface pit scale information;The scale factor is the corresponding relationship between the road surface laser point cloud and the reconstruction point cloud calibration data.The application can improve the accuracy of road surface pit detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, device and storage medium for automatic detection and depth estimation of road surface potholes. Background Technology

[0002] With the development of intelligent driving technology, people have placed higher demands on the driving comfort of intelligent vehicles. As roads gradually age, cracks, damage, potholes, and other issues that affect driving comfort are likely to appear. Therefore, during the driving process of intelligent vehicles, it is necessary to accurately identify whether potholes exist on the road ahead and their depth information to provide data support for whether the intelligent vehicle should take evasive action.

[0003] Currently used methods for detecting road potholes have drawbacks such as poor real-time performance and low detection accuracy, which cannot meet the requirements of rapid detection of road potholes during intelligent vehicle operation. Summary of the Invention

[0004] The purpose of this invention is to provide an automatic method, device, and storage medium for detecting and estimating the depth of road potholes, which can improve the accuracy of road pothole detection.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] An automatic method for detecting and estimating the depth of road potholes includes:

[0007] The acquisition devices are deployed at fixed intervals and angles at the front of the vehicle, and the acquisition devices are used to acquire road surface images and road surface laser point clouds; the acquisition devices include cameras and 2D LiDAR.

[0008] The YOLOv5 deep learning model and DeepSORT target tracker are used to detect and track the pothole regions in the road surface image to obtain road surface pothole region images that change over time.

[0009] The SFM algorithm was used to reconstruct the image of the road surface pothole area to obtain a three-dimensional reconstructed point cloud;

[0010] The reconstructed point cloud is post-processed to obtain reconstructed point cloud calibration data; the point cloud post-processing includes road plane fitting, point cloud 3D rotation, point cloud completion, and point cloud outlier detection.

[0011] The coordinate systems of the road surface laser point cloud and the reconstructed point cloud calibration data are converted into pixel coordinate systems, and the scale factor is calculated based on the grid search algorithm to determine the scale information of road surface potholes; the scale factor is the correspondence between the road surface laser point cloud and the reconstructed point cloud calibration data.

[0012] Optionally, the step of using the YOLOv5 deep learning model and the DeepSORT object tracker to detect and track the pothole regions in the road surface image to obtain a road surface pothole region image that changes over time specifically includes:

[0013] The YOLOv5 deep learning model is used to detect the pixel positions of the potholes in the image to obtain an image of the pothole area on the road surface.

[0014] The DeepSORT target tracker is used for position tracking to obtain images of road surface pothole areas that change over time.

[0015] Optionally, the YOLOv5 deep learning model is used to detect the pixel location of the pit in the image. The specific process is as follows:

[0016] The road surface image is convolved using multiple downsampling layers to generate a first feature map.

[0017] By performing transpose convolution on the first feature map through an upsampling layer corresponding to each of the downsampling layers, a second feature map with the same size as the original road surface image is obtained.

[0018] The second feature map is predicted by using a Softmax layer and a regression task, and the number of duplicate detection boxes for road potholes is reduced by using a non-maximum suppression method to obtain the final image of the road pothole region.

[0019] Optionally, the DeepSORT target tracker is used for position tracking, and the specific process is as follows:

[0020] The appearance features of the road surface pothole area image are extracted, and the road surface pothole is tracked based on the Kalman filter algorithm. The position of the road surface pothole in the image is predicted in the next frame image, and the position of the pothole is updated when the pothole is detected in the next frame.

[0021] The Hungarian algorithm is used to determine whether the road surface potholes in the current frame and the previous frame are the same target. A unique ID is assigned to each road surface pothole, and the detection box is expanded to twice its original size with the center of the box as the reference, so as to obtain the road surface pothole region image according to the time sequence.

[0022] Optionally, the three-dimensional reconstructed point cloud is subjected to point cloud post-processing, the specific process of which is as follows:

[0023] In the coordinate system of the three-dimensional reconstructed point cloud, the least squares method is used to fit all three-dimensional points to obtain the coefficients of the road plane equation.

[0024] Based on the road plane equation coefficients, the point cloud is rotated in three dimensions to obtain the calibrated three-dimensional point cloud coordinates.

[0025] The calibrated 3D point cloud coordinates are completed based on the membership cloud theory, resulting in the completed point cloud coordinates, expressed as follows:

[0026] x i =Enx·t1+x

[0027] y i =Eny·t2+y

[0028] z i =Enz·t3+z

[0029] Among them, (x i y i , z i (x, y, z) represents the coordinates of the completed point cloud, (x, y, z) represents the coordinates of the three-dimensional point cloud, t1, t2, and t3 are one-dimensional standard normal random numbers, and Enx, Eny, and Enz are the entropies of the three-dimensional point cloud in the x, y, and z directions, respectively, representing the degree of dispersion of the point cloud.

[0030] Outlier detection based on isolated forest is performed on the completed point cloud coordinates to eliminate the influence of noise points on the estimation of negative obstacle depth on the road surface.

[0031] Optionally, the calculation process of the scaling factor is as follows:

[0032] Based on the grid search algorithm, a 2D LiDAR point with the highest reflection intensity and whose corresponding pixel position is located within the pit detection frame is selected, and the pixel position corresponding to the 2D LiDAR point is denoted as C. A 21×21 pixel grid is constructed with point C as the center. Within the range of the pixel grid, the pixel position corresponding to the reconstructed point cloud with the smallest Euclidean distance from point C is denoted as K. The ratio of the depth of point C in the world coordinate system to the depth of point K in the 3D reconstructed point cloud coordinate system is determined as the scaling factor.

[0033] Optionally, the calculation process for the pavement pothole size information is as follows:

[0034] Multiply the scaling factor by the Z-axis value of the point cloud in the 3D reconstructed point cloud coordinate system to obtain the true depth of each point cloud in the world coordinate system, thereby determining the scale information of road surface potholes.

[0035] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the above-described method for automatic detection and depth estimation of road surface potholes.

[0036] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for automatic detection and depth estimation of road surface potholes as described above.

[0037] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0038] This invention discloses an automatic method, device, and storage medium for detecting and estimating the depth of road potholes. The method includes acquiring a road surface image and a road surface laser point cloud; detecting and tracking pothole regions in the road surface image using a YOLOv5 deep learning model and a DeepSORT target tracker; reconstructing the road surface pothole region image using the SFM algorithm to obtain a three-dimensional reconstructed point cloud; performing point cloud post-processing on the three-dimensional reconstructed point cloud to obtain reconstructed point cloud calibration data; converting the coordinate systems of the road surface laser point cloud and the reconstructed point cloud calibration data into a pixel coordinate system, and calculating a scaling factor based on a grid search algorithm to determine the scale information of the road potholes; the scaling factor represents the correspondence between the road surface laser point cloud and the reconstructed point cloud calibration data. This invention can improve the accuracy of road pothole detection and the driving comfort of intelligent vehicles. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the automatic detection and depth estimation method for road surface potholes according to the present invention.

[0041] Figure 2 This is a schematic diagram of the installation of the data acquisition device in this embodiment;

[0042] Figure 3 This is a schematic diagram illustrating the principle of using SFM for 3D reconstruction in this embodiment;

[0043] Figure 4 This is a schematic diagram of the electronic device in this embodiment.

[0044] Reference numerals: 201, detection vehicle; 202, road surface to be detected; 203, pothole; 204, camera; 205, 2D lidar; 501, memory; 502, processor; 503, computer program. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The purpose of this invention is to provide an automatic method, device, and storage medium for detecting and estimating the depth of road potholes, which can improve the accuracy of road pothole detection.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the present invention provides an automatic detection and depth estimation method for road surface potholes, comprising:

[0049] Step 100: Deploy the acquisition equipment at fixed intervals and angles at the front of the vehicle, and use the acquisition equipment to acquire road surface images and road surface laser point clouds; the acquisition equipment includes a camera and a 2D LiDAR.

[0050] Step 200: Use the YOLOv5 deep learning model and DeepSORT object tracker to detect and track the pothole regions in the road surface image to obtain a road surface pothole region image that changes over time.

[0051] The required road surface information can be extracted using cameras and 2D LiDAR sensors. The YOLOv5-based deep learning model and DeepSORT target tracker can segment road surface pothole images, which is beneficial for the SFM algorithm to quickly reconstruct 3D point clouds without processing all image information. In addition, based on the accurate ranging capability of 2D LiDAR, the depth information of road surface potholes can be accurately estimated.

[0052] Step 300: Reconstruct the image of the road surface pothole area using the SFM algorithm to obtain a three-dimensional reconstructed point cloud.

[0053] Step 400: Due to the complexity of road conditions, the 3D reconstructed point cloud obtained by the SFM algorithm contains noisy points and the angle of the 3D reconstructed point cloud is tilted. Therefore, it is necessary to perform point cloud post-processing on the 3D reconstructed point cloud to obtain reconstructed point cloud calibration data. Point cloud post-processing includes road plane fitting, 3D rotation of point cloud, point cloud completion, and point cloud outlier detection.

[0054] Step 500: Convert the coordinate systems of the road surface laser point cloud and the reconstructed point cloud calibration data into pixel coordinate systems, and calculate the scale factor based on the grid search algorithm to determine the scale information of road surface potholes; the scale factor is the correspondence between the road surface laser point cloud and the reconstructed point cloud calibration data.

[0055] As a specific implementation method, step 500 specifically includes the following processing procedures:

[0056] 1) Determining the virtual depth of the pit: After post-processing of the 3D reconstructed point cloud, the point with the smallest depth along the Z-axis of the 3D reconstructed point cloud coordinate system is the maximum depth of the pit. Since the 3D reconstructed point cloud obtained by the SFM algorithm has no scale features, the maximum depth of the pit is called the maximum virtual depth of the pit. The values ​​corresponding to other 3D reconstructed point cloud points on the Z-axis of the 3D reconstructed point cloud coordinate system are the virtual depths of those points.

[0057] 2) Determining the scaling factor: 2D LiDAR has accurate ranging capability. The true depth of the road surface pothole is obtained by using 2D LiDAR, and grid search is used to match the LiDAR points with the three-dimensional reconstructed point cloud points, thereby obtaining the scaling factor between the virtual depth of the road surface pothole and the true depth of the road surface pothole.

[0058] The joint calibration of 2D LiDAR and camera enables them to work together in the same coordinate system by calibrating the spatial relationship between the two systems. The joint calibration also allows us to obtain the rotation and translation relationship between the 2D LiDAR coordinate system and the camera coordinate system.

[0059] The camera intrinsic and extrinsic parameters are calibrated using the computer vision toolbox in MATLAB. This process obtains the spatial relationship between the camera's pixel coordinate system and the world coordinate system. The camera's extrinsic parameters include the rotation and translation relationship between the camera coordinate system and the world coordinate system. The camera's intrinsic parameters include information such as the camera's focal length, optical center position, pixel scaling factor, and distortion coefficient.

[0060] Based on the joint calibration of the 2D LiDAR and the camera, as well as the calibration of the camera's intrinsic and extrinsic parameters, the position of the 2D LiDAR point in the camera pixel coordinate system is calculated.

[0061] Based on the pixel positions of the 2D LiDAR points in the camera coordinate system and the pixel positions of the 3D reconstructed point cloud points in the camera coordinate system, a 2D LiDAR point with the maximum reflection intensity and whose corresponding pixel position is located within the pit detection frame is selected, and its corresponding pixel position is denoted as C. A 21×21 pixel grid is constructed with this point as the center. Within this range, the pixel position corresponding to the 3D reconstructed point cloud point with the smallest Euclidean distance from point C is found, and this point is denoted as K. At this time, it is assumed that the depths of points C and K in the world coordinate system are the same.

[0062] The scaling factor is determined by the ratio of the depth of point C in the world coordinate system to the depth of point K in the 3D reconstructed point cloud coordinate system.

[0063] 3) Estimation of the true depth of road surface potholes: The true depth of potholes corresponding to all three-dimensional reconstructed point cloud points is calculated based on the scaling factor.

[0064] The scale of road potholes is restored by multiplying the scaling factor with the Z-axis value of the point cloud in the 3D reconstructed point cloud coordinate system, which is the true depth corresponding to all 3D reconstructed point cloud points in the world coordinate system, thereby realizing the restoration of the scale information of road potholes.

[0065] Based on the above technical solution, the following is provided: Figures 2-4 The example shown.

[0066] like Figure 2 The installation method of the camera and 2D LiDAR on the detection vehicle specifically includes: the detection vehicle 201 is any vehicle that can drive normally on the road surface; the road surface to be detected 202 is any road surface with one or more potholes 203 that allows vehicles to drive on; the data acquisition equipment includes: a camera 204 and a 2D LiDAR 205, which are deployed on the detection vehicle according to fixed requirements.

[0067] The fixed requirements for the data acquisition equipment can be as follows: the vertical distance Hc between the camera and the road surface is 1.5m, and the camera's shooting angle is parallel to the road plane, i.e., θ is 0°; the vertical distance Hc between the 2D lidar and the road surface is... l The height is 1.25m, and the angle δ between the laser plane emitted by the 2D LiDAR and the road plane is 10°. Under this setting, the 2D LiDAR can detect the road surface height at approximately 7.1m from the vehicle's front. Assuming that the coordinates of one of the point cloud points detected by the 2D LiDAR in the LiDAR coordinate system are (x, y), the vertical distance of this point from the road plane can be calculated using the following formula:

[0068] The vertical height of a point cloud point (x, y) above the ground: d = x * sin(δ) - H l H l This indicates the vertical distance between the 2D lidar and the road surface.

[0069] Based on the above formula, the vertical height of all 2D LiDAR point cloud points from the road plane can be calculated.

[0070] This embodiment also introduces the steps for segmenting and tracking road surface pothole images based on the YOLOv5 deep learning model and the DeepSORT object tracker, specifically including:

[0071] S1: Acquire a series of road surface pothole images based on the data acquisition equipment installed on the vehicle, and use Labelimg software to annotate the road surface pothole areas.

[0072] S2: Train the labeled image data based on the YOLOv5 deep learning model and obtain the trained weight file.

[0073] S3: Based on the trained weight file, identify the pothole region of the road image and obtain the pothole detection box.

[0074] S4: Based on the DeepSORT target tracker, the pit detection box is tracked and the recognition box is enlarged to twice its original size with the center of the box as the reference, so as to provide a data foundation for subsequent SFM 3D reconstruction.

[0075] Furthermore, this embodiment illustrates the main process of 3D reconstruction of a series of road surface pit images acquired by the DeepSORT target tracker using SFM (Structure from Motion), specifically including:

[0076] Camera intrinsic and extrinsic parameter calibration; assuming the camera coordinate system is the world coordinate system, the camera extrinsic parameters do not need further calibration. Here, the rotation matrix in the camera extrinsic parameters is denoted as R, and the translation matrix is ​​denoted as T; the focus is still on the calibration of the camera intrinsic parameters; using a series of checkerboard images with different poses captured by the camera, the camera intrinsic parameter K is automatically calculated using the calibration tools built into MATLAB or OpenCV.

[0077] The expression for the camera intrinsic parameter K is: Among them, f x The focal length, expressed in pixels along the x-axis, is represented by 's', which is the tangent parameter used when the image coordinate axes are not perfectly orthogonal. c x f represents the x-coordinate of the camera's principal point. y c represents the focal length in pixels along the y-axis. y This represents the y-coordinate of the camera's principal point.

[0078] The determination of the fundamental matrix F during camera movement is crucial. The fundamental matrix reflects the relative motion between cameras and forms the basis for 3D reconstruction of road surface pits. It is typically calculated using corresponding points in two images, with the most common method being the 8-point method. This method includes:

[0079] like Figure 3 As shown, assuming there exists a corresponding pair of points in the two images, with pixel coordinates (x1, y1) for point p1 in img1 and (x2, y2) for point p2 in img2, the following system of linear equations can be constructed:

[0080] The relationship between points p1 and p2 is as follows:

[0081] The system of equations is solved using the 8-point method. This system has 8 equations and 9 unknowns, so the fundamental matrix F can be obtained by the least squares method.

[0082] In the process of calculating corresponding points between two images, there are a variety of algorithms to choose from, such as Speed-Up Robust Feature Transform (SURF), Speed-Up Segmented Test Feature Transform (FAST), Scale-Invariant Feature Transform (SIFT), and the recently proposed BEBLID feature. Among them, the BEBLID feature descriptor has excellent performance in terms of both computation speed and accuracy. Therefore, the BEBLID feature descriptor is used to extract image feature points for matching.

[0083] Furthermore, based on the camera intrinsic parameters K and the fundamental matrix F, the essential matrix E during camera motion can be calculated; the formula for calculating the essential matrix E is: E = K T FK.

[0084] Once the motion relationship between the cameras is determined, triangulation can be used to calculate the three-dimensional spatial coordinates of the target point, specifically including:

[0085] The essential matrix E is decomposed using SVD to extract the rotation matrix r and translation vector t between cameras, where E = UΣV T Here, U represents an orthogonal matrix, and the column vectors of U are E·E T Let E be an eigenvector, Σ denote a diagonal matrix with singular values ​​of matrix E on the diagonal, and V denote an orthogonal matrix whose column vectors are E. T • Eigenvectors of E.

[0086] The formula for calculating the rotation matrix r and the translation vector t is: r = UWV T , t=Ut z .

[0087] Where t z The translation vector is extracted using SVD, where W is a fixed matrix defined as:

[0088]

[0089] Converting the image points to a normalized coordinate system, the normalized camera coordinates of image coordinates p1 and p2 are:

[0090] x1'=K -1 x1

[0091] x2'=K -1 x2

[0092] The 3D point P = (x, y, z) is calculated using triangulation. The formula is: P = λ1x1' = λ2(rx2' + t), where λ1 and λ2 are scale factors (representing the distance from the point to the camera), obtained by minimizing the reprojection error.

[0093] Post-processing of 3D reconstructed point clouds specifically includes:

[0094] Road plane fitting: In the 3D reconstructed point cloud coordinate system, let the road plane equation be expressed as: ax + by + cz + d = 0. For all 3D points in the 3D reconstructed point cloud coordinate system, the least squares method is used to fit the road plane equation coefficients a, b, c, and d.

[0095] The point cloud is rotated in 3D, and the point cloud is calibrated using the following calculation formula:

[0096]

[0097]

[0098] Among them, u x u y Intermediate variables set up for ease of calculation.

[0099] The calibrated 3D point cloud coordinates are: P1 = RP.

[0100] Point cloud completion processing based on membership cloud theory uses the 3D coordinates of a point (x, y, z) as the expected value of each dimension of the 3D membership cloud model. A 3D normal cloud generator produces three one-dimensional standard normal random numbers t1, t2, and t3. The formula for calculating the coordinates of the points generated by the cloud generator is as follows:

[0101] x i =Enx·t1+x

[0102] y i =Eny·t2+y

[0103] z i =Enz·t3+z

[0104] Furthermore, outlier detection based on isolated forest is performed on the completed point cloud to eliminate the influence of noise points on the estimation of negative obstacle depth on the road surface.

[0105] The specific steps for restoring the dimensions of road potholes are as follows:

[0106] The joint calibration of the 2D LiDAR and camera yields the rotation matrix R when transforming the 2D LiDAR coordinate system to the camera coordinate system. lc Translation matrix T lcThe transformation relationship from a point in a 2D LiDAR to the camera pixel coordinate system can be expressed as:

[0107]

[0108] Where u represents the pixel x-coordinate, v represents the pixel y-coordinate, and x wl The x-axis represents the world coordinate of the lidar point, and the y-axis represents the world coordinate of the lidar point. wl The z-axis represents the world coordinates of the lidar point. wl This represents the actual z-axis coordinate of the lidar point, which is 0 in this case.

[0109] Based on the transformation relationship obtained from joint calibration, the SFM algorithm can transform both 2D LiDAR points and 3D reconstructed point clouds to the pixel coordinate system. Assuming that the pixel positions of the 2D LiDAR points and the 3D reconstructed point cloud points in the camera coordinate system are known, a 2D LiDAR point with the maximum reflection intensity and whose corresponding pixel position is located within the pit detection box is selected, and its corresponding pixel position is denoted as C. A 21×21 pixel grid is constructed with this point as the center. Within this range, the pixel position corresponding to the 3D reconstructed point cloud point with the smallest Euclidean distance from point C is found, and this point is denoted as K. At this time, it is assumed that the depths of points C and K in the world coordinate system are the same.

[0110] The scaling factor is the ratio of the depth of point C in the world coordinate system to the depth of point K in the 3D reconstructed point cloud coordinate system.

[0111] The scale of road potholes is restored by multiplying the scale factor by the Z-axis value of the point cloud points in the 3D reconstructed point cloud coordinate system, which is the true depth of these point cloud points in the world coordinate system, thus realizing the restoration of the scale information of road potholes.

[0112] Based on the scaling factor, the true depth corresponding to the point with the smallest Z-axis direction in the 3D reconstructed point cloud is the estimated depth of the road surface pothole.

[0113] Furthermore, the present invention also provides, as Figure 4 An electronic device is shown, including a memory 501 and a processor 503. The memory 501 stores a computer program 502, and the processor 503 runs the computer program 502 to enable the electronic device to perform the automatic detection and depth estimation method for road potholes as described above.

[0114] The present invention also provides a computer-readable storage medium storing a computer program 502, which, when executed by a processor, implements the method for automatic detection and depth estimation of road surface potholes as described above.

[0115] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0116] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for automatic detection and depth estimation of potholes in a road surface, characterized in that, The method comprises the following steps: deploying a collection device at a fixed interval and angle at the vehicle head position, and acquiring road surface images and road surface laser point clouds by using the collection device; the collection device comprises a camera and a 2D laser radar; detecting and tracking the pothole area in the road surface image by using a YOLOv5 deep learning model and a DeepSORT target tracker to obtain road surface pothole area images according to the time sequence change; reconstructing the road surface pothole area image by using an SFM algorithm to obtain a three-dimensional reconstructed point cloud; performing point cloud post-processing on the three-dimensional reconstructed point cloud to obtain reconstructed point cloud calibration data; the point cloud post-processing comprises road plane fitting, point cloud three-dimensional rotation, point cloud completion and point cloud outlier detection; converting the coordinate system of the road surface laser point cloud and the reconstructed point cloud calibration data into a pixel coordinate system, and calculating a scale factor based on a grid search algorithm to determine road surface pothole scale information; the scale factor is the corresponding relationship between the road surface laser point cloud and the reconstructed point cloud calibration data; the method of detecting and tracking the pothole area in the road surface image by using the YOLOv5 deep learning model and the DeepSORT target tracker to obtain road surface pothole area images according to the time sequence change specifically comprises: detecting the pixel position of the pothole in the image by using the YOLOv5 deep learning model to obtain a road surface pothole area image; tracking the position by using the DeepSORT target tracker to obtain road surface pothole area images according to the time sequence change; the process of detecting the pixel position of the pothole in the image by using the YOLOv5 deep learning model specifically comprises: performing convolution operation on the road surface image through multiple downsampling layers to generate a first feature map; performing transpose convolution operation on the first feature map through the upsampling layer corresponding to each downsampling layer to obtain a second feature map with the same size as the original road surface image; performing prediction on the second feature map through a Softmax layer and a regression task, and reducing the number of repeated detection boxes of the road surface pothole by using a non-maximum suppression method to obtain the final road surface pothole area image; the process of tracking the position by using the DeepSORT target tracker specifically comprises: extracting the appearance feature of the road surface pothole area image, and tracking the road surface pothole based on a Kalman filtering algorithm to predict the position of the road surface pothole in the next frame of image, and updating the position of the pothole when the pothole is detected in the next frame; judging whether the road surface pothole targets in the current frame and the previous frame are the same target by using the Hungarian algorithm, assigning a unique ID to each road surface pothole, and expanding the detection box to twice the original size based on the center of the box to obtain road surface pothole area images according to the time sequence change; the process of performing point cloud post-processing on the three-dimensional reconstructed point cloud specifically comprises: performing least square fitting on all three-dimensional points in the coordinate system of the three-dimensional reconstructed point cloud to obtain road plane equation coefficients; performing point cloud three-dimensional rotation based on the road plane equation coefficients to obtain calibrated three-dimensional point cloud coordinates; Based on the membership cloud theory, the calibrated three-dimensional point cloud coordinates are completed to obtain completed point cloud coordinates, denoted as: ； wherein (x, y, z) is the point cloud coordinate, x i , y i , z i ) is the completed point cloud coordinate, (x, y, z) is the point cloud coordinate, x , y , z ) is the point cloud coordinate, t 1 , t 2 and t 3 is a one-dimensional standard normal random number, Enx , Eny , Enz are the entropies of the three-dimensional point cloud in x, y, z directions, respectively, indicating the dispersion degree of the point cloud; The completed point cloud coordinates are subjected to outlier detection based on isolated forest to exclude the influence of noise points on road surface negative obstacle depth estimation.

2. The method of claim 1, wherein, The calculation process of the scale factor is as follows: Based on the grid search algorithm, a 2D laser radar point with maximum reflection intensity and pixel position located in the pit detection frame is selected, and the pixel position corresponding to the 2D laser radar point is recorded as C. A 21*21 pixel grid is constructed with point C as the center. In the range of the pixel grid, the pixel position corresponding to the reconstructed point cloud with the minimum Euclidean distance from point C is recorded as K. The ratio of the depth of point C in the world coordinate system to the depth of point K in the three-dimensional reconstructed point cloud coordinate system is determined as the scale factor.

3. The method of claim 1, wherein, The calculation process of the road pit scale information is as follows: The scale factor is multiplied by the Z-axis value of the point cloud in the three-dimensional reconstructed point cloud coordinate system to obtain the real depth of each point cloud in the world coordinate system, thereby determining the road pit scale information.

4. An electronic device, comprising: The electronic device comprises a memory and a processor. The memory is used to store a computer program. The processor runs the computer program to enable the electronic device to perform the road pit automatic detection and depth estimation method according to any one of claims 1-3.

5. A computer readable storage medium, characterized in that, The computer program is stored in the memory and is executed by the processor to realize the road pit automatic detection and depth estimation method according to any one of claims 1-3.

Images