A method and system for calculating road pothole depth based on monocular camera images

Abstract

The present invention belongs to the technical fields of road engineering, municipal engineering, and highway engineering, and relates to a method and system for calculating pothole depth based on monocular camera images. The method comprises: acquiring an image to be detected, extracting the vertical coordinate of the vanishing point of the image, as well as 3D information and height of the pothole; combining the 3D information and height of the pothole based on the vertical coordinate of the vanishing point to obtain the actual distance from the center point of the pothole top surface to the camera and the pixel distance of the pothole depth in a horizontal view; and combining the actual distance and pixel distance with the distance from the monocular camera imaging plane to the camera to obtain the actual pothole depth. The present invention can intelligently detect pothole locations while rapidly calculating pothole depth information. It also features low implementation cost, low computational overhead, and high operational efficiency.

Description

Technical Field

[0001] The present invention relates to the technical fields of road engineering, municipal engineering and highway engineering, and in particular to a method and system for calculating road pothole depth based on monocular camera images. Background Art

[0002] With the acceleration of urbanization, road maintenance and management are becoming increasingly important. Potholes are a common form of road damage, increasing vehicle wear and tear and compromising driving safety. Traditional pothole detection methods rely on manual inspections, which are inefficient and susceptible to subjective factors, resulting in inaccurate results. Recent advances in computer vision technology have provided new solutions for automated detection. Using a monocular camera to capture road images and combining it with deep learning object detection technology can automatically identify potholes.

[0003] Among the existing monocular camera-based pothole detection methods, there are mainly the following technical routes:

[0004] Methods based on traditional image processing: These methods usually use edge detection, morphological operations, and other methods to extract pit features. However, these methods are less effective in pit recognition against complex backgrounds and cannot directly obtain pit depth information:

[0005] Instability in feature extraction: Traditional image processing methods rely on hand-crafted features, such as edge detection and threshold segmentation. In real-world applications, road conditions are complex and variable, with shadows, reflections, and texture variations. These factors can interfere with the accurate extraction of pothole edges, significantly reducing feature extraction accuracy in complex scenarios and making false and missed detections more likely. For example, on a road exposed to direct sunlight, pothole edges can be blurred by reflections, making it difficult for edge detection-based methods to accurately locate the pothole boundaries.

[0006] Limitations of depth information acquisition: These methods only process two-dimensional images and cannot directly obtain pothole depth information. However, pothole depth is crucial for road maintenance decisions. Potholes of different depths require different maintenance measures and priorities. Lack of depth information can make maintenance work less targeted and timely.

[0007] Methods based on binocular or multi-camera detection: Although binocular or multi-camera detection can obtain depth information through parallax calculation, the equipment cost is high and a complex calibration process is required. In addition, the installation and maintenance of multi-camera systems in practical applications are also difficult:

[0008] High equipment costs: Binocular or multi-camera systems require multiple cameras, which not only increases hardware costs but also requires additional equipment such as mounting brackets and devices to ensure camera synchronization. This overall high equipment cost makes it unaffordable for some road maintenance projects with limited budgets.

[0009] Complex calibration process: To ensure the accuracy of the relative positions of multiple cameras, a complex calibration process is required. This involves numerous parameter adjustments, and even the slightest deviation can lead to significant errors in depth calculations. Furthermore, if the device moves during use due to vehicle vibration or other reasons, recalibration is necessary, which undoubtedly increases the difficulty and cost of system maintenance.

[0010] Difficulty in installation and maintenance: The installation of a multi-camera system requires careful planning of the position and angle of each camera to ensure consistent coverage and viewing angle, enabling accurate depth information acquisition. However, achieving this requirement is not easy in actual installation environments. If a camera fails or performance degrades, the entire system may need to be readjusted and optimized, which is complex and time-consuming.

[0011] Deep learning-based methods: In recent years, deep learning has made significant progress in image recognition. By training neural network models, efficient pothole detection can be achieved. However, most existing methods only provide pothole location information on a two-dimensional image and have difficulty directly calculating pothole depth data.

[0012] Lack of Depth Information: Most current deep learning-based pothole detection methods focus primarily on detecting objects in two-dimensional images. While they excel at identifying pothole locations, they lack depth data. This makes it difficult to determine appropriate maintenance measures and priorities based on pothole depth during road maintenance, potentially leading to delayed maintenance, increased repair costs, and increased driving risks. For example, deep potholes, if not discovered and repaired promptly, can cause greater damage to vehicles and even lead to accidents.

[0013] In summary, existing monocular camera-based pothole detection methods face varying degrees of challenges in feature extraction, depth information acquisition, equipment cost, calibration complexity, generalization, real-time performance, and computational resource requirements. In particular, due to the timely maintenance requirements for potholes of varying depths, pothole depth information is essential for actual maintenance. Therefore, developing a method that can both efficiently and accurately detect pothole locations and precisely calculate their depths is a pressing technical challenge. Summary of the Invention

[0014] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to propose a method and system for calculating the pothole depth of a road surface based on monocular camera images. While intelligently detecting the pothole location, the pothole depth information can be quickly calculated. The present invention has the characteristics of low implementation cost, low computational cost, and high operating efficiency.

[0015] To achieve the above object, the present invention provides the following solutions:

[0016] A method for calculating pothole depth based on monocular camera images, comprising:

[0017] Acquire an image to be detected, and extract the vertical coordinate of the vanishing point, the 3D information of the pit, and the pit height of the image to be detected;

[0018] Obtaining, according to the vanishing point vertical coordinate, the actual distance from the center point of the top surface of the pit to the camera and the pixel distance of the pit depth in the horizontal view in combination with the pit 3D information and the pit height;

[0019] The actual depth of the pit is obtained by using the actual distance and the pixel distance in combination with the distance from the imaging plane of the monocular camera to the camera.

[0020] Optionally, extracting the vertical coordinate of the vanishing point of the image to be detected includes:

[0021] The image to be detected is input into an image vanishing point detection model to obtain the vertical coordinate of the vanishing point; the image vanishing point detection model is trained using a first training set, wherein the first training set includes images marked with vanishing point information.

[0022] Optionally, extracting the pit 3D information and the pit height includes:

[0023] The image to be detected is input into a pit 3D detection model to obtain the pit 3D information and the pit height; the pit 3D detection model is trained using a second training set, where the second training set includes: image data containing pit 3D information; the Depth Block module in the pit 3D detection model is used to extract target depth feature information, and a target depth information representation is added to an output layer.

[0024] Optionally, obtaining the actual distance from the center point of the top surface of the pit to the camera includes:

[0025] Calculating the vertical coordinate of the vanishing point to obtain the current pitch angle, obtaining a mapping matrix from the image to the overhead image based on the current pitch angle and a constant matrix for the image to the overhead image, and further using the vertical coordinate of the vanishing point to obtain the distance from the bottom edge of the image to the camera;

[0026] Based on the 3D information of the pit, the coordinates of the center point of the top surface of the pit in the image are obtained, and the coordinates of the center point of the bottom surface of the pit in the image are obtained according to the height of the pit;

[0027] Calculating a mapping matrix from the image to the top-view image and the coordinates of the center point of the upper top surface of the pit in the image to obtain the coordinates of the center point of the upper top surface of the pit in the image in the top-view image, and calculating the coordinates of the center point of the upper top surface of the pit in the image in the top-view image to obtain the actual distance from the center point of the upper top surface of the pit to the lower edge of the original image;

[0028] The actual distance from the center point of the top surface of the pit to the lower edge of the original image is added to the distance from the lower edge of the image to the camera to obtain the actual distance from the center point of the top surface of the pit to the camera.

[0029] Optionally, obtaining a constant matrix from the image to the overhead image includes:

[0030] Calibrate a monocular camera to determine the camera pitch angle, and obtain a rotation matrix based on the camera pitch angle; the rotation matrix includes: a first rotation matrix and a second rotation matrix:

[0031]

[0032] in, is the rotation matrix of the y-axis rotation angle relative to the world coordinate system, is the pitch angle of the camera;

[0033] Acquire an original image through a calibrated monocular camera, perform calibration calculation on the original image, and obtain a mapping matrix from the original image to the overhead image;

[0034] The first rotation matrix is combined with the mapping matrix from the original image to the overhead image to obtain a constant matrix from the image to the overhead image:

[0035]

[0036] in, is the mapping matrix from image to overhead image, is the first rotation moment, is the constant matrix from image to overhead image.

[0037] Optionally, obtaining the actual distance from the center point of the top surface of the pit to the lower edge of the original image includes:

[0038]

[0039] in, is the height of the top view, is the ordinate of point P in the top view, is the actual distance from point P to the lower edge of the top view.

[0040] Optionally, obtaining the pixel distance of the pit depth in the horizontal view includes:

[0041] Obtaining an image-to-horizontal-view mapping matrix based on the current pitch angle and a constant matrix for image-to-horizontal-view, and obtaining coordinates of the center point of the upper top surface and the center coordinates of the lower bottom surface of the pit in the image based on the coordinates of the center point of the upper top surface and the center coordinates of the lower bottom surface of the pit in the image in combination with the image-to-horizontal-view mapping matrix;

[0042] The pixel distance of the pit depth in the horizontal view is acquired based on the coordinates of the center point of the upper top surface in the horizontal view and the coordinates of the center point of the lower bottom surface in the horizontal view.

[0043] Optionally, acquiring a constant matrix of the image to a horizontal view includes:

[0044] Perform checkerboard calibration calculation on the original image to obtain the mapping matrix from the original image to the horizontal view. Use the second rotation matrix in combination with the mapping matrix from the original image to the horizontal view to obtain the constant matrix from the image to the horizontal view:

[0045]

[0046] in, is the mapping matrix from image to horizontal view, is the second rotation matrix, A constant matrix that converts the image to a horizontal view.

[0047] Optionally, obtaining the actual depth of the pit includes:

[0048]

[0049] in, is the actual depth of the pit, is the pixel distance of the ground pit depth in the horizontal view, is the distance from the imaging plane to the monocular camera position, is the position from the pit to the camera.

[0050] To achieve the above objectives, the present invention further provides a road pothole depth calculation system based on monocular camera images, comprising:

[0051] An image acquisition module, used to acquire the image to be detected;

[0052] The cloud storage computing module extracts the vertical coordinate of the vanishing point, the 3D information of the pit, and the pit height of the image to be detected. The module obtains the actual distance from the center point of the top surface of the pit to the camera and the pixel distance of the pit depth in the horizontal view based on the vertical coordinate of the vanishing point in combination with the 3D information of the pit and the pit height. The module then uses the actual distance and the pixel distance in combination with the distance from the imaging plane of the monocular camera to the camera to obtain the actual depth of the pit.

[0053] The beneficial effects of the present invention are:

[0054] This invention can be used in conjunction with an intelligent road surface defect detection system to detect pothole locations and output pothole height information. This system can calculate pothole depth through simple calibration, complementing the current monocular camera-based pothole detection method, which lacks depth information.

[0055] In terms of execution efficiency, compared with the low efficiency and high cost of traditional pit depth measurement, the present invention can basically achieve no manual operation, reduce measurement costs and improve efficiency.

[0056] Compared with binocular or multi-camera based pothole depth calculation, the present invention uses a monocular camera as the acquisition device. Through camera calibration and pixel distance conversion, it can achieve accurate measurement of pothole depth. This not only reduces hardware costs, but also simplifies the calibration process, making the system easier to install and maintain.

[0057] Compared with relying solely on deep learning target detection to achieve road pothole detection, the present invention can not only intelligently detect the location of potholes, but also obtain the depth information of potholes, providing essential data support for intelligent maintenance of potholes.

[0058] The method of detecting image vanishing points and correcting the image mapping matrix in real time introduced in the present invention greatly improves the anti-interference capability of the overall system and the accuracy of pit depth calculation.

[0059] The deep learning 3D pit detection algorithm based on monocular camera images used in the present invention optimizes the traditional deep learning target detection algorithm. While introducing the depth information feature map, it optimizes the network structure and enriches the data samples. This enables the deep learning 3D pit detection algorithm used in the present invention to output both the position information and the depth information of the pit. The algorithm also has strong generalization ability and high accuracy.

[0060] In summary, the present invention can intelligently detect the location of the pit and quickly calculate the pit depth information. The present invention has the characteristics of low implementation cost, low calculation cost, and high operation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 This is a schematic diagram of the installation of an edge acquisition device according to an embodiment of the present invention;

[0063] Figure 2 A schematic diagram of the pitch angle of a camera according to an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of imaging with a monocular camera according to an embodiment of the present invention;

[0065] Figure 4 A calibration original image for obtaining a top-view image mapping matrix according to an embodiment of the present invention;

[0066] Figure 5 A schematic diagram of 3D marking / detection of potholes in a road surface according to an embodiment of the present invention;

[0067] Figure 6 A schematic diagram of the deep learning 3D object detection algorithm flow in an embodiment of the present invention;

[0068] Figure 7 This is a flow chart of the equipment installation and calibration phase according to an embodiment of the present invention;

[0069] Figure 8 This is a flow chart of a method for calculating road pothole depth based on monocular camera images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0072] Explanation of the idea of dynamically correcting the mapping matrix of the camera's original image to the top-view image and the mapping matrix of the original image to the horizontal view: It is known that the mapping matrix of the original image captured by the camera is converted to the top-view image and the pitch angle of the camera (Camera coordinate system and world coordinate system The angle of the y-axis), roll angle (Camera coordinate system and world coordinate system x-axis angle), heading angle (Camera coordinate system and world coordinate system For example, when the roll angle When the pitch angle changes, the image captured by the camera will appear tilted up and down in the vertical direction. When the heading angle changes, the relative position of the object in the image in the vertical direction will change. When the heading angle changes, it determines the left and right rotation of the camera shooting direction. When taking a panoramic photo, by changing the heading angle You can get scenes in different directions.

[0073] The conversion matrix formula from the camera original image to the top view image is as follows:

[0074]

[0075]

[0076] in, is the mapping matrix from the camera original image to the top view image, is the camera's rotation matrix, is the translation matrix, Camera intrinsic parameter matrix; Camera intrinsic parameter matrix The general form is as follows:

[0077]

[0078] in, and They are images The focal lengths in the a-axis and y-axis directions, and It is the coordinate value of the principal point in the image coordinate system (the pixel coordinate corresponding to the intersection of the optical axis and the image plane).

[0079] The general form of the translation vector T is as follows: ;

[0080] The rotation matrix R is a 3x3 matrix that can be composed of the rotation angles of the camera coordinate system relative to the three coordinate axes of the world coordinate system. Multiply them together to get, as shown in formula (2).

[0081] is the rotation matrix of the camera coordinate system relative to the three coordinate axes of the world coordinate system, and the pitch angle, roll angle and yaw angle of the camera are hour:

[0082]

[0083]

[0084]

[0085] It's important to note that, for the purposes of this invention, the purpose of acquiring the top-view image is solely to accurately calculate the actual ground distance between two ground points in the top-view image. Therefore, this process has no direct connection to the location of the coordinates of the world coordinate system's XY plane (ground origin). Therefore, the top-view mapping matrix M of this invention can be constructed without considering the heading angle (i.e., the angle between the camera coordinate system and the z-axis of the world coordinate system). When the camera's roll angle and mounting position remain constant, the camera coordinate system's roll matrix, translation matrix, and intrinsic parameter matrix are also fixed. In this case, the mapping matrix from the original image to the top-view image can be considered a function dependent solely on the pitch angle, as shown below:

[0086]

[0087] Where A is a 3x3 constant matrix.

[0088] Similarly, since the camera is affected by the pitch angle and roll angle when shooting the original image, the image obtained is not a horizontal perspective. The present invention can transform the coordinate system of the original image into an ideal coordinate system equivalent to the camera being in a horizontal state through a series of coordinate transformation operations. In this horizontal view coordinate system, there is a significant feature, that is, the same straight line perpendicular to the ground has the same horizontal coordinate in the horizontal view. Figure 2 As shown, during the camera installation phase, when the camera's pitch angle is the pitch angle value When the camera coordinate system rotates from the initial state to the top view, the rotation angle in the y-axis direction is ( ), the rotation matrix at this time is ; When rotating to a horizontal viewing angle, the rotation angle along the y-axis is , the rotation matrix at this time is Based on the above principles and related formulas, it can be seen that when the camera pitch angle is When , the mapping matrix from the original image to the horizontal view can be expressed as follows:

[0089] (8)

[0090] When the camera installation position and camera roll angle remain unchanged, formula (7) and formula (8) are only related to the change of the camera's pitch angle. The mapping matrix from the camera's original image to the top view image and the mapping matrix from the camera's original image to the horizontal view can be updated in real time by monitoring the change of the camera's pitch angle.

[0091] An explanation of the relationship between the vanishing point of the original image and the camera's pitch angle, as well as the relationship between the vanishing point of the original image and the distance from the bottom edge of the original image to the camera.

[0092] It's known that the vertical coordinate of the vanishing point in the original image is linearly related to the camera's pitch angle and the distance from the bottom edge of the original image to the camera. Therefore, after installing the camera and fixing the mounting position and roll angle, manually adjust the camera's pitch angle, capture original images at different pitch angles, and measure the actual distance D1 from the bottom edge of the original image to the camera at each pitch angle. Then, use a deep learning image vanishing point detection algorithm to detect the vanishing points in the original images and obtain the vertical coordinate of the vanishing point for each original image. This results in a set of data, Data1, consisting of the camera pitch angle and the vertical coordinate of the vanishing point in the original image, and a set of data, Data2, consisting of the vertical coordinate of the vanishing point in the original image and the distance from the bottom edge of the original image to the camera. By performing a linear fit on Data1, we obtain the linear equation for the relationship between the vertical coordinate of the original image's vanishing point and the camera's pitch angle, as shown in Equation 9. By performing a linear fit on Data2, we obtain the linear equation for the relationship between the vertical coordinate of the original image's vanishing point and the distance from the bottom edge of the original image to the camera, as shown in Equation 10.

[0093]

[0094]

[0095] Where x is the vertical coordinate value of the vanishing point of the original image.

[0096] During real-time acquisition, the vanishing point of the original image is detected using a deep learning image vanishing point detection algorithm. The real-time pitch angle of the camera is calculated according to formula (9), and the new mapping matrix can be calculated using formula (7). The distance from the bottom edge of the original image to the camera can also be calculated using formula (10).

[0097] Explanation of the relationship between the height of an object in the horizontal view and the pixel distance in the horizontal view:

[0098] The imaging principle of a monocular camera can be compared to pinhole imaging, such as Figure 3 As shown, the ratio of the actual height RH of the object to the actual distance Da from the object to the camera is equal to the ratio of the pixel distance of the object in the image to the distance Db from the imaging plane to the camera position. The formula is as follows:

[0099]

[0100]

[0101]

[0102] Where Db is the distance from the imaging plane to the monocular camera position, PH is the pixel distance of the object in the horizontal view, RH is the actual height of the object, Da is the actual distance from the object to the camera, Dc is the actual distance from the object to the lower edge of the original image, and Dl is the actual distance from the lower edge of the original image to the camera.

[0103] During the calibration phase, the object's actual height RH and the horizontal distance Da from the object to the camera can be obtained through on-site measurement. The object's pixel distance PH in the horizontal view can be calculated from the object's ground point pixel coordinates and vertex pixel coordinates in the horizontal view. The unknown parameter Db is the parameter that this method requires calibration and calculation. Therefore, during calibration, Db is calculated by collecting the values of Da, PH, and RH. Typically, multiple sets of RH, Da, and PH values are obtained, and multiple Db values are calculated. The average of these Db values is taken as the calculation parameter for this method, resulting in the formula for calculating the object's height in the horizontal view:

[0104]

[0105] Formula (14) shows that when the pixel distance PH of the ground pit depth in the horizontal view, the position Da of the pit to the camera, and the distance Db from the imaging plane to the monocular camera position calculated during calibration are known, the actual depth RH of the pit can be obtained.

[0106] Introduction to deep learning image vanishing point detection algorithm:

[0107] The deep learning image vanishing point detection algorithm is a key technical approach used in this paper to obtain key information such as the camera pitch angle and the distance from the bottom edge of the original image to the camera. Based on deep learning principles, this algorithm is trained on a large number of images with annotated vanishing point information, enabling the model to automatically identify the location of vanishing points in images.

[0108] During training, the algorithm learns the characteristic patterns of vanishing points in images across different scenes and camera poses. For example, for images of road scenes, the algorithm can identify points where the road's extension, building edges, and other features converge in the image—these are called vanishing points. The location of these vanishing points is closely related to the camera's pose (particularly the pitch angle) and also has a certain geometric relationship with the distance from the bottom edge of the image to the camera.

[0109] The algorithm is based on a convolutional neural network (CNN) architecture, leveraging its powerful feature extraction capabilities. It progressively processes the input image through a combination of multiple convolutional, pooling, and fully connected layers. Convolutional layers extract local image features, pooling layers reduce the data dimensionality, and fully connected layers map the extracted features to predicted outputs at the vanishing point.

[0110] To improve the algorithm's accuracy and generalization capabilities, the training data covers images from a variety of road conditions, weather conditions, and time of day. This allows the algorithm to adapt to image variations in different environments and accurately detect the vanishing point, providing a reliable basis for subsequent calculations of the camera's pitch angle and the distance from the bottom edge of the image to the camera.

[0111] Deep learning 3D pothole detection algorithm: The deep learning 3D pothole detection algorithm is one of the core algorithms for calculating pothole depth in this invention. It aims to extract 3D information of potholes from 2D images. Figure 5 As shown, it includes the quadrilateral box information of the top surface of the pit and the pixel height of the pit depth in the image.

[0112] Traditional deep learning 2D object detection algorithms for monocular camera images mainly rely on deep learning neural networks to extract target features from the input RGB image. However, while directly using the 2D object detection algorithm to learn the target's feature expression from the original image, it is difficult to learn the target's depth information features. In order to improve the accuracy of pit depth information prediction, the present invention uses a deep learning pit 3D detection algorithm (such as Figure 6 Compared with traditional deep learning algorithms, the following optimizations are achieved:

[0113] 1. Increase input information: By using a mature depth estimation algorithm to extract full-image depth information, this information is used as an independent feature map and concatenated with the original RGB image to form a 4D input. This provides the deep learning network with more direct and accurate full-image depth information, thereby reducing the difficulty of learning target depth information.

[0114] 2. Optimize network structure: Compared with traditional 2D target detection algorithms, the deep learning 3D pit detection algorithm used in this invention adds a network structure (Depth Block) for fitting target depth information, making it easier to extract target depth feature information, thereby improving the learning ability of target depth information.

[0115] 3. Enhanced output layer information: The depth information representation of the target is added to the output layer of the algorithm, so that the deep learning 3D pit detection algorithm outputs the target depth information at the same time as the target polygon position information, thereby achieving the goal of 3D pit detection.

[0116] When labeling training data, such as Figure 5As shown in the figure, the deep learning 3D pit detection algorithm used in this invention uses quadrilaterals to mark the top surface of the pits. This annotation method can more accurately describe the shape and position of the pits in the image. Furthermore, the pit depth is clearly marked at the pixel height in the image, allowing the algorithm to learn the relationship between pit depth and image features, thereby accurately outputting 3D pit information during the detection process.

[0117] During training, the present invention uses a large amount of manually annotated image data containing 3D information about potholes. This data was collected from actual scenes with different road conditions (such as highways, urban roads, rural roads, etc.), different weather conditions (sunny, rainy, cloudy, etc.), and different times (daytime, nighttime, etc.) to ensure that the algorithm has good generalization capabilities.

[0118] During inference, the deep learning 3D pit detection algorithm used in this invention follows a similar process to traditional deep learning object detection networks. First, a convolutional neural network extracts feature maps of image and depth information. Next, a region proposal network (RPN) is used to generate candidate regions that may contain pits. These candidate regions undergo further classification and regression to determine the precise pit location (rectangular box information) and the pit depth in pixel height within the image.

[0119] Based on the above information, this embodiment discloses a method for calculating pothole depth based on a monocular camera image. The method includes: obtaining an image to be inspected, extracting the vertical coordinate of the vanishing point of the image, as well as the 3D pothole information and height; obtaining the actual distance from the center point of the pothole top surface to the camera and the pixel distance of the pothole depth in a horizontal view based on the vertical coordinate of the vanishing point, combined with the 3D pothole information and the pothole height; and calculating the actual pothole depth using the actual distance and pixel distance combined with the distance from the monocular camera imaging plane to the camera.

[0120] Furthermore, obtaining the actual distance from the center point of the upper top surface of the pit to the camera includes: calculating the vertical coordinate of the vanishing point, obtaining the current pitch angle, obtaining the image-to-top image mapping matrix based on the current pitch angle combined with the constant matrix of the image to the top view image, and further using the vertical coordinate of the vanishing point to obtain the distance from the lower edge of the image to the camera; obtaining the coordinates of the center point of the upper top surface of the pit in the image based on the 3D information of the pit, and obtaining the coordinates of the center point of the lower bottom surface of the pit in the image through the pit height; calculating the mapping matrix from the image to the top view image and the coordinates of the center point of the upper top surface of the pit in the image, obtaining the coordinates of the center point of the upper top surface of the pit in the top view image, and calculating the coordinates of the center point of the upper top surface of the pit in the top view image, and obtaining the actual distance from the center point of the upper top surface of the pit to the lower edge of the original image; adding the actual distance from the center point of the upper top surface of the pit to the lower edge of the original image to the distance from the lower edge of the image to the camera to obtain the actual distance from the center point of the upper top surface of the pit to the camera.

[0121] Obtaining the pixel distance of the pit depth in the horizontal view includes: obtaining a mapping matrix from the image to the horizontal view based on the current pitch angle in combination with a constant matrix from the image to the horizontal view; obtaining the coordinates of the center point of the upper top surface of the pit and the center coordinates of the lower bottom surface of the pit in the image in combination with the mapping matrix from the image to the horizontal view; and obtaining the pixel distance of the pit depth in the horizontal view based on the coordinates of the center point of the upper top surface and the coordinates of the center coordinates of the lower bottom surface in the horizontal view.

[0122] Specifically: installation and calibration of equipment, such as Figure 7 shown.

[0123] S101: Install the equipment, install the monocular camera, edge control machine, and positioning module on the collection vehicle.

[0124] S102: Fitting to obtain the relationship between the vertical coordinate of the vanishing point of the original image and the camera pitch angle , and the relationship between the vertical coordinate of the vanishing point of the original image and the distance from the bottom edge of the original image to the camera The specific method is as follows.

[0125] After fixing the camera mounting position and camera roll angle, adjust the pitch angle of the monocular camera, collect multiple sets of original images at different pitch angles, and measure the distance from the bottom edge of each original image to the camera. Use the deep learning image vanishing point detection algorithm to detect the vertical coordinate of the vanishing point of each original image. Obtain a set of data Data1 consisting of the camera pitch angle and the vertical coordinate of the image vanishing point, and fit the relationship between the vertical coordinate of the vanishing point of the original image and the pitch angle. , get a set of data Data2 consisting of the vertical coordinates of the vanishing point of the original image and the distance from the bottom edge of the original image to the camera, and get the relationship between the vertical coordinates of the vanishing point of the original image and the distance from the bottom edge of the original image to the camera by fitting .

[0126] S103: After adjusting the appropriate pitch angle, fix the camera pitch angle. Measure the current camera pitch angle. , use formula (5) to calculate the rotation matrix at this time and .

[0127] S104: Obtain the mapping matrix M from the original image to the overhead image through calibration calculation.

[0128] The original image to overhead image mapping matrix M can be obtained by knowing the coordinate information of the points in two pixel spaces (the original image pixel space and the overhead image pixel space); the specific method is as follows.

[0129] By laying a chessboard of known size on the road, the lower edge of the chessboard coincides with the lower edge of the original image, and the original image of the monocular camera is collected at this time. Figure 4As shown, the pixel coordinates of the four corner points of the chessboard in the original image are [(xtl, ytl), (xtr, ytr), (xbl, RawImageH), (xbr, RawImageH)], where (xtl, ytl) represents the coordinates of the upper left corner of the chessboard in the original image; (xtr, ytr) represents the coordinates of the upper right corner of the chessboard in the original image; (xbl, RawImageH) represents the coordinates of the lower left corner of the chessboard in the original image, and RawImageH is the height of the original image (the lower edge of the chessboard is flush with the lower edge of the imaging plane); (xbr, RawImageH) represents the coordinates of the lower left corner of the chessboard in the original image. The coordinates of the lower right corner of the checkerboard grid. Since the checkerboard grid does not deform in the top-down image, the grid's length and width are both 2 meters. The distance between each pixel in the top-down image is set to 1 millimeter. This gives the coordinates of the four corner points of the checkerboard grid in the top-down image: [(Xtl, IMGH - 2000), (Xtl + 2000, IMGH - 2000), (Xtl, IMGH), (Xtl + 2000, IMGH)]. Xtl represents the pixel coordinates of the left edge of the checkerboard grid in the top-down image. This can be adjusted as needed. Generally, the checkerboard grid is placed in the center of the top-down image, i.e., Xtl = IMGW / 2 - 1000. IMGW is the width of the top-down image, and IMGH is the height of the top-down image (typically set to the effective acquisition distance for each image). Using the OpenCV getPerspectiveTransform method, inputting the coordinates of the checkerboard grid corner points in the original image and the top-down image, the mapping matrix M can be quickly calculated. Since the mapping matrix M assumes that the distance between every two pixels in the top view image represents 1 mm, the height of the top view image is IMGH, and the bottom edge of the top view image coincides with the bottom edge of the original image, we can know that the actual distance Dc from point P (x, y) in the top view image to the bottom edge of the top view image satisfies the following formula:

[0130]

[0131] Wherein, IMGH is the height of the top view, y is the vertical coordinate of point P in the top view, and Dc is the actual distance from point P to the lower edge of the top view.

[0132] S105: Use the rotation matrix calculated by S103 The mapping matrix M obtained in S104 is calculated according to formula (7) to obtain the constant matrix A from the original image to the overhead image.

[0133] S106: Obtain the mapping matrix from the original image to the horizontal view through checkerboard calibration calculation .

[0134] The specific method is as follows: a chessboard of known size is placed vertically on the road surface, and the original image at this time is collected; the pixel coordinates of the four corner points of the chessboard placed vertically on the ground in the original image are [(xtl1, ytl1), (xtr1, ytr1), (xbl1, ybl1), (xbr11, ybr11)], where (xtl1, ytl1) represents the coordinates of the upper left corner of the chessboard placed vertically on the ground in the original image; (xtr1, ytr1) represents the coordinates of the upper right corner of the chessboard in the original image; (xbl1, ybl1) represents the coordinates of the lower left corner of the chessboard in the original image; (xbr1, ybr1) represents the coordinates of the lower right corner of the chessboard in the original image; since the chessboard does not deform in the horizontal view , let the coordinates of the upper left corner of the chessboard in the horizontal view be consistent with the coordinates of the upper left corner in the original image, and let the width of the chessboard in the horizontal view be consistent with the width in the original image. We can get the coordinates of the four corner points of the chessboard in the horizontal view [(xtl1, ytl1), (xtr1, ytl1), (xtl1, ybl1), (xtr1, ybl1)], where (xtl1, ytl1) represents the coordinates of the upper left corner of the chessboard in the horizontal view; (xtr1, ytl1) represents the coordinates of the upper right corner of the chessboard in the horizontal view; (xtl1, ybl1) represents the coordinates of the lower left corner of the chessboard in the horizontal view; (xtr1, ybl1) represents the coordinates of the lower right corner of the chessboard in the horizontal view. Use the getPerspectiveTransform method in OpenCV, input the coordinate information of the chessboard perpendicular to the ground in the original image and the coordinate information in the horizontal view, and you can calculate the mapping matrix from the original image to the horizontal view. .

[0135] S107: Use the rotation matrix obtained in S103 , and the mapping matrix obtained in S106 According to formula (8), the constant matrix from the original image to the horizontal view is calculated .

[0136] S108: Calculate the distance Db between the monocular camera imaging plane and the camera through calibration.

[0137] The specific steps are to place a calibration pole of known height vertically at different horizontal distances from the camera to collect the original image; obtain multiple sets of grounding point coordinates P1 (x1, y1) and vertex coordinates P2 (x2, y2) of the calibration pole in the original image; use the mapping matrix Mt from the original image to the horizontal view to calculate the coordinates PP1 and PP2 of P1 and P2 in the horizontal view; according to PP1 and PP2, the pixel distance PH of the calibration pole in the horizontal view can be calculated; use a ruler to measure the actual distance Da from the grounding point of the calibration pole to the camera; given the actual height RH of the calibration pole, the required parameter Db can be calculated according to formula (12); multiple sets of Db can be calculated and averaged to obtain Db to reduce measurement errors.

[0138] The second stage is the collection and operation stage, such as Figure 8 shown.

[0139] S201: Run the collection vehicle to collect original road images in real time, match the positioning information and upload it to the cloud server.

[0140] S201: After receiving the information, the cloud service uses a deep learning image vanishing point detection algorithm to detect the vertical coordinate ynew of the vanishing point of the original image.

[0141] S203: Calculate the camera pitch angle when the original image is captured using ynew and formula (9) .

[0142] S204: Use S203 to get the camera pitch angle and S105 to obtain the constant matrix A and the formula At this time, the mapping matrix Mnew from the original image to the overhead image.

[0143] S205: Use S203 to get the camera pitch angle The constant matrix is calculated by S106 And formula (8) calculates the mapping matrix from the original image to the horizontal view at this time .

[0144] S206: Calculate the distance Dlnew from the lower edge of the original image to the camera when the original image is captured using ynew obtained in S201 and formula (10).

[0145] S207: Detect the 3D pit information (including the quadrilateral box [x1, y1, x2, y2, x3, y3, x4, y5] information and pixel height h) in the original image using a deep learning pit 3D detection algorithm.

[0146] S208: Based on the position information box [x1, y1, x2, y2, x3, y3, x4, y5] of the top surface of the pit in the original image detected by S207, the center point coordinates Ptc (xc, yc) of the top surface of the pit in the original image are calculated. Based on the pit pixel height h output by the deep learning pit 3D detection algorithm in S207, the center coordinates Pbc (xc, yc-h) of the bottom surface of the pit in the original image are calculated.

[0147] S209: Based on the original image-to-top view image mapping matrix Mnew and the coordinates of the center point of the top surface of the pit in the original image, Ptc (xc, yc), calculated in S204, use the cv2.perspectiveTransform method in OpenCV to calculate the coordinates of the center point of the top surface of the pit in the original image in the top view image, PFtc (xfc, yfc). Using PFtc (xfc, yfc) and formula (15), calculate the actual distance Dcnew from the center point of the top surface of the pit to the bottom edge of the original image. Add the distance Dlnew from the bottom edge of the original image to the camera, obtained in S206, to Dcnew to obtain the actual distance Danew from the center point of the top surface of the pit to the camera.

[0148] S210: Based on the top center coordinates Ptc (xc, yc) and bottom center coordinates Pbc (xc, yc-h) of the pit in the original image obtained in S208 and the mapping matrix from the original image to the horizontal view obtained in S205 The horizontal coordinates of Ptc (xttc, yttc) and Pbc (xtbc, ytbc) in the original image are calculated using the cv2.perspectiveTransform method in OpenCV. The horizontal pixel distance PHnew of the pit depth is calculated based on the horizontal coordinates PTtc (xtt, ytt) and PTbc (xtb, ytb).

[0149] S211: Based on the distance Danew from the center point of the top surface of the pit to the camera obtained in S209, the pixel distance PHnew of the pit depth in the horizontal view obtained in S210, and Db obtained in the calibration stage S108, the actual depth RHnew of the pit is calculated using formula (14).

[0150] This embodiment also provides a road pothole depth calculation system based on monocular camera images, comprising: an image acquisition module for acquiring an image to be detected; and a cloud storage and calculation module for extracting the vertical coordinate of the vanishing point of the image to be detected, as well as 3D pothole information and pothole height. Based on the vertical coordinate of the vanishing point, the module combines the 3D pothole information and pothole height to determine the actual distance from the center point of the pothole top surface to the camera and the pixel distance of the pothole depth in a horizontal view. The module then uses the actual distance and pixel distance, combined with the distance from the monocular camera imaging plane to the camera, to determine the actual pothole depth.

[0151] Specifically: The present invention discloses a road pothole depth calculation system based on monocular camera images. The system architecture mainly consists of two parts: a vehicle-mounted acquisition device and a cloud storage and calculation module. Among them, the specific structure of the vehicle-mounted acquisition device is as follows: Figure 1 As shown, the system includes key components such as a monocular camera for image acquisition, a GPS / Beidou positioning module with precise positioning capabilities, and an edge central industrial computer. This on-board acquisition equipment is cleverly installed on the vehicle. Its core task is to capture raw road images, quickly match them with corresponding precise positioning information, and upload this data to the cloud module in real time.

[0152] The cloud storage computing module mainly includes several important sub-modules: the first is a storage module specifically used to store the original images uploaded by edge devices, ensuring that massive image data can be properly preserved; the second is a database module responsible for storing original image information, such as acquisition time, positioning information and other structured data, providing a data basis for subsequent analysis; the third is the image vanishing point detection module, which can detect the vanishing point position of the original image in real time, providing key parameters for subsequent calculations; the fourth is the pit position and depth information detection module, which uses a deep learning 3D target detection algorithm to detect the pit position information and depth information in the image; the fifth is the pit depth calculation module, which accurately calculates the pit depth based on the data collected and analyzed in the early stage.

[0153] The specific workflow of the entire system is mainly divided into two key stages: the first is the equipment installation and calibration stage, during which the on-board acquisition equipment must be carefully installed and accurately calibrated; the second is the real-time acquisition operation stage, during which the on-board acquisition equipment and the cloud storage and computing module work closely together. The former continuously collects and uploads data, while the latter quickly processes and analyzes it, ultimately achieving accurate calculation of the depth of potholes in the road surface.

[0154] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for calculating pothole depth based on monocular camera images, characterized in that: include: Acquire an image to be detected, and extract the vertical coordinate of the vanishing point, the 3D information of the pit, and the pit height of the image to be detected; Obtaining, according to the vanishing point vertical coordinate, the actual distance from the center point of the top surface of the pit to the camera and the pixel distance of the pit depth in the horizontal view in combination with the pit 3D information and the pit height; Obtaining the actual distance from the center point of the top surface of the pit to the camera includes: Calculating the vertical coordinate of the vanishing point to obtain the current pitch angle, obtaining a mapping matrix from the image to the overhead image based on the current pitch angle and a constant matrix for the image to the overhead image, and further using the vertical coordinate of the vanishing point to obtain the distance from the bottom edge of the image to the camera; Based on the 3D information of the pit, the coordinates of the center point of the top surface of the pit in the image are obtained, and the coordinates of the center point of the bottom surface of the pit in the image are obtained according to the height of the pit; Calculating a mapping matrix from the image to the top-view image and the coordinates of the center point of the upper top surface of the pit in the image to obtain the coordinates of the center point of the upper top surface of the pit in the image in the top-view image, and calculating the coordinates of the center point of the upper top surface of the pit in the image in the top-view image to obtain the actual distance from the center point of the upper top surface of the pit to the lower edge of the original image; Adding the actual distance from the center point of the top surface of the pit to the lower edge of the original image and the distance from the lower edge of the image to the camera to obtain the actual distance from the center point of the top surface of the pit to the camera; The constant matrix for obtaining the image to the overhead image includes: Calibrate a monocular camera to determine the camera pitch angle, and obtain a rotation matrix based on the camera pitch angle; the rotation matrix includes: a first rotation matrix and a second rotation matrix: in, is the rotation matrix of the y-axis rotation angle relative to the world coordinate system, is the pitch angle of the camera; Acquire an original image through a calibrated monocular camera, perform calibration calculation on the original image, and obtain a mapping matrix from the original image to the overhead image; The first rotation matrix is combined with the mapping matrix from the original image to the overhead image to obtain a constant matrix from the image to the overhead image: in, is the mapping matrix from image to overhead image, is the first rotation matrix, is the constant matrix from image to overhead image; Obtaining the actual distance from the center point P on the top surface of the pit to the lower edge of the original image includes: in, is the height of the top view, is the ordinate of point P in the top view, is the actual distance from point P to the lower edge of the top view; The actual depth of the pit is obtained by using the actual distance and the pixel distance in combination with the distance from the imaging plane of the monocular camera to the camera.

2. The method for calculating pothole depth based on monocular camera images according to claim 1, characterized in that: Extracting the vertical coordinate of the vanishing point of the image to be detected includes: The image to be detected is input into an image vanishing point detection model to obtain the vertical coordinate of the vanishing point; the image vanishing point detection model is trained using a first training set, wherein the first training set includes images marked with vanishing point information.

3. The method for calculating pothole depth based on monocular camera images according to claim 1, characterized in that: Extracting the pit 3D information and the pit height includes: The image to be detected is input into a mature depth estimation algorithm to extract depth information of the entire image. The depth information is then spliced together with the image to be detected as an independent feature map to form a 4-dimensional input pit 3D detection model to obtain the 3D pit information and the pit height. The pit 3D detection model is trained using a second training set, which includes image data containing 3D pit information. The Depth Block module in the pit 3D detection model is used to extract target depth feature information, and the target depth information representation is added to the output layer.

4. The method for calculating pothole depth based on monocular camera images according to claim 1, characterized in that: Obtaining the pixel distance of the pit depth in the horizontal view includes: Obtaining an image-to-horizontal-view mapping matrix based on the current pitch angle and a constant matrix for image-to-horizontal-view, and obtaining coordinates of the center point of the upper top surface and the center coordinates of the lower bottom surface of the pit in the image based on the coordinates of the center point of the upper top surface and the center coordinates of the lower bottom surface of the pit in the image in combination with the image-to-horizontal-view mapping matrix; The pixel distance of the pit depth in the horizontal view is acquired based on the coordinates of the center point of the upper top surface in the horizontal view and the coordinates of the center point of the lower bottom surface in the horizontal view.

5. The method for calculating pothole depth based on monocular camera images according to claim 4, characterized in that: The constant matrix for getting the image into a horizontal view includes: Perform checkerboard calibration calculation on the original image to obtain the mapping matrix from the original image to the horizontal view. Use the second rotation matrix in combination with the mapping matrix from the original image to the horizontal view to obtain the constant matrix from the image to the horizontal view: in, is the mapping matrix from image to horizontal view, is the second rotation matrix, A constant matrix that converts the image to a horizontal view.

6. The method for calculating pothole depth based on monocular camera images according to claim 1, characterized in that: Obtaining the actual depth of the pit includes: in, is the actual depth of the pit, is the pixel distance of the ground pit depth in the horizontal view, is the distance from the imaging plane to the monocular camera position, is the distance from the pit to the camera.

7. A road pothole depth calculation system applied to the road pothole depth calculation method based on monocular camera images according to any one of claims 1 to 6, characterized in that: include: An image acquisition module, used to acquire the image to be detected; The cloud storage computing module extracts the vertical coordinate of the vanishing point, the 3D information of the pit, and the pit height of the image to be detected. The module obtains the actual distance from the center point of the top surface of the pit to the camera and the pixel distance of the pit depth in the horizontal view based on the vertical coordinate of the vanishing point in combination with the 3D information of the pit and the pit height. The module then uses the actual distance and the pixel distance in combination with the distance from the imaging plane of the monocular camera to the camera to obtain the actual depth of the pit.

Images