Monocular vision depth estimation method and computer readable storage medium

Abstract

This invention provides a monocular vision depth estimation method and a computer-readable storage medium, comprising the following steps: for each distance value D in a pre-defined distance sequence D i Obtain the two-dimensional coordinates W of its point set in the pixel coordinate system. i,j And perform curve fitting to obtain the curve sequence f i ; Identify the target bounding box of the obstacle in the image data, with the midpoint of the lower edge of the target bounding box being the grounding point B of the obstacle; In the curve sequence f i Find the grounding point as B(u) b ,v b Two adjacent fitted curves f k with f k+1 The distance value of ground point B is interpolated between two fitted curves. This monocular vision depth estimation method based on lidar adopts a calculation method of first fitting and then interpolating in real time. Since the large amount of computational fitting process is performed before running, it does not affect the real-time performance of the system. During the operation, it is only necessary to find two adjacent curves and perform simple distance interpolation, which can greatly improve the depth estimation speed of the system during operation.

Description

Technical Field

[0001] This invention relates to the field of visual image processing technology, specifically to a monocular visual depth estimation method and a computer-readable storage medium. Background Technology

[0002] In autonomous driving applications in mining scenarios, LiDAR and millimeter-wave radar often fail to effectively detect small stones, especially when identifying obstacles at a distance. Furthermore, due to factors such as detection distance, obstacle surface material, and weather conditions, LiDAR and millimeter-wave radar suffer from low detection rates for specific obstacles. Based on these issues, a feasible and generally applicable solution is to compensate for the aforementioned shortcomings of LiDAR and millimeter-wave radar through visual augmentation.

[0003] It's important to note that camera-captured image data cannot include obstacle depth information. Therefore, even if obstacle bounding boxes can be obtained from image data through machine learning or other methods, it's insufficient to provide effective auxiliary information for autonomous driving systems. Thus, obtaining obstacle depth information from camera-captured image data is one of the key issues that needs to be addressed in autonomous driving systems utilizing image data. It's important to note that when discussing depth information in image data, it's usually equivalent to three-dimensional coordinates in the world coordinate system. In mining applications, the loading area, due to extensive excavation, has a relatively flat surface with minimal slope, except for minor local undulations.

[0004] Currently, commonly used methods for depth estimation of image data include monocular visual depth estimation based on similar triangles, depth estimation based on binocular visual disparity, and visual depth estimation based on deep learning. Among these methods, monocular visual depth estimation based on similar triangles provides low-precision depth information, making it difficult to meet practical needs; binocular visual disparity-based depth estimation requires high positional accuracy between the two cameras and involves complex software algorithms, consuming significant computational resources; and deep learning-based visual depth estimation is computationally intensive, complex, and difficult to implement. Generally speaking, current depth estimation methods are either too complex, time-consuming, and inefficient; or too complex and computationally time-consuming, making it difficult to meet the real-time requirements of autonomous driving; or have poor depth estimation accuracy, failing to meet the precision requirements of autonomous driving. All of these pose significant challenges to the safety of autonomous driving. Summary of the Invention

[0005] To address the problem that existing visual depth estimation methods cannot simultaneously meet the accuracy and efficiency requirements of autonomous driving scenarios, this invention provides a monocular visual depth estimation method and a computer-readable storage medium.

[0006] The technical solution of this invention provides a monocular vision depth estimation method based on lidar, comprising the following steps:

[0007] S1. Preparation steps:

[0008] In the world coordinate system, a distance sequence D is predefined, where each distance value is denoted as Di. i Let i be the number of items. For each distance value D... i Select multiple ground points that are equidistant from the reference point to form a point set.

[0009] For each point set Calculate the two-dimensional coordinates W of all points in the pixel coordinate system. i,j (u j ,v j ), where j is the count of points in the point set. For each point set, the two-dimensional coordinates W of all points in the pixel coordinate system are... i,j (u j ,v j Curve fitting is performed to obtain the curve sequence f. i (u,v), where i is the count of the fitted curve;

[0010] S2. Steps for determining the grounding point of an obstacle:

[0011] The target bounding box of the obstacle is identified in the image data. The midpoint of the lower edge of the target bounding box is taken as the ground point of the obstacle, and its coordinates are denoted as B(u). b ,v b );

[0012] S3. Coordinate interpolation steps

[0013] For grounding point B(u) b ,v b Obstacles in the curve sequence f i (u,v) finds the grounding point as B(u) b ,v b Two adjacent fitted curves f k (u,v) and f k+1 (u,v);

[0014] By fitting the curve f k (u,v) and the fitted curve f k+1 The grounding point B(u) is obtained by interpolating the distance between (u,v). b ,v b Distance value

[0015] Preferably, the S1 preparation step includes:

[0016] S12. Point selection steps:

[0017] Predefine a distance sequence D, for each distance sequence D... i Determine the point set P in the world coordinate system. Di So that for any ground point All have L p =D i L p Let be the distance from point p to the reference point.

[0018] S13. Point set mapping steps:

[0019] For each point set The coordinates of each ground point in the lidar coordinate system are obtained by mapping each point to the lidar coordinate system using the first extrinsic parameter matrix. Then, the coordinates W of the corresponding point in the pixel coordinate system are obtained using the second extrinsic parameter matrix and the camera's intrinsic parameter matrix. i,j (u j ,v j ), where j is the count of the points in the set.

[0020] S14. Curve Fitting Steps

[0021] For sets belonging to the same point All points below correspond to coordinates W in the pixel coordinate system. i,j (u j ,v j By fitting, a fitting curve f is obtained for all pixel coordinates. i (u,v), and then a set of fitted curve sequences f(u,v) corresponding to the aforementioned distance sequence D are obtained.

[0022] Preferably, the first extrinsic parameter matrix in the S13 point set mapping step is obtained by calibrating the lidar and the world coordinate system; the second extrinsic parameter matrix is ​​obtained by jointly calibrating the lidar and the monocular camera.

[0023] Preferably, the fitted curve in the S14 curve fitting step is a polynomial curve, and the interpolation method in the S3 coordinate interpolation step is linear interpolation.

[0024] Preferably, the ground point in the S12 point selection step is selected within a semi-circular area within a 90-degree range to the left and right of the camera.

[0025] Preferably, the distance sequence D i The step size is 10m, and the sampling interval of the ground points in the point set is 0.01m.

[0026] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the monocular vision depth estimation method based on lidar described in any of the above claims.

[0027] The monocular vision depth estimation method based on lidar of the present invention adopts a calculation method of fitting first and then interpolating in real time. Since the large amount of computation in the fitting process is performed before running, the calculation time of this part does not affect the real-time performance of the system. During the operation, it is only necessary to find two adjacent curves and perform simple distance interpolation. Therefore, the real-time computation is small, the method is simple and easy to implement, and it can greatly improve the depth estimation speed during system operation. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the system principle of the present invention;

[0029] Figure 2 This is a system control flowchart of the present invention. Detailed Implementation

[0030] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. In this specification, the dimensions of the drawings do not represent the actual dimensions. They are only used to illustrate the relative positional and connection relationships between the components. Components with the same name or the same reference numeral represent similar or identical structures and are limited to illustrative purposes.

[0031] Existing autonomous driving systems require obstacle information from the environment as one of the decision-making bases for the autonomous driving system. Although sensors such as LiDAR or millimeter-wave radar can be used for obstacle recognition, the detection rate remains low for specific obstacles. Therefore, machine vision-assisted obstacle recognition can be considered. Monocular camera systems are simple and easy to install and maintain. Moreover, considering that sensors with depth information, such as LiDAR, already exist, they can be used as a basis for recovering depth information. Therefore, this invention uses a monocular vision system. For monocular vision systems, when a point in the three-dimensional world (world coordinate system) is transformed to the pixel coordinate system of the image, depth information is lost. Although it is possible to uniquely determine a point in the pixel coordinate system given a point in the world coordinate system, this process is irreversible. For a point in the pixel coordinate system, it is impossible to determine the three-dimensional coordinates of the corresponding point in the world coordinate system solely from the image data. Corresponding to these three-dimensional coordinates, the distance information from the point to the origin of the camera coordinate system or the world coordinate system, i.e., the depth, can be obtained. Therefore, the depth information in this invention is equivalent to the three-dimensional coordinates of a point in the world coordinate system.

[0032] This article involves multiple coordinate systems, and the world coordinate system and the vehicle coordinate system are equivalent. Generally, the LiDAR coordinate system is established with the origin of the LiDAR, the camera coordinate system is established with the optical center of the camera as the origin, and the pixel coordinate system is established with the top left corner of the image as the origin.

[0033] This invention is based on the logic that, in obstacle recognition using a monocular vision system, although the image data from the monocular vision system does not contain depth information, the data from other sensors such as LiDAR does contain depth information, or rather, three-dimensional spatial information of points. Therefore, as long as the coordinate relationship between the two sensors can be established, the depth of the corresponding point in the image data can be inferred. However, as mentioned earlier, the problem is that reverse calculation is not feasible; due to a lack of conditions, it is impossible to obtain the true value of the corresponding point through reverse calculation of the image data. To address this, the solution of this invention is to obtain the distribution curves of points at multiple different depths in the image coordinate system through forward calculation, and then obtain the estimated value of the depth of a point in the image coordinate system by interpolation between the distribution curves.

[0034] The specific steps of this invention are as follows:

[0035] S1. Preparation Steps

[0036] In the world coordinate system, a distance sequence D is predefined, where each distance value is denoted as Di. i Let i be the number of items. For each distance value D... i Select multiple ground points that are equidistant from the reference point to form a point set.

[0037] For each point set Calculate the two-dimensional coordinates W of all points in the pixel coordinate system. i,j (u j ,v j ), where j is the count of points in the point set. For each point set, the two-dimensional coordinates W of all points in the pixel coordinate system are... i,j (u j ,v j Curve fitting is performed to obtain the curve sequence f. i (u,v), where i is the count of the fitted curve. Clearly, f i (u,v) and D i It's a one-to-one correspondence.

[0038] S2. Steps for determining the grounding point of an obstacle

[0039] Identifying obstacle bounding boxes in image data typically employs methods such as machine learning, which are common in existing technologies. For ground obstacles, since the obstacle is considered to be on the ground, the midpoint of the bottom edge of the bounding box can be considered the nearest ground point of the obstacle, denoted as B(u). b ,v b ).

[0040] S3. Coordinate interpolation steps

[0041] For grounding point B(u) b ,v b Obstacles in the curve sequence f i (u,v) finds the grounding point as B(u) b ,v b Two adjacent fitted curves f k (u,v) and f k+1 (u,v), where k is a specific value of i, and its specific value is determined by B(u b ,v b As determined by ).

[0042] By analyzing the fitted curve f k (u,v) and the fitted curve f k+1 (u,v) interpolation yields the grounding point B(u). b ,v b Distance value Since it is obtained through distance interpolation, the distance value is obviously... Between the first distance D k Distance D from the second point k+1 The interpolation method is linear interpolation, that is, based on the grounding point B(u). b ,v b ) to the fitted curve f k (u,v) and the fitted curve f k+1 The grounding point B(u) is obtained by linear interpolation of the graphical distance between (u,v). b ,v b Distance value to the origin of the world coordinate system

[0043] Typically, the S1 preparation step is completed in advance. For a specific scenario, only the corresponding distance sequence D and curve sequence f need to be saved. i (u,v), only the S2 step of determining the obstacle contact point and the S3 step of coordinate interpolation need to be performed in real time during vehicle movement, thus improving the system's real-time response speed. Specifically, the S1 preparation step can be performed as follows:

[0044] S11. Equipment Calibration Procedure

[0045] The lidar is calibrated to obtain the extrinsic parameter matrix between the lidar and the world coordinate system, denoted as the first extrinsic parameter matrix. The lidar and monocular camera are then jointly calibrated to obtain the extrinsic parameter matrix between the monocular camera and the lidar, denoted as the second extrinsic parameter matrix. This joint calibration process typically involves multiple calibration boards. Feature matching is performed between the features of the calibration boards in the point cloud data obtained from the lidar and the image features obtained from the image data, thus obtaining the second extrinsic parameter matrix between the two.

[0046] S12. Point Selection Steps

[0047] Predefine a distance sequence D, for each distance sequence D... i Determine the set of points in the world coordinate system So that for any ground point All have L p =D i L p Let D be the distance from point p to the reference point. Each distance sequence D... i Corresponding point set Within the world coordinate system, points are distributed in a roughly circular shape around the vehicle body. For accuracy, points can be selected sequentially with a certain step size. Since the camera's field of view generally does not exceed 180 degrees, the selection of the point set can exclude areas outside the camera's field of view, thus improving processing efficiency during the preparation phase. In specific applications, a distance sequence D = {D...} can be selected within a range of 10-100 meters with a step size of 10 meters. i |D i =10·i, i∈(1,2,...10)}, corresponding to each distance value D i The interval between points is 0.01m. During this process, points outside the camera's field of view can be excluded as much as possible. That is, the semi-circular area within 90 degrees to the left and right of the camera is used as the selection area, and all points in each point set are determined within this range.

[0048] S13. Point set mapping steps

[0049] For each point set By mapping each ground point to the lidar coordinate system using the first external parameter matrix, the coordinates V of its corresponding point in the lidar coordinate system are obtained. i,j =(x j ,y j ,z j ), where j is the count of the points in the set. The coordinates W of the corresponding point in the LiDAR coordinate system are obtained from the second extrinsic parameter matrix and the camera's intrinsic parameter matrix. i,j (u j ,v j Its value can be calculated in homogeneous coordinates by the following formula, where T3×4 With M 3×3 These are the second extrinsic parameter matrix and the intrinsic parameter matrix of the camera, respectively.

[0050]

[0051] S14. Curve Fitting Steps

[0052] For sets belonging to the same point All points below correspond to coordinates W in the pixel coordinate system. i,j (u j ,v j By using a suitable fitting method, a fitting curve f is obtained for all the pixel coordinates mentioned above. i (u,v), obviously f i (u,v) and D i It is a one-to-one correspondence. Therefore, a set of fitting curve sequences f(u,v) can be obtained corresponding to the aforementioned distance sequence D. In actual mining scenarios, since the ground in the loading area is flat and has little undulation after loading operations, polynomial fitting can achieve a good fitting effect. After actual testing, a fifth-order polynomial is preferred as the fitting curve.

[0053] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0054] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A monocular vision depth estimation method based on lidar, characterized in that, The method comprises the following steps: S1. A preparation step: A distance sequence D is preset in the world coordinate system, wherein each distance value is denoted as D i , i is a count, and each distance value D i is selected for each point set solving all points in the point set in the two-dimensional coordinates W i,j (u j ,v j ) in the pixel coordinate system, where j is the count of points in the point set; performing curve fitting on all points in the two-dimensional coordinates W i,j (u j ,v j ) in the pixel coordinate system to obtain a curve sequence f i (u,v), i is the count of the fitted curve; S2. A step of determining the ground point of the obstacle: The lower edge midpoint of the target frame in the image data is identified as the grounding point of the obstacle, and the coordinates are denoted as B(u b ,v b ). S3. A coordinate interpolation step: For an obstacle with a ground point B(u b ,v b ), two fitting curves f i (u,v) and f b (u,v) adjacent to the ground point B(u b ,v k ) are found in the curve sequence f k+1 (u,v). The distance value of the grounding point B(u k ,v k+1 ) is obtained by making distance interpolation between the fitting curve f b (u,v) and the fitting curve f b (u,v) 2. The LIDAR-based monocular vision depth estimation method of claim 1, wherein, The S1 preparation step comprises: S12. A point taking step: A sequence of distances D is predetermined, for each distance D in D i The point set is determined in the world coordinate system such that for any ground point L p = D i where L p is the distance from the point p to the reference point; S13. A point set mapping step: For each point set By mapping each ground point in the point set to its corresponding point in the laser radar coordinate system through the first extrinsic matrix, the coordinates of the corresponding point in the laser radar coordinate system are obtained. By using the second extrinsic matrix and the intrinsic matrix of the camera, the coordinates of the corresponding point in the laser radar coordinate system are mapped to the coordinates of the corresponding point in the pixel coordinate system W i,j (u j ,v j ), where j is the count of points in the point set. S14. A curve fitting step: All pairs of points from the same point set correspond to coordinates W in the pixel coordinate system i,j (u j ,v j ) fitted to obtain a fitted curve f i (u,v) of all pixel coordinates, and further corresponding to the aforementioned distance sequence D to obtain a fitted curve sequence f(u,v).

3. The LIDAR-based monocular vision depth estimation method of claim 2, wherein, The first extrinsic parameter matrix in the S13 point set mapping step is obtained by calibrating the laser radar and the world coordinate system; and the second extrinsic parameter matrix is obtained by jointly calibrating the laser radar and the monocular camera.

4. The LIDAR-based monocular vision depth estimation method of claim 2, wherein, The fitting curve in the S14 curve fitting step is a polynomial curve, and the interpolation mode in the S3 coordinate interpolation step is linear interpolation.

5. The LIDAR-based monocular vision depth estimation method of claim 2, wherein, The ground point in the S12 point taking step is selected in a semicircular area within a 90-degree range left and right of the camera pointing direction.

6. The LIDAR-based monocular vision depth estimation method of claim 5, wherein, The distance sequence D i The step length is 10 m, and the point interval of the ground points in the point set is 0.01 m.

7. A computer readable storage medium characterized in that, A computer program is stored, and the computer program is executed by a processor to enable the processor to perform the laser radar-based monocular vision depth estimation method according to any one of claims 1-6.

Images