Three-dimensional reconstruction method based on lidar and vision multi-information fusion

Abstract

Provided in the present invention is a three-dimensional reconstruction method based on LiDAR and vision multi-information fusion. The method comprises: calibrating a coordinate system of a LiDAR detection result and a visual detection coordinate system to unify the coordinate systems; obtaining sparse point cloud data of a LiDAR, and projecting data points onto a visual image; performing feature point extraction and depth estimation on an image collected by a monocular camera, so as to perform feature point extraction and depth estimation on a planar image, and thereby densifying a point cloud; fusing feature points, which are extracted from the image, with LiDAR point cloud data to form a dense point cloud, and mapping the dense point cloud to a three-dimensional space; and performing fast triangulation processing on the processed point cloud data to obtain a three-dimensional plane, performing spatial matching on image pixels and the three-dimensional plane, and restoring colors. In the method of the present invention, the LiDAR and the monocular camera are combined to complement each other, so that device costs are reduced, advantages in both robustness and stability are achieved, and the quality of point cloud data can be effectively improved.

Description

A 3D reconstruction method based on laser radar and visual multi-information fusion Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction, and in particular relates to a three-dimensional reconstruction method based on the fusion of laser radar and visual multi-information. Background Art

[0002] 3D reconstruction technology has been widely used in various fields, such as scene simulation, scene reconstruction, intelligent driving, simulation experiments, model design, etc. Different fields have different requirements for 3D reconstruction performance, such as accuracy and speed.

[0003] Traditional 3D reconstruction can achieve a high level of model accuracy, but it usually requires a significant amount of time to collect 3D point cloud data. Existing methods are mostly based on high-precision point cloud sensors such as LiDAR, Kinect, and binocular vision. These methods are not only costly but also difficult to handle in application scenarios requiring real-time 3D reconstruction, such as intelligent driving and real-time simulation. These scenarios usually do not require extremely high-precision models, but require the timely reflection of the contours of the reconstructed objects. Traditional reconstruction methods focus more on accuracy and the reproduction of model details, but are still lacking in aspects such as fast reconstruction and real-time reconstruction. In addition, high-precision reconstruction also brings higher equipment and computing costs. Therefore, this method proposes a 3D reconstruction method that can effectively improve reconstruction efficiency and reduce equipment and computing costs. This method uses LiDAR and a monocular vision system to fuse to generate relatively dense point cloud data, and then performs 3D reconstruction based on this data. This method can effectively reduce the equipment cost of the existing technology pipeline and has a faster reconstruction speed, which is of great significance.

[0004] Deficiencies in the existing technology: Existing three-dimensional reconstruction methods mostly use binocular or multi-view vision or laser scanning to generate dense point clouds. For some long-distance targets, there are also methods that use inverse synthetic aperture radar, such as the "Target Three-Dimensional Reconstruction Method Based on Multi-View Radar Images" (Publication No.: CN114067064A) published by Zhejiang University of Technology. Its scope of application does not overlap with the present invention. The patent application document "A Method and Device for Environmental Reconstruction" (Publication No.: CN113506372A) published by Northwestern Polytechnical University proposes a method of using laser radar combined with laser radar to generate pseudo point cloud images, and combining binocular vision images with a deep learning model to generate feature maps, and then performing Poisson reconstruction to generate a three-dimensional surface. Although this method provides a higher-precision reconstruction effect, its equipment cost is high and its timeliness is poor, making it difficult to achieve real-time and rapid three-dimensional reconstruction.

[0005] Currently, there are relatively few cases of combining LiDAR with monocular vision for 3D reconstruction. The advantages of LiDAR are its compactness, flexibility, low cost, and high speed. These characteristics significantly increase the speed of 3D reconstruction while reducing costs, enabling real-time reconstruction to a certain extent. Therefore, it is necessary to develop an effective method for 3D reconstruction combining LiDAR with monocular vision.

[0006] Summary of the Invention

[0007] In response to the defects and problems existing in the existing technology, the present invention proposes a three-dimensional reconstruction solution based on lidar and computer vision, which solves the problems of high equipment cost and low robustness in the existing three-dimensional reconstruction solutions.

[0008] The solution adopted by the present invention to solve its technical problem is: a three-dimensional reconstruction method based on the fusion of laser radar and visual multi-information, including the following steps: S1. Calibrate the coordinate system of the radar detection result and the visual detection coordinate system, unify the coordinate system, and map the image and the point cloud.

[0009] S2. Obtain sparse point cloud data from the lidar and project the data points onto the visual image, projecting the lidar point cloud from three-dimensional space to two-dimensional space of the image row to facilitate image depth estimation.

[0010] S3. Extract feature points from the image captured by the monocular camera and perform depth estimation to achieve feature point extraction and depth estimation for the planar image, thereby thickening the point cloud.

[0011] S4. The feature points extracted from the image are fused with the radar point cloud data to form a dense point cloud, which is then mapped into three-dimensional space. This completes the initial fusion of radar and visual information by mapping the feature points with depth information into the three-dimensional point cloud space.

[0012] S5. Perform rapid triangulation on the processed point cloud data to obtain a three-dimensional plane.

[0013] S6. Spatially match the image pixels with the three-dimensional plane to restore the colors.

[0014] Furthermore, step S1 includes the following steps: S1.1 When calibrating the coordinate system of the radar detection result and the visual detection coordinate system for data space fusion, the radar original signal adopts the polar coordinate system, and the formula for converting it to the world coordinate system is as follows:

[0015] In the above formula (1), x w ,y w ,z wrepresents the world coordinate, R represents the distance from the radar coordinate point to the origin of the radar coordinate system, and α is the target azimuth of the radar, which is the angle formed by the line connecting the target point and the origin and the horizontal axis.

[0016] S1.2 The formula for converting the world coordinate system to the image pixel coordinate system is as follows:

[0017] In the above formula (2), u and v represent the converted pixel coordinates, dx and dy are the dimensions of each pixel in the x and y directions of the pixel coordinate system, s′ represents the camera imaging tilt factor, u0 and v0 represent the coordinates of the origin of the pixel coordinate system in the image coordinate system, f represents the focal length of the camera, R is the rotation matrix, and t is the translation vector.

[0018] Furthermore, step S2 includes projecting the three-dimensional point cloud acquired by the radar onto the xz axis of the image plane and retaining the depth information.

[0019] Furthermore, step S3 includes the following steps:

[0020] S3.1 uses the ORB algorithm to extract feature points from the current image.

[0021] S3.2 traverses the feature points extracted in the previous step, and draws a circle with r as the radius on the image plane with it as the center. min <r<r max ; r min is the lower limit of the diameter change, r max is the upper limit of the radius.

[0022] S3.3 For an image feature point, calculate the sum of the depth values ​​of all point clouds contained in its circle and find the average, which is used as the depth value of the feature point.

[0023] S3.4 If a certain image feature point circle does not contain any point cloud, then expand the radius r of the feature point circle. If the upper limit r is reached max , then the point is considered as an isolated point and discarded.

[0024] S3.5 Repeat the above steps S3.1-3.4 until all image feature points are assigned depth values.

[0025] Furthermore, step S4 includes the following steps: S4.1 traverses the feature points assigned depth values ​​in the previous step, and uses the depth values ​​as y values.

[0026] S4.2 In the xyz three-dimensional point cloud coordinate system, all feature points are mapped to the space. At this time, the feature points are combined with the original point cloud as an expanded point cloud to form a dense point cloud.

[0027] Furthermore, step S5 includes the following steps: S5.1 constructing a super tetrahedron in the three-dimensional space where the point cloud data is located so that all points are included.

[0028] S5.2 randomly selects a point and connects it to the four vertices of the super tetrahedron to form four new tetrahedrons.

[0029] S5.3 randomly select a second point that must fall in one of the four new tetrahedrons, connect the point to the four vertices of the tetrahedron containing it, and divide the tetrahedron into four new tetrahedrons.

[0030] S5.4 circumscribes each of the four new tetrahedrons divided in step S5.3. If the sphere does not contain any other points, the tetrahedron is considered to conform to the Delaunay triangulation, and the judgment continues for the next new tetrahedron; if the circumscribed sphere contains any other points, the tetrahedron is considered to not conform to the Delaunay triangulation, and the edge flipping operation is performed until all new tetrahedrons satisfy the local Delaunay.

[0031] S5.5 Based on step S5.2, randomly insert the next point and repeat the above step 5.3, and make it and the generated tetrahedron satisfy local Delaunay until all point clouds are added to the super tetrahedron.

[0032] S5.6 removes the vertices of the super tetrahedron and all internal points, retaining only the surface triangle mesh. At this point, the 3D reconstruction process ends.

[0033] Furthermore, step S6 includes the following steps: S6.1 aligning the generated three-dimensional surface with the original image so that their orientations are unified and the horizontal and vertical coordinates of each point in the point cloud correspond one-to-one with the horizontal and vertical coordinates of the pixel points in the original image.

[0034] S6.2 matches the x and z coordinates of all corresponding points on the three-dimensional surface and finds all image pixels with the same x and z coordinates.

[0035] S6.3 attaches the RGB information of the corresponding image pixels to the point cloud to complete color matching.

[0036] The present invention's beneficial effects include: The proposed 3D reconstruction method, which utilizes the fusion of lidar and monocular vision, combines and complements the two, reducing the time and computational complexity of traditional 3D reconstruction methods and significantly lowering the equipment cost. Compared to single-sensor methods, this method offers advantages in robustness and stability, effectively improving point cloud data quality and adapting to a variety of environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a block diagram of the relationship between the lidar and vision system;

[0038] FIG2 is a process diagram of the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and examples.

[0040] Example 1: Existing three-dimensional reconstruction methods mostly use lidar, Kinect, multi-view images, binocular vision and other methods to obtain point cloud data. However, in outdoor scenes, Kinect, multi-view images and binocular vision are not suitable. LiDAR, as a point cloud data acquisition device, has the advantages of low cost and all-time and all-weather. But its disadvantages are also obvious. The point cloud data collected by the lidar is relatively sparse, and conventional three-dimensional reconstruction methods are difficult to reconstruct its surface. The point cloud data obtained by the monocular camera also has problems such as a single spatial dimension. Therefore, this embodiment combines the two and complements each other, which can effectively improve the quality of point cloud data, reduce the equipment and computing costs of three-dimensional reconstruction, and can adapt to a variety of environments.

[0041] Specifically, in combination with FIG1 and FIG2 , a three-dimensional reconstruction method based on the fusion of laser radar and visual multi-information is provided, which includes the following steps.

[0042] S1. Calibrate the LiDAR and vision system to unify the coordinate system.

[0043] The purpose of step S1 is to calibrate the LiDAR and vision system and unify the coordinate system to facilitate mapping between images and point clouds. The specific steps are as follows:

[0044] S1.1 calibrates the coordinate system of the radar detection result and the visual detection coordinate system, unifies the coordinate system, and performs data spatial fusion. The original radar signal uses the polar coordinate system, and the formula for converting it to the world coordinate system is as follows:

[0045] S1.1.1 In the above formula, (x w ,y w ,z w ) represents the world coordinate, R represents the distance from the radar coordinate point to the origin of the radar coordinate system, and α is the target azimuth of the radar, which is the angle formed by the line connecting the target point and the origin and the horizontal axis.

[0046] S1.2 The formula for converting the world coordinate system to the image pixel coordinate system is as follows:

[0047] S1.2.1 In the above formula, (u, v) represents the transformed pixel coordinates, dx and dy are the dimensions of each pixel in the x and y directions of the pixel coordinate system. s′ represents the camera imaging tilt factor, u0 and v0 represent the coordinates of the pixel coordinate system origin in the image coordinate system, f represents the focal length of the camera, R is the rotation matrix, and t is the translation vector.

[0048] S2. Obtain sparse point cloud data from the lidar and project the data points onto the visual image.

[0049] The purpose of step S2 is to project the radar point cloud from the three-dimensional space to the two-dimensional space of the image line to facilitate image depth estimation, which includes the following steps:

[0050] S2.1 projects the three-dimensional point cloud acquired by the radar onto the xz axis of the image plane and retains the depth information.

[0051] S3. Extract feature points from the image captured by the monocular camera and perform depth estimation.

[0052] The purpose of step S3 is to extract feature points from the plane image and estimate the depth, thereby thickening the point cloud. It includes the following steps:

[0053] S3.1 uses the ORB algorithm to extract feature points from the current image.

[0054] S3.2 traverses the feature points extracted in the previous step, and uses them as the center of the circle on the plane with r(r min <r<r max ) as the radius to draw a circle.

[0055] S3.3 For an image feature point, calculate the sum of the depth values ​​of all point clouds contained in its circle and find the average, which is used as the depth value of the feature point.

[0056] S3.4 If a certain image feature point circle does not contain any point cloud, then expand the radius r of the feature point circle. If the upper limit r is reached max , then the point is considered as an isolated point and discarded.

[0057] S3.5 Repeat the above steps until all image feature points are assigned depth values.

[0058] S4. The feature points extracted from the image will be fused with the radar point cloud data to form a dense point cloud and mapped into three-dimensional space.

[0059] The purpose of step S4 is to map the feature points with depth information to the 3D point cloud space, completing the first fusion of radar visual information. The specific steps are:

[0060] S4.1 traverses the feature points assigned depth values ​​in the previous step and uses their depth values ​​as y values.

[0061] S4.2 In the xyz three-dimensional point cloud coordinate system, all feature points are mapped to the space. At this time, the feature points are combined with the original point cloud as an expanded point cloud to form a dense point cloud.

[0062] S5. Perform rapid triangulation on the processed point cloud data to obtain a three-dimensional plane.

[0063] The purpose of step S5 is to perform Delaunay triangulation on the processed dense point cloud to generate a three-dimensional surface. The specific steps are as follows:

[0064] S5.1 constructs a super tetrahedron in the three-dimensional space where the point cloud data is located, so that all points are included in it.

[0065] S5.2 randomly selects a point and connects it to the four vertices of the super tetrahedron to form four new tetrahedrons.

[0066] S5.3 randomly select a second point that must fall within a tetrahedron, connect the point to the four vertices of the tetrahedron containing it, and divide the tetrahedron into four new tetrahedrons.

[0067] S5.4 Make a circumscribed sphere for each of the four tetrahedrons divided in the previous step. If the sphere does not contain any other points, the tetrahedron is considered to conform to the Delaunay triangulation, and the judgment continues for the next tetrahedron; if the circumscribed sphere contains any other points, the tetrahedron is considered to not conform to the Delaunay triangulation, and the edge flipping operation is performed until all tetrahedrons satisfy the local Delaunay.

[0068] S5.5 randomly inserts the next point, repeats the above steps, and makes it and the resulting tetrahedron satisfy the local Delaunay until all point clouds are added to the super tetrahedron.

[0069] S5.6 removes the vertices of the super tetrahedron and all internal points, retaining only the surface triangle mesh. At this point, the 3D reconstruction process ends.

[0070] S6. Spatially match the image pixels with the three-dimensional plane to restore the colors.

[0071] The purpose of step S6 is to restore the color information of the three-dimensional surface pixels, and the specific steps are as follows:

[0072] S6.1 aligns the generated three-dimensional surface with the original image so that their orientations are consistent and the horizontal and vertical coordinates of each point in the point cloud correspond one-to-one with the horizontal and vertical coordinates of the pixels in the original image.

[0073] S6.2 matches the x,z (horizontal and vertical coordinates) coordinates of all corresponding points on the three-dimensional surface, and finds all image pixels with the same x,z coordinates.

[0074] S6.3 attaches the RGB information of the corresponding image pixels to the point cloud to complete color matching.

[0075] This 3D reconstruction method combines LiDAR with monocular vision for 3D reconstruction. It uses feature point extraction to densify point cloud data, data dimensionality reduction to estimate depth from image feature points, Delaunay triangulation to reconstruct the 3D surface from the dense point cloud generated by LiDAR and monocular vision, and spatial matching to restore the model's color information. This method significantly reduces the equipment cost for 3D reconstruction, lowering the time and computational complexity of traditional 3D reconstruction methods. Compared to single-sensor methods, this method offers advantages in robustness and stability.

[0076] The above specific embodiments of the present invention are merely illustrative or illustrative of the principles of the present invention and do not constitute limitations of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on laser radar and visual multi-information fusion, characterized in that: The steps include: S1. Calibrate the coordinate system of the radar detection result and the visual detection coordinate system to unify the coordinate system so as to map the image and the point cloud; S2. Obtain sparse point cloud data of the laser radar and project the data points onto the visual image, projecting the radar point cloud from the three-dimensional space to the two-dimensional space of the image row to facilitate image depth estimation; S3. Extract feature points from the image captured by the monocular camera and perform depth estimation, so as to extract feature points from the plane image and estimate the depth, thereby thickening the point cloud; S4. The feature points extracted from the image will be fused with the radar point cloud data to form a dense point cloud and mapped to the three-dimensional space. The feature points with depth information will be mapped to the three-dimensional point cloud space, completing the first fusion of radar visual information; S5. quickly triangulate the processed point cloud data to obtain a three-dimensional plane; S6. Spatially match the image pixels with the three-dimensional plane to restore the colors.

2. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 1 is characterized in that: Step S1 comprises the following steps: S1.1 When the coordinate system of the radar detection result is calibrated with the visual detection coordinate system for data space fusion, the original radar signal adopts the polar coordinate system, and the formula for converting it to the world coordinate system is as follows: In the above formula (1), x w ,y w ,z w represents the world coordinates, R represents the distance from the radar coordinate point to the origin of the radar coordinate system, α is the target azimuth of the radar, which is the angle formed by the line connecting the target point and the origin and the horizontal axis; S1.2 The formula for converting the world coordinate system to the image pixel coordinate system is as follows: In the above formula (2), u, v represent the converted pixel coordinates, dx, dy are the dimensions of each pixel in the x-axis and y-axis directions of the pixel coordinate system, s′ represents the camera imaging tilt factor, u0, v0 represent the coordinates of the origin of the pixel coordinate system in the image coordinate system, f represents the focal length of the camera, R is the rotation matrix, and t is the translation vector.

2. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 2 is characterized in that step S2 includes projecting the three-dimensional point cloud acquired by the radar to the xz axis of the image plane and retaining the depth information.

3. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 1 or 2, characterized in that: Step S3 includes the following steps: S3.1 uses ORB algorithm to extract feature points of the current image; S3.2 Traverse the feature points extracted in the previous step, and use them as the center to draw a circle with r as the radius on the image plane. min <r<r max ; r min is the lower limit of the diameter change, r max is the upper limit of the radius; S3.3 For an image feature point, calculate the sum of the depth values ​​of all point clouds contained in its circle and find the average value, and the average value is used as the depth value of the feature point; S3.4 If a certain image feature point circle does not contain any point cloud, then expand the feature The radius r of the point circle, if it reaches the upper limit r max , then the point is considered as an isolated point and discarded; S3.5 Repeat the above steps S3.1-3.4 until all image feature points are assigned depth values.

4. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 3 is characterized in that: Step S4 comprises the following steps: S4.1 traverses the feature points assigned depth values ​​in the previous step, and uses their depth values ​​as y values; S4.2 In the xyz three-dimensional point cloud coordinate system, all feature points are mapped to the space. At this time, the feature points are combined with the original point cloud as an expanded point cloud to form a dense point cloud.

5. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 1 is characterized in that: Step S5 comprises the following steps: S5.1 construct a super tetrahedron in the three-dimensional space where the point cloud data is located, so that all points are included; S5.2 randomly selects a point and connects it to the four vertices of the super tetrahedron to form four new tetrahedrons; S5.3 randomly select a second point that must fall in one of the four new tetrahedrons, connect the point to the four vertices of the tetrahedron containing it, and divide the tetrahedron into four new tetrahedrons; S5.4 Make circumscribed balls for each of the four tetrahedrons divided in step S5.

3. If the balls do not contain any other points, the tetrahedron is considered to conform to the Delaunay triangulation, and the judgment is continued for the next new tetrahedron. If the circumscribed ball contains any other points, the tetrahedron is considered to not conform to the Delaunay triangulation, and the edges are flipped. Operation until all new tetrahedrons satisfy local Delaunay; S5.5 Based on step S5.2, randomly insert the next point, repeat the above step 5.3, and make it and the generated tetrahedron satisfy the local Delaunay until all point clouds are added to the super tetrahedron; S5.6 removes the vertices of the super tetrahedron and all internal points, retaining only the surface triangle mesh, and the three-dimensional reconstruction process is completed at this time.

6. The three-dimensional reconstruction method based on laser radar and visual multi-information fusion according to claim 5 is characterized in that: Step S6 comprises the following steps: S6.1 aligns the generated three-dimensional surface with the original image so that their orientations are unified and the horizontal and vertical coordinates of each point in the point cloud correspond one to one with the horizontal and vertical coordinates of the pixel points in the original image; S6.2 matches the x and z coordinates of all corresponding points on the three-dimensional surface, and finds all image pixels with the same x and z coordinates; S6.3 attaches the RGB information of the corresponding image pixel to the point cloud to complete the color matching.

Images