A method for reconstructing large-scale cable tunnel environment model

Abstract

The present invention discloses a large-scale cable tunnel environment model reconstruction method, comprising the steps of: recording the forward fisheye lens of a panoramic camera as an F camera, recording the backward fisheye lens as an R camera, using the F camera and the R camera to respectively collect checkerboard images at different angles and fisheye images with different exposures; using the panoramic camera to advance at a uniform speed to collect a panoramic video S of the cable tunnel environment, and dividing the panoramic video S into Q segments of a panoramic image sequence I; for each segment of the panoramic image sequence I, obtaining the external parameters and sparse three-dimensional point cloud of the F camera and the R camera, and densely reconstructing a dense three-dimensional point cloud based on a spherical scanning algorithm; performing triangulation and texture mapping on the three-dimensional point cloud to obtain a large-scale cable tunnel environment three-dimensional model. The present invention effectively solves the problem of a small field of view of a pinhole camera; and also solves the speed and accuracy problems faced by a traditional SFM algorithm. The method has practical value in image-based three-dimensional reconstruction technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a large-scale cable tunnel environment model reconstruction method. Background Art

[0002] Image-based 3D reconstruction uses computers to identify, analyze, and simulate pre-collected 2D images of real-world objects and then build corresponding models to create virtual scenes. Structure From Motion (SFM) is a 3D reconstruction method that uses 2D image information to restore camera motion parameters and scene structure. This method is widely used in geographic information systems, locomotive navigation systems, virtual reality scene roaming systems, etc. In order to more efficiently obtain environmental information, increase the authenticity and immersion of virtual roaming scenes, solve the problems of large scale, long distance, complex scenes (for example, weak texture, no texture, moving objects, dark light) in 3D model reconstruction, and improve the accuracy of model reconstruction, the use of images collected by wide-view consumer-grade 360 ​​panoramic cameras (for example, Samsung Gear 360, LG 360, etc.) for 3D model reconstruction has a wide range of application value.

[0003] Based on the unordered spherical panoramic image dataset, Alain Pagani et al. used approximate geometry to simplify the calculation of each step of the standard SFM, and input the generated sparse point cloud into the multi-view stereo system (Multiple View Stereo, MVS) to obtain a dense map. Since the approximate calculation and MVS are mainly applicable to planar images, although this method improves the computational efficiency, the map accuracy is lacking. Sunghoon et al. used a spherical panoramic camera (SPC) to collect ordered panoramic images of small movements, the KLT algorithm was used for Harris corner tracking, and the spherical scanning algorithm was used for dense reconstruction of large-scale scenes. Compared with the standard SFM, this method has improved running speed, but is only applicable to short-distance scenes. The present invention proposes a large-scale cable tunnel environment model reconstruction method, which is based on the panoramic camera to collect large-scale, long-distance, and complex scene cable tunnel environment panoramic video, and reconstruct the panoramic video sequence in segments. Each panoramic image sequence uses an improved SFM method to obtain the panoramic camera trajectory and sparse three-dimensional information of the environment, and the spherical scanning algorithm is used to establish a dense point cloud of the environment. Then, the nonlinear optimization method is used to fuse the information of each dense point cloud to obtain a complete cable tunnel model. The traditional SFM algorithm usually uses a pinhole camera as the only visual sensor. First, the image SIFT features are extracted for matching. Then, the transformation matrix (including rotation matrix and translation vector) of each frame image is calculated using epipolar geometry and PnP algorithm. The three-dimensional points are reconstructed by triangulation. Finally, the bundle adjustment (BA) is used to optimize the transformation matrix and three-dimensional point cloud as a whole. The panoramic camera used in the improved SFM method includes two front and rear fisheye lenses. The field of view of each fisheye lens covers 200°, which can collect richer environmental information. The unified camera model (UCM) is used to establish an accurate geometric projection model for fisheye lens imaging. For each panoramic image sequence, the dynamic range of the panoramic image is restored based on the Debevec method, and the KLT optical flow algorithm is used to track Harris corners on the high dynamic range panoramic image, avoiding the calculation and matching of descriptors, saving a lot of time, and improving the accuracy of Harris corner tracking. Then, BA based on the unified camera model is used to optimize the external parameters of the two fisheye lenses in the panoramic camera and the three-dimensional map point cloud. Compared with the standard SFM algorithm and the algorithm proposed by Sunghoon et al., our method greatly improves the efficiency and accuracy of constructing 3D models of large-scale, long-distance, and complex environments.

[0004] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0005] In view of the above technical problems in the related art, the present invention proposes a large-scale cable tunnel environment model reconstruction method, which can solve the above problems.

[0006] To achieve the above technical objectives, the technical solution of the present invention is implemented as follows:

[0007] A large-scale cable tunnel environment model reconstruction method, characterized in that it includes the steps of:

[0008] S1: The forward fisheye lens of the panoramic camera is denoted as F camera, and the backward fisheye lens is denoted as R camera. The F camera and the R camera are used to collect checkerboard images at different angles and fisheye images with different exposures respectively;

[0009] S2: Use a panoramic camera to move forward at a constant speed to collect a panoramic video S of the cable tunnel environment, and divide the panoramic video S into Q segments of panoramic image sequence I;

[0010] S3: For each panoramic image sequence I, obtain the external parameters and sparse 3D point cloud of the F camera and the R camera, and densely reconstruct the dense 3D point cloud based on the spherical scanning algorithm;

[0011] S4: Triangulate and texture the 3D point cloud to obtain a large-scale 3D model of the cable tunnel environment.

[0012] Furthermore, the step S1 specifically includes:

[0013] S11: The forward fisheye lens of the panoramic camera is denoted as F camera, and the backward fisheye lens is denoted as R camera. 12 checkerboard images of different angles are collected using F camera and R camera respectively. The internal parameters of the panoramic camera are calibrated based on the unified camera model, including the internal parameters K of the F camera. F ,ξ F , the internal parameters K of the R camera R ,ξ R , and the internal parameters P between the F camera and the R camera, where K F ,K R is a 3×3 matrix, ξ F ,ξ R is a constant, P is a 3×4 matrix;

[0014] S12: Use the F camera and the R camera to collect 5 fisheye images with different exposure values, and calibrate the camera response curves of the F camera and the R camera based on the Debevec method. The nonlinear camera response functions are denoted as f F () and f R ().

[0015] Furthermore, the step S2 specifically includes:

[0016] S22: Fix the exposure time Δt of the F camera and the R camera F and Δt R , a panoramic camera is used to move forward at a constant speed to collect a panoramic video S of the cable tunnel environment with a resolution of 1920×1080; the panoramic video S is divided into Q segments of panoramic image sequences I, each segment of the panoramic image sequence I contains 900 frames of panoramic images, and adjacent segments of the panoramic image sequence overlap 90 frames of panoramic images.

[0017] Furthermore, the step S3 specifically includes:

[0018] S31: extract a panoramic image frame every 30 frames from the panoramic image sequence I to form a panoramic image set I(i) of N=30 frames (i=1, 2, ..., N), and the panoramic image resolution is 1920×1080; the panoramic image set I(i) includes N fisheye images IF(i) (i=1, 2, ..., N) acquired by the F camera and N fisheye images IR(i) (i=1, 2 ..., N) acquired by the R camera, and the fisheye image resolution is 960×1080;

[0019] S32: Using the camera response function f F () and f R () Convert the fisheye images IF(i) and IR(i) into high dynamic range fisheye images IFH(i) and IRH(i) respectively;

[0020] S33: Extract the Harris corner points of the first frame fisheye images IF(1) and IR(1), and use the KLT algorithm to track the Harris corner points extracted on the high dynamic range fisheye images IFH(i) and IRH(i) respectively; when the corner point tracking is lost or the corner point is more than 900 pixels away from the image center, the corner point is filtered out; the image coordinates of the j-th Harris corner point in the i-th frame after filtering are recorded as and The number of corner points is recorded as NF and NR respectively;

[0021] S34: Calculate image coordinates based on the unified camera model The corresponding spherical coordinates are Then the spherical coordinates of the jth corner point in the first frame are and The corresponding 3D point coordinates in the spherical coordinate system can be calculated by inverse depth:

[0022] in, is the inverse depth of the jth corner point in the first frame of camera F, yes The Euclidean norm of , is the inverse depth of the jth corner point in the first frame of the R camera, yes The Euclidean norm of .

[0023] S35: Using the rotation vector r i ,(r i ∈r) and the translation vector t i ,(t i ∈t) the 3D point The 3D coordinates converted to the spherical coordinate system of the i-th frame are And normalize it to get the spherical coordinates:

[0024] S36: Construct the BA formula to minimize the distance of spherical coordinates, expressed as

[0025]

[0026] Among them, |||| H is the Huber kernel function, which is used to enhance the robustness to outliers. The external parameters P of the F camera and the R camera are transformed into i F ,P i R Expressed in the same reference system, the conversion relationship is

[0027]

[0028] in, Indicates that the rotation vector is converted into a rotation matrix using the Rodrigues formula;

[0029] S37: Initialize the conversion matrix P i F =[I 3×3 0 3×1 ], inverse depth rice;

[0030] S38: Use Ceres optimizer to solve formula (1) and obtain the external parameters P of F camera and R camera i F ,P i R , sparse three-dimensional point cloud Point;

[0031] S39: The panoramic image sequence I and the external parameters P of the panoramic camera are i F and sparse 3D point cloud Point as input, and densely reconstruct dense 3D point cloud Points based on the spherical scanning algorithm.

[0032] Furthermore, the step S4 specifically includes:

[0033] S41: The 28th, 29th and 30th panoramic images of the adjacent segment q in the panoramic image sequence I are overlapped with the 1st, 2nd and 3rd panoramic images of the q+1st frame (q∈Q), and the common Harris corner points in these 6 panoramic images are extracted. The 3D coordinates of the corresponding point clouds Point(q) and Point(q+1) are recorded as X(q) and X(q+1) respectively; the world coordinate system where the first panoramic image sequence I is located is used as the global coordinate system, and the transformation matrix T(q) from the 3D point X(q) to the 3D point X(1) is calculated in sequence using a nonlinear optimization method; wherein T(q) is a 3×4 matrix, and homogeneous coordinates are used for 3D point transformation;

[0034] S42: Use the transformation matrix T(q) to successively merge the point cloud Points(q) into Points(1) to obtain a globally consistent dense three-dimensional point cloud;

[0035] S43: Perform triangulation and texture mapping on the 3D point cloud to obtain a large-scale cable tunnel environment 3D model.

[0036] Beneficial effects of the invention: The positive effect of the invention is to reconstruct a large-scale cable tunnel environment model based on a panoramic camera, effectively solving the problem of a small field of view of a pinhole camera; and also solving the speed and accuracy problems faced by the traditional SFM algorithm. The method has practical value in image-based three-dimensional reconstruction technology, and can be promoted and used in the fields of reverse engineering, computer vision, virtual reality / augmented reality, etc. It has the advantages of low algorithm complexity and high reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 For the unified camera model.

[0039] Figure 2 The relationship between the internal and external parameters of the Ricoh Theta S panoramic camera. DETAILED DESCRIPTION

[0040] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0041] like Figure 1-2 As shown, a large-scale cable tunnel environment model reconstruction method according to an embodiment of the present invention is characterized in that: a Ricoh Theta S panoramic camera is connected to a computer via a cable, and the specific method is implemented according to the following steps:

[0042] Step 1: Unify the camera model to simulate the two projection processes of fisheye lens imaging, such as Figure 1 As shown, firstly, the 3D point X = (X, Y, Z) in the space is projected onto the unit sphere of the spherical coordinate system, and then the projection point on the sphere is projected again onto the image plane of the camera coordinate system formed by moving the center of the sphere along the Z axis by a distance of -ξ to form a pinhole projection. The mathematical expression is:

[0043]

[0044] in, f x ,f y is the focal length, [c x ,c y ] T is the image center coordinate. That is, the internal parameters of the fisheye lens use a 3×3 matrix and a constant ξ. The inverse projection function of the unified camera model is

[0045]

[0046] Among them, u=[u,v] T is the image coordinate, are spherical coordinates, is the inverse depth.

[0047] The forward fisheye lens of the Ricoh Theta S panoramic camera is denoted as F camera, and the backward fisheye lens is denoted as R camera. The F camera and the R camera are used to collect 12 checkerboard images at different angles, and the internal parameters of the Ricoh Theta S panoramic camera are calibrated based on the unified camera model, including the internal parameters K of the F camera. F ,ξ F , the internal parameters K of the R camera R ,ξ R , and the internal parameters P between the F camera and the R camera, where K F ,K R is a 3×3 matrix, ξ F ,ξ R is a constant and P is a 3×4 matrix.

[0048] Step 2: The Debevec method uses multiple low dynamic range images with different exposures to calculate the camera response curve. The camera response curve describes the nonlinear function of exposure X and pixel value I as follows:

[0049] I=f(X)=f(EΔt) (3)

[0050] Where E is the ambient irradiance and Δt is the exposure time. The image pixel value I increases monotonically with the increase of exposure X, so the camera response function f() is reversible, and formula (3) can be modified as follows:

[0051] lnf -1 (I) = lnE + lnΔt (4)

[0052] Formula (4) shows that when the exposure time Δt is constant, the camera response function f() is used to convert the pixel brightness I into a high-range ambient irradiance E, so that the image has a higher dynamic range.

[0053] Five fisheye images with different exposures were collected using the F camera and the R camera, and the camera response curves of the F camera and the R camera were calibrated based on the Debevec method. The nonlinear camera response functions are denoted as f F () and f R ().

[0054] Step 3: Fix the exposure time Δt of the F camera and the R camera F and Δt R , use Ricoh Theta S panoramic camera to capture panoramic video S of cable tunnel environment at a constant speed, with a resolution of 1920×1080. Segment the panoramic video S into Q panoramic image sequences I, each panoramic image sequence I contains 900 panoramic images, and adjacent panoramic image sequences overlap 90 panoramic images. For each panoramic image sequence I, use steps 4 to 12 to obtain the external parameters P of F camera and R camera i F (q),P i R (q), sparse 3D point cloud Point(q) and dense 3D point cloud Points(q).

[0055] Step 4: Extract a panoramic image every 30 frames from the panoramic image sequence I to form a panoramic image set I(i) of N=30 frames (i=1, 2, ..., N), with a panoramic image resolution of 1920 × 1080. The panoramic image set I(i) includes N fisheye images IF(i) (i=1, 2, ..., N) captured by the F camera and N fisheye images IR(i) (i=1, 2 ..., N) captured by the R camera, with a fisheye image resolution of 960 × 1080.

[0056] Step 5: Use the camera response function f F () and f R () Convert the fisheye images IF(i) and IR(i) into high dynamic range fisheye images IFH(i) and IRH(i) respectively.

[0057] Step 6: Extract the Harris corner points of the first frame fisheye images IF(1) and IR(1), and use the KLT algorithm to track the Harris corner points extracted on the high dynamic range fisheye images IFH(i) and IRH(i). When the corner point tracking is lost or the corner point is more than 900 pixels away from the image center, the corner point is filtered out. The image coordinates of the jth Harris corner point in the i-th frame after filtering are recorded as and The number of corner points is denoted as NF and NR respectively.

[0058] Step 7: Calculate image coordinates based on the unified camera model The corresponding spherical coordinates are Then the spherical coordinates of the jth corner point in the first frame are and The corresponding 3D point coordinates in the spherical coordinate system can be calculated by inverse depth:

[0059] Step 8: Use the rotation vector r i ,(r i ∈r) and the translation vector t i ,(t i ∈t) the 3D point The 3D coordinates converted to the spherical coordinate system of the i-th frame are And normalize it to get the spherical coordinates:

[0060] Step 9: Construct the BA formula to minimize the distance of spherical coordinates, expressed as

[0061]

[0062] Among them, |||| H It is the Huber kernel function, which is used to enhance the robustness to outliers. The external parameters P of the F camera and the R camera are transformed into i F ,P i R Expressed in the same reference system, the conversion relationship is

[0063]

[0064] in, Indicates that the rotation vector is converted into a rotation matrix using the Rodrigues formula.

[0065] Step 10: Initialize the transformation matrix P i F =[I 3×3 0 3×1 ], inverse depth rice.

[0066] Step 11: Use Ceres optimizer to solve formula (5) and obtain the external parameters P of F camera and R camera. i F ,P i R , sparse three-dimensional point cloud Point.

[0067] Step 12: The panoramic image sequence I and the external parameters P of the Ricoh Theta S panoramic camera are i F and sparse 3D point cloud Point as input, densely reconstructing dense 3D point cloud Points based on the Sphere Sweeping Algorithm.

[0068] Step 13: The 28th, 29th, and 30th panoramic images of the adjacent segment q in the panoramic image sequence I overlap with the 1st, 2nd, and 3rd panoramic images of q+1 (q∈Q), and the common Harris corner points in these 6 panoramic images are extracted. The 3D coordinates of the corresponding point clouds Point(q) and Point(q+1) are recorded as X(q) and X(q+1), respectively. The world coordinate system where the first panoramic image sequence I is located is used as the global coordinate system, and the transformation matrix T(q) from the 3D point X(q) to the 3D point X(1) is calculated in sequence using a nonlinear optimization method. T(q) is a 3×4 matrix, and homogeneous coordinates are used for 3D point transformation.

[0069] Step 14: Use the transformation matrix T(q) to successively integrate the point cloud Points(q) into Points(1) to obtain a globally consistent dense three-dimensional point cloud.

[0070] Step 15: Triangulate and texture map the three-dimensional point cloud to obtain a three-dimensional model of a large-scale cable tunnel environment.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large-scale cable tunnel environment model reconstruction method, characterized in that: Includes steps: S1: The forward fisheye lens of the panoramic camera is denoted as F camera, and the backward fisheye lens is denoted as R camera. The F camera and the R camera are used to collect checkerboard images at different angles and fisheye images with different exposures respectively; S2: Use a panoramic camera to move forward at a constant speed to collect a panoramic video S of the cable tunnel environment, and divide the panoramic video S into Q segments of panoramic image sequence I; S3: For each panoramic image sequence I, obtain the external parameters and sparse 3D point cloud of the F camera and the R camera, and densely reconstruct the dense 3D point cloud based on the spherical scanning algorithm; The step S3 specifically includes: S31: extract a panoramic image frame every 30 frames from the panoramic image sequence I to form a panoramic image set I(i) of N=30 frames, i=1,2,…,N; the panoramic image resolution is 1920×1080; the panoramic image set I(i) includes N fisheye images IF(i) captured by the F camera, i=1,2,…,N; and N fisheye images IR(i) captured by the R camera, i=1,2,…,N; the fisheye image resolution is 960×1080; S32: Using the camera response function f F () and f F () Convert the fisheye images IF(i) and IR(i) into high dynamic range fisheye images IFH(i) and IRH(i) respectively; S33: Extract the Harris corner points of the first frame fisheye images IF(1) and IR(1), and use the KLT algorithm to track the Harris corner points extracted on the high dynamic range fisheye images IFH(i) and IRH(i) respectively; when the corner point tracking is lost or the corner point is more than 900 pixels away from the image center, the corner point is filtered out; the image coordinates of the j-th Harris corner point in the i-th frame after filtering are recorded as and The number of corner points is recorded as NF and NR respectively; S34: Calculate image coordinates based on the unified camera model The corresponding spherical coordinates are Then the spherical coordinates of the jth corner point in the first frame are and The corresponding 3D point coordinates in the spherical coordinate system are calculated by inverse depth: in, is the inverse depth of the jth corner point in the first frame of camera F, yes The Euclidean norm of , is the inverse depth of the jth corner point in the first frame of the R camera, yes The Euclidean norm of ; S35: Using the rotation vector r i , r i ∈r and the translation vector t i , t i ∈t 3D point The 3D coordinates converted to the spherical coordinate system of the i-th frame are And normalize it to get the spherical coordinates: S36: Construct the BA formula to minimize the distance of spherical coordinates, expressed as Among them, || || H is the Huber kernel function, which is used to enhance the robustness to outliers. The external parameters P of the F camera and the R camera are transformed into i F ,P i R Expressed in the same reference system, the conversion relationship is in, Indicates that the rotation vector is converted into a rotation matrix using the Rodrigues formula; S37: Initialize the conversion matrix P i F =[I 3×3 0 3×1 ], inverse depth S38: Use Ceres optimizer to solve formula (1) and obtain the external parameters P of F camera and R camera i F ,P i R , sparse 3D point cloud; S39: The panoramic image sequence I and the external parameters P of the panoramic camera are i F And sparse 3D point cloud Point as input, densely reconstruct dense 3D point cloud based on spherical scanning algorithm; S4: Triangulate and texture the 3D point cloud to obtain a large-scale 3D model of the cable tunnel environment.

2. The large-scale cable tunnel environment model reconstruction method according to claim 1 is characterized in that: The step S1 specifically includes: S11: Use the F camera and the R camera to collect 12 checkerboard images at different angles, and calibrate the internal parameters of the panoramic camera based on the unified camera model, including the internal parameters K of the F camera. F ,ξ F , the internal parameters K of the R camera R ,ξ R , and the internal parameters P between the F camera and the R camera, where K F ,K R is a 3×3 matrix, ξ F ,ξ R is a constant, P is a 3×4 matrix; S12: Use the F camera and the R camera to collect 5 fisheye images with different exposure values, and calibrate the camera response curves of the F camera and the R camera based on the Debevec method. The nonlinear camera response functions are denoted as f F ( ) and f R ( ).

3. The large-scale cable tunnel environment model reconstruction method according to claim 2 is characterized in that: The step S2 specifically includes: S22: Fix the exposure time Δt of the F camera and the R camera F and Δt R , a panoramic camera is used to move forward at a constant speed to collect a panoramic video S of the cable tunnel environment with a resolution of 1920×1080; the panoramic video S is divided into Q segments of panoramic image sequences I, each segment of the panoramic image sequence I contains 900 frames of panoramic images, and adjacent segments of the panoramic image sequence overlap 90 frames of panoramic images.

4. The large-scale cable tunnel environment model reconstruction method according to claim 1 is characterized in that: The step S4 specifically includes: S41: Segment q and segment q+1 in the panoramic image sequence I are adjacent segments, and the 28th, 29th, and 30th panoramic images of segment q overlap with the 1st, 2nd, and 3rd panoramic images of segment q+1, respectively, where q∈Q, and the common Harris corner points in these 6 panoramic images are extracted, and the 3D coordinates of the corresponding point clouds Point(q) and Point(q+1) are recorded as X(q) and X(q+1), respectively; the world coordinate system where the first panoramic image sequence I is located is used as the global coordinate system, and the transformation matrix T(q) from the 3D point X(q) to the 3D point X(1) is calculated in sequence using a nonlinear optimization method; where T(q) is a 3×4 matrix, and homogeneous coordinates are used for 3D point transformation; S42: Use the transformation matrix T(q) to successively merge the point cloud Points(q) into Points(1) to obtain a globally consistent dense three-dimensional point cloud; S43: Perform triangulation and texture mapping on the 3D point cloud to obtain a large-scale cable tunnel environment 3D model.

Images