Panorama camera and 3D laser scanner mobile measurement matching method

Abstract

According to the panorama camera and 3D laser scanner mobile measurement matching method, distortion correction, camera calibration and multi-lens relative orientation matching of the panorama camera are carried out based on the characteristics of the panorama camera for a monocular camera and the panorama camera; establishing collaborative matching between the laser data and the image data and a matching and fusion basis between the laser data and the image data; based on a pinhole imaging model, all parameters of camera calibration are introduced to complete camera calibration, and the relative orientation relation between the laser scanner and the panoramic camera is deduced according to a cylindrical or spherical projection method; a cooperative calibration method based on plane constraint is adopted, initial values of rotation and translation vectors between a checkerboard plane and a distance are solved by utilizing a constraint relation between the normal vector and the distance of the checkerboard plane in a laser and camera coordinate system, and finally a global optimal solution is obtained by iteratively optimizing a target equation. The method is simpler to operate and does not need too much manual intervention. And the matching precision is high, the speed is high, and the robustness is good.

Description

Technical Field

[0001] The present application relates to a mobile measurement calibration method, and in particular to a mobile measurement calibration method for a panoramic camera and a 3D laser scanner, and belongs to the technical field of multi-source photogrammetry. Background Art

[0002] With the development of technologies such as virtual reality and 3D modeling, 3D laser scanners have been widely used in many fields, and 3D point cloud data is increasingly needed as data support. However, 3D laser scanners only obtain the reflection intensity and distance information of objects, but cannot obtain the visual information of objects, so it is necessary to rely on imaging devices such as cameras to provide real visual information of the surrounding environment. In order to fuse and match the 3D laser point cloud data with the image data provided by the camera, obtain the visual and depth information of the surrounding environment, and provide reliable data support for subsequent research such as object recognition, the camera and 3D laser scanner need to be accurately calibrated in relative orientation, that is, to obtain the coordinate transformation between the two coordinate systems.

[0003] Compared with other traditional measurement technologies and 3D digitization technologies, 3D laser scanning technology has a very fast speed of collecting high-resolution point clouds and 3D reconstruction, and has been widely used in 3D modeling, architecture, civil surveying, urban topography, mining and other fields. With the development of virtual reality technology, 3D laser scanning technology can be used in entertainment such as animation or movie action scene simulation.

[0004] On the other hand, although 3D laser scanning technology provides people with 3D data of the surface of the target object, it cannot obtain its true texture information. Traditional image data is still needed to obtain the texture information of the real world. Panoramic lenses have been widely used in mobile measurement systems, security monitoring, virtual reality, etc. due to their large field of view, integrated imaging, and rotation invariance. However, their disadvantage is that the image after imaging has a large deformation, and the distortion in the panoramic camera needs to be corrected during the panoramic imaging process.

[0005] Since the depth information and image information provided by 3D laser scanners and panoramic cameras are naturally complementary, they have been used in a variety of robots and application systems, such as mobile robots, drones, and unmanned driving systems. If the camera image and 3D point cloud can be fused, the real 3D coordinates of the surface of the object and its texture information can be obtained, which helps to better understand the world around us. One of the basic tasks of current mobile measurement is to automatically create a 3D map of the environment. However, to create a real 3D map, it is necessary to obtain visual information (such as color, texture) of the surrounding environment, and the visual information needs to be accurately matched with the distance information. It is necessary to obtain the relative orientation relationship between the camera and the 3D laser scanner, and to coordinate the two, the rigid body transformation between their respective reference coordinate systems must be estimated. By obtaining the rigid body transformation between the two, the 3D point cloud and the image in the laser coordinate system are matched, laying the foundation for data fusion.

[0006] The problems that need to be solved in the calibration of panoramic cameras and 3D laser scanners in the prior art and the key technical difficulties of this application include:

[0007] (1) How to obtain 3D information of the surrounding environment more quickly and effectively is the research and development direction of mobile measurement. The speed and accuracy of laser scanners in obtaining 3D information of the surrounding environment are also constantly improving, while panoramic cameras can obtain visual information of the surrounding environment. Since the data of the two are complementary, laser scanners and cameras can be integrated into mobile platforms. In order to obtain the matching relationship between laser data and image data, it is necessary to calibrate the relative orientation relationship between the two. The existing solution method based on feature points requires high laser scanning accuracy and is easily restricted by experimental conditions. There is a lack of calibration methods for 3D laser scanners and panoramic cameras based on 3D plane geometric constraints for monocular cameras and panoramic cameras. There is a lack of calibration of the relative orientation relationship between 3D laser scanners and panoramic cameras. The relative orientation relationship between laser scanners and panoramic cameras cannot be derived based on cylindrical or spherical projection methods; there is a lack of collaborative calibration methods based on plane constraints. Traditional calibration methods are complex to operate, require too much manual intervention, and have low calibration accuracy.

[0008] (2) The 3D laser scanner only obtains the reflection intensity and distance information of the object, but cannot obtain the visual information of the object, so it is necessary to rely on imaging devices such as cameras to provide real visual information of the surrounding environment. The existing technology lacks the data fusion and matching of 3D laser point cloud data with the image data provided by the camera to obtain the visual and depth information of the surrounding environment. The camera and 3D laser scanner lack the precise relative orientation relationship calibration, and the coordinate transformation between the two coordinate systems cannot be obtained. The relative orientation relationship between the 3D laser scanner and the camera is not established. There is a lack of plane constraint methods through calibration planes, and the rotation and translation vectors between the laser coordinate system and the camera coordinate system are not obtained. The panoramic camera and 3D laser scanner cannot be accurately fused and calibrated. The accuracy and efficiency of mobile measurement calibration are relatively low, and there are many obstacles in practical applications.

[0009] (3) The depth information and image information provided by 3D laser scanners and panoramic cameras are naturally complementary. To create a real 3D map, it is necessary to obtain visual information of the surrounding environment (such as color and texture), and the visual information needs to be accurately matched with the distance information. The existing technology cannot accurately obtain the relative orientation relationship between the camera and the 3D laser scanner, and cannot perform collaborative calibration between the two. It lacks the ability to estimate the rigid body transformation between their respective reference coordinate systems. In order to obtain the matching relationship between 3D laser point cloud data and images, camera calibration is required. The existing technology matches the laser point cloud with the feature points on the image, and thereby solves the rotation and translation relationship between the two coordinate systems. This feature point matching-based method requires high laser scanning accuracy and is easily restricted by experimental conditions. It also requires manual selection and matching of feature points. The accuracy of feature point selection is easily affected by human errors, resulting in poor robustness of the calibration results. Summary of the invention

[0010] This application aims at monocular cameras and panoramic cameras, and establishes a calibration method for 3D laser scanners and panoramic cameras based on the geometric constraints of 3D planes. The relative orientation relationship between the 3D laser scanner and the panoramic camera is established, and distortion correction, camera calibration and multi-lens relative orientation calibration of the panoramic camera are performed based on the characteristics of the panoramic camera, and the collaborative calibration between the two and the matching and fusion basis between the laser data and the image data are established; based on the pinhole imaging model, various parameters of the camera calibration are introduced to complete the camera calibration, and the relative orientation relationship between the laser scanner and the panoramic camera is derived according to the cylindrical or spherical projection method; a collaborative calibration method based on plane constraints is adopted, and the constraint relationship between the normal vector and distance of the chessboard plane in the laser and camera coordinate systems is used to solve the initial value of the rotation and translation vector between the two, and finally the target equation is iteratively optimized to obtain the global optimal solution. Compared with the traditional calibration method, this method is simpler to operate and does not require too much manual intervention. The calibration has high accuracy, fast speed and good robustness.

[0011] In order to achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0012] The mobile measurement calibration method of panoramic camera and 3D laser scanner establishes the relative orientation relationship between 3D laser scanner and camera, and obtains the rotation and translation vectors between laser coordinate system and camera coordinate system through the plane constraint method of calibration plane; establishes the solution model of multi-lens panoramic relative orientation relationship in panoramic camera, shoots indoor close-up with two cameras and obtains the image coordinates of the marker points by manually selecting marker points, and then performs spatial resection respectively, so as to solve the relative orientation relationship between two cameras; establishes the collaborative calibration method of 3D laser scanner and camera based on plane constraint, based on the specific calibration plane, obtains the constraint relationship between the normal vector of the calibration plane and the laser point on the calibration plane through camera calibration and plane fitting method, obtains the initial value of the rotation and translation vector between laser coordinate system and camera coordinate system, and finally obtains the optimal solution through iterative global optimization;

[0013] 1) Establish the relative orientation relationship between the 3D laser scanner and the panoramic camera, perform distortion correction, camera calibration and relative orientation calibration of the multi-lens of the panoramic camera based on the characteristics of the panoramic camera, establish the collaborative calibration between the two and the matching and fusion basis between the laser data and the image data;

[0014] 2) Based on the pinhole imaging model, various parameters of camera calibration are introduced to complete the camera calibration, and the relative orientation relationship between the laser scanner and the panoramic camera is derived according to the cylindrical or spherical projection method;

[0015] 3) Collaborative calibration of laser scanner and panoramic camera based on 3D constraints: First, the camera panoramic mobile calibration is performed. Then, the plane orientation model is solved, the three-dimensional translation vector is solved, the rotation 3D matrix is ​​estimated, and the global optimization of the calibration parameters is used to complete the relative orientation relationship calibration based on 3D plane constraints. The target equation is iteratively optimized to obtain the global optimal solution to complete the mobile measurement calibration.

[0016] Preferably, multi-lens panoramic relative orientation calibration: An important step in obtaining panoramic images is to calibrate the relative orientation relationship between the panoramic cameras. When the camera principal point coordinates (x0, y0) and the camera focal length f are obtained through camera calibration, they are Taylor expanded into linear equations according to the collinearity condition equation expression to obtain the error equation:

[0017]

[0018] Where (X s , Y s ,Z s) is the 3D coordinate of the camera optical center in the custom control coordinate system, ω, k are the three rotation angles between the image space coordinate system and the image space auxiliary coordinate system;

[0019] For each pair of image and object point coordinates, two error equations are listed according to Formula 1. If n control points are selected on an image, 2n error equation expressions are listed, as shown in Formula 2:

[0020] V=AX-L Formula 2

[0021] in:

[0022]

[0023] L = [x - (x) y - (y)] T Formula 3

[0024] Among them, (x) and (y) in the constant term L are the image coordinates of the control points obtained by substituting the approximate values ​​of the exterior orientation elements into the collinearity condition equation to obtain the solution of the correction number of the approximate values ​​of the exterior orientation elements:

[0025] X=(A T A) -1 A T L type 4

[0026] Get the correction number of the exterior orientation element and compare it with the preset tolerance. If the tolerance requirement is not met, substitute the correction number into the error equation and solve the error equation repeatedly until the equation converges.

[0027] The relative orientation relationship between the two cameras is solved through the exterior orientation elements of the two cameras. In the exterior orientation elements obtained by spatial resection, R is the rotation between the image space S-xyz and the image space auxiliary coordinates S-XYZ, and T is the translation between the image space auxiliary and the object space coordinates, and we get:

[0028] X1=R1*C1

[0029] X2=R2*C2

[0030] X1+T1=X2+T2 Formula 5

[0031] C1 is the image space coordinate of camera 1, C2 is the image space coordinate of camera 2, T1 and T2 are the translations of cameras 1 and 2 in the object coordinate system:

[0032] X1=R1 -1 *X2+R1 -1 (T2-T1) Formula 6

[0033] Equation 6 is the relationship between the image space coordinates of cameras 1 and 2.

[0034] Preferably, the panoramic camera moving measurement projection model: 6 cameras are inlaid on a cylinder to form a panoramic camera, of which 5 cameras are evenly distributed around the cylinder and one camera is on the top of the cylinder. A panoramic image is obtained by projecting each image into a unified coordinate system. When there is an object point A (X, Y, Z) in space, it is projected onto a spherical surface with a radius of r. There will be an intersection point with the sphere. Assume that the intersection point is point a. The 3D coordinates (x, y, z) of point a in the spherical coordinate system are obtained. Then its spherical coordinates are expressed as Convert 3D coordinates to spherical coordinates;

[0035] Obtain the relative orientation relationship between multiple lenses in the panoramic camera, and transform each image from a separate image coordinate system to a unified spherical coordinate system. If R r Represents the rotation matrix between the camera coordinate system and the spherical coordinate system, t r Represents the translation vector between the two coordinate systems, then the camera coordinate system is transformed into the spherical coordinate system:

[0036] X′=R r *X+t r Formula 7

[0037] Then, the spherical coordinates corresponding to the image points in each image are obtained through the spherical projection formula. After the relative orientation relationship between multiple lenses and the collaborative calibration of the laser scanner with a certain camera, the relative orientation relationship between the laser scanner and other cameras is calculated, and the relative orientation relationship with the panoramic spherical coordinate system is obtained at the same time, completing the collaborative calibration of the laser scanner and the panoramic camera.

[0038] Preferably, the camera panoramic motion calibration is performed: a chessboard plane is used as the calibration plane, and then the camera shoots the chessboard plane in different orientations and postures to complete the data acquisition work, and the corner points of the chessboard plane in each photo are extracted, and the homography matrix between the image and the world coordinate system where the defined chessboard plane is located is calculated, and the camera's intrinsic parameter matrix is ​​derived to obtain the extrinsic parameter parameters between the camera coordinate system and the world coordinate system, and the optimal solution of each parameter is obtained by a nonlinear iterative optimization method.

[0039] Preferably, the plane orientation model is solved by first manually selecting laser points on the checkerboard plane on the 3D laser point cloud data, and then using these selected laser points as sample data sets, thereby serving as input for screening and eliminating abnormal data, and then obtaining the plane equation of the checkerboard plane. The specific process is as follows:

[0040] Step 1: Randomly select three laser points from the data set;

[0041] Step 2: Determine the plane equation of the plane where the three space points are located;

[0042] Step 3: Find the inner point in the data set corresponding to the plane;

[0043] Step 4: Repeat the above process until the optimal plane equation is found, that is, the equation includes the most points in the data set, that is, the interior points in the plane, or the error is the smallest;

[0044] The equation of the plane is expressed as follows:

[0045] θx-α=0 Equation 8

[0046] Where x represents the coordinates of an arbitrary laser point on the plane, θ is a 3*1 unit vector, whose direction is the normal direction between the origin of the laser coordinate system and the chessboard plane, and α represents the normal distance between the origin of the laser coordinate system and the chessboard plane. For each chessboard plane, the plane equation of the chessboard plane is fitted to obtain its unit normal vector and normal distance, and the point cloud data is sparsely processed or multiple sets of points are used to solve the plane equation.

[0047] Preferably, solve the three-dimensional translation vector: establish a step-by-step solution method, and divide the solution process of the rotation and translation vector between the laser coordinate system and the camera coordinate system into two stages: in the first stage, decompose the process of solving the external parameter into two processes of solving the rotation and translation vectors respectively, so as to obtain the initial values ​​of the rotation and translation vectors; in the second stage, transform the rotation and translation vectors obtained by the internal point on the chessboard in the laser coordinate system into the camera coordinate system through the previous step, so that the sum of the squares of the difference between the product of the unit normal vector of the chessboard plane in the camera coordinate system and the corresponding distance is minimized, establish an error equation for global optimization, and obtain the optimal solution between the laser coordinate system and the camera coordinate system;

[0048] The camera calibration obtains the rotation R and translation vector t between the camera and each chessboard posture in the camera coordinate system. The component Z of the world coordinate system where the chessboard plane is located is 0. The direction of the third column vector of the rotation matrix is ​​the direction of the unit normal vector of the chessboard plane, which is represented by 0. At the same time, α is used to represent the distance between the chessboard plane and the coordinate origin. Through the translation vector t between the laser coordinate system and the camera coordinate system, the vertical distance between the origin in the laser coordinate system and the chessboard plane should be equal to List the target error equation for the translation vector t:

[0049]

[0050] Writing it in the form of the following matrix gives a simpler expression:

[0051] θc =[θ c，1 θ c，2 … θ c，n ] θ l =[θ l，1 θ l，2 … θ l，n ]

[0052] α c =[α c，1 α c，2 … α c，n ] T α l =[α l，1 α l，2 … α l，n ] T Formula 10

[0053] Where n is the number of postures on the chessboard plane, l represents the number, and the target equation of the translation vector t is written as follows:

[0054]

[0055] Then the closed-form solution of t is:

[0056]

[0057] Get the three-dimensional translation vector.

[0058] Preferably, the rotation 3D matrix is ​​estimated: after obtaining the initial value of the translation vector between the laser coordinate system and the camera coordinate system, the rotation 3D matrix between the two coordinate systems is determined. The rotation matrix does not change the mutual orientation relationship between the vectors. The unit normal vector of the chessboard in the laser coordinate system is equivalent to the unit normal vector in the laser coordinate system after the rotation transformation. The target equation of the rotation matrix is ​​listed as follows:

[0059]

[0060] The rotation matrix is ​​an orthogonal matrix, which satisfies the properties of the orthogonal matrix and obtains the following formula:

[0061] R T R=l

[0062] det(R)=1 Formula 14

[0063] The equation is equivalent to the following form:

[0064]

[0065] The equation solution is a typical OPP problem, which is specified as:

[0066]

[0067] Where R is the rotation matrix, and the corresponding Lagrangian function is obtained, as shown in Equation 17:

[0068]

[0069] Where L and g are the relevant Lagrange multipliers, and the derivative of the rotation matrix R yields:

[0070] M T +R(L T + L) + gR = 0

[0071] R T M T =-(L T +L)-g=MR Formula 18

[0072] The intermediate expression is symmetrical, and we get:

[0073] M T =RMR

[0074] M T M=RMRR T M T R T =RMM T R T Formula 19

[0075] If the matrix M is subjected to singular value decomposition, we get M = USV T , rewrite the above expression:

[0076] VS 2 V T =RUD 2 U T R T Formula 20

[0077] Get the closed-form solution of the rotation matrix:

[0078] R=VU T Formula 21

[0079] According to the solution equation of the rotation matrix, let After singular value decomposition, it is substituted into equation 21 to obtain the initial value of the rotation matrix.

[0080] Preferably, global optimization of calibration parameters: the rotation and translation vectors of the 3D plane in the laser coordinate system are obtained by solving, and the vectors are transformed into the camera coordinate system. After the transformation, the 3D coordinates of the 3D plane in the camera coordinate system are obtained. The product of the coordinates and the unit normal vector of the 3D plane in the camera coordinate system is equal to the vertical distance of the chessboard plane in the camera coordinate system. The obtained rotation and translation vectors between the laser coordinate system and the camera coordinate system are used as initial values, and the error equations of the rotation and translation vectors are listed. Then, the optimal solution is obtained by solving the error equations through iterative optimization method.

[0081] Assume x 1，i is the point cloud coordinate in the chessboard plane of the i-th laser data collected, expressed as follows:

[0082]

[0083] where x 1，i is a 3×1 matrix, m=m(i) is the serial number of the point in the checkerboard plane in the i-th laser data, and the objective equation of global optimization is listed as shown in Equation 23:

[0084]

[0085] Perform plane fitting to obtain the inner points, transform the rotation matrix into quaternion and solve it, then transform it into matrix form, and obtain the optimal solution of the rotation and translation vectors between the laser coordinate system and the camera coordinate system through nonlinear optimization iteration method;

[0086] Nonlinear iterative optimization minimizes the error equation. First, an initial value β of a parameter is provided. When there is only one minimum value, when the initial value is close to the optimal solution, it avoids falling into the local minimum and converges to the global minimum. For each iteration, the parameter vector β is replaced by the new parameter β+δ, and the correction number δ of the parameter β is determined, and the following formula is obtained:

[0087] f(x i ,β+δ)≈f(x i ,β)+J i δ Equation 24

[0088] in, is the i-th row of the Jacobian matrix J. Based on the function error between the parameter vector β and the new parameter β+δ after iteration, we get:

[0089] S(β+δ)≈∑(y i -f(x i ,β)-J i δ) 2 Formula 25

[0090] By taking the derivative of δ through equation 25 and setting the right side to 0 to minimize the error, we obtain:

[0091] (J T J)δ=J T [yf(β)] Equation 26

[0092] Add a small amount on the diagonal so that J T J becomes J T J+λI makes the matrix reversible, and we get:

[0093] (J T J+λI)δ=J T [yf(β)] Equation 27

[0094] Where I is the identity matrix, and λ is adjusted in each iteration. If the value of the error equation decreases rapidly, λ uses a smaller value. If the iteration process causes the error to increase, the value of λ increases. By solving the initial values ​​of the rotation vector R and the translation vector t between the laser coordinate system and the camera coordinate system, and then according to the error equation, the optimal collaborative calibration result is obtained.

[0095] Preferably, the collaborative calibration process between the laser scanner and the camera based on 3D plane constraints is as follows: first, the laser and image data of the chessboard in different postures are collected to ensure the accuracy of the subsequent calibration results; then, the point cloud data on the chessboard plane is selected from the laser point cloud data, and the plane equation of the chessboard plane in the laser coordinate system is solved to obtain the direction of its normal vector and the vertical distance between the origin of the laser coordinate system; then, through camera calibration, the camera internal parameters and the rotation and translation relationship between the camera coordinate system and the chessboard plane are obtained, and then the normal vector of the chessboard plane in the camera coordinate system and the vertical distance from the plane to the origin of the camera coordinate system are obtained; then, the initial values ​​of the translation vector and the rotation vector between the laser coordinate system and the camera coordinate system are solved; finally, the parameters are globally optimized according to the global optimization equation to obtain the global optimal solution. According to the calibration results, the one-to-one correspondence between the laser point cloud data and the image data is obtained, and the point cloud data is rendered.

[0096] Compared with the prior art, the innovations and advantages of this application are:

[0097] (1) This application aims at monocular cameras and panoramic cameras, and establishes a calibration method for 3D laser scanners and panoramic cameras based on the geometric constraints of 3D planes. Taking full advantage of the complementary characteristics between the panoramic camera and 3D laser scanner data, the laser scanner and camera are integrated into a mobile platform, and the relative orientation relationship between the two is calibrated to obtain the matching relationship between the laser data and the image data, and the relative orientation relationship between the 3D laser scanner and the panoramic camera is calibrated. Based on the characteristics of the panoramic camera, distortion correction, camera calibration, and relative orientation calibration of the multi-lens of the panoramic camera are performed to establish a collaborative calibration between the two and a basis for matching and fusion between the laser data and the image data; based on the pinhole imaging model, various parameters of camera calibration are introduced to complete the camera calibration, and the relative orientation relationship between the laser scanner and the panoramic camera is derived according to the cylindrical or spherical projection method, which solves the problem that the traditional solution method based on feature points requires high laser scanning accuracy and is easily restricted by experimental conditions. The calibration of the panoramic camera and the 3D laser scanner is accurate and fast.

[0098] (2) This application adopts a collaborative calibration method based on plane constraints, using the constraint relationship between the normal vector and distance of the checkerboard plane in the laser and camera coordinate systems to solve the initial values ​​of the rotation and translation vectors between the two, and finally iteratively optimizes the target equation to obtain the global optimal solution. Compared with the calibration method of the prior art, this method is simpler to operate and does not require too much manual intervention. Finally, the accuracy of the relative orientation relationship between the solved laser coordinate system and the camera coordinate system is evaluated and analyzed. Through the relative relationship between the two coordinate systems, the corresponding relationship between the laser point cloud data and the image data can be obtained, and the laser point cloud is given the corresponding grayscale information by rendering, or the 3D point cloud data is projected onto a two-dimensional image by point cloud projection, thereby visually verifying the accuracy of the collaborative calibration between the 3D laser scanner and the camera. The panoramic camera and the 3D laser scanner can be accurately fused and calibrated, and the accuracy and efficiency of mobile measurement calibration are relatively high, which brings huge advantages to the actual measurement application of panoramic cameras and 3D laser scanners.

[0099] (3) This application makes full use of the complementary characteristics between the depth information and image information provided by the 3D laser scanner and the panoramic camera respectively, fuses the image and the 3D point cloud, obtains the real 3D coordinates of the object surface and its texture information, and thus helps to better understand the world around us. When creating a real 3D map, the visual information of the surrounding environment (such as color, texture) is obtained, and the visual information is accurately matched with the distance information, and the relative orientation relationship between the camera and the 3D laser scanner is accurately obtained. The two are coordinated and calibrated, and the rigid body transformation between their respective reference coordinate systems is estimated. By obtaining the rigid body transformation between the two, the 3D point cloud in the laser coordinate system is matched with the image, which overcomes the shortcomings of the existing camera calibration method based on feature point matching, which requires high laser scanning accuracy and is easily restricted by experimental conditions. At the same time, it requires manual selection and matching of feature points. The accuracy of feature point selection is easily affected by human errors, and the calibration results have poor robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 It is a flowchart of the collaborative calibration of the laser scanner and camera with 3D plane constraints.

[0101] Figure 2 It is a set of collected laser data and its corresponding image data

[0102] Figure 3 These are 12 sets of camera data collected at different postures and positions.

[0103] Figure 4 This is the effect of chessboard distortion correction.

[0104] Figure 5 These are the rendering effects of chessboards in different postures and their corresponding point cloud data.

[0105] Figure 6 It is the effect of projecting the point cloud onto the corresponding image.

[0106] Figure 7 It is a rendering of point cloud and panoramic image.

[0107] Figure 8 It is the point cloud and panoramic image projection of an object.

[0108] Fig. 9 It is the projection of the point cloud and panoramic image of the entire scene. DETAILED DESCRIPTION

[0109] The following, in conjunction with the accompanying drawings, further describes the technical solution of the panoramic camera and 3D laser scanner mobile measurement and calibration method provided by the present application, so that technicians in this field can better understand the present application and implement it.

[0110] In the current increasingly complex working environment, how to obtain 3D information of the surrounding environment more quickly and effectively is the direction of continuous exploration and research and development of mobile measurement systems. With the development of laser scanning technology, the speed and accuracy of laser scanners in obtaining 3D information of the surrounding environment are also constantly improving, while panoramic cameras can obtain visual information of the surrounding environment. Due to the complementary characteristics between the data of the two, laser scanners and cameras are increasingly integrated into mobile platforms or in them. In order to obtain the matching relationship between laser data and image data, it is essential to calibrate the relative orientation relationship between the two. The traditional solution method based on feature points requires high laser scanning accuracy and is easily restricted by experimental conditions. This application aims at monocular cameras and panoramic cameras, and establishes a calibration method for 3D laser scanners and panoramic cameras based on the geometric constraints of the 3D plane.

[0111] Establish the relative orientation relationship between the 3D laser scanner and the panoramic camera, perform distortion correction, camera calibration and relative orientation calibration of the multi-lens of the panoramic camera based on the characteristics of the panoramic camera, establish the collaborative calibration between the two and the matching and fusion basis between the laser data and the image data;

[0112] Based on the pinhole imaging model, various parameters of camera calibration are introduced to complete the camera calibration, and the relative position relationship between the laser scanner and the panoramic camera is derived according to the cylindrical or spherical projection method;

[0113] A collaborative calibration method based on plane constraints is adopted. The constraint relationship between the normal vector and distance of the checkerboard plane in the laser and camera coordinate systems is used to solve the initial values ​​of the rotation and translation vectors between the two. Finally, the global optimal solution is obtained by iteratively optimizing the target equation. Compared with the traditional calibration method, this method is simpler to operate and does not require too much manual intervention. Finally, the accuracy evaluation and analysis of the relative orientation relationship between the solved laser coordinate system and the camera coordinate system are carried out. Through the relative relationship between the two coordinate systems, the corresponding relationship between the laser point cloud data and the image data can be obtained. The laser point cloud is given the corresponding grayscale information by rendering, or the 3D point cloud data is projected onto the two-dimensional image by point cloud projection, thereby visually verifying the accuracy of the collaborative calibration between the 3D laser scanner and the camera.

[0114] 1. Multi-lens panoramic relative orientation calibration

[0115] An important step in obtaining panoramic images is to calibrate the relative orientation relationship between the two panoramic cameras. When the camera principal point coordinates (x0, y0) and the camera focal length f are obtained through camera calibration, they are Taylor expanded into linear equations according to the collinearity condition equation expression to obtain the error equation:

[0116]

[0117] Where (X s ,Y s ,Z s ) is the 3D coordinate of the camera optical center in the custom control coordinate system, ω, k are the three rotation angles between the image space coordinate system and the image space auxiliary coordinate system;

[0118] For each pair of image and object point coordinates, two error equations are listed according to Formula 1. If n control points are selected on an image, 2n error equation expressions are listed, as shown in Formula 2:

[0119] V=AX-L Formula 2

[0120] in:

[0121]

[0122] L = [x-(x) y-(y)] T Formula 3

[0123] Among them, (x) and (y) in the constant term L are the image coordinates of the control points obtained by substituting the approximate values ​​of the exterior orientation elements into the collinearity condition equation to obtain the solution of the correction number of the approximate values ​​of the exterior orientation elements:

[0124] X=(A T A) -1 A T L type 4

[0125] The correction number of the exterior orientation element is obtained and compared with the preset limit. If the limit requirement is not met, the correction number is substituted into the error equation and the error equation is repeatedly solved until the equation converges.

[0126] The relative orientation relationship between the two cameras is solved through the exterior orientation elements of the two cameras. In the exterior orientation elements obtained by spatial resection, R is the rotation between the image space S-xyz and the image space auxiliary coordinates S-XYZ, and T is the translation between the image space auxiliary and the object space coordinates, and we get:

[0127] X1=R1*C1

[0128] X2=R2*C2

[0129] X1+T1=X2+T2 Formula 5

[0130] C1 is the image space coordinate of camera 1, C2 is the image space coordinate of camera 2, T1 and T2 are the translations of cameras 1 and 2 in the object coordinate system:

[0131] X1=R1 -1 *X2+R1 -1(T2-T1) Formula 6

[0132] Equation 6 is the relationship between the image space coordinates of cameras 1 and 2.

[0133] 2. Panoramic Camera Mobile Measurement Projection Model

[0134] Six cameras are embedded in a cylinder to form a panoramic camera, of which five cameras are evenly distributed around the cylinder and one camera is on the top of the cylinder. A panoramic image is obtained by projecting each image into a unified coordinate system. When there is an object point A (X, Y, Z) in space, it is projected onto a spherical surface with a radius of r. There will be an intersection point with the sphere. Assume that the intersection point is point a. The 3D coordinates (x, y, z) of point a in the spherical coordinate system are obtained. Then its spherical coordinates are expressed as Convert 3D coordinates to spherical coordinates;

[0135] Obtain the relative orientation relationship between multiple lenses in the panoramic camera, and transform each image from a separate image coordinate system to a unified spherical coordinate system. If R r Represents the rotation matrix between the camera coordinate system and the spherical coordinate system, t r Represents the translation vector between the two coordinate systems, then the camera coordinate system is transformed into the spherical coordinate system:

[0136] X′=R r *X+t r Formula 7

[0137] Then, the spherical coordinates corresponding to the image points in each image are obtained through the spherical projection formula. After the relative orientation relationship between multiple lenses and the collaborative calibration of the laser scanner with a certain camera, the relative orientation relationship between the laser scanner and other cameras is calculated, and the relative orientation relationship with the panoramic spherical coordinate system is obtained at the same time, completing the collaborative calibration of the laser scanner and the panoramic camera.

[0138] 3. Collaborative calibration of laser scanner and panoramic camera based on 3D constraints

[0139] First, the camera is calibrated, and then the 3D plane constraint method is used to solve the initial values ​​of the rotation and translation vectors between the camera coordinate system and the laser coordinate system, and finally the optimal solution is obtained through global optimization.

[0140] 1. Camera Panoramic Motion Calibration

[0141] A chessboard plane is used as the calibration plane, and the camera then shoots the chessboard plane in different orientations and postures to complete data collection. The corner points of the chessboard plane in each photo are extracted, and the homography matrix between the image and the world coordinate system where the defined chessboard plane is located is calculated. The camera's intrinsic parameter matrix is ​​then derived, and the extrinsic parameter parameters between the camera coordinate system and the world coordinate system are obtained. Finally, the optimal solution for each parameter is obtained through a nonlinear iterative optimization method.

[0142] Through camera calibration, we can obtain the camera intrinsic parameters and the rotation and translation relationship between the camera coordinate system and the world coordinate system. Through the rotation and translation vectors between the camera coordinate system and the world coordinate system where the chessboard plane is located, we can obtain the normal vector of the chessboard plane in the camera coordinate system and the vertical distance from the plane to the origin of the camera coordinate system. In the next step of the coordinated calibration of the laser scanner and the camera, the external parameters obtained by the camera calibration will lay the foundation for solving the initial values ​​of the rotation and translation vectors between the laser coordinate system and the camera coordinate system.

[0143] In the traditional collaborative calibration work between laser scanners and cameras, it is necessary to manually extract artificial landmarks, which takes a lot of time. This application adopts a collaborative calibration method between laser and panoramic camera based on 3D plane constraints. By fitting the plane equation of the calibration plane and camera calibration, the rotation and translation relationship between the laser coordinate system and the camera coordinate system is obtained.

[0144] (II) Relative orientation alignment based on 3D plane constraints

[0145] A simple chessboard plane is used as the calibration plane, and the constraint relationship between the normal vector and distance of the chessboard plane in the laser and camera coordinate systems is used to calibrate the relative orientation relationship. At the same time, the rotation and translation vectors are optimized and solved separately, so that the error between the 3D points on the chessboard and the original points after transformation to the camera coordinate system is minimized. It is assumed that the origin of the laser coordinate system coincides with the actual laser center to avoid the error caused by the misalignment of the laser center and the origin of the laser coordinate system. At the same time, the difference between the laser center and the origin of the laser coordinate system is small, and this error can be ignored in high-precision laser scanning systems.

[0146] 1. Solve the plane orientation model

[0147] When solving the relative orientation relationship between the laser coordinate system and the camera coordinate system, we must first determine the orientation and posture of the checkerboard plane in the laser coordinate system. In the process of solving the checkerboard plane equation, we must first screen and eliminate abnormal data: Step 1, randomly select a sample subset containing the smallest data item from the input data set, and use the data of the sample subset to calculate the mathematical model and the corresponding model parameters; Step 2, first check which data in the entire data set are consistent with the instantiation model of the estimated model parameters obtained in Step 1, and use them as internal points; if the error between a certain data and the estimated model exceeds the critical value, it is regarded as an external point, and these two steps are repeated until the number of internal points reaches a sufficient number or the error of the model is minimized, thereby finding the optimal model.

[0148] First, the laser points on the checkerboard plane are manually selected from the 3D laser point cloud data. Then, these selected laser points are used as sample data sets as input for screening and eliminating abnormal data, and the plane equation of the checkerboard plane is obtained. The specific process is as follows:

[0149] Step 1: Randomly select three laser points from the data set;

[0150] Step 2: Determine the plane equation of the plane where the three space points are located;

[0151] Step 3: Find the inner point in the data set corresponding to the plane;

[0152] Step 4: Repeat the above process until the optimal plane equation is found, that is, the equation includes the most points in the data set, that is, the interior points in the plane, or the error is the smallest;

[0153] The equation of the plane is expressed as follows:

[0154] θx-α=0 Equation 8

[0155] Where x represents the coordinates of an arbitrary laser point on the plane, θ is a 3*1 unit vector, whose direction is the normal direction between the origin of the laser coordinate system and the checkerboard plane, and α represents the normal distance between the origin of the laser coordinate system and the checkerboard plane. For each checkerboard plane, the plane equation of the checkerboard plane is fitted to obtain its unit normal vector and normal distance. However, in fact, due to the high accuracy of the laser scanner, there are many laser points on the checkerboard plane, which leads to a slow algorithm solution speed. Here, this application performs sparse processing on the point cloud data or uses multiple sets of points to solve the plane equation.

[0156] 2. Solve for the 3D translation vector

[0157] When solving the relative orientation relationship between the laser coordinate system and the camera coordinate system, the optimal external parameter should minimize the observation error of the chessboard. This application establishes a step-by-step solution method, and divides the process of solving the rotation and translation vectors between the laser coordinate system and the camera coordinate system into two stages: in the first stage, the process of solving the external parameter is decomposed into two processes of solving the rotation and translation vectors respectively, so as to obtain the initial values ​​of the rotation and translation vectors; in the second stage, the internal point on the chessboard in the laser coordinate system is transformed into the camera coordinate system through the rotation and translation vectors obtained by the previous step, so that the sum of the squares of the difference between the product of the unit normal vector of the chessboard plane in the camera coordinate system and its corresponding distance is minimized, and the error equation is established for global optimization to obtain the optimal solution between the laser coordinate system and the camera coordinate system.

[0158] Through the camera calibration in the previous part, we get the rotation R and translation vector t between the camera and each chessboard posture in the camera coordinate system. The component Z of the world coordinate system where the chessboard plane is located is 0. The direction of the third column vector of the rotation matrix is ​​the direction of the unit normal vector of the chessboard plane, which is represented by 0. At the same time, α is used to represent the distance between the chessboard plane and the coordinate origin. Through the translation vector t between the laser coordinate system and the camera coordinate system, the vertical distance between the origin in the laser coordinate system and the chessboard plane should be equal to List the target error equation for the translation vector t:

[0159]

[0160] Writing it in the form of the following matrix gives a simpler expression:

[0161] θ c =[θ c，1 θ c，2 … θ c，n ] θ l =[θ l，1 θ l，2 … θ l，n ]

[0162] α c =[α c，1 α c，2 …α c，n ] T α l =[α l，1 α l，2 … α l，n ] T Formula 10

[0163] Where n is the number of postures on the chessboard plane, l represents the number, and the target equation of the translation vector t is written as follows:

[0164]

[0165] Then the closed-form solution of t is:

[0166]

[0167] Get the three-dimensional translation vector.

[0168] 3. Estimating the rotation 3D matrix

[0169] After obtaining the initial value of the translation vector between the laser coordinate system and the camera coordinate system, determine the rotation 3D matrix between the two coordinate systems. The rotation matrix does not change the mutual orientation relationship between the vectors. The unit normal vector of the chessboard in the laser coordinate system is equivalent to its unit normal vector in the laser coordinate system after the rotation transformation. The target equation of the rotation matrix is ​​listed as follows:

[0170]

[0171] The rotation matrix is ​​an orthogonal matrix, which satisfies the properties of the orthogonal matrix and obtains the following formula:

[0172] R T R=I

[0173] det(R)=1 Formula 14

[0174] The equation is equivalent to the following form:

[0175]

[0176] The equation solution is a typical OPP problem, which is specified as:

[0177]

[0178] Where R is the rotation matrix, and the corresponding Lagrangian function is obtained, as shown in Equation 17:

[0179]

[0180] Where L and g are the relevant Lagrange multipliers, and the derivative of the rotation matrix R yields:

[0181] M T +R(L T + L) + gR = 0

[0182] R T M T =-(L T +L)-g=MR Formula 18

[0183] The intermediate expression is symmetrical, and we get:

[0184] M T=RMR

[0185] M T M=RMRR T M T R T =RMM T R T Formula 19

[0186] If the matrix M is subjected to singular value decomposition, we get M = USV T , rewrite the above expression:

[0187] VS 2 V T =RUD 2 U T R T Formula 20

[0188] Get the closed-form solution of the rotation matrix:

[0189] R=VU T Formula 21

[0190] According to the solution equation of the rotation matrix, let After singular value decomposition, it is substituted into equation 21 to obtain the initial value of the rotation matrix.

[0191] 4. Global optimization of calibration parameters

[0192] The rotation and translation vectors of the 3D plane in the laser coordinate system are obtained by solving, and then transformed into the camera coordinate system. After the transformation, the 3D coordinates of the 3D plane in the camera coordinate system are obtained. The product of the coordinates and the unit normal vector of the 3D plane in the camera coordinate system is equal to the vertical distance of the chessboard plane in the camera coordinate system. The rotation and translation vectors between the laser coordinate system and the camera coordinate system are used as initial values, and the error equations of the rotation and translation vectors are listed. Then, the optimal solution is obtained by solving the equations through iterative optimization.

[0193] Assume x 1，i is the point cloud coordinate in the chessboard plane of the i-th laser data collected, expressed as follows:

[0194]

[0195] where x 1，i is a 3×1 matrix, m=m(i) is the serial number of the point in the checkerboard plane in the i-th laser data, and the objective equation of global optimization is listed as shown in Equation 23:

[0196]

[0197] Perform plane fitting to obtain the inner points, transform the rotation matrix into quaternion and solve it, then transform it into matrix form, and obtain the optimal solution of the rotation and translation vectors between the laser coordinate system and the camera coordinate system through nonlinear optimization iteration method;

[0198] Nonlinear iterative optimization minimizes the error equation. First, an initial value β of a parameter is provided. When there is only one minimum value, when the initial value is close to the optimal solution, it avoids falling into the local minimum and converges to the global minimum. For each iteration, the parameter vector β is replaced by the new parameter β+δ, and the correction number δ of the parameter β is determined, and the following formula is obtained:

[0199] f(x i ,β+δ)≈f(x i ,β)+J i δ Equation 24

[0200] in, is the i-th row of the Jacobian matrix J. Based on the function error between the parameter vector β and the new parameter β+δ after iteration, we get:

[0201] S(β+δ)≈∑(y i -f(x i ,β)-J i δ) 2 Formula 25

[0202] By taking the derivative of δ through equation 25 and setting the right side to 0 to minimize the error, we obtain:

[0203] (J T J)δ=J T [yf(β)] Equation 26

[0204] Add a small amount on the diagonal so that J T J becomes J T J+λI makes the matrix reversible, and we get:

[0205] (J T J+λI)δ=J T [yf(β)] Equation 27

[0206] Where I is the unit matrix, and λ is adjusted in each iteration. If the value of the error equation decreases rapidly, λ uses a smaller value. If the iteration process causes the error to increase, the value of λ increases. By solving the initial values ​​of the rotation vector R and the translation vector t between the laser coordinate system and the camera coordinate system, and then according to the error equation, the optimal collaborative calibration result is obtained.

[0207] (III) 3D Constraint Collaborative Calibration Process

[0208] The flowchart of the collaborative calibration between the laser scanner and the camera based on 3D plane constraints is as follows: Figure 1 shown.

[0209] First, the laser and image data of the chessboard in different postures are collected to ensure the accuracy of the subsequent calibration results; then, the point cloud data on the chessboard plane is selected from the laser point cloud data, and the plane equation of the chessboard plane in the laser coordinate system is solved to obtain the direction of its normal vector and the vertical distance between the origin of the laser coordinate system; then, through camera calibration, the camera internal parameters and the rotation and translation relationship between the camera coordinate system and the chessboard plane are obtained, and then the normal vector of the chessboard plane in the camera coordinate system and the vertical distance from the plane to the origin of the camera coordinate system are obtained; then, the initial values ​​of the translation vector and rotation vector between the laser coordinate system and the camera coordinate system are solved; finally, the parameters are globally optimized according to the global optimization equation to obtain the global optimal solution. According to the calibration results, the one-to-one correspondence between the laser point cloud data and the image data is obtained, and the point cloud data is rendered to visually evaluate the accuracy of the calibration parameters.

[0210] 4. Experimental Results and Analysis

[0211] 1. Data preparation and preprocessing

[0212] For the acquisition of laser data, a ground-based 3D laser scanner is used to obtain point cloud data of the surrounding environment. The requirements for laser data acquisition are as follows: a. The data noise is small; b. There must be sufficient scanning resolution; c. The laser scanner is fixed on a stable base, and the origin of the laser coordinate system is close to or at the center of the laser emission; d. The laser coordinate system is a right-handed coordinate system.

[0213] First, fix the calibration plane to a certain position so that it is within the field of view of the camera and the 3D laser scanner at the same time, and then collect laser data and image data. When collecting laser data, each calibration plane returns at least 20 to 30 laser points, and the corner points of the chessboard captured by the camera are clearly identified in the image. After taking a set of data, repeat the above process and change the position and posture of the calibration plane to obtain laser data and image data at different positions and postures, completing the collection of a set of valid data.

[0214] Figure 2It is a set of collected laser data and its corresponding image data, where the laser data is the part of the original laser data that corresponds to the image shooting range. After collecting the laser and image data, the camera calibration is first performed. By selecting the checkerboard corner points of each photo to calculate the homography matrix, the camera's intrinsic parameter matrix is ​​obtained, and then the external parameters between the camera coordinate system and the world coordinate system where the checkerboard is located are obtained. Finally, the optimal solution of all parameters is obtained through global optimization. At this point, the preparation and preprocessing work of the relative orientation relationship calibration of the 3D laser scanner and the camera is completed.

[0215] 2. Relative calibration of 3D laser scanner and single camera

[0216] 1. Data Acquisition

[0217] The experiment used a RIGEL-VZ400 3D laser scanning system, and set the point cloud scanning accuracy to 8mm. The camera used was a Yi motion camera.

[0218] First, fix the 3D laser scanner on a tripod, and fix the camera in a certain position at the same time. Then fix the chessboard plane in a certain position so that it is within the field of view of the camera and the laser scanner at the same time. Then collect laser data and image data at the same time, so as to complete an effective laser-image data collection. Then change the posture and orientation of the chessboard, collect laser and camera data for each chessboard after the transformation, and repeat the process. Avoid having two chessboards on the same plane or parallel planes, and try to make the chessboard fill the frame of the image to ensure the accuracy of the distortion coefficient solution, and ensure that the chessboard image on the camera is clear, avoid image blur, and make the corner points of the chessboard on the image clearly identified.

[0219] After collecting the laser data and image data of the calibration plane according to the above process, the laser-image data of the chessboard with different postures and orientations are obtained. In the calibration experiment of the 3D laser scanner and the monocular camera of this application, a total of 15 sets of effective laser-image data are collected to ensure the accuracy of the final calibration result. Figure 3 These are 12 sets of camera data collected.

[0220] 2. Camera Calibration

[0221] After obtaining multiple sets of laser data and their corresponding image data, the camera calibration is first performed. First, the corner points of the chessboard are selected, using an 11*10 chessboard, where the size of each square is 65mm*65mm. In each photo, the four corner points of the chessboard are manually selected, and then the accuracy of the chessboard corner point positioning is achieved at the sub-pixel level based on the Harris corner point detection method, ensuring the accuracy of the subsequent solution of the homography matrix between the image coordinate system and the plane where the chessboard is located, thereby ensuring the accuracy of the entire camera calibration result; after the corner point selection is completed, the image coordinates of the chessboard corner points are obtained, and then the homography matrix H of the world coordinate system and the image coordinate system where the chessboard is located is obtained by the least squares method, and then the observed corner points are detected by the projection method to see if they match the projected corner points.

[0222] After the homography matrix H is obtained, the homography matrix can be further solved to obtain the camera intrinsic parameters, the extrinsic parameters between the camera coordinate system and the world coordinate system where the chessboard is located, and the camera distortion parameters. Finally, the optimal solution is obtained through iterative optimization.

[0223] After completing the camera calibration and obtaining the various parameters of the camera, it is also necessary to evaluate the accuracy of the obtained calibration parameters. This application performs error analysis on the back-projected corner point coordinates and the corner point coordinates on the original image to obtain the accuracy evaluation of the calibration parameters. The back-projected and original checkerboard corner point coordinates are displayed on the image.

[0224] For each calibration parameter obtained by projecting, the difference between the corner point coordinates obtained and the corner point coordinates of the chessboard is used to calculate the error of the pixel offset of the camera calibration.

[0225] The radial and tangential distortion parameters are obtained through camera calibration, and the distortion parameters are used to correct the original image to evaluate the effect of distortion correction. Figure 4 shown.

[0226] 3. Collaborative Proofreading

[0227] After completing the camera calibration, the coordinated calibration of the 3D laser scanner and the camera is performed. First, the orientation of the chessboard is determined in the laser coordinate system, and the 3D laser data points in the chessboard plane are manually selected. The color depth of the laser data represents the distance between the laser point and the origin. For each set of laser data, the laser points in the calibration plane need to be manually selected to provide point cloud data for the subsequent solution of the plane equation of the chessboard plane in the laser coordinate system.

[0228] After manually selecting the laser points on the chessboard plane, plane fitting is performed to obtain the plane equation of the chessboard plane in the laser coordinate system, thereby obtaining the normal vector of the chessboard plane in the laser coordinate system and the vertical distance between the origin of the coordinates. After obtaining the parameters of the chessboard plane, the initial values ​​of the rotation and translation vectors between the camera coordinate system and the laser coordinate system are obtained. After obtaining the initial parameters, the obtained parameters are sent to the error equation and iteratively optimized to obtain the optimal solution of the external parameters between the laser coordinate system and the camera coordinate system.

[0229] After obtaining the external parameters between the laser coordinate system and the camera coordinate system through iterative optimization, the one-to-one correspondence between the laser point cloud data and the pixels on the image is obtained according to the camera imaging model. Since the original laser point cloud data does not have grayscale information, the laser point cloud is projected onto the image according to the relative orientation relationship between the two to obtain the corresponding grayscale information, thereby obtaining the laser point cloud data with grayscale information. The laser data with grayscale information obtained after projection is registered with the original image to visually check the calibration accuracy of the 3D laser scanner and the camera.

[0230] In the experiment of this application, for the convenience of display, the 3D point cloud data with grayscale information is written as a VRML file. Figure 5 It is a display effect diagram of two chessboards in different postures and their corresponding point cloud data after rendering.

[0231] From the effect diagram of the registration result of 3D laser point cloud and image, after the grayscale information of the original checkerboard is assigned to the corresponding 3D laser point cloud, the laser point cloud can restore the texture information of the checkerboard image relatively well, thus indirectly proving the correctness of the external parameters between the obtained laser coordinate system and the camera coordinate system. At the same time, in order to further verify the progress of the registration, the laser point clouds corresponding to the two checkerboard images of different postures are cut out and projected onto the corresponding image, such as Figure 6 As shown. Figure 6 It can be seen that the point cloud corresponding to the chessboard image can be globally projected onto the chessboard on the image.

[0232] (III) Relative calibration of 3D laser instrument and panoramic camera

[0233] After obtaining the calibration results between the laser scanner and the Ladybug5 camera No. 0, the relative position relationship between the laser coordinate system and the Ladybug5 panoramic coordinate system is naturally obtained. The point cloud data can be transformed into the panoramic coordinate system. According to the panoramic camera spherical projection formula introduced in the third chapter, the one-to-one correspondence between the point cloud data and the panoramic image can be obtained. Therefore, the image data of the entire scene can be rendered with the point cloud, and the rendering effect diagram of the point cloud and the panoramic image can be obtained, such as Figure 7shown.

[0234] From the rendering of the point cloud and the panoramic image, we can see that the point cloud image is consistent with the real scene of the surrounding environment as a whole, but due to the occlusion and sparseness of the point cloud, it cannot be completely consistent with the real scene. In order to further verify the accuracy of the calibration, the point clouds of some typical objects are respectively cut out and projected onto the corresponding panoramic image. At the same time, all the point clouds are projected. The results are as follows Figure 8 and Fig. 9 shown.

[0235] From the rendering of the local point cloud and the panoramic image, it can be seen that the point cloud of a certain object can be well projected onto the corresponding image in detail, which can indirectly reflect the calibration accuracy of the laser scanner and the panoramic camera. However, due to the high accuracy of the point cloud of the laser scanner and the low resolution of the panoramic image, there is a large overlap problem in the projection of the point cloud on the panoramic image. However, in some details, it can still be seen that the projection effect of the point cloud and the panoramic image is consistent with the actual situation, which can visually show the accuracy of the joint calibration of the laser scanner and the panoramic camera.

Claims

1. A method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner, characterized in that: The relative orientation relationship between the 3D laser scanner and the camera is established, and the rotation and translation vectors between the laser coordinate system and the camera coordinate system are obtained through the plane constraint method of the calibration plane; a solution model for the relative orientation relationship of the multi-lens panoramic view in the panoramic camera is established, and the indoor close-up is photographed by two cameras and the image coordinates of the marker points are obtained by manually selecting the marker points, and then the spatial rear intersection is performed respectively, so as to solve the relative orientation relationship between the two cameras; a collaborative calibration method of the 3D laser scanner and the camera based on plane constraints is established. Based on a specific calibration plane, the constraint relationship between the normal vector of the calibration plane and the laser point on the calibration plane is obtained through camera calibration and plane fitting methods, and the initial values ​​of the rotation and translation vectors between the laser coordinate system and the camera coordinate system are obtained, and finally the optimal solution is obtained through iterative global optimization; 1) Establish the relative orientation relationship between the 3D laser scanner and the panoramic camera, perform distortion correction, camera calibration and relative orientation calibration of the multi-lens of the panoramic camera based on the characteristics of the panoramic camera, establish the collaborative calibration between the two and the matching and fusion basis between the laser data and the image data; 2) Based on the pinhole imaging model, various parameters of camera calibration are introduced to complete the camera calibration, and the relative orientation relationship between the laser scanner and the panoramic camera is derived according to the cylindrical or spherical projection method; 3) Collaborative calibration of laser scanner and panoramic camera based on 3D constraints: First, the camera panoramic mobile calibration is performed. Then, the plane orientation model is solved, the three-dimensional translation vector is solved, the rotation 3D matrix is ​​estimated, and the global optimization of the calibration parameters is used to complete the relative orientation relationship calibration based on 3D plane constraints. The target equation is iteratively optimized to obtain the global optimal solution to complete the mobile measurement calibration.

2. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Multi-lens panoramic relative orientation calibration: An important step in obtaining panoramic images is to calibrate the relative orientation relationship between the two panoramic cameras. When the camera principal point coordinates (x0, y0) and the camera focal length f are obtained through camera calibration, they are Taylor expanded into linear equations according to the collinearity condition equation expression to obtain the error equation: Where (X s ,Y s ,Z s ) is the 3D coordinate of the camera optical center in the custom control coordinate system, ω, k are the three rotation angles between the image space coordinate system and the image space auxiliary coordinate system; For each pair of image and object point coordinates, two error equations are listed according to Formula 1. If n control points are selected on an image, 2n error equation expressions are listed, as shown in Formula 2: V=AX-L Formula 2 in: L = [x - (x) y - (y)] T Equation 3 Among them, (x) and (y) in the constant term L are the image coordinates of the control points obtained by substituting the approximate values ​​of the exterior orientation elements into the collinearity condition equation to obtain the solution of the correction number of the approximate values ​​of the exterior orientation elements: X = (A T A) -1 A T Formula L4 Get the correction number of the exterior orientation element and compare it with the preset tolerance. If the tolerance requirement is not met, substitute the correction number into the error equation and solve the error equation repeatedly until the equation converges. The relative orientation relationship between the two cameras is solved through the exterior orientation elements of the two cameras. In the exterior orientation elements obtained by spatial resection, R is the rotation between the image space S-xyz and the image space auxiliary coordinates S-XYZ, and T is the translation between the image space auxiliary and the object space coordinates, and we get: X1=R1*C1 X2=R2*C2 X1+T1=X2+T2 Formula 5 C1 is the image space coordinate of camera 1, C2 is the image space coordinate of camera 2, T1 and T2 are the translations of cameras 1 and 2 in the object coordinate system: X1 = R1 -1 *X2 + R1 -1 (T2 - T1) Formula 6 Equation 6 is the relationship between the image space coordinates of cameras 1 and 2.

3. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Panoramic camera mobile measurement projection model: Six cameras are inlaid on a cylinder to form a panoramic camera, of which five cameras are evenly distributed around the cylinder and one camera is on the top of the cylinder. A panoramic image is obtained by projecting each image into a unified coordinate system. When there is an object point A (X, Y, Z) in space, it is projected onto a spherical surface with a radius of r. There will be an intersection point with the sphere. Assume that the intersection point is point a. The 3D coordinates (x, y, z) of point a in the spherical coordinate system are obtained. Then its spherical coordinates are expressed as Convert 3D coordinates to spherical coordinates; Obtain the relative orientation relationship between multiple lenses in the panoramic camera, and transform each image from a separate image coordinate system to a unified spherical coordinate system. If R r Represents the rotation matrix between the camera coordinate system and the spherical coordinate system, t r Represents the translation vector between the two coordinate systems, then the camera coordinate system is transformed into the spherical coordinate system: X′=R r ×X+t r Formula 7 Then, the spherical coordinates corresponding to the image points in each image are obtained through the spherical projection formula. After the relative orientation relationship between multiple lenses and the collaborative calibration of the laser scanner with a certain camera, the relative orientation relationship between the laser scanner and other cameras is calculated, and the relative orientation relationship with the panoramic spherical coordinate system is obtained at the same time, completing the collaborative calibration of the laser scanner and the panoramic camera.

4. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Camera panoramic motion calibration: A checkerboard plane is used as the calibration plane, and the camera then shoots the checkerboard plane in different orientations and postures to complete data collection. The corner points of the checkerboard plane in each photo are extracted, and the homography matrix between the image and the world coordinate system where the defined checkerboard plane is located is calculated. The camera's intrinsic parameter matrix is ​​derived, and the extrinsic parameter parameters between the camera coordinate system and the world coordinate system are obtained. The optimal solution for each parameter is obtained through nonlinear iterative optimization methods.

5. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Solving the plane orientation model: First, manually select laser points on the checkerboard plane from the 3D laser point cloud data, and then use these selected laser points as sample data sets, which are used as input for screening and eliminating abnormal data, and then obtain the plane equation of the checkerboard plane. The specific process is as follows: Step 1: Randomly select three laser points from the data set; Step 2: Determine the plane equation of the plane where the three space points are located; Step 3: Find the inner point in the data set corresponding to the plane; Step 4: Repeat the above process until the optimal plane equation is found, that is, the equation includes the most points in the data set, that is, the interior points in the plane, or the error is the smallest; The equation of the plane is expressed as follows: θx-α=0 Formula 8 Where x represents the coordinates of an arbitrary laser point on the plane, θ is a 3*1 unit vector, whose direction is the normal direction between the origin of the laser coordinate system and the chessboard plane, and α represents the normal distance between the origin of the laser coordinate system and the chessboard plane. For each chessboard plane, the plane equation of the chessboard plane is fitted to obtain its unit normal vector and normal distance, and the point cloud data is sparsely processed or multiple sets of points are used to solve the plane equation.

6. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Solving the three-dimensional translation vector: Establish a step-by-step solution method, and divide the solution process of the rotation and translation vector between the laser coordinate system and the camera coordinate system into two stages: In the first stage, the process of solving the external parameter is decomposed into two processes of solving the rotation and translation vectors respectively, so as to obtain the initial values ​​of the rotation and translation vectors; In the second stage, the rotation and translation vectors obtained by the internal point on the chessboard in the laser coordinate system are transformed into the camera coordinate system through the previous step, so that the sum of the squares of the difference between the product of the unit normal vector of the chessboard plane in the camera coordinate system and its corresponding distance is minimized, and the error equation is established for global optimization to obtain the optimal solution between the laser coordinate system and the camera coordinate system; The camera calibration obtains the rotation R and translation vector t between the camera and each chessboard posture in the camera coordinate system. The component Z of the world coordinate system where the chessboard plane is located is 0. The direction of the third column vector of the rotation matrix is ​​the direction of the unit normal vector of the chessboard plane, which is represented by 0. At the same time, α is used to represent the distance between the chessboard plane and the coordinate origin. Through the translation vector t between the laser coordinate system and the camera coordinate system, the vertical distance between the origin in the laser coordinate system and the chessboard plane should be equal to List the target error equation for the translation vector t: Writing it in the form of the following matrix gives a simpler expression: i c =[θ c，1 i c，2 …i c，n ] θ l =[θ l，1 i l，2 …i l，n ] a c =[θ c，1 i c，2 …a c，n ] T a l =[a l，1 a l，2 …a l，n ] T formula 10 Where n is the number of postures on the chessboard plane, l represents the number, and the target equation of the translation vector t is written as follows: Then the closed-form solution of t is: Get the three-dimensional translation vector.

7. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Estimate the rotation 3D matrix: After obtaining the initial value of the translation vector between the laser coordinate system and the camera coordinate system, determine the rotation 3D matrix between the two coordinate systems. The rotation matrix does not change the mutual orientation relationship between the vectors. The unit normal vector of the chessboard in the laser coordinate system is equivalent to its unit normal vector in the laser coordinate system after the rotation transformation. List the target equation of the rotation matrix: The rotation matrix is ​​an orthogonal matrix, which satisfies the properties of the orthogonal matrix and obtains the following formula: R T R=I det(R)=1 Formula 14 The equation is equivalent to the following form: The equation solution is a typical OPP problem, which is specified as: Where R is the rotation matrix, and the corresponding Lagrangian function is obtained, as shown in Equation 17: Where L and g are the relevant Lagrange multipliers, and the derivative of the rotation matrix R yields: M T +R(L T +L)+gR=0 R T M T = -(L T + L) - g = MR Equation 18 The intermediate expression is symmetrical, and we get: M T =RMR M T M = RMRR T M T R T = RMM T R T Equation 19 If the matrix M is subjected to singular value decomposition, we get M = USV T , rewrite the above expression: VS 2 V T =RUD 2 U T R T formula 20 Get the closed-form solution of the rotation matrix: R=VU T Formula 21 According to the solution equation of the rotation matrix, let After singular value decomposition, it is substituted into equation 21 to obtain the initial value of the rotation matrix.

8. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: Global optimization of calibration parameters: The rotation and translation vectors of the 3D plane in the laser coordinate system are obtained by solving them, and then transformed into the camera coordinate system. After the transformation, the 3D coordinates of the 3D plane in the camera coordinate system are obtained. The product of the coordinates and the unit normal vector of the 3D plane in the camera coordinate system is equal to the vertical distance of the chessboard plane in the camera coordinate system. The rotation and translation vectors between the laser coordinate system and the camera coordinate system are used as initial values, and the error equations of the rotation and translation vectors are listed. Then, the optimal solution is obtained by iterative optimization method. Assume that x1,i is the point cloud coordinate in the checkerboard plane of the i-th laser data collected, which can be expressed as follows: Where x1,i is a 3×1 matrix, m=m(i) is the serial number of the point in the checkerboard plane in the i-th laser data, and the objective equation of global optimization is listed as shown in Equation 23: Perform plane fitting to obtain the inner points, transform the rotation matrix into quaternion and solve it, then transform it into matrix form, and obtain the optimal solution of the rotation and translation vectors between the laser coordinate system and the camera coordinate system through nonlinear optimization iteration method; Nonlinear iterative optimization minimizes the error equation. First, an initial value β of a parameter is provided. When there is only one minimum value, when the initial value is close to the optimal solution, it avoids falling into the local minimum and converges to the global minimum. For each iteration, the parameter vector β is replaced by the new parameter β+δ, and the correction number δ of the parameter β is determined, and the following formula is obtained: f(x i ,β+δ)≈f(x i ,b)+J i d formula 24 in, is the i-th row of the Jacobian matrix J. Based on the function error between the parameter vector β and the new parameter β+δ after iteration, we get: S(β+δ)≈Σ(y i -f(x i ,b)-J i d) 2 formula 25 By taking the derivative of δ through equation 25 and setting the right side to 0 to minimize the error, we obtain: (J T J)δ = J T [y - f(β)] Equation 26 Add a small amount on the diagonal so that J T J becomes J T J+λI makes the matrix reversible, and we get: (J T (J + λI)δ = J T [y - f(β)] Equation 27 Where I is the identity matrix, and λ is adjusted in each iteration. If the value of the error equation decreases rapidly, λ uses a smaller value. If the iteration process causes the error to increase, the value of λ increases. By solving the initial values ​​of the rotation vector R and the translation vector t between the laser coordinate system and the camera coordinate system, and then according to the error equation, the optimal collaborative calibration result is obtained.

9. The method for mobile measurement and calibration of a panoramic camera and a 3D laser scanner according to claim 1, characterized in that: The process of collaborative calibration between laser scanner and camera based on 3D plane constraint: first, collect laser and image data of chessboard in different postures to ensure the accuracy of subsequent calibration results; then select point cloud data on chessboard plane from laser point cloud data, solve the plane equation of chessboard plane in laser coordinate system, obtain the direction of its normal vector and the vertical distance between it and the origin of laser coordinate system; then, through camera calibration, obtain camera internal parameters and the rotation and translation relationship between camera coordinate system and chessboard plane, and then obtain the normal vector of chessboard plane in camera coordinate system and the vertical distance between the plane and the origin of camera coordinate system; then, solve the initial value of translation vector and rotation vector between laser coordinate system and camera coordinate system; finally, optimize the parameters globally according to global optimization equation to obtain global optimal solution. According to the calibration results, obtain the one-to-one correspondence between laser point cloud data and image data, and render the point cloud data.

Images