Camera self-calibration method based on Delaunay triangulation network

Abstract

The invention discloses a camera self-calibration method based on a Delaunay triangulation network, and relates to the technical field of camera calibration, and the method comprises the following steps: S1, collecting an image in an experiment region, carrying out the corner detection of the experiment image through employing a Harris algorithm, and eliminating false points through employing a Gaussian smoothing function; and S2, constructing a Delaunay triangle by using the feature angular points extracted in the S1, and optimizing and removing'ill-conditioned triangles'. Computer vision and point cloud data processing are combined, the Delaunay triangulation network is constructed to simulate topographic feature information, feature matching of a three-dimensional point cloud and a two-dimensional image is achieved, camera self-calibration under the complex topographic condition is achieved, calibration precision is high, robustness is good, and the method is not prone to being affected by noise and initial values; by constructing the Delaunay triangulation network of the image and the point cloud to represent the scene topographic features, the requirement of camera calibration for the geometric features of the scene ground features is reduced, camera self-calibration under the complex topographic condition is realized, the calibration steps are simplified, and the flexibility and universality of camera calibration are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a camera self-calibration method based on Delaunay triangulation. Background Art

[0002] Camera calibration technology uses image information to restore a camera model with metric significance, and is widely used in 3D reconstruction, video surveillance and other fields. Camera intrinsic parameters and optical distortion coefficients are used to describe the camera's own characteristics, which is the premise for accurately obtaining the 3D coordinates of the object being measured. Most of the existing camera calibration methods are limited by the data source, requiring specific calibration objects and controlling the camera to perform special movements.

[0003] The camera self-calibration method starts from the perspective of projective geometry theory and proves that there is a certain constraint relationship between every two images in a multi-image sequence. This relationship is not related to the content of the captured scene and the movement of the camera, but only depends on the intrinsic parameters of the camera itself. Therefore, no calibration block is required. The camera is directly calibrated by establishing the corresponding relationship between multiple images during the movement process. It overcomes the shortcomings of traditional calibration methods and has strong flexibility. It has gradually become the mainstream of current research. However, there are still some problems in the current research on camera self-calibration methods:

[0004] 1. Currently, most camera self-calibration methods still have common problems such as low calibration accuracy, poor robustness, sensitivity to noise and initial values, and easy to fall into local optimal solutions.

[0005] 2. The corresponding geometric constraint relationship in the image is easily affected by the environment, and the calibration steps are relatively cumbersome.

[0006] 3. The camera self-calibration process is restricted by the scene, especially in large field of view and long-distance scenes with complex terrain conditions. The feasibility is poor and cannot meet the needs of fast self-calibration.

[0007] Based on this, a camera self-calibration method based on Delaunay triangulation is now provided, which can eliminate the drawbacks of the existing device. Summary of the invention

[0008] The purpose of the present invention is to provide a camera self-calibration method based on Delaunay triangulation to solve the shortcomings of modern products in the background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The camera self-calibration method based on Delaunay triangulation includes the following steps:

[0011] S1: Collect images in the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points;

[0012] S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles";

[0013] S3: construct a triangle region feature descriptor using the Delaunay triangle constructed in step S2, complete the rough matching and fine matching of the image, and obtain a feature point set;

[0014] S4: voxelize and rasterize the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map;

[0015] S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points;

[0016] S6: Using the feature point set obtained in step S3, establish the feature point relationship equations of the three sets of matching images, calculate the basic matrix and the pole coordinates, derive the Kruppa equation, and calculate the camera intrinsic parameters;

[0017] S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

[0018] On the basis of the above technical solution, the present invention also provides the following optional technical solution:

[0019] In an optional solution: in step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization, and after merging adjacent subsets, "pathological" triangles are further eliminated.

[0020] In an optional solution: in step S3, the Delaunay triangle obtained in step S2 is used to construct a triangle descriptor, a support region is established for the region formed by the three sides of any triangle, and a rotationally invariant feature vector of each pixel in the support region is constructed.

[0021] In an optional solution: in step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled, and on this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

[0022] In an optional solution: in step S5, the intersection ratio of common point line bundles in the projective invariant property is used to identify incorrect matching pairs and perform precise matching.

[0023] In an optional solution: in step S6, the property of epipolar geometry is used to establish a relationship equation between feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention combines computer vision with point cloud data processing, and simulates terrain feature information by constructing a Delaunay triangulation network, thereby achieving feature matching between three-dimensional point clouds and two-dimensional images, and realizing camera self-calibration under complex terrain conditions. The calibration has high accuracy, good robustness, and is not easily affected by noise and initial values.

[0026] 2. The present invention constructs a Delaunay triangulation network of images and point clouds to represent the terrain features of the scene, thereby reducing the requirements of camera calibration for the geometric features of the scene objects, realizing camera self-calibration under complex terrain conditions, simplifying the calibration steps, and improving the flexibility and universality of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a principle block diagram of the present invention.

[0028] Figure 2 The flowchart of the present invention is the resampling of point cloud data after voxelization.

[0029] Figure 3 This is a schematic diagram of the point cloud rasterization of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0031] In one embodiment, Figure 1-Figure 3 As shown, the camera self-calibration method based on Delaunay triangulation includes the following steps:

[0032] S1: Collect images in the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points;

[0033] S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles";

[0034] S3: construct a triangle region feature descriptor using the Delaunay triangle constructed in step S2, complete the rough matching and fine matching of the image, and obtain a feature point set;

[0035] S4: voxelize and rasterize the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map;

[0036] S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points;

[0037] S6: Using the feature point set obtained in step S3, establish the feature point relationship equations of the three sets of matching images, calculate the basic matrix and the pole coordinates, derive the Kruppa equation, and calculate the camera intrinsic parameters;

[0038] S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

[0039] In one embodiment, Figure 1 and Figure 2 As shown, in step S1, the extracted feature point set needs to be segmented and sorted.

[0040] The extracted feature point set is segmented and sorted to divide the feature point set into several non-intersecting subsets, and this process is repeated until the number of feature points in all feature point subsets is less than 4.

[0041] In one embodiment, Figure 1 and Figure 2 As shown, in step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization, and after merging adjacent subsets, "morbid" triangles are further eliminated.

[0042] Each feature point subset after segmentation in step S1 is used to construct a Delaunay triangle and perform transformation optimization to ensure that each triangle conforms to the properties of the Delaunay triangle, namely, the maximum and minimum angles and the empty circumscribed circle properties. After merging adjacent subsets, the "pathological" triangles are further eliminated to obtain the final Delaunay triangle.

[0043] In one embodiment, Figure 1 and Figure 2 As shown, in step S3, the Delaunay triangle obtained in step S2 is used to construct a triangle descriptor, a support region is established for the region formed by the three sides of any triangle, and a rotationally invariant feature vector of each pixel in the support region is constructed.

[0044] The similarity of triangles is used to complete the rough matching of Delaunay triangles. The intersection ratio of common point line bundles in the projective invariant property is further used to identify incorrect matching pairs and perform fine matching to obtain the final matching triangles.

[0045] In one embodiment, Figure 1 and Figure 2 As shown, in step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled, and on this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

[0046] In step S4, in order to make the point cloud evenly distributed and reduce the computational complexity of point cloud conversion, it is necessary to establish a uniform three-dimensional voxel grid according to the distribution space of the point cloud and resample it. On this basis, establish a mapping relationship between the point cloud and the image, and then obtain the projection of the point cloud on the two-dimensional plane. Finally, obtain the depth map by determining the grayscale value of the image.

[0047] In one embodiment, Figure 1 and Figure 2 As shown, in step S5, the intersection ratio of common point line bundles in the projective invariant property is used to identify incorrect matching pairs and perform precise matching.

[0048] In step S5, the Harris algorithm in step S1 is used to extract corner points of the depth map, and the Gaussian smoothing function is used to eliminate pseudo points to obtain a feature point set, and the extracted feature point set is segmented and sorted to obtain a feature point subset; step S2 is used to construct a Delaunay triangle in each feature point subset and perform transformation optimization, and after merging adjacent subsets, "ill-conditioned" triangles are further eliminated to obtain the final Delaunay triangle; after obtaining the final Delaunay triangle in step S3, a triangle descriptor is constructed, a support region is established for the region formed by the three sides of any triangle, and a rotationally invariant feature vector of each pixel in the support region is constructed. The similarity of triangles is used to complete the rough matching of Delaunay triangles; the intersection ratio of common point bundles in the projective invariant property is further used to identify incorrect matching pairs, and fine matching is performed to obtain the final matching triangle.

[0049] In one embodiment, Figure 1 and Figure 2 As shown, in step S6, the property of epipolar geometry is used to establish a relationship equation between the feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

[0050] On this basis, the Kruppa equation is derived and solved by conjugate iteration method to obtain the internal parameter matrix.

[0051] In step S7, based on the matching feature point set obtained in step S5, a coordinate relationship equation is constructed and a camera projection matrix is ​​constructed, the feature matching point set is introduced, the three-dimensional coordinates are matched one-to-one with the two-dimensional coordinates, and the equations are solved by simultaneous transformation to obtain the camera extrinsic parameters.

[0052] The above embodiment discloses a camera self-calibration method based on Delaunay triangulation, and its specific working principle and process are as follows:

[0053] The first step is to use the Harris algorithm to detect corners in the image to be matched. After the corner detection is completed, it needs to be segmented and sorted to divide the feature point set into several non-intersecting subsets, and repeat this cycle until the number of feature points in all feature point subsets is less than 4.

[0054] The second step is to construct a Delaunay triangle in each feature point subset after segmentation and perform transformation optimization to ensure that each triangle meets the properties of the Delaunay triangle, namely the maximum and minimum angles and the empty circumscribed circle properties. Although the maximum minimum angle property of the Delaunay triangle can avoid the appearance of "pathological" triangles as much as possible, it is still necessary to further eliminate "pathological" triangles after merging adjacent subsets.

[0055] Step 3: The area formed by the three sides of any triangle is called the support area. Calculate the average gradient (Adx, Ady) of each point in the support area. ij ,y ij ), whose gradient vector is (Gx ij , Gy ij ), where i = 1, 2, ..., M, j = 1, 2, ..., N, M is the number of sub-regions divided according to the grayscale information of the image, and N is the number of pixels in the sub-region. Calculate the vectors (Adx, Ady) and (Gx ij , Gy ij ), perform scalar multiplication and vector multiplication on these two vectors respectively, and use them to construct the rotation-invariant eigenvector V for each pixel in the support region.

[0056]

[0057] Among them, IP(X) and EP(X) are the inner product and outer product respectively, V ij is a four-dimensional rotation invariant vector. The direction of each dimension of the feature vector V is defined, and the feature description is performed in the same direction to achieve rotation invariance. As shown in formula (3), the rotation invariant vector value of each dimension is judged respectively to obtain the rotation invariant feature vector V.

[0058]

[0059] In each sub-region, the description vectors of the pixels that are rotationally invariant are arranged in matrix form, as shown in formula (4):

[0060]

[0061] Among them, V i1 、V i2 ,…,V iN The four-dimensional rotation invariant vector constructed for N pixels. Due to the difference in the number of pixels contained in different sub-regions, the dimensions of the description matrices of different sub-regions are also different. In order to obtain a descriptor that is independent of the size of the sub-region, the mean vector (MV Mean Vector) and standard deviation vector (SDV Standard Deviation Vector) of each column of the description matrix in the sub-region are calculated.

[0062] On this basis, the mean vector and standard deviation vector of each sub-region are classified and sorted to obtain the mean vector MV (T Δ ) and the corresponding standard deviation description vector SDC(T Δ ), and finally combine them into a single description matrix and normalize them to obtain triangle description vectors with the same dimensions:

[0063]

[0064] The Delaunay triangulation constructed by the set of corner points of the same scene can uniquely represent the geometric features of its terrain. After the characteristic triangle descriptor is established, the similarity between descriptors is measured by Euclidean distance. A pair of similar triangles is randomly selected from two images, and the three vertices of the triangles are matched to obtain the similarity of the triangles.

[0065]

[0066] In the formula, the value of ∠A is a, the value of ∠A′ is a′, and the similarity I of the other two angles is calculated using formula (6): b ,I c , the similarity of ΔABC and ΔA′B′C′ is the average similarity of the three vertices. By calculating the similarity of all triangle pairs in the Delaunay triangulation, the triangle pairs with a similarity greater than 0.75 are screened out to complete the rough matching of the image. Since the triangle matching pairs selected by the rough matching include both correct and incorrect matching pairs, the incorrect matching pairs must be eliminated from the rough matching set. In the present invention, the intersection ratio of the common point line bundle in the projective invariant property is used to distinguish the incorrect matching pairs.

[0067] The fourth step is to make the point cloud evenly distributed and reduce the amount of calculation for point cloud conversion. A uniform three-dimensional voxel grid is established according to the distribution space of the point cloud to complete the point cloud rasterization. On the basis of point cloud resampling, the mapping relationship between the point cloud and the image is established, and then the projection of the point cloud on the two-dimensional plane is obtained, that is, the two-dimensional image of the point cloud.

[0068] ① Construct a plane pixel coordinate system. First, create an xyo plane two-dimensional image coordinate system, where the x-axis is the horizontal axis, the coordinate range is: 0-(maxx-minx), maxx and minx are the maximum and minimum values ​​of the point cloud x-axis respectively; the y-axis is the vertical axis, the coordinate range is: 0-(maxy-miny), where maxy and miny are the maximum and minimum values ​​of the point cloud y-axis respectively. The interval of the coordinate system is equivalent to the size of the point cloud voxel grid unit.

[0069] ② Two-dimensional projection of point cloud. According to the values ​​of each point cloud on the x-axis and y-axis, its position on the two-dimensional plane is determined. When the rasterized point falls into the determined pixel coordinate system grid, a point is distributed to four pixels using the bilinear interpolation method, such as Figure 3 As shown in the figure, a and b represent the distance between the point cloud and the grid. When a point falls into the grid (i, j), it will be bilinearly interpolated to four grids: (i, j), (i, j+1), (i+1, j), and (i+1, j+1).

[0070] According to the range of the point cloud and the resolution of the desired image, the actual length L represented by a single pixel is calculated. Traverse the point cloud and divide the difference between the coordinates of the point and the coordinates of the minimum point by L, which is the pixel coordinate of the point in the two-dimensional image, as shown in equations (7) and (8).

[0071]

[0072] ③ Determine the grayscale value of the image. Determine the value of each grid according to the value of the z-axis of each point cloud, that is, the grayscale value, as shown in formula (9):

[0073]

[0074] Among them, color (i，j) is the gray value of pixel (i, j), z (i，j) is the value of the pixel (i, j) on the z-axis in the grid. The maximum and minimum values ​​of the z-axis in the point cloud are represented as maxz and minz respectively. When the grayscale value is 255, it means there is no point cloud in the grid.

[0075] Step 5: Since the 2D image is directly generated from the 3D point cloud, the pixels in the image have a one-to-one correspondence with the points in the point cloud. Use the 2D depth image obtained after the point cloud is rasterized in the above steps to construct a Delaunay triangulation network, and perform triangle matching with the Delaunay triangulation network constructed by the 2D image to obtain the corresponding triangle matching pairs. On this basis, using the mapping relationship between the 2D image and the point cloud space, search for the 3D space points corresponding to the 2D image feature points in the point cloud space and obtain their 3D coordinates.

[0076] The sixth step is to obtain the Delaunay triangulation similar triangle matching pairs after completing the image matching by extracting features from the two images and constructing Delaunay triangles. Using the obtained Delaunay triangle matching pairs, the feature point relationship equation of the two images to be matched is established, and the basic matrix F and the pole coordinates e are calculated. On this basis, the Kruppa equation is derived:

[0077]

[0078] Among them, F represents the basic matrix between the two images, e is the pole, is the antisymmetric matrix of the poles, and λ is a non-zero constant. The equation is solved using the conjugate iteration method, and ω is decomposed to obtain the internal parameter matrix.

[0079] The camera's external parameter matrix represents the translation and rotation of the camera coordinate system and the world coordinate system, which is determined by the spatial relationship between the two coordinate systems. The coordinate system relationship equation is constructed based on the point cloud and image feature matching point set, as shown in formula (11):

[0080] O c =[R|t]O w (11)

[0081] The world coordinate origin is O w The three orthogonal coordinate axes are X w , Y w , Z w The camera is located in the world coordinate system, and the origin of the camera coordinate system is O c , the lens direction coordinate is Z c , the other two coordinate axes Y c , X c Perpendicular to the direction of the lens. Any point p is mapped from the world coordinate system to the origin of the camera coordinate system O c The formula is:

[0082]

[0083] Where R is the rotation matrix, represents the position of point p in the world coordinate system, Indicates the camera origin O c The position in the world coordinate system, Represents the position of point p in the camera coordinate system. Construct the camera's projection matrix M:

[0084] M=K[R|t] (13)

[0085] In the formula, K is the camera intrinsic matrix, R is the rotation matrix, and t is the translation vector. Using the midpoint X in the point cloud = [x, y, z] T Its corresponding image point U = [u, v] T , in the homogeneous coordinate system, the three-dimensional coordinates are matched one-to-one with the two-dimensional coordinates and transformed simultaneously.

[0086]

[0087] From equation (15), we can see that the parameter equations of any three points in the point cloud data and their corresponding image points have three constraints. By introducing the matching point pairs of any three images and point clouds to solve the equations, we can get the camera extrinsic parameters:

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. The camera self-calibration method based on Delaunay triangulation is characterized by: The following steps are involved: S1: Collect images in the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points; S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles"; S3: construct a triangle region feature descriptor using the Delaunay triangle constructed in step S2, complete the rough matching and fine matching of the image, and obtain a feature point set; S4: voxelize and rasterize the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map; S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points; S6: Using the feature point set obtained in step S3, establish the feature point relationship equations of the three sets of matching images, calculate the basic matrix and the pole coordinates, derive the Kruppa equation, and calculate the camera intrinsic parameters; S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

2. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In the step S1, the extracted feature point set also needs to be segmented and sorted.

3. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization, and after merging adjacent subsets, "morbid" triangles are further eliminated.

4. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S3, a triangle descriptor is constructed using the Delaunay triangle obtained in step S2, a support region is established for the region formed by the three sides of any triangle, and a rotationally invariant feature vector of each pixel in the support region is constructed.

5. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled, and on this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

6. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S5, the intersection ratio of common point line bundles in the projective invariant property is used to identify wrong matching pairs and perform precise matching.

7. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S6, the property of epipolar geometry is used to establish a relationship equation between the feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

Images