Camera self-calibration method based on Delaunay triangulation

Abstract

The present invention discloses a camera self-calibration method based on Delaunay triangulation, which relates to the field of camera calibration technology. The method comprises the following steps: S1: collecting images within an experimental area, detecting corners of the experimental images using the Harris algorithm, and eliminating pseudo points using a Gaussian smoothing function; S2: constructing Delaunay triangles using the characteristic corner points extracted in S1, and optimizing and eliminating "pathological triangles". The present invention combines computer vision with point cloud data processing, simulates terrain feature information by constructing a Delaunay triangulation, achieves feature matching between three-dimensional point clouds and two-dimensional images, and realizes camera self-calibration under complex terrain conditions. The calibration method has high calibration accuracy, good robustness, and is not easily affected by noise and initial values. By constructing a Delaunay triangulation of images and point clouds to represent scene terrain features, the requirements for camera calibration on the geometric features of scene objects are reduced, camera self-calibration under complex terrain conditions is realized, the calibration steps are simplified, and the flexibility and universality of camera calibration are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a camera self-calibration method based on Delaunay triangulation. Background Art

[0002] Camera calibration technology uses image information to restore a metrically meaningful camera model and is widely used in fields such as 3D reconstruction and video surveillance. Camera intrinsic parameters and optical distortion coefficients are used to describe the camera's characteristics and are a prerequisite for accurately obtaining the 3D coordinates of the measured object. However, existing camera calibration methods are mostly limited by the data source, requiring specific calibration objects and controlling the camera to perform specific motions.

[0003] Camera self-calibration methods, based on projective geometry theory, prove that a certain constraint relationship exists between every two images in a multi-image sequence. This relationship is independent of the captured scene content or the camera's motion, but depends solely on the camera's intrinsic parameters. Therefore, no calibration block is required. Calibration is performed directly by simply establishing the corresponding relationship between multiple images during the camera's motion. This method overcomes the shortcomings of traditional calibration methods, is more flexible, and is gradually becoming the mainstream of current research. However, current research on camera self-calibration methods still faces some challenges:

[0004] 1. Currently, most camera self-calibration methods still have common problems such as low calibration accuracy, poor robustness, sensitivity to noise and initial values, and easy to fall into local optimal solutions.

[0005] 2. The corresponding geometric constraints in the image are easily affected by the environment, and the calibration steps are relatively cumbersome.

[0006] 3. The camera self-calibration process is restricted by the scene, especially in large field of view and long-distance scenes with complex terrain conditions. The feasibility is poor and cannot meet the needs of fast self-calibration.

[0007] Based on this, a camera self-calibration method based on Delaunay triangulation is now provided, which can eliminate the drawbacks of existing devices. Summary of the Invention

[0008] The object of the present invention is to provide a camera self-calibration method based on Delaunay triangulation to solve the shortcomings of modern products in the background technology.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The camera self-calibration method based on Delaunay triangulation includes the following steps:

[0011] S1: Collect images within the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points;

[0012] S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles";

[0013] S3: Use the Delaunay triangle constructed in step S2 to construct a triangular region feature descriptor, complete the coarse matching and fine matching of the image, and obtain the feature point set;

[0014] S4: Perform voxelization and rasterization on the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map;

[0015] S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points;

[0016] S6: Using the feature point set obtained in step S3, establish the feature point relationship equation of the three sets of matching images, calculate the basic matrix and the pole coordinates, and derive the Kruppa equation to calculate the camera intrinsic parameters;

[0017] S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

[0018] On the basis of the above technical solutions, the present invention also provides the following optional technical solutions:

[0019] In an optional solution: in step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization, and after merging adjacent subsets, "pathological" triangles are further eliminated.

[0020] In an optional solution: in step S3, the Delaunay triangle obtained in step S2 is used to construct a triangle descriptor, a support region is established for the region formed by the three sides of any triangle, and a rotation-invariant feature vector of each pixel in the support region is constructed.

[0021] In an optional solution: in step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled, and on this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

[0022] In an optional solution: in step S5, the intersection ratio of common point line bundles in the projective invariant property is used to identify incorrect matching pairs and perform precise matching.

[0023] In an optional solution: in step S6, the properties of epipolar geometry are used to establish a relationship equation between the feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention combines computer vision with point cloud data processing. By constructing a Delaunay triangulation to simulate terrain feature information, it achieves feature matching between three-dimensional point clouds and two-dimensional images, and realizes camera self-calibration under complex terrain conditions. The calibration is highly accurate, robust, and not easily affected by noise and initial values.

[0026] 2. The present invention constructs a Delaunay triangulation network of images and point clouds to represent the terrain features of the scene, which reduces the requirements of camera calibration for the geometric features of the scene objects, realizes camera self-calibration under complex terrain conditions, simplifies the calibration steps, and improves the flexibility and universality of camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a principle block diagram of the present invention.

[0028] Figure 2 This is a flowchart of resampling point cloud data after voxelization according to the present invention.

[0029] Figure 3 This is a schematic diagram of the point cloud rasterization principle of the present invention. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0031] In one embodiment, Figure 1-Figure 3 As shown in FIG, the camera self-calibration method based on Delaunay triangulation includes the following steps:

[0032] S1: Collect images within the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points;

[0033] S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles";

[0034] S3: Using the Delaunay triangle constructed in step S2 to construct a triangular region feature descriptor, complete the coarse matching and fine matching of the image, and obtain a feature point set;

[0035] S4: Perform voxelization and rasterization on the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map;

[0036] S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points;

[0037] S6: Using the feature point set obtained in step S3, establish the feature point relationship equation of the three sets of matching images, calculate the basic matrix and the pole coordinates, and derive the Kruppa equation to calculate the camera intrinsic parameters;

[0038] S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

[0039] In one embodiment, Figure 1 and Figure 2 As shown, in step S1, the extracted feature point set needs to be segmented and sorted.

[0040] The extracted feature point set is segmented and sorted to divide the feature point set into several non-overlapping subsets, and this process is repeated until the number of feature points in all feature point subsets is less than 4.

[0041] In one embodiment, Figure 1 and Figure 2 As shown, in step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization. After merging adjacent subsets, "morbid" triangles are further eliminated.

[0042] Each feature point subset after segmentation in step S1 is used to construct a Delaunay triangle and perform transformation optimization to ensure that each triangle conforms to the properties of the Delaunay triangle, namely the maximum and minimum angles and the empty circumscribed circle properties. After merging adjacent subsets, "pathological" triangles are further eliminated to obtain the final Delaunay triangle.

[0043] In one embodiment, Figure 1 and Figure 2 As shown, in step S3, the Delaunay triangle obtained in step S2 is used to construct a triangle descriptor, a support region is established for the region formed by the three sides of any triangle, and a rotation-invariant feature vector of each pixel in the support region is constructed.

[0044] The similarity of triangles is used to complete the rough matching of Delaunay triangles. The intersection ratio of common point bundles in the projective invariant property is further used to identify incorrect matching pairs and perform fine matching to obtain the final matching triangles.

[0045] In one embodiment, Figure 1 and Figure 2 As shown, in step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled, and on this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

[0046] In step S4, in order to make the point cloud evenly distributed and reduce the computational complexity of point cloud conversion, it is necessary to establish a uniform three-dimensional voxel grid based on the distribution space of the point cloud and perform resampling. On this basis, a mapping relationship between the point cloud and the image is established, and then the projection of the point cloud on the two-dimensional plane is obtained. Finally, the depth map is obtained by determining the grayscale value of the image.

[0047] In one embodiment, Figure 1 and Figure 2 As shown, in step S5, the intersection ratio of the common point line bundles in the projective invariant property is used to identify the wrong matching pairs and perform fine matching.

[0048] In step S5, the Harris algorithm in step S1 is used to extract corner points from the depth map, and the Gaussian smoothing function is used to eliminate pseudo points to obtain a feature point set. The extracted feature point set is then segmented and sorted to obtain a feature point subset. In step S2, a Delaunay triangle is constructed in each feature point subset and transformed and optimized. After merging adjacent subsets, "ill-conditioned" triangles are further eliminated to obtain the final Delaunay triangle. After obtaining the final Delaunay triangle in step S3, a triangle descriptor is constructed, a support region is established for the region formed by the three sides of any triangle, and a rotation-invariant feature vector for each pixel in the support region is constructed. The coarse matching of the Delaunay triangle is completed using the similarity of the triangles. The intersection ratio of the common point bundles in the projective invariant property is further used to identify incorrect matching pairs, and a fine matching is performed to obtain the final matching triangle.

[0049] In one embodiment, Figure 1 and Figure 2 As shown, in step S6, the properties of epipolar geometry are used to establish a relationship equation between the feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

[0050] On this basis, the Kruppa equation is derived and solved using the conjugate iteration method to obtain the internal parameter matrix.

[0051] In step S7, based on the matching feature point set obtained in step S5, a coordinate relationship equation is constructed and a camera projection matrix is ​​constructed. The feature matching point set is introduced, and the three-dimensional coordinates are matched one-to-one with the two-dimensional coordinates and the equations are solved by simultaneous transformation to obtain the camera extrinsic parameters.

[0052] The above embodiment discloses a camera self-calibration method based on Delaunay triangulation. The specific working principle and process are as follows:

[0053] The first step is to use the Harris algorithm to detect corners in the image to be matched. After corner detection, the image is segmented and sorted to divide the feature point set into several non-overlapping subsets. This process is repeated until the number of feature points in all feature point subsets is less than 4.

[0054] The second step is to construct a Delaunay triangle within each segmented feature point subset and perform a transformation optimization to ensure that each triangle conforms to the Delaunay triangle's properties, namely, maximum and minimum angles and an empty circumcircle. Although the Delaunay triangle's maximization of minimum angles minimizes the occurrence of "pathological" triangles, further elimination of "pathological" triangles is still necessary after merging adjacent subsets.

[0055] The third step is to calculate the average gradient (Adx, Ady) of each point in the support region. ij ,y ij ), its gradient vector is (Gx ij , Gy ij ), where i = 1, 2, ..., M, j = 1, 2, ..., N, M is the number of sub-regions divided according to the grayscale information of the image, and N is the number of pixels in the sub-region. Calculate the vectors (Adx, Ady) and (Gx ij , Gy ij ), perform scalar multiplication and vector multiplication on these two vectors respectively, and use them to construct the rotation-invariant eigenvector V of each pixel in the support area.

[0056]

[0057] Among them, IP(X) and EP(X) are inner product and outer product respectively, V ij is a four-dimensional rotation-invariant vector. The direction of each dimension of the feature vector V is defined, and the feature description is performed in the same direction to achieve rotation invariance. As shown in formula (3), the rotation-invariant vector value of each dimension is judged separately to obtain the rotation-invariant feature vector V.

[0058]

[0059] In each sub-region, the description vectors of the pixel points that are rotationally invariant are arranged in matrix form, as shown in formula (4):

[0060]

[0061] Among them, V i1 、V i2 ,…,V iN A four-dimensional rotationally invariant vector constructed for N pixels. Because different subregions contain varying numbers of pixels, the dimensions of the subregion description matrices also vary. To obtain a descriptor that is independent of subregion size, the mean vector (MV) and standard deviation vector (SDV) of each column in the subregion description matrix are calculated.

[0062] On this basis, the mean vector and standard deviation vector of each sub-region are classified and sorted to obtain the mean vector MV (T Δ ) and the corresponding standard deviation description vector SDC(T Δ ), and finally combine them into a single description matrix and normalize them to obtain triangle description vectors with the same dimensions:

[0063]

[0064] The Delaunay triangulation constructed from a set of corner points in the same scene can uniquely represent the geometric features of the terrain. Once the characteristic triangle descriptors are established, the similarity between descriptors is measured using Euclidean distance. A pair of similar triangles is randomly selected from the two images, and the three vertices of the triangles are matched to calculate the triangle similarity.

[0065]

[0066] In the formula, take the value of ∠A as a, the value of ∠A′ as a′, and use formula (6) to continue to calculate the similarity of the other two angles I b , I c , the similarity of ΔABC and ΔA′B′C′ is the average similarity of the three vertices. By calculating the similarity of all triangle pairs in the Delaunay triangulation, the triangle pairs with a similarity greater than 0.75 are screened out to complete the image coarse matching. Since the triangle matching pairs selected by the coarse matching include both correct and incorrect matching pairs, the incorrect matching pairs must be eliminated from the coarse matching set. In the present invention, the intersection ratio of the common point bundle in the projective invariant property is used to distinguish incorrect matching pairs.

[0067] In the fourth step, to evenly distribute the point cloud and reduce the computational complexity of point cloud conversion, a uniform 3D voxel grid is created based on the point cloud's distribution space to complete point cloud rasterization. Based on point cloud resampling, a mapping relationship between the point cloud and the image is established, and the projection of the point cloud on a 2D plane is obtained, i.e., the 2D image of the point cloud.

[0068] ① Construct a planar pixel coordinate system. First, create an xyo planar 2D image coordinate system, where the x-axis is the horizontal axis and has a coordinate range of 0-(maxx-minx), where maxx and minx are the maximum and minimum x-axis values ​​of the point cloud, respectively. The y-axis is the vertical axis and has a coordinate range of 0-(maxy-miny), where maxy and miny are the maximum and minimum y-axis values ​​of the point cloud, respectively. The interval of the coordinate system is equivalent to the size of the point cloud voxel grid unit.

[0069] ② Two-dimensional projection of point cloud. According to the value of each point cloud on the x-axis and y-axis, its position on the two-dimensional plane is determined. When the rasterized point falls into the determined pixel coordinate system grid, the bilinear interpolation method is used to distribute one point to four pixels, such as Figure 3 As shown in the figure, a and b represent the distance between the point cloud and the grid. When a point falls into the grid (i, j), it will be bilinearly interpolated to the four grids (i, j), (i, j+1), (i+1, j), and (i+1, j+1).

[0070] According to the range of the point cloud and the desired image resolution, the actual length L represented by a single pixel is calculated. Traverse the point cloud and divide the difference between the coordinates of the point and the coordinates of the minimum point by L, which is the pixel coordinate of the point in the two-dimensional image, as shown in Equations (7) and (8).

[0071]

[0072] ③ Determine the image grayscale value. Determine the value of each grid according to the z-axis value of each point cloud, that is, the grayscale value, as shown in formula (9):

[0073]

[0074] Among them, color (i，j) is the gray value of pixel (i, j), z (i，j) is the z-axis value of pixel (i, j) in the grid. The maximum and minimum z-axis values ​​in the point cloud are denoted as maxz and minz, respectively. A grayscale value of 255 indicates that there are no point clouds in the grid.

[0075] In the fifth step, since the 2D image is directly generated from the 3D point cloud, there is a one-to-one correspondence between the pixels in the image and the points in the point cloud. A Delaunay triangulation is constructed using the 2D depth image obtained after rasterizing the point cloud in the previous step. Triangles are then matched with the Delaunay triangulation constructed from the 2D image to obtain corresponding triangle matching pairs. Based on this, the mapping relationship between the 2D image and point cloud space is used to search for 3D points in the point cloud space that correspond to the 2D image feature points and obtain their 3D coordinates.

[0076] In the sixth step, after image matching is completed by extracting features from the two images and constructing Delaunay triangles, similar triangle matching pairs of the Delaunay triangulation are obtained. Using the obtained Delaunay triangle matching pairs, the relationship equation of the feature points of the two images to be matched is established, and the basic matrix F and the pole coordinates e are calculated. Based on this, the Kruppa equation is derived:

[0077]

[0078] Among them, F represents the basic matrix between the two images, e is the pole, is the antisymmetric matrix of the poles, and λ is a nonzero constant. The equation is solved using the conjugate iteration method, and ω is decomposed to obtain the internal parameter matrix.

[0079] The camera's extrinsic matrix represents the translation and rotation of the camera coordinate system and the world coordinate system, which is determined by the spatial relationship between the two coordinate systems. The coordinate system relationship equation is constructed based on the point cloud and image feature matching point set, as shown in Equation (11):

[0080] O c =[R|t]O w (11)

[0081] Among them, the world coordinate origin is O w , the three orthogonal coordinate axes are X w , Y w , Z w The camera is located in the world coordinate system, and the origin of the camera coordinate system is O c , the lens direction coordinate is Z c , the other two coordinate axes Y c , X c Perpendicular to the direction of the lens. Any point p is mapped from the world coordinate system to the origin of the camera coordinate system O c The formula is:

[0082]

[0083] Where R is the rotation matrix, represents the position of point p in the world coordinate system, Indicates the camera origin O c The position in the world coordinate system, Represents the position of point p in the camera coordinate system. Construct the camera's projection matrix M:

[0084] M=K[R|t] (13)

[0085] Where K is the camera intrinsic matrix, R is the rotation matrix, and t is the translation vector. Using the midpoint X in the point cloud = [x, y, z] T Its corresponding image point U = [u, v] T , in the homogeneous coordinate system, the three-dimensional coordinates are matched one-to-one with the two-dimensional coordinates and transformed simultaneously.

[0086]

[0087] From formula (15), we can see that there are three constraints on the parametric equations of any three points in the point cloud data and their corresponding image points. By introducing the matching point pairs of any three images and point clouds to solve the equations, we can get the camera extrinsic parameters:

[0088] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. The camera self-calibration method based on Delaunay triangulation is characterized by: The following steps are involved: S1: Collect images within the experimental area, use the Harris algorithm to detect corners of the experimental images, and use the Gaussian smoothing function to eliminate pseudo points; It is also necessary to segment and sort the extracted feature point set; S2: Use the feature corner points extracted in step S1 to construct Delaunay triangles and optimize and eliminate "pathological triangles"; S3: Use the Delaunay triangle constructed in step S2 to construct a triangular region feature descriptor, complete the coarse matching and fine matching of the image, and obtain the feature point set; S4: Perform voxelization and rasterization on the point cloud data of the experimental area, convert it into a depth map, and extract the feature points of the depth map; S5: construct a Delaunay triangle using the feature points extracted in step S4 and match it with the Delaunay triangle constructed in step S2 to obtain a set of matching feature points; S6: Using the feature point set obtained in step S3, establish the feature point relationship equation of the three sets of matching images, calculate the basic matrix and the pole coordinates, and derive the Kruppa equation to calculate the camera intrinsic parameters; S7: Based on the matching feature point set obtained in step S5, a coordinate relationship equation and a camera projection matrix are constructed, and the feature matching point set is introduced to solve the equation to obtain the camera extrinsic parameters.

2. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S2, each feature point subset segmented in step S1 is used to construct a Delaunay triangle and perform transformation optimization. After merging adjacent subsets, "morbid" triangles are further eliminated.

3. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S3, a triangle descriptor is constructed using the Delaunay triangle obtained in step S2, a support region is established for the region formed by the three sides of any triangle, and a rotation-invariant feature vector of each pixel in the support region is constructed.

4. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S4, a uniform three-dimensional voxel grid is established according to the distribution space of the point cloud and resampled. On this basis, a mapping relationship between the point cloud and the image is established, the projection of the point cloud on the two-dimensional plane is obtained, and the grayscale value of the image is determined.

5. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S5, the intersection ratio of common point line bundles in the projective invariant property is used to identify incorrect matching pairs and perform precise matching.

6. The camera self-calibration method based on Delaunay triangulation according to claim 1, characterized in that: In step S6, the properties of epipolar geometry are used to establish a relationship equation between the feature points of the two images to be matched, a two-dimensional matching point set is introduced, and an eight-point method is used for basic solution.

Images