A calibration method for 3D reconstruction using bundle adjustment based on calibration sphere rotation

Abstract

The present invention discloses a calibration method for three-dimensional reconstruction based on bundle adjustment of calibration sphere rotation. The present invention proposes a method for establishing residuals according to the correspondence between marker points before and after the rotation of the calibration sphere, roughly estimating the three-dimensional coordinates of the marker points and camera parameters as initial values ​​through prior information, and performing nonlinear least squares iterative optimization. The present invention solves the problems of complex operation and expensive equipment in the calibration process, and reduces the economic cost and labor cost in the calibration process of the three-dimensional reconstruction system. Theoretical verification and experimental results show that the present invention establishes the association between multiple groups of data through rotation, solves the problem of insufficient constraints on a small amount of data and susceptibility to noise influence leading to insufficient accuracy, and the obtained results can withstand experimental verification of artificial data and real data. It has a huge application space and high universality when used in three-dimensional reconstruction systems.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction in computer graphics, and in particular to a calibration method for three-dimensional reconstruction based on bundle adjustment of calibration sphere rotation. Background Art

[0002] Three-dimensional reconstruction is an important research topic in the fields of computer graphics and computer vision, and is widely needed in fields such as virtual reality and augmented reality.

[0003] Currently, most 3D reconstruction systems use calibration plates for calibration. However, calibration plates are expensive, and calibration of multi-camera systems often requires taking multiple sets of photos from multiple angles, which results in high economic and labor costs. Other calibration methods, however, may not be accurate enough, resulting in a certain deviation between the reconstruction results and the real object, especially when reconstructing faces, which can create a sense of unreality. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide a calibration method for a bundle adjustment 3D reconstruction system based on the rotation of a calibration sphere.

[0005] The object of the present invention is achieved through the following technical solution: a calibration method for a bundle adjustment 3D reconstruction system based on calibration sphere rotation, characterized in that it comprises the following steps:

[0006] (1) A calibration sphere is placed at the center of the 3D reconstruction system, at the same position as the object being reconstructed. The delay parameters of each camera are adjusted so that each camera captures images of the calibration sphere at different angles at the same time. The size of the calibration sphere is close to the object being reconstructed so that a clear image can be captured without adjusting the focus. The color of the calibration sphere is set to be slightly different from the background color to perform foreground segmentation. Pairs of markers are randomly attached to the calibration sphere. The distance between the centers of each pair of markers is fixed, and there are no other markers with the same distance. The centers of the markers are a special color, and the surrounding circular areas are the same as the grid points on a chessboard.

[0007] (2) Keeping the position of the calibration ball unchanged, the calibration ball is rotated by 30° to 60° according to the required number of groups, and the rotation is random to ensure that the calibration ball does not return to the original position after several rotations. Repeat the shooting operation of step (1) several times to obtain several groups of calibration ball images at different angles;

[0008] (3) For each set of images taken at different angles of the calibration sphere, a coordinate system is established with the center of the sphere as the origin based on the size and color of the calibration sphere and the rough focal length and field of view angle obtained from the image EXIF ​​file. The three-dimensional coordinates of the marker points on the calibration sphere in the coordinate system as well as the position and orientation of the camera are estimated.

[0009] (3.1) Segment the calibration ball from the background based on the prior information of the calibration ball’s color and fit the circular shape of the calibration ball.

[0010] Contour, obtain the position of the center of the calibration ball on the image;

[0011] (3.2) Find the position of the center of the marker point according to the prior information of the center color of the marker point, and

[0012] The surrounding chessboard pattern accurately positions the center of the marker to sub-pixel coordinates;

[0013] (3.3) According to the prior information of the landmark point pairs, find the closest landmark point for each landmark point to match;

[0014] (3.4) According to the image coordinates of the center of the marker point, the image coordinates of the center of the calibration ball, and the position of the calibration ball on the image

[0015] Radius, the three-dimensional coordinate of the center of the estimated marker point in the coordinate system with the center of the calibration ball as the origin;

[0016] (3.5) Estimate the camera's position in the coordinate system with the center of the calibration sphere as the origin based on the radius of the calibration sphere on the image, the prior information of the calibration sphere radius, and the rough focal length of the camera obtained from the image EXIF;

[0017] (4) Using the random sampling consensus algorithm RANSAC, calculate the rotation transformation relationship between the coordinate systems established for each set of calibration sphere images at different angles in step (3), unify the coordinate system of each set of calibration sphere images, and merge the same points captured at different angles;

[0018] (4.1) Choose a random pair of landmarks in each coordinate system and calculate the rotation transformation of the two pairs of landmarks:

[0019] SVD(P T Q)=UΣV T (1)

[0020] R=VU T (2)

[0021] in v1 is the 3D coordinate of the first marker point, and v2 is the 3D coordinate of the second marker point.

[0022] Similarly, Q is the matrix generated in the coordinate system;

[0023] (4.2) Apply the obtained rotation matrix to the three-dimensional coordinates of the center of each marker point in the first coordinate system, determine whether there is a corresponding point in the second coordinate system, calculate the proportion of marker points that can find corresponding points, and repeat the steps

[0024] The process in (4.1) continues until the ratio is above a certain threshold;

[0025] (5) Use the random sampling consistency algorithm again to calculate the rotation transformation of the calibration balls between different groups, find the same points in the calibration ball images of different groups and establish the corresponding relationship;

[0026] (6) Based on the correspondence between the landmarks in the same group and the correspondence between the landmarks in different groups obtained in steps (4) and (5), the residual is established for each camera's observation of a landmark in a group:

[0027] min∑‖uK(R c R g Xt)‖(3)

[0028] Where u is the screen coordinate of the marker point, K, R c ,t is the camera's internal and external parameters, R g The rotation matrix from the first group to this group

[0029] Calculate the rotation transformation between different groups, the three-dimensional coordinates of the landmark points, and the position and orientation information of the camera as initial values, and use the nonlinear least squares method to iteratively optimize the parameters until the residual meets the requirements;

[0030] (7) Determine the origin of the 3D reconstruction coordinate system and the direction of the coordinate axes, and scale the parameters to be consistent with the physical prior.

[0031] The beneficial effect of the present invention is that it uses low-cost consumer-grade hardware equipment to solve the problem of under-constraint in facial highlight and roughness reconstruction by first calculating the regional mean and then using the regional mean as a constraint to calculate the pixel-by-pixel value within the region. The reconstruction process of this method does not require the addition of average material constraints and smoothness constraints. This reconstruction method can be tested through synthetic experiments (i.e., using an image rendered with this material as the target image for material reconstruction to reconstruct the material map used for rendering), proving the correctness of the reconstruction result. The rendering result can be felt that the reconstruction result can reflect the real facial material of the reconstructed user, and the error between the reconstructed image and the real image is significantly reduced compared to the original method. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flowchart of this method;

[0033] Figure 2 Schematic diagram of the calibration sphere. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] Attachment Figure 1 Schematic diagram of the process of this method.

[0036] The general process of the present invention is to first estimate the 3D coordinates of the landmarks in each group and the rough intrinsic and extrinsic parameters of the camera from a single set of images. Then, the rotation transformation between the different groups is calculated. Finally, the residual is established based on the corresponding relationship established by the rotation. The estimated results are used as the initial value for nonlinear least squares iterative optimization. The specific process of the present invention is described in detail below.

[0037] Step 1: Capture the calibration sphere image

[0038] like Figure 2 As shown, the calibration sphere is placed at the center of the 3D reconstruction system, at the same position as the reconstructed object, and the delay parameters of each camera are adjusted so that each camera captures images of the calibration sphere at different angles at the same time.

[0039] The calibration sphere is close in size to the reconstructed object so that a clear image can be captured without adjusting the focus. The color of the calibration sphere is set to be slightly different from the background color to facilitate foreground segmentation. Pairs of markers are randomly affixed to the calibration sphere. The distance between the centers of each pair of markers is fixed, and there are no other markers with the same distance. The centers of the markers are a special color, and the surrounding circular areas are the same as the grid points on a chessboard.

[0040] Step 2: Rotate the calibration sphere

[0041] Keep the calibration sphere in the same position and rotate it by 30° to 60° according to the required number of groups. Rotate it randomly to ensure that the calibration sphere does not return to its original position after several rotations. Repeat the first step of shooting several times to obtain several groups of calibration sphere images at different angles.

[0042] Step 3: Estimate the 3D coordinates of the markers on the calibration sphere and the position and orientation of the camera using prior information

[0043] For each set of images taken at different angles of the calibration sphere, a coordinate system is established with the center of the sphere as the origin based on the size and color of the calibration sphere and the rough focal length and field of view angle obtained from the image EXIF ​​file. The three-dimensional coordinates of the markers on the calibration sphere in the coordinate system as well as the position and orientation of the camera are estimated.

[0044] The calibration sphere is segmented from the background according to the prior information of its color, and the circular contour of the calibration sphere is fitted to obtain the position of the center of the calibration sphere on the image.

[0045] The position of the center of the marker point is found based on the prior information of the center color of the marker point, and the center position of the marker point is accurately determined to sub-pixel coordinates through the chessboard pattern around the center of the marker point.

[0046] According to the prior information of the landmark point pairs, the nearest landmark point is found for each landmark point for matching.

[0047] According to the image coordinates of the center of the marker point, the image coordinates of the center of the calibration ball, and the radius of the calibration ball on the image, estimate the three-dimensional coordinates of the center of the marker point in the coordinate system with the center of the calibration ball as the origin:

[0048]

[0049] x 2 +y 2 +z 2 =R 2 (9)

[0050] Where (x, y, z) is the three-dimensional coordinate of the center of the calibration point, (x screen ,y screen ) is the coordinate of the center of the calibration point on the image, f is the rough focal length of the camera obtained from the image EXIF, and R is the prior information of the radius of the calibration sphere.

[0051] We can get:

[0052]

[0053] x=ax screen ,y=ay screen ,z=af (11)

[0054] Based on the radius of the calibration sphere on the image, the prior information of the calibration sphere radius, and the rough focal length of the camera obtained from the image EXIF, the position of the camera in the coordinate system with the center of the calibration sphere as the origin is estimated:

[0055]

[0056] Among them, (x c ,y c ,z c ) is the 3D coordinate of the camera, f is the rough focal length of the camera obtained from the image EXIF, c x ,c y is a rough estimate of the camera optical center, r m is the prior information of the calibration sphere radius, r p is the radius of the calibration sphere in the image.

[0057] Step 4: Use RANSAC to calculate the rotation relationship between cameras

[0058] Use the random sampling consensus algorithm RANSAC to calculate the rotation transformation relationship between the coordinate systems established in the third step for each set of calibration sphere images at different angles, unify the coordinate system of each set of calibration sphere images, and merge the same points captured at different angles.

[0059] For each random pair of landmarks in the two coordinate systems, calculate the rotation transformation of the two pairs of landmarks:

[0060] SVD(P T Q)=UΣV T (15)

[0061] R=VU T (16)

[0062] in v1 is the three-dimensional coordinate of the first landmark point, v2 is the three-dimensional coordinate of the second landmark point, and Q is the same, which is the matrix generated in the coordinate system.

[0063] Apply the obtained rotation matrix to the three-dimensional coordinates of the center of each landmark point in the first coordinate system, determine whether there is a corresponding point in the second coordinate system, calculate the proportion of landmark points that can find corresponding points, and repeat the process of random landmark point pairs until the proportion is higher than a certain threshold.

[0064] Step 5: Use RANSAC to calculate the rotation transformation of the calibration sphere between different groups

[0065] The random sampling consistency algorithm is used again to calculate the rotation transformation of the calibration spheres between different groups, find the same points in the calibration sphere images of different groups and establish the corresponding relationship.

[0066] Step 6: Establish the residual according to the corresponding relationship and perform nonlinear least squares iterative optimization

[0067] According to the correspondence between the landmarks in the same group and the correspondence between the landmarks in different groups obtained in the fourth and fifth steps, the residual is established for each camera's observation of a landmark in a certain group:

[0068] min∑‖uK(R c R g Xt)‖(17)

[0069] Where u is the screen coordinate of the marker point, K, R c ,t is the camera's internal and external parameters, R g The rotation matrix from the first group to this group

[0070] The rotation transformation between different groups, the three-dimensional coordinates of the landmark points, and the position and orientation information of the camera are calculated as initial values, and the parameters are iteratively optimized using the nonlinear least squares method until the residual meets the requirements.

[0071] Step 7: Scale to real scale and determine the coordinate system

[0072] Determine the origin of the 3D reconstruction coordinate system and the direction of the coordinate axes, and scale the parameters to be consistent with the physical prior.

[0073] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0074] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A calibration method for 3D reconstruction based on bundle adjustment of calibration sphere rotation, characterized in that: The following steps are involved: (1) Place the calibration sphere at the center of the 3D reconstruction system, at the same position as the reconstructed object, and adjust the delay parameters of each camera so that each camera captures images of the calibration sphere at different angles at the same time; (2) Keeping the position of the calibration ball unchanged, the calibration ball is rotated by 30° to 60° according to the required number of groups, and the rotation is random to ensure that the calibration ball does not return to the original position after several rotations. Repeat the shooting operation of step (1) several times to obtain several groups of calibration ball images at different angles; (3) For each set of images taken at different angles of the calibration sphere, a coordinate system is established with the center of the sphere as the origin based on the size and color of the calibration sphere and the rough focal length and field of view angle obtained from the image EXIF ​​file. The three-dimensional coordinates of the marker points on the calibration sphere in the coordinate system as well as the position and orientation of the camera are estimated. (4) Using the random sampling consensus algorithm RANSAC, calculate the rotation transformation relationship between the coordinate systems established for each set of calibration sphere images at different angles in step (3), unify the coordinate system of each set of calibration sphere images, and merge the same points captured at different angles; (5) Use the random sampling consistency algorithm again to calculate the rotation transformation of the calibration balls between different groups, find the same points in the calibration ball images of different groups and establish the corresponding relationship; (6) Based on the correspondence between the landmarks in the same group and the correspondence between the landmarks in different groups obtained in steps (4) and (5), the residual is established for each camera's observation of a landmark in a group: min∑‖u-K(R c R g X-t)‖(1) Where u is the screen coordinate of the marker point, K, R c ,t is the camera's internal and external parameters, R g The rotation matrix from the first group to this group Calculate the rotation transformation between different groups, the three-dimensional coordinates of the landmark points, and the position and orientation information of the camera as initial values, and use the nonlinear least squares method to iteratively optimize the parameters until the residual meets the requirements; (7) Determine the origin of the 3D reconstruction coordinate system and the direction of the coordinate axes, and scale the parameters to be consistent with the physical prior.

2. The calibration method for bundle adjustment 3D reconstruction based on calibration sphere rotation according to claim 1, characterized in that: The step (1) is specifically as follows: The calibration sphere is close in size to the reconstructed object so that a clear image can be captured without adjusting the focus. The color of the calibration sphere is set to be slightly different from the background color to facilitate foreground segmentation. Pairs of markers are randomly affixed to the calibration sphere. The distance between the centers of each pair of markers is fixed, and there are no other markers with the same distance. The centers of the markers are a special color, and the surrounding circular areas are the same as the grid points on a chessboard.

3. The calibration method for bundle adjustment 3D reconstruction based on calibration sphere rotation according to claim 1, characterized in that: The step (3) is specifically as follows: (3.1) Segment the calibration sphere from the background based on the prior information of the calibration sphere’s color, fit the circular contour of the calibration sphere, and obtain the position of the calibration sphere’s center on the image; (3.2) Find the center of the marker point based on the prior information of the center color of the marker point, and use the checkerboard pattern around the center of the marker point to accurately locate the center of the marker point to sub-pixel coordinates; (3.3) According to the prior information of the landmark point pairs, find the closest landmark point for each landmark point to match; (3.4) Estimate the three-dimensional coordinates of the center of the marker point in the coordinate system with the center of the calibration ball as the origin based on the image coordinates of the center of the marker point, the image coordinates of the center of the calibration ball, and the radius of the calibration ball on the image; (3.5) Estimate the position of the camera in the coordinate system with the center of the calibration sphere as the origin based on the radius of the calibration sphere on the image, the prior information of the calibration sphere radius, and the rough focal length of the camera obtained from the image EXIF.

4. The calibration method for bundle adjustment 3D reconstruction based on calibration sphere rotation according to claim 1, characterized in that: The RANSAC in step (4) is specifically as follows: (4.1) Choose a random pair of landmarks in each coordinate system and calculate the rotation transformation of the two pairs of landmarks: SVD(P T Q)=UΣV T (2) R=VU T (3) in v1 is the three-dimensional coordinate of the first marker point, v2 is the three-dimensional coordinate of the second marker point, and Q is the matrix generated in the coordinate system; (4.2) Apply the obtained rotation matrix to the three-dimensional coordinates of the center of each landmark point in the first coordinate system, determine whether there is a corresponding point in the second coordinate system, calculate the proportion of landmark points that can find corresponding points, and repeat the process in step (4.1) until the proportion is higher than a certain threshold.

Images