A sparse-view camera pose estimation and three-dimensional surface reconstruction method

Abstract

This invention relates to a sparse-view camera pose estimation and 3D surface reconstruction method, belonging to the field of 3D reconstruction technology. The method includes estimating the absolute camera pose of each image from a 2D image sequence and dividing each image into a grid region. Using bilinear interpolation of the 2D grid, alignment constraints are transformed into weighted linear equations of grid vertices. Combined with projection constraints and local constraints, a sparse linear equation system is constructed in the form of hard geometric relationships. The sparse linear equation system is solved using the least squares conjugate gradient method to directly obtain the optimal surface satisfying all constraints, generating the final 3D surface mesh. This one-step solution directly obtains the optimal surface satisfying all constraints, avoiding the accumulated errors caused by multiple steps such as "reconstruction, alignment, and smoothing." Compared to complex nonlinear optimization, this invention ultimately transforms into an efficient solution of sparse linear equations, achieving improvements in both accuracy and efficiency overall.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, specifically to a method for sparse viewpoint camera pose estimation and 3D surface reconstruction. Background Technology

[0002] Recovering the three-dimensional surface structure of an object from a sequence of two-dimensional images is an important research problem in the field of computer vision. It involves using multiple two-dimensional images captured by a camera and estimating the camera's position and pose to map the two-dimensional information in the images into a three-dimensional surface in the real world. This technology has wide applications in reverse engineering, digital twins, digital preservation of cultural heritage, and medical image reconstruction.

[0003] In existing technologies, Structure from Motion (SfM) is a commonly used method for camera pose estimation, capable of recovering the spatial position and orientation of the camera from an image sequence. However, when the number of viewpoints is limited and the overlap area between viewpoints is small, it is difficult to obtain reasonable camera parameters using traditional SfM methods. Furthermore, obtaining only the camera pose is insufficient to directly generate high-quality 3D surfaces. During the mapping from 2D image pixels to 3D surfaces, due to image noise, feature matching errors, and the lack of depth information, the reconstructed surface often suffers from deformation, discontinuities, or inconsistencies with the original image. How to stably and accurately solve for 3D surfaces from a set of 2D images while simultaneously introducing effective constraint mechanisms, while better solving for camera pose, remains a technical challenge that has not yet been fully resolved. Summary of the Invention

[0004] The purpose of this invention is to provide a sparse viewpoint camera pose estimation and 3D surface reconstruction method to solve the problems mentioned in the background art.

[0005] To solve the above technical problems, the present invention adopts the following technical solution: a sparse viewpoint camera pose estimation and three-dimensional surface reconstruction method, which estimates the absolute camera pose corresponding to each image from a two-dimensional image sequence and divides each image into a grid region;

[0006] By using bilinear interpolation of two-dimensional meshes, alignment constraints are transformed into weighted linear equations of mesh vertices, and sparse linear equations are constructed in the form of hard geometric relations by combining projection constraints and local constraints.

[0007] The sparse linear equations are solved using the least squares conjugate gradient method, directly obtaining the optimal surface that satisfies all constraints, and generating the final three-dimensional surface mesh.

[0008] Preferably, based on the two-dimensional image sequence, all image pairs with overlapping relationships are identified, and the optimal geometric model is adaptively selected for each image pair;

[0009] Starting with the reference camera, the global rotation of each camera is initialized through relative rotation propagation. Then, the rotation-averaged linear system is solved iteratively to update the global rotation until convergence.

[0010] Based on global rotation and matching point pairs, the relative translation direction is estimated by solving homogeneous equations, and the correct sign is determined by triangulation verification, and the relative translation vector of each image pair is updated.

[0011] Transform all relative translations to the world coordinate system, construct a linear system using orientation and scale constraints, and solve for the initial value of the camera's global position;

[0012] Using the global position as a variable, a nonlinear least squares problem containing parallel constraints and orientation consistency constraints is constructed based on relative direction observations, and the global position of the camera is obtained through optimization.

[0013] Using all camera poses and sparse 3D points as initial values, we perform bundle adjustment and jointly optimize camera parameters to minimize reprojection error.

[0014] Preferably, the essential matrix, fundamental matrix, and homography matrix are estimated for each pair of overlapping images as candidate geometric models;

[0015] The essential matrix is ​​decomposed into four candidate relative poses by singular value decomposition, and the pose with the most inliers is selected as the final relative pose by triangulation and positive depth test. Similarly, the homography matrix is ​​normalized and decomposed and optimized.

[0016] For all image pairs, relative pose recovery and triangulation are performed using the essential matrix and homography matrix as models, respectively. The triangulation success rate and reprojection error are statistically analyzed, and an adaptive geometric model selection strategy is introduced to obtain the optimal geometric model.

[0017] Preferably, the adaptive geometric model selection strategy is as follows:

[0018] Compare the triangulation success rates of the homography matrix and the essential matrix. If the triangulation success rate of either matrix is ​​more than 1.1 times that of the other matrix, choose the matrix with the higher triangulation success rate; otherwise;

[0019] Compare the average reprojection errors of the homography matrix and the essential matrix. If the average reprojection error of either matrix is ​​more than 0.5 pixels smaller than that of the other matrix, select the matrix with the smaller average reprojection error; otherwise;

[0020] Compare the interior percentages of the homography matrix and the essential matrix. If the interior percentage of the homography matrix exceeds 15% of that of the essential matrix, choose the homography matrix; otherwise, choose the essential matrix.

[0021] If the ratio of inliers is greater than 0.8, the homography matrix is ​​forced to be used; if the ratio is less than 0.55, the essential matrix is ​​forced to be used.

[0022] Preferably, in the sparse linear equation system, projection constraints directly constrain the geometric relationship between the three-dimensional vertex coordinates and their projected two-dimensional coordinates; alignment constraints directly constrain the weighted sum of the corresponding three-dimensional vertex coordinates under different viewpoints to be equal; and local constraints directly constrain the smooth relationship between the vertex and its neighbors.

[0023] Preferably, in the sparse linear equation system, the right-hand side of the projection constraint is the corresponding non-zero constant term, and the right-hand side of the alignment constraint and the local constraint are both zero; the unknowns are the three-dimensional coordinates of all grid vertices in all images, and the total number of variables is approximately the number of images × the number of grid vertices × 3.

[0024] Preferably, for each image, a projection matrix is ​​constructed using its camera intrinsic parameter matrix and camera absolute pose; each two-dimensional vertex in the image grid is traversed, and the coordinates of the three-dimensional vertex corresponding to the two-dimensional vertex are taken as unknowns and expanded according to the projection equation to obtain two linear constraints. The projection constraints are added to the sparse linear equation system with preset projection constraint weights.

[0025] Preferably, the interior point matching between each pair of overlapping images is obtained, and the matching points are interpolated on their respective two-dimensional grids to obtain the vertex weights of the corresponding polygons; for each pair of matching points, equality constraints are established on the three components of the three-dimensional space, and the weighted sum of the three-dimensional coordinates of the relevant vertices of the polygons corresponding to the matching points in any two images is limited to be equal; and the alignment constraints are added to the equation system with preset alignment weights.

[0026] Preferably, vertex adjacency relationships are established for the grid of each image, effective vertices are selected based on the number of neighboring vertices, and smoothing constraints are applied to the three components of the three-dimensional space of the effective vertices respectively, so that the coordinate value of the current vertex is equal to the arithmetic mean of the coordinate values ​​of all its neighboring vertices; and the smoothing constraints are added to the equation system with local constraint weights.

[0027] Preferably, the sparse linear equation system is solved using the least squares conjugate gradient method. After convergence, three consecutive components of each group are extracted from the solution vector in the order of the images as vertex coordinates to construct a three-dimensional surface mesh corresponding to each image. The three-dimensional surface meshes of each image are then stitched together and merged to output a complete three-dimensional surface reconstruction result.

[0028] Beneficial effects: In the motion recovery structure stage, this invention adopts an adaptive geometric model selection strategy, which selects the optimal model between the essential matrix and the homography matrix based on the triangulation success rate and reprojection error. At the same time, it combines rotation averaging, translation averaging and global bundle adjustment optimization to realize the estimation of camera pose under sparse viewpoints, which significantly improves the accuracy and robustness of camera pose and provides good initial values ​​for downstream surface reconstruction.

[0029] This invention, by jointly constructing alignment constraints, projection constraints, and local constraints, unifies the modeling of image matching consistency, projective geometric relationships, and local surface smoothness. This effectively overcomes the problems of surface deformation and discontinuity in traditional methods, restoring a high-precision, globally consistent 3D surface. Specifically, this invention utilizes bilinear interpolation of a 2D mesh to transform alignment constraints into a weighted linear equation of mesh vertices. This transformation allows alignment constraints to directly use the same unknowns as projection and local constraints—the 3D mesh vertices—without introducing any intermediate variables, ensuring the linearity and sparsity of the unified model. Furthermore, this invention transforms the conflicting relationships among the three constraints into finding the optimal solution within a weighted least squares framework through preset weights, requiring experimental balance among them. This unified modeling and solution directly obtains the optimal surface satisfying all constraints in a single solution, avoiding the accumulated errors caused by multiple steps such as "reconstruction, alignment, and smoothing." Compared to complex nonlinear optimization, this invention ultimately transforms into an efficient solution of sparse linear equations, simultaneously improving both accuracy and efficiency overall.

[0030] This method can recover three-dimensional surfaces using only ordinary two-dimensional image sequences, without the need for depth sensors or special acquisition equipment. It has a wide range of applications and can be flexibly deployed in three-dimensional reconstruction tasks of various objects. Attached Figure Description

[0031] Figure 1 This is a flowchart of the sparse viewpoint camera pose estimation and 3D surface reconstruction method of the present invention;

[0032] Figure 2 This is a flowchart illustrating the motion recovery structure for estimating camera pose according to the present invention.

[0033] Figure 3 This is a flowchart of the adaptive geometry model selection strategy of the present invention;

[0034] Figure 4 This is a schematic diagram illustrating the projection constraint construction principle of the present invention;

[0035] Figure 5 This is a schematic diagram illustrating the principle of alignment constraint construction in this invention;

[0036] Figure 6 This is a schematic diagram illustrating the principle of local constraint construction in this invention. Detailed Implementation

[0037] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0038] A sparse-view camera pose estimation and 3D surface reconstruction method estimates the absolute camera pose of each image from a 2D image sequence and divides each image into a grid region. Using bilinear interpolation of the 2D grid, alignment constraints are transformed into weighted linear equations of grid vertices. Combined with projection constraints and local constraints, a sparse linear equation system is constructed in the form of hard geometric relationships. The least squares conjugate gradient method is used to solve the sparse linear equation system, directly obtaining the optimal surface that satisfies all constraints, generating the final 3D surface mesh. In one embodiment, a reference is made... Figure 1 The process includes the following steps:

[0039] Step 1: Estimate the absolute camera pose for each image in the 2D image sequence; for example... Figure 2 As shown, it specifically includes:

[0040] Step 1.1: For the input two-dimensional image sequence, firstly, feature point extraction and matching are performed to identify all image pairs with overlapping relationships; for each pair of overlapping images, their essential matrix is ​​estimated. Fundamental matrix Homography matrix As candidate geometric models, the interior point mask and the number of interior points corresponding to each model are recorded;

[0041] Step 1.2, for the essential matrix First, perform singular value decomposition on it to obtain the left singular vector matrix. and right singular vector matrix ;like or If it is negative, then multiply it by To ensure the orthogonality of the rotation matrices;

[0042] Introducing matrices:

[0043] ;

[0044] Construct two candidate relative rotation matrices from the decomposition results:

[0045] ;

[0046] in, The meaning is that the images i and j correspond to the relative rotation between cameras i and j. The subscript ij represents any two overlapping images i and j, and the superscript T represents the matrix transpose.

[0047] and take The third column, after normalization, is used as the translation direction vector. Four possible relative pose assumptions can be derived:

[0048] ;

[0049] For each pose hypothesis, triangulation is performed using the camera rays corresponding to the two images. The reconstructed 3D points are checked to see if they satisfy the positive depth constraint (i.e., they are located in front of the two cameras). Points that satisfy the condition are counted as interior points of the hypothesis. The pose hypothesis with the most interior points is selected as the final relative pose, and the corresponding triangulated 3D points are output.

[0050] For the fundamental matrix First, use the camera intrinsic parameters to convert it into an essential matrix:

[0051] ;

[0052] Then process it according to the above essential matrix process;

[0053] For homography matrix First, the camera intrinsic parameter matrices corresponding to the two viewpoints are used. , Normalization is performed to obtain the normalized homography matrix:

[0054] ;

[0055] right Perform singular value decomposition, normalize the elements by the second singular value, and ensure that the determinant is positive to avoid reflection transformation;

[0056] Construct the matrix:

[0057] ;

[0058] Where I is the identity matrix, with 1s on the main diagonal and 0s on the rest;

[0059] Determine whether its infinity norm is less than a preset threshold (e.g.) If it is less than 1, it is considered as an approximately pure rotation case, and is directly applied. As a rotation matrix, both the translation vector and the plane normal vector are set to zero; if they are greater than zero, a general decomposition is performed.

[0060] In general decomposition, calculation The cofactors of each order and their corresponding sign functions, according to The branch with the largest absolute value of the diagonal elements is selected for computation, resulting in two candidate plane normal vectors. , By combining auxiliary quantities such as the matrix trace, the corresponding translation vectors are derived, and two sets of rotation matrices and corresponding translation vectors are recovered, forming four sets of candidate relative poses:

[0061] ;

[0062] and its corresponding plane normal vector.

[0063] Triangulation reconstruction is performed on each group of candidate poses. Based on positive depth constraints and reprojection error, the final relative pose, planar normal vector, and 3D points are selected.

[0064] Step 1.3: For all image pairs, perform relative pose restoration and triangulation using the essential matrix and homography matrix as models respectively. Statistically calculate the triangulation success rate and reprojection error, referring to... Figure 3 As shown, firstly, all overlapping image pairs are initially configured as homography matrix H, their relative poses are restored and triangulated, and the triangulation success rate (the ratio of effective 3D points to the number of input matches) and average reprojection error of each image pair are calculated; then, all image pairs are switched to the essential matrix E, their poses are restored and various indicators are calculated, and the optimal geometric model is selected for each image pair.

[0065] In one embodiment, an adaptive geometric model selection strategy is introduced to obtain the optimal geometric model, specifically:

[0066] If the triangulation success rate of the homography matrix H is more than 1.1 times that of the essential matrix E, then choose the homography matrix H; otherwise, if the triangulation success rate of the essential matrix E is more than 1.1 times that of the homography matrix H, then choose the essential matrix E.

[0067] If the success rate difference between the two is within 10%, then compare the average reprojection error: if the homography matrix H error is more than 0.5 pixels smaller than the essential matrix E, choose the homography matrix H; otherwise, if the essential matrix E error is more than 0.5 pixels smaller than the homography matrix H, choose the essential matrix E.

[0068] If the error difference is also within 0.5 pixels, then compare the proportion of interior points with a heavy projection error of less than 3 pixels: if the proportion of interior points of the homography matrix H exceeds 15% of that of the essential matrix E, then choose the homography matrix H; otherwise, choose the essential matrix E.

[0069] After initial selection, a forced correction is performed using the pre-calculated H / E inlime ratio: if the ratio is greater than 0.8, the homography matrix H is forced to be used; if the ratio is less than 0.55, the essential matrix E is forced to be used. Finally, the optimal geometric model, inlime mask, and number of inlimes for each pair of images are determined.

[0070] Step 1.4: Starting with the reference camera, initialize the global rotation of each camera through relative rotation propagation, then iteratively solve the rotation-averaged linear system, updating the global rotation until convergence; specifically including:

[0071] Initialization Phase: Initialize the global rotation matrices of all cameras to the identity matrix, and designate a reference camera (such as the camera in the first image) as the starting point; traverse all relative rotation edges, and for camera i with known global rotation and its adjacent camera j with unknown rotation, use relative rotation... To spread:

[0072] If i is known but j is unknown, then:

[0073] ;

[0074] If j is known but i is unknown, then:

[0075] ;

[0076] Repeat the propagation until all cameras have obtained the initial global rotation;

[0077] Iterative optimization phase: Based on the initialization, multiple iterations are performed; the goal of each iteration is to solve a set of global rotations. This makes the calculated relative rotation and the observed relative rotation... Minimize total error:

[0078] The objective function is:

[0079] ;

[0080] Here, d is the distance metric on SO(3);

[0081] By utilizing the exponential / logarithmic mapping between the SO(3) Lie group and the SO(3) Lie algebra, the nonlinear rotation error is transformed into a linear Lie algebra space for solution.

[0082] ;

[0083] In an ideal error-free scenario, if ,but ,at this time ;

[0084] Therefore, the original optimization objective can be transformed into minimizing the sum of squares of all Lie algebra residuals:

[0085] ;

[0086] Iterative updates are performed using a left perturbation model, introducing a small perturbation δi∈so(3) for each camera, and the updated rotation is... ; SO(3): The current global rotation matrix (Lie group elements) for the i-th camera. so(3): for The applied perturbation (Lie algebra) is a 3x3 matrix, where e is the exponential mapping over the Lie group, representing the perturbation in the Lie algebra space. This maps back to a tiny rotation increment in the Lie group space; For a small perturbation The generated tiny rotation matrix.

[0087] Using the BCH approximation of SO(3) to linearize the residuals, we obtain the linearization constraints for each pair of image pairs:

[0088] ;

[0089] Multiple iterations are performed based on the initialization; in each iteration, the Lie algebraic representation of the relative rotation error of each side is calculated, and a linear system of equations A·δ=b is constructed.

[0090] ;

[0091] In this system, matrix A consists of positive and negative identity matrix blocks corresponding to the camera positions, and vector b is the negative value of the corresponding rotation error vector. The rotation of the reference camera is fixed, and the increment vector δ is solved by column pivot QR decomposition. The increment is restored to the rotation matrix through exponential mapping (Rodriguez formula), and the global rotation of each camera is updated by left multiplication. After the iteration converges, the global rotation is assigned to each camera, and the relative rotation matrix of all image pairs is updated.

[0092] Step 1.5: Based on the global rotation and matching point pairs, estimate the relative translation direction by solving the homogeneous equation, and use triangulation to verify the correct sign, then update the relative translation vector of each image pair; specifically:

[0093] For each pair of overlapping images, calculate the relative rotation based on the global rotation obtained in the previous step. Extract the interior point matches of the image pair, and remove the camera intrinsic parameters from the coordinates of the matched points to obtain normalized coordinates;

[0094] Using epipolar geometry constraints:

[0095] ;

[0096] Where the subscript x indicates opposition to this matrix; A three-dimensional displacement vector; t1, t2, t3 are three elements of this vector. ;

[0097] make:

[0098]

[0099] in, It is a three-dimensional vector;

[0100] All interior points are matched and stacked into a homogeneous linear system of equations. Each line Perform singular value decomposition on matrix A, take the right singular vector corresponding to the smallest singular value as the initial estimate of the translation direction, and normalize it to a unit vector.

[0101] To determine the positive and negative directions of the translation, respectively using and- The two hypotheses are triangulated, and the proportion of effective 3D points for each is statistically analyzed. The direction with the higher proportion of effective points is selected as the final translation direction and stored as the relative translation vector for the image pair.

[0102] Step 1.6: Transform all relative translations to the world coordinate system, construct a linear system using orientation and scale constraints, and solve for the initial value of the camera's global position; specifically:

[0103] First, transform each pair of relative translations from the local coordinate system to the world coordinate system; then use the global rotation matrix of the image pair. Transform it into a relative direction vector in the world system. And normalize;

[0104] A linear constraint system is constructed using the 3D world positions of all cameras as variables to be determined; orientation constraint: for each pair of images, the camera position difference vector should be parallel to the relative orientation of the world system, i.e., the antisymmetric matrix of the orientation vector multiplied by the position difference is zero, weighted by the weight w_dir:

[0105] ;

[0106] Where C represents the position coordinates of the camera center in the world coordinate system;

[0107] Scale constraint: For each pair of images, each component of its camera position difference vector is equal to the product of the scale factor and the orientation component, weighted by the weight w_scale.

[0108] ;

[0109] Reference fixed constraints: Select a reference camera and strongly constrain its position to the origin, using high weights to ensure the system is nonsingular; combine the three constraints into a system of equations. The initial global positions of all cameras are obtained by using column pivoting QR decomposition.

[0110] Step 1.7: Using the global position as a variable, construct a nonlinear least squares problem with parallel constraints and orientation consistency constraints based on relative direction observations, and optimize to obtain the camera's global position; specifically:

[0111] Using the camera global position obtained in the previous step as the initial value, a nonlinear least squares problem is constructed for optimization; the relative world system orientation of each pair of overlapping images is used as the observation, and a residual function is constructed:

[0112] ;

[0113] Among them: the first three terms of the residual are directional parallelism (cross product) constraints, which force the position difference vector of the two cameras to be parallel to the observation direction vector; the fourth term of the residual is directional consistency constraints, with weights... It is encouraged that the position difference vector and the observation direction vector are in the same direction to avoid the occurrence of reverse solutions; This is a local constant used to prevent the denominator from being zero and to ensure the stability of numerical calculations.

[0114] The Huber loss function is used to reduce the impact of outliers. The reference camera position is kept fixed, and the Ceres solver is used for optimization to obtain the optimized accurate global camera position.

[0115] Step 1.8: Using all camera poses and sparse 3D points as initial values, perform bundle adjustment and jointly optimize camera parameters to minimize reprojection error; specifically:

[0116] First, the global rotation matrix of each camera... Convert to axis-angle representation; perform Rodrigues decomposition on the rotation matrix to obtain the unit vector of the rotation axis. With rotation angle , forming a three-dimensional vector , , That is, all forms like The space formed by three-dimensional vectors; the axis angle vector and the translation vector Together they constitute the 6-dimensional optimization quantity for each camera. ;

[0117] For each three-dimensional point In its visible first The projection process involves first transforming the 3D points to the camera coordinate system using camera extrinsic parameters, and then projecting them to the pixel plane using intrinsic parameters.

[0118] ;

[0119] ;

[0120] in, For the axis angle The rotation matrix recovered using the Rodriguez formula For camera Internal reference;

[0121] Note this point in the perspective. The actual observed pixel coordinates on are The reprojection residual is then:

[0122] ;

[0123] The objective function is the sum of squared reprojection errors from all visible viewpoints, and a Huber robust kernel function is employed. Reduce the impact of mismatches:

[0124] ;

[0125] The Huber loss function is defined as follows:

[0126] ;

[0127] The pose of the fixed reference camera (such as the first camera) remains unchanged, i.e., its position is constrained. , To eliminate the normed degrees of freedom and ensure the uniqueness of the solution; iterative optimization is performed using the sparse normal Cholesky solver;

[0128] After optimization, the axis-angle vectors of each camera are... The rotation matrix is ​​recovered using Rodriguez's formula. ;

[0129] Then the relative rotation and relative translation of all image pairs were recalculated:

[0130] ;

[0131] ;

[0132] The updated relative pose will be used for downstream 3D surface reconstruction.

[0133] Step 2: For each image, construct a projection matrix using its camera intrinsic parameter matrix and camera absolute pose; traverse each 2D vertex in the image mesh, taking the coordinates of the corresponding 3D vertex as unknowns, expand according to the projection equation to obtain two linear constraints, and add the projection constraints to the sparse linear equation system with preset projection constraint weights; (Refer to...) Figure 4 As shown, specifically:

[0134] For each image, use its camera intrinsic parameter matrix and the absolute pose obtained in step 1 Build Projection matrix:

[0135] ;

[0136] Divide each image into Regular grids (such as) Each grid vertex corresponds to a two-dimensional image coordinate. and a three-dimensional coordinate system to be solved According to the projection equation, two linear constraints can be written for each vertex:

[0137] ;

[0138] ;

[0139] Where P is the projection matrix, used to project points in the world coordinate system onto the image coordinate system; the 3x3 rotation matrix R is placed on the left; the 3x1 translation matrix is ​​placed on the right, and so on, to obtain a 3x4 matrix, which is then multiplied by the intrinsic parameter matrix K to obtain the final projection matrix P; the subscript of P indicates the row and column values ​​(starting from 0). Represents the pixel coordinates in the image coordinate system, where u and v correspond to x and y in the 2D coordinate system, respectively;

[0140] Iterate through all grid vertices of all images, using preset projection constraint weights. The above constraints are added to the overall linear equation system, with the unknowns being the three-dimensional coordinates of each vertex; the physical meaning of the projection constraint is to force the three-dimensional vertices to maintain the same position as the two-dimensional vertices on the original image after being projected by the camera.

[0141] Step 3: Obtain the interior point matching between each pair of overlapping images, interpolate the matching points on their respective 2D meshes to obtain the vertex weights of the corresponding polygons; for each pair of matching points, establish equality constraints on the three components of 3D space, and stipulate that the weighted sum of the 3D coordinates of the relevant vertices of the polygons corresponding to the matching points in any two images is equal. This transformation allows the alignment constraints to directly use the 3D mesh vertices without introducing any intermediate variables, ensuring the linearity and sparsity of the unified model; furthermore, the alignment constraints are added to the equation system with preset alignment weights; see reference. Figure 5 As shown, specifically:

[0142] For each pair of images with overlapping regions, obtain the interior point matching relationship selected in step 1; each matching point pair corresponds to the same point in three-dimensional space, and its two-dimensional position on its respective image is represented by bilinear interpolation through the image grid; the interpolation process determines the polygon where the matching point is located and the interpolation weights of each vertex;

[0143] Let the image The set of vertices of the two-dimensional mesh is The corresponding coordinates of the three-dimensional vertex to be solved are:

[0144] ;

[0145] In one specific embodiment, the interpolation process determines that the matching point is located in a quadrilateral; for an image Matching points on First, determine the quadrilateral of the grid it belongs to (index is...). The four vertices of the quadrilateral are indexed as follows: The interpolation weights of the point at each vertex are calculated using the diagonal area weighting method.

[0146] ;

[0147] in, , Furthermore, the interpolation uses the diagonal vertex mapping order (to ensure correct normalization of area weights); this weight satisfies ;

[0148] Similarly, point In the image The corresponding grid quadrilateral index is The vertex index is The interpolation weights are ;

[0149] For each pair of matching points, establish equality constraints on the X, Y, and Z components respectively: the weighted sum of the three-dimensional coordinates P of the four vertices of the polygon corresponding to the matching point in Image 1 is equal to the weighted sum of the three-dimensional coordinates P of the four vertices of the polygon corresponding to the matching point in Image 2.

[0150] ;

[0151] Such constraints are added to the overall linear equation system with a preset alignment weight w_align, and the right-hand side term is zero; the alignment constraint ensures that the coordinates of the three-dimensional points corresponding to the same image points are consistent under different viewpoints;

[0152] Step 4: Establish local smoothing constraints for each vertex of the 3D surface and its neighboring vertices to maintain surface continuity; the smoothing constraints are added to the equation system with local constraint weights; refer to... Figure 6 As shown, specifically:

[0153] For each image's 2D regular grid, the adjacency relationships of each vertex are pre-constructed; for internal vertices, their neighbors are typically the four adjacent vertices (up, down, left, and right) (or more, depending on the grid topology); for boundary or corner vertices, if the number of effective neighbors is insufficient, they are skipped, and no smoothing constraint is applied; the grid vertices are recorded. The effective neighbor set is Number of neighbors ;

[0154] For the X, Y, and Z components of each valid vertex, the following local smoothing constraint is applied: the coordinate value of the current vertex should be equal to the arithmetic mean of the coordinate values ​​of all its neighboring vertices;

[0155] In the dimension( The constraint equations on the ) can be uniformly written as:

[0156] ;

[0157] In the global system of linear equations, this constraint corresponds to a single line of equations: current vertex The coefficient is Each neighboring vertex The coefficient is The coefficients of non-neighboring vertices are 0; the right-hand side is 0; where, These are preset local constraint weights used to adjust the smoothing intensity;

[0158] For those without effective neighbors ( ) or only one label is The corner vertices of invalid neighbors are skipped directly without adding constraints;

[0159] Step 5: Combine the projection constraints, alignment constraints, and local constraints constructed in steps 2, 3, and 4 into a unified sparse linear equation system. The right-hand side of the projection constraints consists of corresponding non-zero constant terms, while the right-hand side of the alignment and local constraints are both zero. The unknowns are the 3D coordinates of all grid vertices in all images, and the total number of variables is approximately the number of images × the number of grid vertices × 3. The least squares conjugate gradient method is used to efficiently iteratively solve this sparse linear equation system. After convergence, three consecutive components are extracted from the solution vector in image order as the (X, Y, Z) coordinates of each vertex, constructing the corresponding 3D surface mesh for each image. This achieves unified modeling and solving, directly obtaining the optimal surface satisfying all constraints in a single solution, avoiding the accumulated errors caused by multiple steps such as "reconstruction, alignment, and smoothing." Finally, the 3D surface meshes of each image are stitched and merged to output a complete high-precision 3D surface reconstruction result. Compared to complex nonlinear optimization, this is ultimately transformed into an efficient sparse linear equation system solution, achieving an overall improvement in both accuracy and efficiency.

[0160] The embodiments of the present invention have been described in detail above with reference to the examples. However, the present invention is not limited to the above embodiments. For those skilled in the art, after learning the contents described in the present invention, several equivalent changes and substitutions can be made without departing from the principle of the present invention. These equivalent changes and substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A method for sparse viewpoint camera pose estimation and 3D surface reconstruction, characterized in that: The absolute camera pose of each image is estimated from the two-dimensional image sequence, and each image is divided into a grid region; By using bilinear interpolation of two-dimensional meshes, alignment constraints are transformed into weighted linear equations of mesh vertices, and sparse linear equations are constructed in the form of hard geometric relations by combining projection constraints and local constraints. The sparse linear equations are solved using the least squares conjugate gradient method, directly obtaining the optimal surface that satisfies all constraints, and generating the final three-dimensional surface mesh.

2. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 1, characterized in that: Based on a two-dimensional image sequence, all image pairs with overlapping relationships are identified, and the optimal geometric model is adaptively selected for each image pair. Starting with the reference camera, the global rotation of each camera is initialized through relative rotation propagation. Then, the rotation-averaged linear system is solved iteratively to update the global rotation until convergence. Based on global rotation and matching point pairs, the relative translation direction is estimated by solving homogeneous equations, and the correct sign is determined by triangulation verification, and the relative translation vector of each image pair is updated. Transform all relative translations to the world coordinate system, construct a linear system using orientation and scale constraints, and solve for the initial value of the camera's global position; Using the global position as a variable, a nonlinear least squares problem containing parallel constraints and orientation consistency constraints is constructed based on relative direction observations, and the global position of the camera is obtained through optimization. Using all camera poses and sparse 3D points as initial values, we perform bundle adjustment and jointly optimize camera parameters to minimize reprojection error.

3. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 2, characterized in that: For each pair of overlapping images, estimate the essential matrix, fundamental matrix, and homography matrix as candidate geometric models; The essential matrix is ​​decomposed into four candidate relative poses by singular value decomposition, and the pose with the most inliers is selected as the final relative pose by triangulation and positive depth test. Similarly, the homography matrix is ​​normalized and decomposed and optimized. For all image pairs, relative pose recovery and triangulation are performed using the essential matrix and homography matrix as models, respectively. The triangulation success rate and reprojection error are statistically analyzed, and an adaptive geometric model selection strategy is introduced to obtain the optimal geometric model.

4. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 3, characterized in that: The adaptive geometry model selection strategy is as follows: Compare the triangulation success rates of homography matrices and essential matrices. If the triangulation success rate of one matrix is ​​more than 1.1 times that of the other matrix, choose the matrix with the higher triangulation success rate. otherwise; Compare the average reprojection errors of the homography matrix and the essential matrix. If the average reprojection error of either matrix is ​​more than 0.5 pixels smaller than that of the other matrix, select the matrix with the smaller average reprojection error. otherwise; Compare the percentage of interior points in the homography matrix and the essential matrix. If the percentage of interior points in the homography matrix exceeds 15% of that in the essential matrix, choose the homography matrix; otherwise, choose the essential matrix. as well as If the ratio of inliers is greater than 0.8, the homography matrix is ​​forced to be used; if the ratio is less than 0.55, the essential matrix is ​​forced to be used.

5. A sparse viewpoint camera pose estimation and 3D surface reconstruction method according to any one of claims 1-4, characterized in that: In the sparse linear equation system, the geometric relationship between the coordinates of the three-dimensional vertex and the two-dimensional coordinates of its projection is directly constrained by the projection constraint; the weighted sum of the coordinates of the corresponding three-dimensional vertex under different viewpoints is directly constrained by the alignment constraint; and the smooth relationship between the vertex and its neighbors is directly constrained by the local constraint.

6. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 5, characterized in that: In the sparse linear equation system, the right-hand side of the projection constraint is the corresponding non-zero constant term, and the right-hand side of the alignment constraint and the local constraint are both zero; the unknowns are the three-dimensional coordinates of all grid vertices in all images, and the total number of variables is approximately the number of images × the number of grid vertices × 3.

7. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 6, characterized in that: For each image, a projection matrix is ​​constructed using its camera intrinsic parameter matrix and camera absolute pose. Each two-dimensional vertex in the image grid is traversed, and the coordinates of the corresponding three-dimensional vertex of the two-dimensional vertex are taken as unknowns. The projection equation is expanded to obtain two linear constraints. The projection constraints are added to the sparse linear equation system with preset projection constraint weights.

8. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 6, characterized in that: The interior point matching between each pair of overlapping images is obtained, and the matching points are interpolated on their respective two-dimensional grids to obtain the vertex weights of the corresponding polygons. For each pair of matching points, equality constraints are established on the three components of the three-dimensional space, and the weighted sum of the three-dimensional coordinates of the relevant vertices of the polygons corresponding to the matching points in any two images is required to be equal. Alignment constraints are added to the equation system with preset alignment weights.

9. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 6, characterized in that: For each image's grid, establish vertex adjacency relationships, filter valid vertices based on the number of neighboring vertices, apply smoothing constraints to the three components of the valid vertex's 3D space, making the current vertex's coordinates equal to the arithmetic mean of the coordinates of all its neighboring vertices; and add the smoothing constraints to the equation system with local constraint weights.

10. The sparse viewpoint camera pose estimation and 3D surface reconstruction method according to claim 1, characterized in that: The sparse linear equation system is solved using the least squares conjugate gradient method. After convergence, three consecutive components of each group are extracted from the solution vector in the order of the images to form vertex coordinates, and a three-dimensional surface mesh corresponding to each image is constructed. The three-dimensional surface meshes of each image are then stitched together and merged to output a complete three-dimensional surface reconstruction result.

Images