Dynamic 3d reconstruction method under smooth camera motion scenario

Abstract

The method for dynamic three-dimensional reconstruction under smooth camera motion scene based on compressive sensing belongs to the field of three-dimensional reconstruction.The method (1) combines the principle of compressive sensing, and the sparse representation of two-dimensional projection in the discrete cosine transform domain can be used to accurately restore three-dimensional point trajectories.In order to strengthen the coefficient sparsity, the L1 norm constraint is applied to the coefficient as a penalty term, a new reconstruction function is formed, and the dynamic three-dimensional reconstruction accuracy is improved;(2) the new reconstruction function is changed from the original underdetermined equation set to the simplified problem of convex function, and a new threshold adjustment link is designed in the convex optimization algorithm, so that the sparse solution can be quickly and accurately obtained in cooperation with the L1 norm penalty term;(3) by defining the reconstruction accuracy judgment standard, a new cross-validation link is designed, which can adaptively select the optimal trajectory basis vector combination and sparse constraint weight for each reconstruction point, meet the good reconstruction relationship, and realize the optimal reconstruction under the reconstruction function.

Description

Technical Field

[0001] This invention designs a dynamic 3D reconstruction method based on L1 norm constraints and convex optimization, aiming to improve the reconstruction accuracy in smooth camera motion scenes. Background Technology

[0002] With the continuous development of science and technology in recent years, cutting-edge technologies such as autonomous driving and virtual reality are increasingly becoming part of people's lives, and computer vision technology is highly integrated with reality. Among these, dynamic 3D reconstruction plays an important role in cutting-edge technologies, and it may be found in pedestrian avoidance systems for autonomous vehicles, augmented reality imaging, and trajectory reconstruction for athletes' scientific training. Technological development and future life have many applications that require accurate 3D reconstruction, and the complexity and multiple solutions to this problem make it a challenging and hot topic in the field of computer vision.

[0003] The reconstruction method based on compressed sensing has the widest application scenarios and the most research results. It applies the signal decompression process of compressed sensing to dynamic 3D reconstruction (Akhter I, Sheikh Y, Khan S, et al. Nonrigid structure from motion in trajectory space[J]. Advances in neural information processing systems,2008,21.). It demonstrates that trajectory basis has advantages such as strong independence and small computational cost compared with shape basis, and defines discrete cosine transform basis for reconstruction, pioneering this type of method. (Park HS, Shiratori T, Matthews I, et al. 3D trajectory reconstruction under perspective projection[J]. International Journal of Computer Vision,2015,115(2):115-135.). On this basis, the number of trajectory basis is adaptively selected through iterative training, and the reconstructability is calculated through 2D reprojection error, but it only achieves the improvement of reconstruction accuracy in complex camera motion scenes. (Zhu Y, Cox M, Lucey S. 3D motion reconstruction for real-world camera motion[C] / / CVPR 2011.IEEE,2011:1-8.) An improvement was made to address this deficiency by incorporating some ground truth values ​​into the reconstruction process and then applying a type-1 norm constraint to the trajectory basis coefficients to ensure their sparsity. However, the feasibility of incorporating costly ground truth values ​​is limited under real-world conditions. This type of method improves reconstruction results by applying reasonable constraints and defining new bases, but its difficulty lies in the fact that the infinity of solutions and the complex camera motion required for accurate reconstruction contradict reality. This is a prerequisite for compressed sensing to achieve signal recovery and also limits the performance of this type of method.

[0004] To address the challenges of reconstruction based on compressed sensing methods and achieve accurate reconstruction in real-world scenarios with smooth camera motion, this paper proposes a new reconstruction function. This function employs L1 norm constraints on the solution space and utilizes a convex optimization algorithm for the first time to find sparse solutions. Furthermore, by defining a new criterion for reconstruction accuracy and designing an adaptive cross-validation mechanism to select the optimal sparse weights, the paper ensures that all variables satisfy the optimal reconstruction relationship. Summary of the Invention

[0005] This invention, based on traditional compressed sensing methods, designs a dynamic 3D reconstruction method for smooth camera motion scenarios. This overcomes the inherent limitation of traditional methods, which only achieve high accuracy under complex camera motion, and improves the accuracy of dynamic 3D reconstruction in the most common real-world scenario (smooth camera motion), making the reconstruction practically meaningful. Specifically, firstly, an L1 norm penalty term is added to the original objective function to constrain the solution space, making the 2D projection as sparse as possible in the discrete cosine transform domain, thus more accurately recovering the 3D points. Since the L1 norm constraint transforms the objective function from a nonlinear function to a convex function, a threshold adjustment step is designed in the convex optimization algorithm to solve the objective function. A new iterative step is created, utilizing a newly defined 2D reprojection error to adaptively select the number of basis vectors and coefficient weights. For each point, the optimal combination of basis vectors and sparse weights are selected to obtain the sparse optimal solution of the 2D projection in the transform domain, thereby achieving the most accurate recovery of the 3D points.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A dynamic 3D reconstruction method based on L1 norm and convex optimization, the method comprising:

[0008] Step 1: Constrain the solution space of the objective function using the L1 norm to construct a new objective function;

[0009] Step 2: Design a new threshold adjustment step in the fast convex optimization algorithm and use it to solve the new objective function;

[0010] Step 3: Define 2D reprojection error as the criterion for determining reconstruction accuracy;

[0011] Step 4: Design an iterative process to adaptively select the optimal combination of basis vectors and coefficient weights;

[0012] The dynamic 3D reconstruction method specifically includes the following steps:

[0013] Step 1): Construct a new objective function by using L1 norm constraints and the solution space of the objective function;

[0014] Reconstruction is performed point-by-point. For the i-th time sample in the time series, the camera projection matrix is ​​known. and two-dimensional projected coordinates x i =[x i y i ] T Solve for the true value X of a 3D point. i =[X i Y i Z i ] TThe two-dimensional projection is the result of projecting the three-dimensional ground truth onto a two-dimensional plane through the camera projection matrix, as shown below:

[0015]

[0016] in[·] × This is the antisymmetric representation of the cross product, which is written here as a nonhomogeneous equation.

[0017]

[0018] Where P i，1∶3 and P i，4 Let the first three and fourth columns of the camera projection matrix for the i-th frame be represented, respectively. Further simplification of the equations leads to the basic structure of a dynamic 3D reconstruction problem:

[0019]

[0020]

[0021] camera matrix A new two-dimensional projection with a rank of 2. The rank is 1, and these are the first two rows of the cross product. Each frame transform satisfies Q. i X i =q i , is an underconstrained system. The orthogonal matrix of each frame of the simulated camera is arranged diagonally to form the camera matrix space Q. The true value of the three-dimensional point X and the new two-dimensional projection q are arranged in a column. F is the sequence length. The underconstrained system of each three-dimensional point reconstruction sequence is shown in (3).

[0022] For equation system (3), the augmented matrix is Its rank is 2F, and it has infinitely many solutions. To recover the accurate 3D point trajectory, the solution space of the 3D point X to be recovered is constrained by fitting the point trajectory using a linear combination of trajectory basis vectors.

[0023]

[0024] in These are the trajectory basis vectors. The matrix formed by the trajectory basis. Let K be the trajectory basis coefficient vector, which is also the solution (sparse solution) of the equation. K is the number of trajectory basis vectors in each dimension. The 3D point trajectory is recovered by constructing the following trilinear system.

[0025] QΘβ=q. (6)

[0026] This allows us to derive the complete reconstruction equation.

[0027]

[0028] Using the predefined trajectory basis vector Θ DCT In the discrete cosine transform domain, equation (7) is reduced to a bilinear form.

[0029]

[0030] By adding a penalty term, the solution space is constrained using the L0 norm. The L0 norm is used to calculate the number of non-zero elements in β, i.e., to find the sparsest solution β. λ is the weight of the additional term (L0 norm constraint weight, sparsity weight), which measures the percentage of the degree of L0 norm constraint.

[0031]

[0032] The L1 norm and the L0 norm are equivalent. In the field of computer science, there are many proofs and applications using the L1 norm to constrain sparsity. Since equation (9) is an NP-hard problem, replacing the L0 norm with the L1 norm makes equation (9) solvable. λ is the L1 norm constraint weight.

[0033]

[0034] Step 2): Design a new threshold adjustment step using the fast convex optimization algorithm, and use it to solve the new objective function:

[0035] In this invention, equation (10) includes the L1 norm constraint on β and the reconstruction fitting function, which meets the solution objective of the iterative shrinkage threshold algorithm. The specific analysis is as follows:

[0036] min F(β),where F(β)=||QΘ DCT β-q|| 2 +λ||β||1 (11)

[0037] ||QΘ DCT β-q|| 2 If the Lipschitz gradient is continuous, then its derivative is Lipschitz continuous.

[0038]

[0039] f(β) = ||QΘ DcT β-q|| 2 (13)

[0040] L(f) is the Lipsitz constant, which is the gradient of the fitted function. The smallest positive number that satisfies equation (12) is then Satisfying the Lipsitz continuity condition, β x β yThese are any two solutions that satisfy equation (10). Equation (12) shows that the local variation of the derivative of the fitted function f(β) cannot exceed L(f). As mentioned above, the function f(β) satisfies the Lipsitz gradient continuity.

[0041]

[0042] Removing the absolute value, we obtain the maximum value of f(β).

[0043]

[0044] By iteratively selecting sparse solutions β that make the function value of equation (10) smaller, the solution β is gradually selected. k Until the sparse solution β corresponding to the minimum function value is obtained, β k For the sparse solution of the current iteration, β k-1 This is the sparse solution from the previous iteration.

[0045]

[0046] t k Let t be the step size, when t k Pick sequence beta k The convergence rate is sublinear. The sparse solution β in equation (17) k By minimizing f(β) x The conclusion is:

[0047]

[0048] Since this is a minimization problem, the constant term is ignored.

[0049]

[0050] Add the additional term to equation (10),

[0051]

[0052] As an algorithm for solving the objective function.

[0053] Simplify equation (19),

[0054]

[0055]

[0056] Where c k It is the operator for selecting the new sparse solution β in gradient descent, 2(QΘ) DCT ) T (QΘ DCT β k-1-q) is the gradient of the fitting function in the (k-1)th iteration. Since the L1 norm and L2 norm in equation (21) are both sums of several non-negative terms, the sparse solution β in the k-th iteration is... k This can be calculated using the following k-fold contraction steps.

[0057]

[0058] γ α (c)| k =max(|c k |-α,0)sgn(c k ) (twenty three)

[0059] in The soft thresholding operation function is a shrinkage operator, γ. α (c)| k This is the sparse solution β obtained in the k-th iteration under the k-th contraction step. k The shrinkage (soft threshold) operation function, c k It is the gradient descent operator, |c k | is the absolute value of the operator, sgn(c k ) is the return gradient descent operator c k The function returns the sign of the symbol (positive, negative, or 0). As the threshold of the gradient descent operator and c k The difference is determined by c. k Can it be used to calculate a new β? k This invention amplifies the threshold α by a factor of ten, i.e., α = α × 10 n Based on the order-of-magnitude difference between the camera matrix Q and the two-dimensional projection q in different sequences, the ||QΘ in equation (10) is maintained. DcT β-q|| 2 The ratio of ||β||1 is between 1:1 and 10:1. The new threshold α can make the bandwidth coverage of the reconstruction equation (10) reach the global minimum when the coefficient weight λ ranges from 0 to 1, thus achieving the optimal reconstruction under this reconstruction condition. In each iteration, β is updated through a shrinkage / soft thresholding operation. k By iteratively calculating equations (21), (22), and (23), the sparse solution β that minimizes the error of equation (20) is obtained.

[0060] Step 3): Define 2D reprojection error as the criterion for determining reconstruction accuracy:

[0061]

[0062] λ k The sparse weight is one case of the additional weight λ in equation (10), and step 4 gives λ(λ kThe range of values ​​for ). Where, e j (K i , λ k ) represents the number of basis vectors K. i Coefficient weight λ k The two-dimensional reprojection error under x s y s Let P be the coordinates of the true two-dimensional projection, and P be the camera projection matrix. The rows of the camera projection matrix in the test set... Let K be the number of corresponding basis vectors. i And coefficient weights λ k The reconstructed 3D points of the test set are the number of corresponding basis vectors K. i and sparse weights λ k Below is the trajectory basis matrix The training set coefficients obtained in steps 1 and 2 The reconstructed 3D points of the test set S (as described in step 1) are used with the corresponding camera matrix P. s Project onto a two-dimensional plane and normalize: and The difference between the actual value and the corresponding two-dimensional projection is used as a criterion for judging the reconstruction accuracy when the three-dimensional true value is lacking.

[0063] Step 4): Design an iterative process to adaptively select the optimal combination of basis vectors and coefficient weights.

[0064]

[0065] In each basis vector number K i Next, an iterative step is added, which minimizes the two-dimensional reprojection error e through the outer iteration. j Choose an optimal sparse constraint weight λ for each point k . The minimum two-dimensional reprojection error e of the test set over multiple iterations i The sum of the corresponding basis vectors and sparse weights. Equation (25) also explains K. i and λ k The range of values ​​for .

[0066] This invention proposes a dynamic 3D reconstruction method based on L1 norm constraints and convex optimization, overcoming the shortcomings of traditional methods. First, a new objective function is proposed by imposing sparsity constraints on the solution space. Second, the new objective function is transformed from an underdetermined system of equations into a convex optimization problem. A new step is designed within the convex optimization algorithm to solve the new objective function. Finally, to achieve optimal reconstruction, a 2D reprojection error is defined, and an iterative step is designed to select the optimal combination of basis vectors and coefficient weights. This method overcomes the shortcomings of traditional methods in reconstructing accuracy in real-world scenes with smooth camera motion, improving overall reconstruction accuracy and enabling the reconstruction of realistic scenes. Attached Figure Description

[0067] Figure 1 This is a flowchart of dynamic 3D reconstruction based on L1 norm constraints and convex optimization provided by the present invention;

[0068] Figure 2 This is a reconstruction effect diagram of dynamic 3D reconstruction based on L1 norm constraints and convex optimization provided by the present invention;

[0069] Figure 3 The figure shows the experimental results of dynamic three-dimensional reconstruction based on L1 norm constraints and convex optimization provided by this invention.

[0070] Above: Comparison of 3D reconstruction errors in synthetic dataset experiments (with ground truth 3D points).

[0071] Below: Comparison of 2D reprojection errors in real dataset experiments (without 3D point ground truth). Detailed Implementation

[0072] The purpose of this invention is to overcome the shortcomings of traditional methods in reconstructing accuracy in scenes with smooth camera motion, and to achieve more accurate reconstruction. A penalty term with the L1 norm is added to constrain the solution space of the objective function; an improved fast convex optimization algorithm is used to solve the objective function; a two-dimensional reprojection error is defined; and an iterative process is designed to adaptively select the optimal combination of basis vectors and coefficient weights, thus achieving more accurate reconstruction.

[0073] The present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the described embodiments are intended only to facilitate the understanding of the present invention and are not intended to limit it in any way.

[0074] Figure 1 This is a flowchart of dynamic 3D reconstruction based on L1 norm constraints and convex optimization provided by the present invention; Figure 2 This is a reconstruction effect diagram of the dynamic three-dimensional reconstruction based on L1 norm constraints and convex optimization provided by the present invention. It can be seen that the present invention can accurately reconstruct the human motion sequence. Figure 3The figure shows the experimental results of dynamic 3D reconstruction based on L1 norm constraints and convex optimization provided by the present invention. It can be seen that the present invention has greatly improved the reconstruction accuracy compared with traditional methods.

[0075] The dynamic 3D reconstruction method based on L1 norm constraints and convex optimization provided by this invention specifically includes:

[0076] Step 1: Add an L1 norm penalty term to construct a new objective function;

[0077] To overcome the inherent limitation of traditional methods in achieving poor reconstruction accuracy in scenes with smooth camera motion, this invention analyzes the solution space of the reconstruction function. Based on one of the prerequisites of compressed sensing—sparseness—in dynamic 3D reconstruction tasks, 2D projection must be sparsely represented in the discrete cosine transform domain to achieve accurate 3D point trajectory reconstruction. Therefore, a penalty term with the L1 norm is chosen as a sparsity constraint to constrain the solution space, searching for the sparse solution that achieves optimal reconstruction from infinitely many solutions.

[0078] In the implementation of this invention, the basic idea of ​​reconstruction was first determined: trajectory reconstruction based on each point. All two-dimensional projections are grouped by point, with only weak temporal constraints between different times for the same point, breaking the strong physical constraints. A single reconstruction completes the position of a point across all time periods, which is a trajectory-based reconstruction method. Because the temporal constraints between different times for each point are low, a wider range of basis vectors can be selected to represent the trajectory of each point, reducing computational complexity while maintaining strong generalization ability.

[0079] Step 2: Improve the convex optimization algorithm to solve the objective function;

[0080] This invention improves upon existing iterative threshold shrinkage algorithms to better suit dynamic 3D reconstruction scenarios. Specifically, it addresses the issue of insufficient bandwidth for selecting independent variables due to the difference in absolute values ​​between the fitted objective function and the additional terms. Through continuous experimentation and summarizing the mathematical relationship between the fitted function and the additional terms, a suitable range for the threshold α is chosen: 0.001 ≤ α ≤ 0.1. This ensures that the absolute values ​​of the fitted function and the additional terms are on the same order of magnitude, facilitating the setting of appropriate coefficient weight ranges in subsequent steps. This allows the function values ​​to cover a wider bandwidth, thereby selecting the optimal coefficient weights for the most accurate reconstruction.

[0081] Step 3: Define the two-dimensional reprojection error;

[0082] In the implementation of this invention, to achieve optimal reconstruction, iterative verification is employed to select the most suitable parameters to construct the best reconstruction conditions. After solving for the coefficients of the training set, they are substituted into the test set for verification. The two-dimensional reprojection error is defined as the criterion for judging reconstruction accuracy, so as to select the optimal combination of basis vectors and coefficient weights, thereby achieving optimal reconstruction.

[0083] Step 4: Design an iterative process to adaptively select the optimal combination of basis vectors and coefficient weights.

[0084] To achieve the optimal reconstruction under this objective function, we perform a reconstruction once by selecting each basis vector combination and coefficient weight, calculate the corresponding two-dimensional reprojection error, and select the basis vector combination and coefficient weight corresponding to the minimum error for a final reconstruction, which is the optimal reconstruction.

[0085] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications and substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dynamic 3D reconstruction in scenes with smooth camera motion, characterized in that, Includes the following steps: Step 1: Add a penalty term with the L1 norm A new objective function is constructed. In many real datasets and real-world scenarios, in order to obtain a good shooting perspective, the camera mainly uses smooth motion, and the camera motion law and the 3D point motion law have a non-linear relationship. Step 2: Design a new threshold adjustment step in the fast convex optimization algorithm - iterative shrinkage threshold method, so that it can solve the objective function and accurately realize dynamic 3D reconstruction; Step 3: Define 2D reprojection error as a new criterion for determining reconstruction accuracy in order to construct the optimal reconstruction relationship and achieve optimal 3D reconstruction; Step 4: Design a cross-validation iteration process. Combining the two-dimensional reprojection error from Step 3, select the optimal combination of trajectory basis vectors and coefficient weights to achieve the optimal reconstruction result. The specific calculation steps for step 1 are as follows: Using the L1 norm to constrain the solution of the reconstructed equation in the form of additional terms We construct a new objective function from the solution space and solve it in the discrete cosine transform domain: (1)； Among them, camera matrix Predefined discrete cosine transform trajectory basis Two-dimensional projection Given quantities, solve for the coefficients. To achieve the reconstruction of three-dimensional points , The L1 norm constraint weights are the percentages that measure the degree of L1 norm constraint. The specific calculation steps for step 2 are as follows: The solution process for equation (1) using the fast convex optimization algorithm is as follows: (2)； (3)； Equations (2) and (3) are the convex optimization solution process of equation (1), the purpose of which is to find the sparsest solution of equation (1). ; Let be the independent variable for the k-th iteration; The step size for the k-th iteration will be dynamically adjusted based on the result of the (k-1)-th iteration. Select a new gradient descent The operator is calculated as shown in equation (3). The fitting function of equation (1) The gradient in the (k-1)th iteration; Because of the additional items L1 norm and fitting function The L2 norm is the sum of several non-negative terms, and the optimal coefficient is... Calculated through k contraction steps, i.e. , (4)； (5)； This is the process of finding the coefficients of the k-th iteration under k contraction steps. The contraction operation function, It is the gradient descent operator. It is the absolute value of the operator. It returns the gradient descent operator. The sign of the symbol returns a function; threshold By expanding the threshold Let n be any positive integer of order of magnitude, such that in equation (2)... and The ratio of values ​​is between 10:1 and 1:1, that is... ; set the threshold And gradient descent to select new operators The absolute value is used to determine the difference. Can it be used to calculate new, better solutions? Equations (4) and (5) provide solutions for equations (2) and (3), and adjust the threshold. Make the bandwidth cover the sparsest solution The shrinkage / soft thresholding operation performed in each iteration Updated and optimized coefficients When the number of iterations k reaches its maximum or the coefficients calculated in two iterations... and When they are the same, they represent the optimal coefficients for that reconstruction. .

2. The dynamic 3D reconstruction method for smooth camera motion scenes according to claim 1, characterized in that, The specific calculation steps for step 3 are as follows: (6)； The sparse weights are the additional weights in equation (1). One of the situations, ;in, , Given the coordinates of the true value of the two-dimensional projection, , , Let the first, second, and third rows of the known camera projection matrix P in the test set S be... The number of corresponding basis vectors and L1 norm constraint weights Below is the trajectory basis matrix The training set coefficients obtained in steps 1 and 2 The reconstructed test set S contains 3D points used to determine the reconstruction quality; these points are then passed through rows of the corresponding camera projection matrix. , , Reprojected onto a two-dimensional plane: , , And perform normalization: For reconstruction point x in a two-dimensional projection For reconstruction point The y-value of the two-dimensional projection is compared with the known true value of the two-dimensional projection. , Find the absolute distance and divide it by the known true value of the 2D projection. , The distance to the origin is the number of corresponding basis vectors. L1 norm constraint weights Two-dimensional reprojection error Equation (6) is the reconstructed three-dimensional point of Equation (1). An indirect determination of reconstruction accuracy is defined, that is, the points to be reconstructed. Through camera matrix Projected onto a two-dimensional plane, and the two-dimensional truth value , Make a comparison.

3. The dynamic 3D reconstruction method for smooth camera motion scenes according to claim 1, characterized in that, Step 4 involves the following specific calculation steps: For each basis vector... Next, an iterative step is added to minimize the two-dimensional reprojection error through external iteration. The sum of the weights is used to select an optimal L1 norm constraint for each point. , (7)； Equation (7) is a specific application of Equation (6) in the reconstruction process of Equation (1), constructing the optimal reconstruction environment for Equation (1); among which, , For multiple iterations Minimum reprojection error of the test set and The corresponding number of basis vectors and L1 norm constraint weights; for each basis vector... L1 norm constraint weights The next reconstruction is completed, and the two-dimensional reprojection error is obtained through equations (6) and (7). The total number of iterations in the process of evaluating whether it is the optimal reconstruction is determined by... , The number of times can be taken is determined, as shown in equation (7). To reconstruct the sequence length, and to make equation (1) solvable, i.e., the number of equations must be greater than the number of unknowns: The number of basis vectors is obtained. The range of values ​​for , As L1 norm constraint weights, they represent the strength of the constraint effect, ranging from 0.01 to 1; to obtain the optimal number of basis vectors. L1 norm constraint weights The total number of iterations is .

Images