Method and system for estimating multi-view camera displacement and three-dimensional scene point coordinates

Abstract

The present invention provides a method and system for estimating the displacement of a multi-view camera and the coordinates of a three-dimensional scene point, including a three-dimensional scene abnormal point identification step and a global displacement and scene recovery step; the three-dimensional scene abnormal point identification step identifies and removes abnormalities for all three-dimensional scene points; the global displacement and scene recovery step constructs LSGT constraints based on image observations, uses Schur decomposition to solve global displacement and three-dimensional scene points, and screens the correct solution. The present invention is applicable to various camera motion conditions, has simple initialization, high robustness, fast calculation speed, and high reconstruction accuracy. The method can be used as a core computing engine for applications such as visual three-dimensional reconstruction, unmanned vehicle / unmanned aerial vehicle visual navigation, and augmented reality.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and photogrammetry, and in particular to a method and system for estimating multi-view camera displacement and three-dimensional scene point coordinates, and in particular to a method for fast estimating multi-view camera displacement and three-dimensional scene point coordinates. Background Art

[0002] Bundle Adjustment (BA) has always been the core part of multi-view geometry solution. BA optimization relies on good global initialization, which is mainly divided into the initialization of global pose, global displacement and 3D scene point coordinates. The main difficulty lies in the initialization of global displacement. When the pose is known, the global displacement and 3D scene point coordinates can be solved by constructing bilinear homogeneous equations (such as singular value decomposition SVD), but it faces the high-dimensional disaster of the solution vector space. For example, for the case of M 3D scene points and N images (usually M is much larger than N), the solution vector dimension is as high as 3M+3N. In addition, the solution of bilinear homogeneous equations is also easily affected by abnormal points in the 3D scene, resulting in abnormal solutions.

[0003] A patent document with publication number CN111161355A discloses a method and system for solving pure pose of multi-view camera pose and scene, including: a pure rotation anomaly identification step: performing pure rotation anomaly identification on all views, marking views with pure rotation anomalies, and obtaining marked views and unmarked views; a global displacement linear calculation step: selecting one unmarked view as a reference view, constructing a constraint tr=0, constructing a global displacement linear constraint, solving the global displacement according to tr and restoring the global displacement of the marked view, and screening the correct solution of the global displacement; a three-dimensional scene analysis and recovery step: analyzing and recovering the coordinates of all 3D points according to the correct solution of the global pose.

[0004] A patent document with publication number CN109357621A discloses a three-dimensional vibration displacement measurement device and method based on a linear array camera and position-sensing stripes. The device includes position-sensing stripes, a linear array camera, a data line and a computer. The method includes the following steps: pasting patch-type position-sensing stripes or spraying position-sensing stripes at the measurement position of the vibration structure, adjusting the imaging position of the linear array camera, so that the center of the position-sensing stripes is imaged in the linear array sensor; the position-sensing stripes on the surface of the structure vibrate with the structure, and the stripe signals imaged in the linear array sensor also change accordingly, and the linear array camera is used to continuously image the stripe signals during the vibration process, and then the collected stripe sequence is transmitted to the computer through the data line; finally, the stripe sequence is processed by the stripe sequence processing software module, and the three-dimensional vibration displacement information of the structure is calculated using the proposed three-dimensional displacement calculation formula.

[0005] A patent document with publication number CN108398091A discloses a method for accurate visual measurement of the three-dimensional displacement of a blade tip. The steps are as follows: affix a marker circle to the blade tip to be measured, set up a three-dimensional measurement device for a visual sensor, calibrate the parameters inside a single camera, convert all camera coordinate systems to the global coordinate system with the help of a laser tracker, use various image algorithms to detect the sub-pixel image coordinates of the center point of the marker circle on the blade tip, and finally obtain the three-dimensional coordinates of the center point of the blade tip marker circle in the global coordinate system before and after static loading by using the principle of stereoscopic vision reconstruction, and calculate the displacement of each dimension of the blade tip using the three-dimensional coordinate values ​​before and after loading.

[0006] With respect to the above-mentioned related technologies, the inventor believes that the current mainstream global displacement estimation algorithm uses the relative displacement of two views as input and optimizes the estimation of global displacement by minimizing the algebraic error. Its disadvantages are that it loses image information and global displacement estimation accuracy, and estimation anomalies occur in cases of pure camera rotation and collinear motion. Therefore, it is necessary to propose a technical solution to improve the above-mentioned technical problems. Summary of the invention

[0007] In view of the defects in the prior art, an object of the present invention is to provide a method and system for estimating multi-view camera displacement and three-dimensional scene point coordinates.

[0008] A method for estimating multi-view camera displacement and three-dimensional scene point coordinates provided by the present invention includes:

[0009] 3D scene outlier point identification step: observe 3D scene points through multiple views, identify and eliminate outliers for all 3D scene points;

[0010] Global displacement and scene restoration steps: Construct LSGT constraints based on image observations, use Schur decomposition to solve global displacement and 3D scene points, and screen the correct solution.

[0011] Preferably, the step of identifying abnormal points in the dimensional scene includes the following steps:

[0012] Step S1: Consider a 3D scene point P, which is observed by n views; select two views i and j from them and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively;

[0013] Step S2: Statistic θ i,jThe maximum value is denoted as

[0014] Step S3: Given a threshold δ, if Mark the 3D scene point as an abnormal point and remove it;

[0015] Step S4: Repeat steps S1 to S3 for all three-dimensional scene points.

[0016] Preferably, in step S1, θ is calculated i,j =||[X j ] × R i,j X i ||, and select the maximum value in S2 As a statistical indicator of abnormal 3D points.

[0017] Preferably, the global displacement and scene restoration step comprises the following steps:

[0018] Step 1: Assume that there are M 3D scene points observed by N cameras. For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [ ] × represents the cross product matrix, t i represents the global displacement of view i, I3 represents the 3D identity matrix;

[0019] Step 2: For all views and 3D scene points, construct The bilinear constraints are and

[0020] Step 3: Multiply the constraints in the previous step by matrix A. T , A T The LSGT constraint is obtained after the A matrix is ​​divided into blocks

[0021] Step 4: Use Schur decomposition to estimate global displacement and 3D scene points, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, using P = - (B3) -1 B4t restores 3D scene points;

[0022] Step 5: Based on the global displacement obtained and 3D scene points pass Filter the correct solution, where

[0023] Preferably, in step 3, the constraint in step 2 is multiplied by A on the left. T , obtain the LSGT constraint, and solve it using Schur decomposition in step 4.

[0024] The present invention also provides a multi-view camera displacement and three-dimensional scene point coordinate estimation system, comprising:

[0025] 3D scene outlier point recognition module: observes 3D scene points through multiple views, identifies and removes outliers from all 3D scene points;

[0026] Global displacement and scene restoration module: construct LSGT constraints based on image observations, use Schur decomposition to solve global displacement and three-dimensional scene points, and screen the correct solution.

[0027] Preferably, the dimensional scene outlier identification module includes the following modules:

[0028] Module M1: Consider a 3D scene point P, which is observed by n views; select any two views i and j from them and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively;

[0029] Module M2: Statistics θ i,j The maximum value is denoted as

[0030] Module M3: Given a threshold δ, if Mark the 3D scene point as an abnormal point and remove it;

[0031] Module M4: Repeat modules M1 to M3 for all 3D scene points.

[0032] Preferably, the module M1 calculates θ i,j =||[X j ] × R i,j X i ||, and select the maximum value in S2 As a statistical indicator of abnormal 3D points.

[0033] Preferably, the global displacement and scene restoration module includes the following modules:

[0034] Module 1: Assume that there are M 3D scene points observed by N cameras. For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [ ] × represents the cross product matrix, t i represents the global displacement of view i, I3 represents the 3D identity matrix;

[0035] Module 2: For all views and 3D scene points, construct The bilinear constraints are and

[0036] Module 3: Multiply the constraints in the previous step by matrix A T , A T The LSGT constraint is obtained after the A matrix is ​​divided into blocks

[0037] Module 4: Estimating global displacement and 3D scene points using Schur decomposition, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, using P = - (B3) -1 B4t restores 3D scene points;

[0038] Module 5: Based on the global displacement obtained and 3D scene points pass Filter the correct solution, where

[0039] Preferably, in module 3, the constraint in module 2 is multiplied by A on the left. T , obtain the LSGT constraints, and solve them using Schur decomposition in module 4.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention is applicable to various camera motion conditions, has simple initialization, high robustness, fast calculation speed, and high reconstruction accuracy for camera displacement and three-dimensional scene point coordinate estimation method;

[0042] 2. The present invention can be used as the core computing engine for applications such as visual 3D reconstruction, unmanned vehicle / unmanned aerial vehicle visual navigation, and augmented reality;

[0043] 3. The present invention can provide good initial values ​​for visual backend optimization and provide a stable and fast initialization process for the visual optimization algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0045] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0047] The present invention aims to use Schur decomposition to propose a new multi-view camera displacement and three-dimensional scene point coordinate estimation method and system, which greatly improves the robustness and calculation speed of global displacement and three-dimensional scene point estimation. The global displacement and three-dimensional scene point coordinate estimation method proposed in the present invention includes: three-dimensional scene point anomaly recognition, linear Schur-based global translation and 3D structure estimation (LSGT) based on Schur decomposition. Compared with the traditional global displacement solution algorithm, the LSGT algorithm starts from the reprojection equation and solves it by constructing equivalent bilinear constraints. It is almost applicable to any motion situation, effectively avoiding the anomaly problems caused by pure rotation and collinear motion of the camera, and can provide robust and high-precision global displacement and three-dimensional scene point coordinate initial values ​​for BA optimization. The present invention considers the situation where the camera has been calibrated, and assumes that the matching point pairs on multiple views have been determined, and the global posture of each view is known.

[0048] The present invention provides a method for estimating multi-view camera displacement and three-dimensional scene point coordinates, comprising a three-dimensional scene abnormal point identification step and a global displacement and scene restoration step; the three-dimensional scene abnormal point identification step identifies and removes abnormalities for all three-dimensional scene points; the global displacement and scene restoration step constructs LSGT constraints based on image observations, uses Schur decomposition to quickly solve global displacement and three-dimensional scene points, and selects the correct solution.

[0049] The three-dimensional scene outlier identification step includes the following steps:

[0050] Step S1: Consider a 3D scene point P, assuming that the point is observed by n views. Select any two views i and j, and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively.

[0051] Step S2: Statistic θ i,j The maximum value is denoted as

[0052] Step S3: Given a threshold δ, if The 3D scene point is marked as an abnormal point and removed.

[0053] Step S4: Repeat steps S1 to S3 for all three-dimensional scene points.

[0054] Considering N views and M 3D scene points, the global displacement and scene restoration steps include the following steps:

[0055] Step 1: For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [ ] × represents the cross product matrix, t i represents the global displacement of view i, and I3 represents the 3D identity matrix.

[0056] Step 2: For all views and 3D scene points, a structure of The bilinear constraints are and

[0057] Step 3: Multiply the constraint by A T And order You can get the LSGT constraint

[0058] Step 4: Use Schur decomposition to estimate global displacement and 3D scene points, that is, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, and then use P = - (B3) -1 B4t restores 3D scene points.

[0059] Step 5: Based on the global displacement that has been solved and 3D scene points (positive and negative signs are ambiguous), through Filter the correct solution, where

[0060] The present invention also provides a multi-view camera displacement and three-dimensional scene point coordinate estimation system, including: a three-dimensional scene abnormal point recognition module: observing three-dimensional scene points through multiple views, identifying and eliminating abnormalities for all three-dimensional scene points; a global displacement and scene recovery module: constructing LSGT constraints based on image observations, using Schur decomposition to solve global displacement and three-dimensional scene points, and screening the correct solution.

[0061] The dimensional scene anomaly recognition module includes the following modules:

[0062] Module M1: Consider a 3D scene point P, which is observed by n views; select any two views i and j from them and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively; calculate θ i,j =||[X j ] × R i,j X i ||, and select the maximum value in S2 As a statistical indicator of abnormal 3D points.

[0063] Module M2: Statistics θ i,j The maximum value is denoted as Module M3: Given a threshold δ, if Mark the 3D scene point as an abnormal point and remove it; Module M4: Repeat Module M1-Module M3 for all 3D scene points.

[0064] The global displacement and scene restoration module includes the following modules:

[0065] Module 1: Assume that there are M 3D scene points observed by N cameras. For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [ ] × represents the cross product matrix, t i represents the global displacement of view i, and I3 represents the 3D identity matrix.

[0066] Module 2: For all views and 3D scene points, construct The bilinear constraints are and

[0067] Module 3: Multiply the constraints in the previous step by matrix A T , A T The LSGT constraint is obtained after the A matrix is ​​divided into blocks

[0068] Module 4: Estimating global displacement and 3D scene points using Schur decomposition, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, using P = - (B3) -1 B4t restores 3D scene points.

[0069] Module 5: Based on the global displacement obtained and 3D scene points pass Filter the correct solution, where

[0070] In module 3, the constraints in module 2 are multiplied by A on the left T , obtain the LSGT constraints, and solve them using Schur decomposition in module 4.

[0071] The present invention is applicable to various camera motion conditions, has simple initialization, high robustness, fast calculation speed and high reconstruction accuracy, and is a camera displacement and 3D scene point coordinate estimation method; it can be used as a core computing engine for applications such as visual 3D reconstruction, unmanned vehicle / unmanned aerial vehicle visual navigation, and augmented reality; it can provide good initial values ​​for visual back-end optimization and provide a stable and fast algorithmic basis for visual optimization algorithms.

[0072] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0073] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for estimating multi-view camera displacement and three-dimensional scene point coordinates, characterized in that: include: 3D scene outlier point identification step: observe 3D scene points through multiple views, identify and eliminate outliers for all 3D scene points; Global displacement and scene restoration steps: construct LSGT constraints based on image observations, use Schur decomposition to solve global displacement and 3D scene points, and select the correct solution; The global displacement and scene restoration steps include the following steps: Step 1: Assume that there are M 3D scene points observed by N cameras. For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [] × represents the cross product matrix, t i represents the global displacement of view i, I3 represents the 3D identity matrix; Step 2: For all views and 3D scene points, construct The bilinear constraints are and Step 3: Multiply the constraints in the previous step by matrix A. T , A T The LSGT constraint is obtained after the A matrix is ​​divided into blocks Step 4: Use Schur decomposition to estimate global displacement and 3D scene points, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, using P = - (B3) -1 B4t restores 3D scene points; Step 5: Based on the global displacement obtained and 3D scene points pass Filter the correct solution, where 2. The method for estimating multi-view camera displacement and three-dimensional scene point coordinates according to claim 1, characterized in that: The step of identifying abnormal points in the dimensional scene includes the following steps: Step S1: Consider a 3D scene point P, which is observed by n views; select two views i and j from them and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively; Step S2: Statistic θ i,j The maximum value is denoted as Step S3: Given a threshold δ, if Mark the 3D scene point as an abnormal point and remove it; Step S4: Repeat steps S1 to S3 for all three-dimensional scene points.

3. The method for estimating multi-view camera displacement and three-dimensional scene point coordinates according to claim 2, characterized in that: In step S1, θ is calculated i,j =||[X j ] × R i,j X i ||, and select the maximum value in S2 As a statistical indicator of abnormal 3D points.

4. The method for estimating multi-view camera displacement and three-dimensional scene point coordinates according to claim 1, characterized in that: In step 3, the constraint in step 2 is multiplied by A on the left. T , obtain the LSGT constraint, and solve it using Schur decomposition in step 4.

5. A system for estimating multi-view camera displacement and three-dimensional scene point coordinates, characterized in that: include: 3D scene outlier point recognition module: observes 3D scene points through multiple views, identifies and removes outliers from all 3D scene points; Global displacement and scene restoration module: construct LSGT constraints based on image observations, use Schur decomposition to solve global displacement and 3D scene points, and select the correct solution; The global displacement and scene restoration module includes the following modules: Module 1: Assume that there are M 3D scene points observed by N cameras. For the jth 3D scene point P j , where 1≤j≤M, using the known N camera intrinsic parameter matrix set K={K i |i=1,...,N} and the global pose matrix set R={R i |i=1,...,N}, construct bilinear constraints on global displacement and 3D scene points in represents the feature point observation of the jth 3D scene point in the i-th image, [] × represents the cross product matrix, t i represents the global displacement of view i, I3 represents the 3D identity matrix; Module 2: For all views and 3D scene points, construct The bilinear constraints are and Module 3: Multiply the constraints in the previous step by matrix A T , A T The LSGT constraint is obtained after the A matrix is ​​divided into blocks Module 4: Estimating global displacement and 3D scene points using Schur decomposition, through (B2-B1(B3) -1 B4) t = 0 to solve the global displacement t, using P = - (B3) -1 B4t restores 3D scene points; Module 5: Based on the global displacement obtained and 3D scene points pass Filter the correct solution, where 6. The system for estimating multi-view camera displacement and three-dimensional scene point coordinates according to claim 5, characterized in that: The dimensional scene outlier recognition module includes the following modules: Module M1: Consider a 3D scene point P, which is observed by n views; select any two views i and j from them and calculate θ i,j =||[X j ] × R i,j X i ||, where 1≤i≠j≤n, R i,j is the relative posture matrix of the view, X i and X j are the normalized image feature point observations of the current 3D point P in views i and j respectively; Module M2: Statistics θ i,j The maximum value is denoted as Module M3: Given a threshold δ, if Mark the 3D scene point as an abnormal point and remove it; Module M4: Repeat modules M1 to M3 for all 3D scene points.

7. The system for estimating multi-view camera displacement and three-dimensional scene point coordinates according to claim 6, characterized in that: The module M1 calculates θ i,j =||[X j ] × R i,j X i ||, and select the maximum value in S2 As a statistical indicator of abnormal 3D points.

8. The system for estimating multi-view camera displacement and 3D scene point coordinates according to claim 5, characterized in that: The module 3 multiplies the constraints in module 2 by A on the left T , obtain the LSGT constraint, and solve it using Schur decomposition in module 4.

Images