A Robust Detection Method for Space Target Geometric Constraint Aided by Multi-Field-of-View

Abstract

The present application proposes a geometric constraint-assisted robust detection method for space targets based on multiple fields of view, which relates to the field of space target detection and includes: establishing a target detection model based on multiple fields of view, and extracting a sequence of multi-field-of-view images according to the target detection model; based on the sequence of multi-field-of-view images, extracting two-dimensional point features and line features, and screening and matching through trifocal tensors to calculate corresponding three-dimensional line sets, estimating a plane set according to the three-dimensional line sets, calculating corresponding homography matrices, and performing mapping of points between images through the homography matrices to establish a matching relationship; based on the matching relationship, managing candidate points to obtain mature points, calculating GDOP weighting factors, and using the mature points and GDOP weighting factors for BA optimization to calculate the optimization result; based on the optimization result, determining the motion parameters of the target and the three-dimensional reconstruction result. The present application can overcome the problems of insufficient point features and too high similarity, and perform 3D reconstruction with high precision.

Description

Technical Field

[0001] This application relates to the field of space target detection, and particularly to a method for geometric constraint-assisted robust detection of space targets based on multi-field of view. Background Art

[0002] Space target detection is the basis for a series of on-orbit operations, including target tracking, spacecraft maintenance, and debris interception. The main task of space target detection is to estimate its three-dimensional geometry and motion parameters, which is a common challenge in current research.

[0003] Machine vision based on optical cameras has been widely used in target detection because it provides rich information and high detection accuracy. However, most of the research and applications of space target detection are based on a single field of view, which is limited by the state-of-the-art technology. For space targets (such as satellites, debris, and asteroids), single-field-of-view detection usually results in insufficient information. With the long baseline and synchronous working ability of a distributed satellite system (DSS), multi-field-of-view data of a specified target can be obtained, thus greatly improving the accuracy and robustness and realizing long-term detection, tracking, reconstruction, and recognition.

[0004] However, there are two problems in feature matching in space applications. First, considering that the viewing angles of different detecting spacecraft vary greatly, the changing patterns are large, and it is difficult to accurately match the features. Second, the target patterns are always simple or repetitive (such as solar array panels). Summary of the Invention

[0005] In view of the above problems, a method for geometric constraint-assisted robust detection of space targets based on multi-field of view is proposed. The method includes:

[0006] Establish a multi-field-of-view based target detection model, and extract a multi-field-of-view image sequence according to the target detection model;

[0007] Based on the multi-field-of-view image sequence, extract two-dimensional point features and line features, and screen and match the corresponding three-dimensional line sets through trifocal tensors. Estimate the plane set according to the three-dimensional line sets, calculate the corresponding homography matrix, and map points between images through the homography matrix to establish a matching relationship;

[0008] Based on the matching relationship, manage candidate points to obtain mature points, and calculate the GDOP weighting factor. Use the mature points and the GDOP weighting factor for BA optimization and calculate the optimization result;

[0009] Based on the optimization result, determine the motion parameters and three-dimensional reconstruction result of the target.

[0010] Optionally, the establishment of the multi-field-of-view based target detection model includes:

[0011] If the camera coordinate system is defined as {S} and the target coordinate system is defined as {G}, then the camera imaging model is

[0012]

[0013] where is the camera frame optical axis coordinate of the feature point, is the pixel coordinate of the k-th feature point, and are the 3D coordinates of the k-th feature point in {G} and {S} respectively, is the transformation matrix between {G} and {S}, is the position vector from the origin {G} to the origin {S}, and M is the intrinsic parameter matrix;

[0014] Based on the above camera imaging model, the multi-field-based target detection model is established. Among them, the camera coordinate system on each satellite is defined as {S i}, i = 1, 2,..., n, and its origin is O i .

[0015] Optionally, based on the multi-field image sequence, two-dimensional point features and line features are extracted, and the corresponding three-dimensional line set is screened and matched through the trifocal tensor. The plane set is estimated according to the three-dimensional line set, the corresponding homography matrix is calculated, and the mapping of points between images is performed through the homography matrix to establish a matching relationship, including:

[0016] After extracting the LSD line features and point features from one camera, the least squares method is used to find the plane containing most lines, and the feature points are used to match the lines in another camera. The RANSAC method is used to find the plane and eliminate the wrong matching relationships, where the matching lines are in the same plane, and the line and the point are respectively and

[0017] Based on the trifocal tensor constraint, the 2D line features are screened and matched. If the 2D line features satisfy formula (2), then the three-dimensional line set {L k} is expressed as Formula (2) is:

[0018]

[0019] where ε is the threshold, and T = [T1, T2, T3] is the trifocal tensor;

[0020] If the 3D line is represented as the direction vector L = (x, y, z) and the point P = (x0, y0, z0) on the line. If the line L is in the plane M, then When two lines define a plane, the Random Sample Consensus (RANSAC) method based on the line set is used to determine the possible plane equation {L k}, where the estimated plane set is represented by ;

[0021] According to formula (3), the homography matrix derived from the plane If the homography matrix constraint is satisfied and the point feature descriptors are similar, the matching points are determined. Formula (3) is as follows:

[0022]

[0023] where is the normal vector of the M plane, and d is the distance to the plane.

[0024] Optionally, based on the matching relationship, the candidate points are managed to obtain the mature points, including:

[0025] When calculating the projection errors of the candidate feature points and their matching points, if the errors gradually converge, these points become the mature points for detection and matching.

[0026] Optionally, the selection of the candidate points needs to meet the preset conditions, which include:

[0027] Uniformly distributed and preferably selected at locations with large gradients;

[0028] When a feature point is observed, another candidate point is searched along the epipolar line. When two points in the camera are matched, the final candidate is selected based on the complete trifocal tensor constraint.

[0029] Optionally, the GDOP weighting factor is calculated, and the mature points and the GDOP weighting factor are used for Bundle Adjustment (BA) optimization to calculate the optimization result, including:

[0030] The continuous q-frame measurements of the camera and the points not in the common field of view are optimized. The state vector is defined as follows:

[0031]

[0032] where are the coordinates of the l points that are detected at least three times in the image sequence but not in the common view area;

[0033] The measurement vector is

[0034]

[0035] where is the measurement of the point, where

[0036] Among them

[0037] If the (m + j)th point is observed, the corresponding measurement value is

[0038] Obtained through the weight formula (4) Among them, the measurement function is expressed as:

[0039]

[0040] The said formula (4) is:

[0041]

[0042] By marginalizing δx point and δx npoint , the said optimized result δz motion = H motion δx motion , among which, δx point and δx npoint Obtained through the mathematical model

[0043] Optionally, the said mathematical model includes:

[0044] If the total number of feature points in the co-visible area at time t is m, the measurement value is defined as:

[0045]

[0046] Among them, Used to represent the coordinates of the jth feature point in all visible cameras;

[0047] Regarding the position attitude and point coordinates as the state vector, we get:

[0048]

[0049] Among them vector is the Rodriguez transformation corresponding to the matrix ;

[0050] According to the said camera imaging model, we get:

[0051]

[0052] Among them, the prior is known;

[0053] According to formula (5), formula (6) and formula (7), the measurement function is zt = f(x t ) is expressed as, and the first-order expansion of the measurement function is:

[0054]

[0055] Partition the matrix to obtain Thus, marginalize δx point , and it is formulated as:

[0056]

[0057] The technical solution provided by the embodiments of the present application at least brings the following beneficial effects:

[0058] Assume that the target consists of multiple planes. The 2D line features are matched based on geometric constraints, and then the 3D planes are estimated. Based on the homography matrix derived from the planes, the point features of different images in multiple fields of view are matched to estimate the initial values of the 3D feature point coordinates and the target motion parameters, realizing the 3D reconstruction result, which can overcome the problems of insufficient point features and too high similarity, and perform 3D reconstruction with high precision.

[0059] The additional aspects and advantages of the present application will be partially given in the following description, partially become obvious from the following description, or be understood through the practice of the present application. Description of the Drawings

[0060] The above-mentioned and / or additional aspects and advantages of the present application will become obvious and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0061] Figure 1 is a flowchart of a method for geometric constraint-assisted robust detection of a spatial target based on multiple fields of view according to an embodiment of the present application;

[0062] Figure 2 is a structural diagram of a target detection model based on multiple fields of view according to an embodiment of the present application;

[0063] Figure 3(a) is an experimental scenario shown according to an embodiment of the present application;

[0064] Figure 3(b) is a 3D model simulation diagram of the experimental scenario shown according to an embodiment of the present application;

[0065] Figure 4(a) is a point matching result based on the Harris operator and the FREAK descriptor shown according to an embodiment of the present application;

[0066] Figure 4(b) is a point matching result based on tensor analysis shown according to an embodiment of the present application;

[0067] Figure 5is the target motion estimation result shown according to the embodiments of the present application;

[0068] FIG. 6(a) is the motion parameter estimation error based on BA shown according to the embodiments of the present application;

[0069] FIG. 6(b) is the motion parameter estimation error based on weighted BA shown according to the embodiments of the present application. Detailed implementation manners

[0070] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as a limitation to the present application.

[0071] Figure 1 is a flowchart of a method for assisting robust detection of a spatial target geometric constraint based on multiple fields of view according to the embodiments of the present application, including:

[0072] Step 101, establish a target detection model based on multiple fields of view, and extract a multi-field-of-view image sequence according to the target detection model.

[0073] In the embodiments of the present application, if the camera coordinate system is defined as {S} and the target coordinate system is defined as {G}, the camera imaging model is

[0074]

[0075] where is the camera frame optical axis coordinate of the feature point, is the pixel coordinate of the k-th feature point, and are the 3D coordinates of the k-th feature point in {G} and {S} respectively, is the transformation matrix between {G} and {S}, is the position vector from the origin {G} to the origin {S}, and M is the intrinsic parameter matrix. Among them, M can be written as:

[0076]

[0077] where, α x and α respectively y are the scale factors of the u-axis and v-axis of the image plane, γ is the non-orthogonal factor of the image plane axis, and (u0, v0) are the pixel coordinates of the camera principal point.

[0078] Based on the imaging model, a target detection model based on multiple fields of view as shown in Figure 2 can be established as follows.

[0079] Among them, the camera coordinate system on each satellite is defined as {S i}, i = 1, 2,..., n, and its origin is O i .

[0080] For any two-dimensional point and all satisfy the epipolar constraint. F is the fundamental matrix and can be expressed as:

[0081]

[0082] Among them, is an anti-symmetric matrix. For any vector [V] × , if the points are concentrated in the same three-dimensional plane M, then their measurements in the ith and jth cameras are and Among them, [V] × is expressed in the form of:

[0083]

[0084] Step 102: Based on the multi-field-of-view image sequence, extract two-dimensional point features and line features, and screen and match through the trifocal tensor to calculate the corresponding three-dimensional line set. Estimate the plane set according to the three-dimensional line set, calculate the corresponding homography matrix, and perform mapping of points between images through the homography matrix to establish a matching relationship.

[0085] In the embodiments of the present application, obtaining sufficiently accurate matching points is the key to target motion parameter estimation and reconstruction. However, the spatial target pattern is always simple or repetitive, and geometric constraints are used to assist point matching in this step.

[0086] The embodiments of the present application adopt the trifocal tensor constraint, and the constraint equation is:

[0087]

[0088]

[0089] Among them, T = [T1, T2, T3] is the trifocal tensor, is a 3×3 matrix. If the relative positions and postures between the three FOVs are known, then T can be determined.

[0090] In addition, according to the definition A = [I 0], the trifocal tensor can be written as

[0091]

[0092] Among them, ~a i is the matrix with specific rows of A omitted, b q , c rThey are the q rows of the C matrix and the r rows of the B matrix respectively.

[0093] Specifically, the trifocal tensor contains all the geometric relationships of the three FOVs that are independent of the scene structure. It can be quickly used for feature matching. In addition, n-FOV (n>3) can be decomposed into groups of three FOVs for analysis.

[0094] In the embodiment of the present application, after extracting the LSD line features and point features from one camera, the least squares method is used to find the plane containing most of the lines, and the feature points are used to match the lines in another camera, and the RANSAC method is used to find the plane and eliminate the wrong matching relationships. Among them, the matching lines are in the same plane, and the lines and points are respectively and

[0095]

[0096] Based on the trifocal tensor constraint, the 2D line features are screened and matched. Among them, if the 2D line feature satisfies formula (2), the three-dimensional line set {L k} is expressed as Formula (2) is:

[0097]

[0098] where ε is the threshold.

[0099] If the 3D line is represented as the direction vector L=(x, y, z) and the point P=(x0, y0, z0) on the line. If the line L is in the plane M, then When two lines determine a plane, the possible plane equation {L k} is determined based on the random sample consensus method of the line set, where the estimated plane set is represented by ;

[0100] The homography matrix derived from the plane according to formula (3) If the homography matrix constraint is satisfied and the feature descriptors of the points are similar, the matching points are found. Formula (3) is:

[0101]

[0102] where is the normal vector of the M plane, and d is the distance to the plane.

[0103] Step 103, based on the matching relationship, manage the candidate points to obtain mature points, calculate the GDOP weighting factor, and use the mature points and the GDOP weighting factor for BA optimization to calculate the optimization result.

[0104] During the process of processing data in step 102, due to the rotation of the target and the spatial distribution of the cameras, it may not be possible to clearly and accurately detect features. The worse the quality of point feature extraction, the worse the estimation and reconstruction results. Therefore, some better methods must be developed.

[0105] In the embodiments of the present application, candidate points refer to points with unstable matching relationships or inaccurate detections. Candidate points in each new frame are extracted, and their projections in other cameras are calculated through a homography matrix. After calculating the projection errors of the candidate points and their matching points, if the errors gradually converge, these points become mature points for detection and matching. When the number of mature feature points reaches a certain value, BA optimization starts using all mature points. This method reduces errors caused by matching errors and missing matches. Among them, the selection of candidate points follows two principles: evenly distributed and preferably selected at places with large gradients; when observing a feature point, another candidate point will be searched along the epipolar line. When two points in the cameras are matched, the final candidate is selected based on the complete trifocal tensor constraint.

[0106] When the feature points move at the next moment, the extracted feature points are tracked by the photometric error method to establish matching relationships, and then the matching points are found in other cameras through the homography matrix. After that, weighted BA optimization is used to estimate the motion of the target and three-dimensional reconstruction.

[0107] In the embodiments of the present application, BA is used for the optimal estimation of motion parameters and the 3D coordinates of feature points, which minimizes the reprojection error. By analyzing the error and weighting the equations, the accuracy of the estimation can be improved. From the camera imaging model, we get:

[0108]

[0109] Therefore, ΔP G =(H T H) -1 H T Δz

[0110] From this, the weight is obtained and defined as

[0111]

[0112] where n is the number of visible cameras.

[0113] In the embodiments of the present application, in order to improve the estimation results, the continuous q-frame measurements of the cameras and points not in the common field of view are optimized. The state vector is defined as follows:

[0114]

[0115] where The coordinates of the l points detected at least three times but not in the common view area in the image sequence;

[0116] The measurement vector is

[0117]

[0118] where is the measurement of the point, where

[0119] where

[0120] If the (m + j)th point is observed, the corresponding measurement value is

[0121] Through the weight formula (4), obtain where the measurement function is expressed as:

[0122]

[0123] By marginalizing δx point and δx npoint , obtain the optimized result δz motion = H motion δx motion , where δx point and δx npoint are obtained through the mathematical model.

[0124] where, for the above-mentioned mathematical model, there is:

[0125] If the total number of feature points in the common view area at time t is m, the measurement value is defined as:

[0126]

[0127] where is used to represent the coordinates of the jth feature point in all visible cameras;

[0128] Regarding the position attitude and point coordinates of the target as the state vector, obtain:

[0129]

[0130] where vector is the Rodriguez transformation corresponding to the matrix ;

[0131] According to the camera imaging model, obtain:

[0132]

[0133] Among them, the prior is known;

[0134] According to formulas (5), (6) and (7), the measurement function is represented by z t = f(x t ), and the first-order expansion of the measurement function is:

[0135]

[0136] Partition the matrix to obtain Thus, marginalize δx point , and the formula is:

[0137]

[0138] Step 104: Based on the optimization result, determine the motion parameters of the target and the 3D reconstruction result.

[0139] In the embodiment of the present application, since the initial value of the state vector and the weighted BA have been obtained, the weighted BA can be used to accurately estimate the target motion parameters and the 3D point coordinates.

[0140] The following details an embodiment of the present application.

[0141] Based on the method proposed in the present application, the following experiment is designed.

[0142] The experiment is carried out on the basis of three FOVs. The target is controlled by a single-axis turntable and a linear displacement stage. As shown in FIGS. 3(a) and 3(b), their resolutions are 0.0125 degrees and 0.01 mm respectively. The target first rotates 90 degrees clockwise, then it moves 20 cm along a straight line, and finally rotates 90 degrees counterclockwise. The experimental camera is a commercial-grade black-and-white camera with a resolution of 1994×2552 pixels, a refresh rate of 1 frame / second, and a field of view angle of 45 degrees.

[0143] Calibrate the internal and external parameters of the camera according to the existing method, and the results are shown in Table 1.

[0144] Table 1 Calibration results of the internal and external parameters of the camera

[0145]

[0146]

[0147] The point matching results based on the above equipment are shown in FIGS. 4(a) and 4(b). As shown in FIG. 4(a), given that there are many similar patterns in the picture, many matches are mismatches, so the correct match cannot be determined. As shown in FIG. 4(b), the correct matching points can be obtained.

[0148] In addition, the controlled motion during the experiment can be observed in three stages. The first stage is circular motion (10 degrees / frame), the second stage is linear motion (20 mm / frame), and the third stage is circular motion (-10 degrees / frame). The least squares method is used as an estimate in the best fit process to evaluate the error of motion parameter estimation, as Figure 5 shown.

[0149] The errors of motion parameter estimation are shown in Fig. 6(a), Fig. 6(b), and Table 2. Compared with BA, the estimation error based on weighted BA is smaller. The three-axis attitude angle error (1σ) decreases from (0.09, 0.012, 0.06) degrees to (0.05, 0.09, 0.06) degrees, and the three-axis position error (1σ) decreases from (0.08, 0.11, 0.21) mm to (0.07, 0.09, 0.13) mm. The reconstruction effect is also improved. In Fig. 6(a), the estimation error based on BA has obvious non-random errors, while in Fig. 6(b), the estimation error based on weighted BA is close to random noise. Therefore, weighted BA can improve the effect of target detection.

[0150] Table 2 Errors of motion parameter estimation and reconstruction

[0151]

[0152] The embodiments of the present application can overcome the problems of insufficient point features and too high similarity, and perform 3D reconstruction with high precision.

[0153] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. No limitation is made herein.

[0154] The above specific embodiments do not constitute a limitation to the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A geometric constraint assisted robust detection method for space targets based on multiple fields of view, characterized in that, Including: Establish a multi-view based object detection model, and extract a multi-view image sequence according to the object detection model; Based on the multi-view image sequence, extract two-dimensional point features and line features, and calculate corresponding three-dimensional line sets through trifocal tensor screening and matching. Estimate plane sets according to the three-dimensional line sets, calculate corresponding homography matrices, and perform mapping of points between images through the homography matrices to establish matching relationships; Based on the matching relationships, manage candidate points to obtain mature points, calculate GDOP weighting factors, perform BA optimization on the mature points and the GDOP weighting factors, and calculate the optimization results; Based on the optimization results, determine the motion parameters and three-dimensional reconstruction results of the object; Wherein, the step of extracting two-dimensional point features and line features based on the multi-view image sequence, calculating corresponding three-dimensional line sets through trifocal tensor screening and matching, estimating plane sets according to the three-dimensional line sets, calculating corresponding homography matrices, and performing mapping of points between images through the homography matrices to establish matching relationships includes: After extracting LSD line features and point features from one camera, the least squares method is used to find the plane containing most lines, and feature points are used to match the lines in another camera, and the RANSAC method is used to find the plane and eliminate incorrect matching relationships, where the matching lines lie in the same plane, and the points and lines are respectively and ; Screen and match 2D line features based on the trifocal tensor constraint. Among them, if the 2D line feature satisfies formula (2), then the three-dimensional line set is expressed as , and the formula (2) is: （2） Among them, is the threshold value, is the trifocal tensor; If the 3D line is represented as a direction vector and a point on the line , the straight line is in the plane . Then , when two lines determine a plane, the random sample consensus method based on the line set is used to determine the possible plane equation , where the estimated plane set is represented by ; Derive the homography matrix from the plane according to Equation (3) Match the points if the homography matrix constraint is satisfied and the feature descriptors of the points are similar. Equation (3) is as follows: ​ Among them, is the normal vector of the plane, and is the distance from the matching point to the plane.

2. The method according to claim 1, characterized in that, The establishment of the multi-view based object detection model includes: If the camera coordinate system is defined as , and the target coordinate system is defined as , then the camera imaging model is （1） Among them is the camera frame optical axis coordinate of the feature point, is the pixel coordinate of the k-th feature point, and respectively are and the 3D coordinates of the k-th feature point in is and the transformation matrix between them, is from the target coordinate system the origin to the camera coordinate system the position vector of the origin, is the intrinsic parameter matrix; Based on the camera imaging model, the multi-field-of-view based target detection model is established, where the camera coordinate system on each satellite is defined as , and its origin is .

3. The method according to claim 1, characterized in that, The management of candidate points based on the matching relationships to obtain mature points includes: When calculating the projection errors of candidate feature points and their matching points, if the errors gradually converge, the candidate points become mature points for detection and matching.

4. The method according to claim 3, characterized in that, The selection of the candidate points needs to meet preset conditions, and the preset conditions include: Distribute the candidate points evenly and select the candidate points in places with large gradients; When observing a candidate point, search for a second candidate point along the epipolar line. When two points in the camera are matched, the selection of the third point is based on tensor constraint conditions.

5. The method according to claim 4, characterized in that, The calculation of the GDOP weighting factors, performing BA optimization on the mature points and the GDOP weighting factors, and calculating the optimization results includes: Optimize the consecutive frame measurements of the camera and the points not in the common field of view. The state vector is defined as follows: , Among them , are the coordinates of points that are detected at least three times in the image sequence but not in the common view area ; The measurement vector is , Among them is the measurement of points, where , wherein If the (m + j)-th point is observed, the corresponding measured value is ; Obtained through the weight formula (4) , where the measurement function is expressed as: The formula (4) is: （4） By marginalization and to obtain the optimized result , where and are obtained through a mathematical model 6. The method according to claim 5, wherein, The mathematical model includes: If at the total number of feature points in the co-visible area at the moment is , the measurement value is defined as: （5） Among them, , which is used to represent the coordinates of the j-th feature point among all visible cameras; The position of the target , attitude and point coordinates are regarded as the state vector, and we get: （6） Among them , , the vector is the Rodriguez transform corresponding to the matrix ; According to the camera imaging model, obtain: Among them, the prior i = 2, 3,..., n is known; According to formulas (5), (6) and (7), the measurement function is represented by and the first-order expansion of the measurement function is as follows: Partition the matrix to obtain , thereby marginalizing , formulated as: 。

Images