Three-Dimensional Reconstruction Method for Non-Cooperative Targets Based on Branch Reconstruction Registration

Abstract

The present invention relates to a three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration, belonging to the field of computer vision. Given a sequence of multi-angle images, the method first classifies the image sequence and uses the structure from motion algorithm to obtain the three-dimensional point cloud data corresponding to each type of image sequence. Then, the scale of each type of point cloud data is unified, and the point cloud registration algorithm is used to register and reconstruct each type of point cloud, thereby realizing the three-dimensional reconstruction of the non-cooperative target in space. The present invention classifies the image sequence and performs parallel reconstruction, thereby reducing the time consumption in the reconstruction process and improving the reconstruction efficiency, and can meet the real-time requirements of spacecraft operations. By registering the point cloud data after various reconstructions, the finally reconstructed point cloud is denser, improving the reconstruction accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and particularly relates to a three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration. Background Art

[0002] In recent years, with the development of space technology and the in-depth exploration of outer space resources by humans, spacecraft have gradually been applied to various fields such as society, military, and economy. However, due to the weak gravitational force of the Earth on spacecraft, after a spacecraft fails or completes its mission and is discarded, it cannot fall into the atmosphere and be destroyed by itself, but continues to float freely within the orbital circle, becoming space debris. Limited by the satellite docking position, it is urgent to clean up the space debris in the orbital circle. In addition, with the development of technology, the structures of various spacecraft have become more precise and complex, and the cost has also increased relatively. To save manufacturing costs as much as possible, extend the service life, and improve the working ability, spacecraft need to have the function of on-orbit maintenance. Obtaining accurate position information of the target and completing space interactive docking are the primary conditions for realizing on-orbit service tasks such as repairing failed satellites and cleaning up space debris. Therefore, as an effective information technology means for obtaining the target position, three-dimensional reconstruction of space non-cooperative targets (spacecraft that cannot provide effective cooperation information and lose cooperation indication) has gradually become a research hotspot.

[0003] With the development of space technology, the capture technology of space cooperative targets has been relatively mature. In the case of having prior knowledge of the target, it is possible to calculate information such as the target attitude and speed, and related technologies have been successfully applied to some on-orbit tasks of spacecraft such as space interactive docking, fuel supply, and cargo transportation.

[0004] However, actual space missions are more non - cooperative, and the motion situation of the target and the information of its position and attitude parameters in the space orbit are not known in advance. Currently, for the 3D reconstruction of space non - cooperative targets, the main technical solutions are divided into two categories: methods based on laser scanning (scanning lidar, TOF flash lidar) and methods based on camera projection geometry (structured light cameras, stereo vision cameras, etc.). Among them, the method based on laser scanning emits a beam from a laser rangefinder to the surface of an object, determines the distance between the object and the laser rangefinder according to the time difference between the transmitted signal and the received signal, and then determines the size and shape of the object. This method has a high accuracy in generating the model, but the obtained point cloud data is relatively large, and then the point cloud data from multiple viewpoints needs to be registered, which takes a long time and cannot meet the real - time requirements of space operations. The typical representative method based on projection geometry is the Structure from Motion (SfM). This method first captures multiple images from multiple viewpoints, then detects the key points in the images, uses feature matching algorithms to obtain the corresponding relationships of pixel points between the images, combines the matching constraints and the principle of triangulation to obtain the 3D coordinate information of space points, and finally reconstructs the 3D information of the object. This type of method has low requirements for images, strong robustness and practical value, but the reconstructed point cloud data is relatively sparse and the reconstruction takes a long time, which cannot meet the real - time requirements. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] In order to avoid the deficiencies of the prior art, the present invention proposes a 3D reconstruction method for non - cooperative targets based on branch reconstruction registration.

[0007] Technical Solution

[0008] A 3D reconstruction method for non - cooperative targets based on branch reconstruction registration, characterized by the following steps:

[0009] S1: Classify the input multi - angle picture sequence.

[0010] S2: For each category of picture sequence, simultaneously use the SfM technology to obtain the corresponding 3D point cloud data.

[0011] S3: Unify the scales of the point cloud data with different scales.

[0012] S4: Use a pairwise registration algorithm to pairwise register the 3D point cloud data obtained from adjacent category picture sequences, and then obtain the reconstructed point cloud.

[0013] A further technical solution of the present invention: The steps of classifying the multi - angle picture sequence according to time sequence in S1 include:

[0014] S11: Number the acquired image sequences in chronological order from 1 to N;

[0015] S12: Set the class size K and the overlap rate r between classes, and calculate the number of classes:

[0016]

[0017] Further technical solution of the present invention: The steps of using SfM technology in S2 to simultaneously obtain the three-dimensional point clouds corresponding to each class of image sequences include:

[0018] S21: Extract feature points and perform feature description on the images, and establish the corresponding relationship between adjacent images;

[0019] S22: According to the corresponding relationship, use the eight-point method to solve the fundamental matrix F;

[0020] S23: Use the fundamental matrix F to estimate the camera matrix M i :

[0021] S24: Use triangulation to solve the coordinates of the three-dimensional point X j of

[0022]

[0023] where X j is the n three-dimensional points to be obtained, j = 1,..., n, x ij represents the pixel coordinates of the three-dimensional point corresponding to m images, i = 1,..., m, M i represents the camera projection matrix corresponding to the i-th image, and

[0024] x ij = M i X j (i = 1,..., m, j = 1,..., n) (3)

[0025] Further technical solution of the present invention: The steps of unifying the scales of various point cloud data in S3 are as follows:

[0026] S31: Denote two adjacent point clouds as the source point cloud P s and the target point cloud P t , and calculate the resolution of each point cloud sequence respectively

[0027]

[0028]

[0029] where n s and n t are the numbers of points in the source point cloud and the target point cloud respectively, disi represents the Euclidean distance between the i-th point and its nearest neighbor in the point cloud;

[0030] S32: Unify the scale and expand the coordinates of each point in the source point cloud P s by times.

[0031] A further technical solution of the present invention: The steps of pairwise registration of the point clouds obtained from various image sequences in S4 are as follows:

[0032] S41: Preprocessing:

[0033] Use the algorithm idea of random sample consensus to register adjacent point clouds. First, preprocess the two point clouds to obtain an initial matching set C = {c i}}, where and respectively represent the key points that match each other in the source point cloud and the target point cloud, and use the initial matching set as the input of the algorithm;

[0034] S42: Calculate the compatibility value between the matches

[0035]

[0036] where t cons is a constant, and D(c i , c j ) represents the distance between the matches, which is defined as follows

[0037]

[0038] S43: Construct a graph and sort the triples according to the compatibility value; abstract each match into a node. If the compatibility value s(c i , c j ) between the nodes is greater than a preset threshold t comp , then connect an edge between the nodes, and finally form a graph from the discrete nodes; calculate the compatibility score of the triangles in the graph

[0039] Comp(COT) = l(e(i,j)) + l(e(i,k)) + l(e(j,k)) (8)

[0040] where l(e(i,j)) = s(c i , c j ), and then sort the triangles according to the compatibility score;

[0041] S44: Sample the triples; perform three-point sampling according to the sorting result of S43, where the three sampled matches are used to calculate the pose

[0042]

[0043] Among which R it , t it respectively represent the rotation transformation matrix and the translation transformation matrix between the point cloud of the it-th iteration scene and the target point cloud;

[0044] S45: Evaluate the quality of the pose using the Mean Absolute Error (MAE) function

[0045]

[0046]

[0047] Among which represents the rotation error, and t he is a constant used to determine whether c j is an inlier;

[0048] S46: Repeat the two steps of S44 and S45, and select the pose R it , t it with the highest score as the final output pose.

[0049] A further technical solution of the present invention: The preprocessing in step 41 includes point cloud downsampling, calculating key points, descriptors, and establishing feature matching.

[0050] Beneficial effects

[0051] A non-cooperative target three-dimensional reconstruction method based on branch reconstruction registration proposed by the present invention. Given a sequence of multi-angle pictures, the method first classifies the picture sequence and uses the Structure from Motion (SfM) algorithm to obtain the three-dimensional point cloud data corresponding to each type of picture sequence. Then, the point cloud data of each type is unified in scale, and the point cloud registration algorithm is used to register and reconstruct each type of point cloud, thereby realizing the three-dimensional reconstruction of the non-cooperative target in space.

[0052] Compared with the prior art, the beneficial effects of the method of the present invention are as follows:

[0053] 1) Classify the picture sequence and perform parallel reconstruction, thereby reducing the time consumption in the reconstruction process, improving the reconstruction efficiency, and meeting the real-time requirements of spacecraft operations.

[0054] 2) Register the point cloud data after reconstruction of each type, and the finally reconstructed point cloud is denser, improving the reconstruction accuracy. Description of the drawings

[0055] The drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference signs denote the same components.

[0056] Figure 1 It is a schematic flow chart of an embodiment of the technical solution of the present invention;

[0057] Figure 2 It is a schematic diagram of parallel reconstruction;

[0058] Figure 3 It is a flow chart of SFM 3D reconstruction;

[0059] Figure 4 It is a flow chart of the point cloud registration algorithm based on RANSAC;

[0060] Figure 5 It is a reconstruction result diagram of the traditional SFM method and the proposed method under different data;

[0061] Table 1 is the quantitative analysis result of the reconstruction results of the proposed method and the traditional SFM method under different data. Specific implementation manners

[0062] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] The non-cooperative target 3D reconstruction method based on branch reconstruction registration provided by the embodiment of the technical solution of the present invention has a process as Figure 1 shown, mainly including four steps: classification, parallel 3D reconstruction, unified scale, and 3D registration reconstruction. Compared with the traditional 3D reconstruction based on structure from motion, the present invention first classifies the initial image sequence according to time sequence; then performs parallel 3D reconstruction on each class; then unifies the scale of the reconstructed point cloud data, and then uses a pairwise registration algorithm for 3D registration reconstruction, and finally obtains the reconstructed target point cloud data.

[0064] The specific solution method is as follows:

[0065] (1) Classify the input multi-angle image sequence. Optional classification strategies include according to perspective, gray value, scale, etc. Considering that the subsequent point cloud registration requires overlap between point cloud data, and there also needs to be an overlapping part between the image sequences of adjacent classes, so the present invention example adopts the Figure 2 time-sequence-based classification method shown. The initial image sequence is numbered 1-N according to time sequence (here the time sequence refers to the moment when the camera acquires the image), the class size K (the number of image sequences included in each class) and the overlap rate r between classes (the overlap rate between adjacent classes, 50% is used in the figure) are set, and the number of classes is calculated

[0066]

[0067] (2) For each category of image sequences, simultaneously use the SfM technique to obtain the corresponding 3D point cloud data. Before reconstruction, it is known that the pixel coordinates x of n 3D points X j (j = 1, …, n) in m images, ij as well as the camera intrinsic parameter matrix K of the m images i (i = 1, …, m), and

[0068] x ij = M i X j = K i [R i T i X j i = 1, …, m, j = 1, …, n (2)

[0069] where m is the number of images, n is the number of 3D points, M i , K i , [R i T i are the corresponding camera projection matrix, intrinsic parameters, and extrinsic parameter matrix respectively. The specific steps of SfM reconstruction include four steps such as Figure 3 establishing feature matching, solving the fundamental matrix, solving the essential matrix, and triangulating to solve the 3D point coordinates as shown

[0070] (2.1) Establishing feature matching. Extract feature points and describe features for the images, and establish the corresponding relationships between adjacent images. Specifically, in this embodiment, sift feature extraction is performed on adjacent images, each feature point is described, and the corresponding relationships between two sets of feature points are established

[0071] (2.2) Solving the fundamental matrix. There may be incorrect matches in the matching relationships obtained in the previous step. Therefore, use the random sample consensus algorithm to eliminate the errors and improve the inlier rate of the matches. Then, based on the corresponding relationships, use the eight-point method to solve the fundamental matrix F

[0072] (2.3) Using the fundamental matrix F and the estimated camera matrix M i .

[0073] (2.4) Calculating the 3D point coordinates. Finally, use triangulation to solve the coordinates of the 3D point X j

[0074]

[0075] where X j is the n 3D points to be obtained, j = 1, …, n, x ij ​Denote the pixel coordinates corresponding to the 3D points in m images, i = 1, …, m, M i Denote the camera projection matrix corresponding to the i-th image, and

[0076] x ij = M i X j (i = 1, …, m, j = 1, …, n) (3)

[0077] Furthermore, the steps of unifying the scale of various point cloud data in S3 are as follows:

[0078] (3) Unify the scale of the multi-temporal 3D point cloud reconstructed in step (2) to facilitate subsequent point cloud registration. The specific operations are as follows.

[0079] (3.1) Denote two adjacent point clouds as the source point cloud P s and the target point cloud P t , and calculate the resolution of each point cloud sequence respectively

[0080]

[0081]

[0082] where n s and n t are the numbers of points in the source point cloud and the target point cloud respectively, and dis i represents the Euclidean distance between the i-th point and its nearest neighbor in the point cloud.

[0083] (3.2) Unify the scale, and expand the coordinates of each point in the point cloud P s by times.

[0084] (4) Perform registration and reconstruction on the 3D point cloud data after unifying the scale in step (3) to obtain complete and dense point cloud data. In this embodiment, the guided three-point sampling consistency algorithm is used for registration. Compared with the traditional random sampling consistency algorithm, this algorithm can sample correct matches in the initial stage of iteration, improving the registration accuracy and efficiency. The algorithm flow chart is as Figure 4 shown, mainly including six steps: preprocessing, calculating compatibility values, constructing graphs, sampling, hypothesis generation, and hypothesis evaluation. The specific operations are as follows.

[0085] (4.1) Use the algorithm idea of random sampling consistency to register adjacent point clouds. Denote two adjacent point clouds as the source point cloud P s and the target point cloud P t . First, preprocess the two point clouds (including point cloud downsampling, calculating key points, descriptors, and establishing feature matches) to obtain the initial matching set C = {c i}}, where and respectively represent the key points that match each other in the source point cloud and the target point cloud. The initial matching set is used as the input of the algorithm.

[0086] (4.2) Calculate the compatibility value between the matches

[0087]

[0088] where t cons is a constant, D9c i , c j ) represents the distance between the matches, which is defined as follows

[0089]

[0090] (4.3) Construct a graph and sort the triangles according to the compatibility value. Each match is abstracted into a node. If the compatibility value s9c i , c j ) between the nodes is greater than the preset threshold t comp , then connect an edge between the nodes, and finally form a graph from the discrete nodes. Calculate the compatibility score of the triangles in the graph

[0091] Comp(COT) = l(e(i,j)) + l(e(i,k)) + l(e(j,k)) (8)

[0092] where l(e(i,j)) = s(c i , c j ), and then sort the triangles according to the compatibility score.

[0093] (4.4) Sample triples. Perform three-point sampling according to the sorting result, and calculate the pose based on the three sampled matches

[0094]

[0095] where R it , t it respectively represent the rotation transformation matrix and the translation transformation matrix between the scene point cloud and the target point cloud in the it-th iteration.

[0096] (4.5) Use the MAE function to evaluate the quality of the pose

[0097]

[0098]

[0099] where represents the rotation error, t he is a constant used to judge c jWhether it is an interior point. S mae (T i ) is the current pose T i 's error. The smaller this error is, the higher the confidence of the current pose.

[0100] (4.6) Repeat iterations of steps (4.4) and (4.5) until the specified number of iterations is reached, then terminate the algorithm and select the pose R with the highest score it ,t it as the final output pose, and use the final pose to complete 3D registration and reconstruction.

[0101] Apply the algorithm of the present invention to the 3D reconstruction of actual non-cooperative targets, and the effects are as Figure 5 shown in and Table 1. By reconstructing multiple groups of spatial non-cooperative targets, it can be found that the algorithm of the present invention is superior to the traditional SFM algorithm in terms of reconstruction timeliness and density.

[0102] Table 1 Quantitative analysis of reconstruction results of the proposed method and the traditional SFM method under different data

[0103]

[0104] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration, characterized in that The steps are as follows: S1: Classify the input multi-angle image sequence, including: S11: Number the obtained image sequence according to the time sequence from 1 to N; S12: Set the class size K and the overlap rate r between classes, and calculate the number of classes: S2: For each class of image sequence, use the SfM technology to obtain the corresponding 3D point cloud data simultaneously; S3: Unify the scales of the point cloud data with different scales; S4: Use the pairwise registration algorithm to pairwise register the 3D point cloud data obtained from adjacent class image sequences, and then obtain the reconstructed point cloud, including: S41: Preprocessing: The algorithm idea of Random Sample Consensus is adopted to register adjacent point clouds. First, the two point clouds are preprocessed to obtain the initial matching set C = {c i}, where and respectively represent the key points that match each other in the source point cloud and the target point cloud. The initial matching set is used as the input of the algorithm; S42: Calculate the compatibility value between the matches where t cons is a constant, D(c i , c j ) represents the distance between matches and is defined as follows S43: Construct and sort triples according to the compatibility value; abstract each match into a node. If the compatibility value s(c i , c j ) between nodes is greater than a pre-set threshold t comp , then connect an edge between the nodes, and finally form a graph from the discrete nodes; calculate the compatibility score of triangles in the graph Comp(COT) = l(e(i,j)) + l(e(i,k)) + l(e(j,k)) (8) where l(e(i,j)) = s(c i , c j ), and then sort the triangles according to the compatibility score; S44: Sample triples; perform three-point sampling according to the sorting result of S43, and calculate the pose based on the three sampled matches where R it , t it represent the rotation change matrix and the translation change matrix between the point cloud of the it-th iteration scene and the target point cloud, respectively; S45: Use the mean absolute error (MAE) function to evaluate the quality of the pose Among them represents the rotation error, and t he is a constant used to determine whether c j is an inlier; S46: Repeat the two steps of S44 and S45 iteratively, and select the pose R with the highest score it ,t it as the final output pose.

2. The three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration according to claim 1, wherein: The steps of using the SfM technology in S2 to simultaneously obtain the 3D point cloud data corresponding to each class of image sequence include: S21: Extract feature points and describe features from the images, and establish the corresponding relationship between adjacent images; S22: Solve the fundamental matrix F using the eight-point method based on the corresponding relationship; S23: Estimate the camera matrix M using the fundamental matrix F i : S24: Solve for the three-dimensional point X j using triangulation for its coordinates Among them, X j is the required n three-dimensional points, j = 1, …, n, x ij represents the pixel coordinates corresponding to the three-dimensional points in m images, i = 1, …, m, M i represents the camera projection matrix corresponding to the i-th image, and x ij = M i X j (i = 1, …, m, j = 1, …, n) (3).

3. A three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration according to claim 1, characterized in that: The steps of unifying the scales of various point cloud data in S3 are as follows: S31: Denote two adjacent point clouds as the source point cloud P s and the target point cloud P t , and calculate the resolution of each point cloud sequence respectively where n s and n t are the numbers of points in the source point cloud and the target point cloud respectively, and dis i represents the Euclidean distance between the i-th point and its nearest neighbor in the point cloud; S32: Unify the scale and expand the coordinates of each point in the source point cloud P s by times.

4. A three-dimensional reconstruction method for non-cooperative targets based on branch reconstruction registration according to claim 1, characterized in that: The preprocessing in step 41 includes point cloud downsampling, calculating key points, descriptors, and establishing feature matches.

Images