A real-time pose estimation and neural reconstruction method without prior model information
By combining the LoFTR algorithm and the iterative nearest point algorithm with neural networks, real-time pose estimation and 3D reconstruction without prior model information are achieved. This solves the problem of pose estimation and reconstruction of target objects without prior models in real space scenarios and is applicable to space debris removal, formation flying and on-orbit rendezvous missions.
Patent Information
- Application Number
- CN202411228198.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2044-09-03
AI Technical Summary
In real-world space scenarios, existing technologies struggle to achieve accurate real-time pose estimation and 3D reconstruction when prior model information of the target object is lacking.
Image frame matching is performed using the LoFTR algorithm and the iterative nearest point algorithm to construct a pose graph and optimize pose parameters. Online neural implicit reconstruction is then performed using a neural network, with CUDA used to accelerate computation. The target body coordinate system is then predicted through the neural network.
Real-time pose estimation and 3D reconstruction of the target were achieved without prior information. The output pose parameters and reconstruction mesh are referenced to the target body coordinate system. It is suitable for space debris removal, formation flying and on-orbit rendezvous missions.
Smart Images
Figure CN119131327B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of aircraft pose and three-dimensional reconstruction, and particularly relates to a real-time pose estimation and neural reconstruction method without prior model information. BACKGROUND
[0002] Pose estimation and three-dimensional reconstruction are crucial in space debris removal, formation flying and on-orbit rendezvous missions. However, the disclosed related methods usually assume known target CAD models or simplified wireframe models and define a reference frame as prior, and in the test, the reference model or image is aligned with the collected data frame to solve the pose. However, in actual space scenarios, it is cumbersome and impractical to obtain the model prior of each target in advance. Therefore, how to accurately and in real time estimate the pose of the target without prior information of the object is a problem to be solved.
[0003] The above information disclosed in the background section is only for the purpose of enhancing the understanding of the background of the present application and therefore can include information that does not constitute the prior art that is already known to those of ordinary skill in the art. SUMMARY
[0004] The present application provides a real-time pose estimation and neural reconstruction method without prior model information, which can realize real-time pose estimation and three-dimensional reconstruction of the target without prior information and significant domain gap.
[0005] A real-time pose estimation and neural reconstruction method without prior model information comprises:
[0006] Step S1, collecting an aircraft data stream, including RGB image data and depth data from a camera and a depth sensor, wherein,
[0007] In step S1.1, the LoFTR algorithm is used to obtain the matching point pairs between the image frames of the image in the aircraft data stream and the previous frame;
[0008] In step S1.2, the iterative closest point algorithm is used to solve the coarse pose parameters of the frame;
[0009] In step S1.3, the coarse pose parameters and the matching point pairs of multiple frames are used to construct and optimize a pose graph to obtain fine pose parameters;
[0010] In step S1.4, the frame with fine pose parameters is put into a memory pool for communication;
[0011] In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model;
[0012] Step S3, solving the target body coordinate system, wherein,
[0013] In step S3.1, frames from multiple memory pools are merged into a complete point cloud according to their corresponding pose parameters;
[0014] In step S3.2, the target body coordinate system is solved using a neural network;
[0015] In step S4, the fine pose parameters obtained in step S1 and the reference coordinate system of the reconstructed aircraft model in step S2 are corrected to the target body coordinate system.
[0016] In the aforementioned real-time pose estimation and neural reconstruction method without prior model information, steps S1, S2, and S3 are run in parallel and accelerated using CUDA. Specifically, step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model at regular intervals, and step S3 solves the target body coordinate system only once globally.
[0017] In the aforementioned real-time pose estimation and neural reconstruction method without prior model information, in step S1.2, the nonlinearity of the iterative nearest point algorithm is reduced by utilizing the matching point pairs. The formula for the iterative nearest point algorithm is as follows: the pose is solved by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}.
[0018]
[0019] in, These are coarse pose parameters. These represent the rotation and translation components in the coarse pose parameters, respectively. and This represents the rotation and translation components of the pose parameters to be optimized. and This indicates the number of points in the current point cloud and the previous point cloud. There are 1 matching point.
[0020] In the aforementioned real-time pose estimation and neural reconstruction method without prior model information, step S1.3 involves constructing a pose graph. The vertices of the pose graph are selected. It consists of the fine pose parameters of the frames in each memory pool and the coarse pose parameters of the current frame. ,in The edges of the pose graph are formed by matching pairs of vertices. ,in This indicates the matching relationship between key points. Indicates the index of a pair of vertices that have a matching relationship. Represents a pair of vertices The index of the matching relationship between them, the pose graph with coarse pose parameters of the current frame. The fine pose parameters are solved by optimization with the objective function of minimizing the Euclidean distance error and the normal vector angle error between the point pairs as the initial value where the Euclidean distance error between the point pairs is defined as:
[0021] ,
[0022] where, represents the vertex set in the and the frame, represents the Huber robust kernel function, which measures the distance between the corresponding 3D feature points and ,
[0023] The normal vector angle error is defined as: ,
[0024] where, represents the unit normal vector corresponding to the point, represents the calculation of the included angle between two vectors, and the coefficient represents a scalar factor, which is set to to keep the same order of magnitude as , which measures the normal vector angle error of the corresponding points.
[0025] In the real-time pose estimation and neural reconstruction method without prior model information, in step S2, the geometry function accepts the 3D point coordinates of the point cloud as input and outputs the signed distance value , and the appearance function accepts the intermediate feature vector of the geometry network , the point normal and the ray direction as input and outputs the color , the unary loss function , and the objective function, the unary loss function measures the point-to-point distance between the current frame point cloud and the neural implicit shape:
[0026] ,
[0027] represents the L1 error between the predicted Euclidean distance at each query point and the ground truth distance to the object surface:
[0028] ,
[0029] wherein, denotes the ground truth distance label, denotes the L2 error between the predicted RGB color and the label color at each query point
[0030] .
[0031] In the real-time pose estimation and neural reconstruction method without prior model information, in step S2, online training of the neural implicit neural network is performed, which uses a part of the most recent in the memory pool for training, and the pose parameters of the reference frame are optimized while learning the geometry function and the appearance function The optimized pose parameters assist subsequent online pose graph optimization.
[0032] In the real-time pose estimation and neural reconstruction method without prior model information, in step S3.2, the body reference coordinate system of the target is predicted by a neural network, wherein a divide-and-conquer training strategy is adopted, the merged aircraft point cloud is first classified, and then the coordinate system correction value is predicted using the corresponding expert regression model The training set comes from the point cloud data of 200 real aircraft models launched historically, and the point cloud of each aircraft model is enhanced to 1000 by using random rotation, random point disturbance and random point clipping. The classification label of the aircraft point cloud is defined according to the design knowledge and physical meaning.
[0033] In the real-time pose estimation and neural reconstruction method without prior model information, in step S3.2, the neural network includes an encoder, a classification decoder and a regression decoder, wherein the encoder first performs feature embedding using MLP, then extracts local features using the farthest point sampling and grouping strategy proposed by PointNet++, and extracts global features using a self-attention mechanism. The local features and global features are fused to obtain the encoding features F, the classifier decodes the maximum pooled encoding features F into classification scores through MLP, and the regressor maximum pools the features F and then regresses the x and y direction vectors through two MLPs, and then obtains the reference system correction value through orthogonalization .
[0034] In the real-time pose estimation and neural reconstruction method without prior model information, in step S4, steps S1 and S2 are in the same coordinate system , the body reference coordinate system is represented as .
[0035] Compared with the prior art, the present application has the following advantages: the present application outputs the pose parameters and the reconstructed grid with the target body coordinate system as the reference system without using the target prior model, and can realize real-time pose estimation and three-dimensional reconstruction of the target without prior information and significant domain gap. BRIEF DESCRIPTION OF DRAWINGS
[0036] Various other advantages and benefits of the present application will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to only illustrate preferred embodiments of the present application and are not to be considered as limiting thereof. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those of ordinary skill in the art. Moreover, the same reference numerals are used to represent the same components throughout the drawings.
[0037] In the drawings:
[0038] Figure 1 A flow chart of a real-time pose estimation and neural reconstruction method without prior model information is proposed for the present application;
[0039] Figure 2 A schematic diagram for solving the coarse pose in step S1.2;
[0040] Figure 3 A schematic diagram for solving the fine pose by pose graph optimization in step S1.3;
[0041] Figure 4 A schematic diagram for the key frame memory pool in step S1.4;
[0042] Figure 5 A flow chart of the whole process of step S2;
[0043] Figure 6 A schematic diagram of the divide-and-conquer strategy of step S3;
[0044] Figure 7 A network structure diagram for solving the modified reference system in step S3.2;
[0045] Figure 8 An explanatory diagram of the aircraft classification defined according to the design knowledge and physical meaning, the classification label of the aircraft model point cloud, according to the structural physical meaning and design knowledge of the aircraft, we mainly divide the aircraft into different categories according to symmetry, and for different categories, different regression models will be used to predict their reference coordinate system.
[0046] The present application will be further explained below in combination with the drawings and examples. DETAILED DESCRIPTION
[0047] Specific embodiments of the present application will now be described in greater detail with reference to the figures. Although specific embodiments of the application are shown in the drawings, it should be understood that the application can be practiced in various forms and should not be limited to the embodiments set forth in the description and / or drawings. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.
[0048] It should be noted that certain terms have been used throughout the description and the claims to refer to particular components. As one skilled in the art will appreciate, components can be referred to by different names and can be used in accordance with a variety of protocols. The description and claims should not be limited to the specific components described herein, but should be given the widest possible interpretation consistent with the principles and features described herein.
[0049] In order to make the embodiments of the present application more comprehensible, the following will be further explained with specific examples in conjunction with the drawings, and each drawing does not constitute a limitation to the embodiments of the present application. Embodiment 1
[0050] As shown in the figure, the real-time pose estimation and neural reconstruction method without prior model information includes the following steps: Figures 1 to 8
[0051] In step S1, the aircraft data stream is collected, wherein,
[0052] In step S1.1, the LoFTR algorithm is used to obtain the matching point pairs between the frame in the aircraft data stream and the previous frame;
[0053] In step S1.2, the iterative closest point algorithm is used to solve the coarse pose parameters of the frame;
[0054] In step S1.3, the coarse pose parameters and matching relationship of multiple frames are used to construct and optimize the pose graph to obtain the fine pose parameters;
[0055] In step S1.4, the frame with fine pose parameters is put into the memory pool for communication;
[0056] In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model;
[0057] In step S3, the target body coordinate system is solved, wherein,
[0058] In step S3.1, frames from multiple memory pools are merged into a complete point cloud according to their corresponding pose parameters;
[0059] In step S3.2, the target body coordinate system is solved using a neural network;
[0060] In step S4, the fine pose parameters obtained in step S1 and the reference coordinate system of the reconstructed aircraft model in step S2 are corrected to the target body coordinate system.
[0061] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, steps S1, S2 and S3 are run in parallel and accelerated by CUDA. Step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model at regular intervals, and step S3 solves the target body coordinate system only once globally.
[0062] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S1.2, the nonlinearity of the iterative nearest point algorithm is reduced by utilizing the matching point pairs. The formula for the iterative nearest point algorithm is as follows: the pose is solved by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}.
[0063]
[0064] in, These are coarse pose parameters. and This indicates the number of points in the current point cloud and the previous point cloud. Matching points, and This represents the rotation and translation components of the pose parameters to be optimized.
[0065] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S1.3, a pose graph is constructed. The vertices of the pose graph are composed of fine pose parameters of frames selected from K memory pools and coarse pose parameters of the current frame. ,in The edges of the pose graph are formed by matching pairs of vertices. ,in This indicates the matching relationship between key points. Indicates the index of a pair of vertices that have a matching relationship. Represents a pair of vertices The index of the matching relationship between them, the pose graph with coarse pose parameters of the current frame. As initial values, fine pose parameters are obtained by optimization with the objective function of minimizing the Euclidean distance error between point pairs and the normal vector angle error where the Euclidean distance error between point pairs is defined as:
[0066]
[0067] where, denotes the set of vertices in the i-th and j-th frame, denotes the Huber robust kernel function, which measures the distance between the corresponding points of 3D features and
[0068] The normal vector angle error is defined as: ,
[0069] where, denotes the unit normal vector corresponding to the point, denotes the calculation of the included angle between two vectors, and the coefficient denotes a scalar factor, which is set to to keep the same order of magnitude as , which measures the normal vector angle error of the corresponding points.
[0070] In the preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S2, the geometry function accepts the 3D point coordinates of the point cloud as input, and outputs the signed distance value , and the appearance function accepts the intermediate feature vector of the geometry network , the point normal and the ray direction as input, and outputs the color , the unary loss function , and the objective function, the unary loss function measures the point-to-point distance between the current frame point cloud and the neural implicit shape:
[0071] ,
[0072] denotes the L1 error between the predicted Euclidean distance at each query point and the ground truth distance to the object surface:
[0073] ,
[0074] where, ground truth distance labels, L2 error between predicted RGB color and label color at each query point
[0075] .
[0076] In the preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S2, online training of the neural implicit neural network is performed, which uses the most recent part in the memory pool for training, while learning the geometry function and the appearance function Optimize the pose parameters of the reference frame, and the optimized pose parameters assist the subsequent online pose graph optimization.
[0077] In the preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S3.2, the body reference coordinate system of the target is predicted by the neural network.
[0078] In one embodiment, the real-time pose estimation and neural reconstruction method without prior model information comprises,
[0079] In step S1.1, the LoFTR algorithm is used to obtain the matching point pairs between the frame and the previous frame. As Figure 2 shown, in step S1.2, the matching results obtained in step S1.1 are used to reduce the nonlinearity of the iterative closest point algorithm, so that the coarse pose parameters have an analytical solution, which is solved by maximizing the alignment degree between the current point cloud {Pt} and the previous frame point cloud {Ps}. As Figure 3 shown, in step S1.3, the pose graph is constructed using the pose parameters and matching relationships of multiple frames, where the vertices of the graph consist of the selected K memory frame fine poses and the coarse pose of the current frame: where The edges of the graph are formed by the pairs of matching points between the vertices: where represents the matching relationship between key points, represents the index of the vertex pair with matching relationship, represents the index of the matching relationship between the vertex pair . The pose graph takes the coarse pose of the current frame as the initial value, and optimizes to obtain the fine pose by minimizing the Euclidean distance and normal vector angle error between the point pairs as the objective function. As Figure 4 shown, in step S1.4, the optimized frame obtained in step S1.3 is put into the memory pool, and communicates with steps S2 and S3 through the memory pool;
[0080] In step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training. A geometry function accepts 3D point coordinates of a point cloud as input and outputs signed distance values A rendering function is defined accepts intermediate feature vectors in the geometry network , point normals and ray directions as input and outputs colors The S2 step flow and reconstruction results are shown in Figure 5 .
[0081] When enough key frames are collected in the memory pool, the S3 step is started. First, step S3.1 merges the depth data of multiple memory frames into a complete point cloud according to the corresponding pose parameters, and then step S3.2 predicts the body reference coordinate system of the target through a neural network. The neural network uses a divide-and-conquer training strategy. The merged aircraft point cloud is first classified, and then the corresponding expert regression model is used to predict the coordinate system correction value as shown in Figure 6 and Figure 7 . The neural network includes an encoder, a classification decoder, and a regression decoder. The encoder first uses an MLP for feature embedding, then uses the farthest point sampling and grouping strategy proposed by PointNet++ to extract local features, and uses a self-attention mechanism to extract global features. The local features and global features are fused to obtain the encoding features F. The classifier decodes the classification scores through an MLP after maximum pooling of the encoding features F. The regressor regresses the x and y direction vectors through two MLPs after maximum pooling of the features F, and then obtains the reference system correction value through orthogonalization. The training set comes from the point cloud data of 200 real aircraft models launched historically. The point cloud of each aircraft model is enhanced to 1000 by random rotation, random point disturbance, and random point clipping. The classification labels of these aircraft point clouds are defined according to design knowledge and physical meaning, as shown in Figure 8 . Finally, the reference coordinate system of the reconstructed model obtained in steps S1 and S2 is corrected to the target ontology coordinate system through step S4. Embodiment 2
[0082] Referring to Figure 5 , in step S2, online neural implicit reconstruction is performed. This step can be implemented as a separate method independent of steps S1 and S3. Since step S2 learns the geometry function and the rendering function pose parameters of the reference frame are optimized at the same time, so the pose parameters can be set as random initial values. Specifically, by streaming the image and depth data of the camera and depth sensor of the mobile platform, a certain number of data frames are used step by step for the reconstruction process. In the reconstruction process, the geometry function accepts 3D point coordinates of the point cloud as input and outputs signed distance values , the appearance function is learned accepts intermediate feature vectors in the geometry network , point normals and ray directions as input and outputs colors . The texture model output of the target is obtained by minimizing the loss function , and . Among them, the unary loss function measures the point-wise distance between the current frame point cloud and the neural implicit shape:
[0083]
[0084] where is different from embodiment 1, which uses random pose parameter initialization instead of the result of S1 optimization, denotes the Huber robust kernel function, denotes the number of data frames selected in this round of training. denotes the L1 error between the predicted Euclidean distance at each query point and the ground-truth distance to the object surface:
[0085]
[0086] where denotes the ground-truth distance label, denotes the number of sampled query points. denotes the L2 error between the predicted RGB color at each query point and the marked color.
[0087]
[0088] where denotes the ground-truth color label, denotes the number of sampled rays.
[0089] In this way, without the pose estimation of step S1 and the coordinate system correction of step S3, S2 can independently reconstruct the texture model of the mobile platform with the first frame of the sensor as the reference coordinate system.
[0090] Although the embodiments of the present application have been described above with reference to the accompanying drawings, the present application is not limited to the above-described specific embodiments and areas of application, and the above-described specific embodiments are merely illustrative and instructive, but are not restrictive. Many modifications can be made by those skilled in the art under the teachings of the present specification and without departing from the scope of the present application as defined by the claims.
Claims
1. A method for real-time pose estimation and neural reconstruction without prior model information, characterized in that, The method comprises the following steps: Step S1, collecting an aircraft data stream, comprising RGB image data and depth data from a camera and a depth sensor, wherein, In step S1.1, the matching point pairs between the image frame and the previous frame of the image in the aircraft data stream are obtained by using the LoFTR algorithm; In step S1.2, the coarse pose parameters of the frame are solved by using the iterative closest point algorithm; In step S1.3, the pose graph is constructed and optimized by using the coarse pose parameters of multiple frames and the matching point pairs, and the fine pose parameters are obtained; In step S1.4, the frame with the fine pose parameters is put into the memory pool for communication; In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model; Step S3, solving a target body coordinate system, wherein, In step S3.1, the frames in multiple memory pools are combined into a complete point cloud according to the corresponding pose parameters; In step S3.2, the target body coordinate system is solved by using a neural network; In step S4, the fine pose parameters obtained in step S1 and the reference coordinate system of the reconstructed aircraft model in step S2 are corrected to the target body coordinate system; wherein, in step S2, the geometry function takes as input the 3D point coordinates of the point cloud and outputs signed distance values , the appearance function takes as input the geometry network intermediate feature vectors , the point normal and the ray direction and outputs the color , the unary loss function , and the objective function, the unary loss function measures the point-wise distance between the current frame point cloud and the neural implicit shape: ， represents the L1 error between the predicted Euclidean distance at each query point and the ground truth distance to the object surface: ， wherein, denotes the ground truth distance label, denotes the L2 error between the predicted RGB color at each query point and the label color; ； In step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training, while learning the geometry function and the appearance function and optimizing the pose parameters of the reference frames, the optimized pose parameters assisting the subsequent online pose graph optimization.
2. The method of claim 1, wherein, In step S1, steps S1, S2 and S3 are run in parallel and use CUDA to accelerate the calculation, wherein step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model once every certain period of time, and step S3 solves the target body coordinate system globally only once.
3. The method of claim 1, wherein, In step S1.2, the matching point pairs are used to reduce the nonlinearity of the iterative closest point algorithm, and the formula of the iterative closest point algorithm is as follows: the pose is solved by maximizing the alignment degree between the current point cloud {Pt} and the point cloud {Ps} of the previous frame, ， in, These are coarse pose parameters. These represent the rotation and translation components in the coarse pose parameters, respectively. and This represents the rotation and translation components of the pose parameters to be optimized. and This indicates the number of points in the current point cloud and the previous point cloud. There are 1 matching point.
4. The method of claim 1, wherein, In step S1.3, a pose graph is constructed wherein the vertices of the pose graph consist of the fine pose parameters of the frames in the selected K memory pools and the coarse fine pose parameters of the current frame, wherein pool ∈ [1, K], the edges of the pose graph are formed by the pairs of point pairs that are pairwise matched between the vertices, wherein denotes the matching relationship between the key points, denotes the index of the vertex pair with the matching relationship, denotes the index of the matching relationship between the vertex pair The pose graph is initialized with the coarse pose parameters of the current frame , and the fine pose parameters are obtained by optimization with the objective function of minimizing the Euclidean distance error and the normal vector angle error between the point pairs, wherein the Euclidean distance error between the point pairs is defined as follows: ， wherein, represents a set of vertices the i-th and j-th frames, represents a Huber robust kernel function, which loss measures the distance between 3D feature correspondences and , The normal vector angle error is defined as follows: , where, represents the unit normal vector corresponding to a point, represents the angle between two vectors, the coefficient represents a scalar factor, set to to keep the same order of magnitude as the loss measure corresponds to the normal vector angle error of the corresponding points.
5. The method of real-time pose estimation and neural reconstruction without prior model information according to claim 1, characterized in that, In step S3.2, the body reference coordinate system of the target is predicted by a neural network, wherein a divide-and-conquer training strategy is adopted, the combined aircraft point cloud is classified first, and then the coordinate system correction value is predicted using the corresponding expert regression model The training set comes from the point cloud data of 200 real aircraft models launched historically, the point cloud of each aircraft model is enhanced to 1000 by random rotation, random point disturbance and random point clipping, and the classification label of the aircraft point cloud is defined according to the design knowledge and physical meaning.
6. The method of real-time pose estimation and neural reconstruction without prior model information according to claim 1, characterized in that, In step S3.2, the neural network comprises an encoder, a classification decoder and a regression decoder, wherein the encoder first uses MLP for feature embedding, then uses the farthest point sampling and grouping strategy proposed by PointNet++ to extract local features, uses a self-attention mechanism to extract global features, and fuses the local features and the global features to obtain the encoding features F; the classifier decodes the classification scores from the encoding features F after maximum pooling by MLP; the regressor regresses the x and y direction vectors respectively by two MLPs after maximum pooling of the features F, and then obtains the reference system correction value by orthogonalization .
7. The method of real-time pose estimation and neural reconstruction without prior model information according to claim 1, characterized in that, In step S4, the steps S1 and S2 are in the same coordinate system In the following, the body reference coordinate system is denoted as .