Real-time pose estimation and nerve reconstruction method without prior model information

Abstract

A real-time pose estimation and neural reconstruction method without prior model information comprises the following steps: collecting an aircraft data stream, and obtaining a matching point pair between a frame in the aircraft data stream and a previous frame by using an LoFTR algorithm; solving a coarse pose parameter of the frame by using an iterative nearest point algorithm; constructing and optimizing a pose map by using the coarse pose parameters of the plurality of frames and the matching relationship to obtain fine pose parameters; the frames with the fine pose parameters are put into a memory pool for communication; performing online neural implicit reconstruction of an aircraft model by using the frame data in the memory pool; solving the coordinate system of the target body, and combining the frames in the plurality of memory pools into a complete point cloud according to the corresponding pose parameters; solving a target body coordinate system by using a neural network; and correcting the fine pose parameters and the reference coordinate system of the reconstructed aircraft model into a target body coordinate system.

Description

Technical Field

[0001] The present invention relates to the technical field of aircraft posture and three-dimensional reconstruction, and in particular to a real-time posture estimation and neural reconstruction method without prior model information. Background Art

[0002] Pose estimation and 3D reconstruction are crucial in space debris removal, formation flying, and on-orbit rendezvous missions. However, the published related methods usually assume a known target CAD model or a simplified wireframe model and define a reference system as a prior. During testing, the reference model or image is aligned with the collected data frame to solve the pose. However, in actual space scenes, it is cumbersome and impractical to obtain the model prior of each target in advance. Therefore, how to accurately estimate the pose and reconstruct the 3D of the target in real time without prior information about the object is a problem to be solved.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0004] The present invention provides a real-time pose estimation and neural reconstruction method without prior model information, which can realize real-time pose estimation and three-dimensional reconstruction of a target without prior information and significant domain gap.

[0005] A real-time pose estimation and neural reconstruction method without prior model information includes: Step S1, collecting aircraft data stream, including RGB image data and depth data from the camera and depth sensor, where: In step S1.1, the LoFTR algorithm is used to obtain matching point pairs between an image frame and a previous frame of an image in an aircraft data stream; In step S1.2, the coarse pose parameters of the frame are solved using an iterative closest point algorithm; In step S1.3, the coarse pose parameters and matching point pairs of multiple frames are used to construct and optimize the pose graph to obtain fine pose parameters; In step S1.4, the frame with fine pose parameters is put into the memory pool for communication; In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model; Step S3, solving the target body coordinate system, where: In step S3.1, the frames in the multiple memory pools are merged into a complete point cloud according to the corresponding pose parameters; In step S3.2, the target body coordinate system is solved using a neural network; In step S4, the detailed pose parameters obtained in step S1 and the reference coordinate system of the aircraft model reconstructed in step S2 are corrected to the target body coordinate system.

[0006] In the real-time pose estimation and neural reconstruction method without prior model information, in step S1, steps S1, S2 and S3 are run in parallel and calculated using CUDA acceleration, wherein step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model at regular intervals, and step S3 solves the target body coordinate system globally only once.

[0007] In the real-time pose estimation and neural reconstruction method without prior model information, in step S1.2, the matching point pairs are used to reduce the nonlinearity of the iterative closest point algorithm. The formula of the iterative closest point algorithm is as follows: the pose is solved by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}.

[0008] in, is the coarse pose parameter, They represent the rotation and translation parts of the coarse pose parameters respectively, and represents the rotation and translation part of the pose parameters to be optimized, and Indicates the current point cloud and the previous point cloud. Matching points.

[0009] In the method for real-time pose estimation and neural reconstruction without prior model information, in step S1.3, a pose graph is constructed. , where the vertices of the pose graph are selected by The fine pose parameters of the frames in the memory pool and the coarse and fine pose parameters of the current frame are composed of ,in , the edges of the pose graph are formed by pairwise matching of vertices, ,in Represents the matching relationship between key points, represents the indices of vertex pairs with matching relationships, Represents a vertex pair The index of the matching relationship between the pose graph and the coarse pose parameters of the current frame As the initial value, the objective function is to minimize the Euclidean distance between point pairs and the normal vector angle error, and then optimize and solve to obtain the fine pose parameters , where the Euclidean distance error between point pairs is defined as follows: , in, Represents a set of vertices The and frame, Represents the Huber robust kernel function, which measures the 3D feature correspondence point and The distance The normal vector angle error is defined as follows: , in, represents the unit normal vector corresponding to the point, Indicates calculating the angle between two vectors, the coefficient represents a scalar factor, set to To maintain Of the same order of magnitude, this loss measures the angle error of the normal vectors of corresponding points.

[0010] In the method for real-time pose estimation and neural reconstruction without prior model information, in step S2, the geometric function Accepts 3D point coordinates of a point cloud As input, and output the signed distance value , appearance function Accepts geometric network intermediate feature vectors , point normal and the ray direction As input, and output color , minimize the unary loss function , And the objective function, the one-dimensional loss function Measure the point cloud of the current frame Point-wise distance from the neural implicit shape: , represents the L1 error between the predicted Euclidean distance and the ground truth distance to the object surface at each query point: , in, represents the ground truth distance label, represents the L2 error between the predicted RGB color and the labeled color at each query point .

[0011] In the method for real-time pose estimation and neural reconstruction without prior model information, in step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training, and in learning the geometric function and appearance function At the same time, the pose parameters of the reference frame are also optimized. The optimized pose parameters assist the subsequent online pose graph optimization.

[0012] In the method for real-time pose estimation and neural reconstruction without prior model information, in step S3.2, the body reference coordinate system of the target is predicted by a neural network, wherein a divide-and-conquer training strategy is adopted, the merged aircraft point cloud is first classified, and then the corresponding expert regression model is used to predict its coordinate system correction value The training set comes from point cloud data of 200 real aircraft models launched historically. The point cloud of each aircraft model is enhanced to 1000 using random rotation, random point perturbation and random point cropping. The classification labels of the aircraft point cloud are defined based on design knowledge and physical meaning.

[0013] In the real-time pose estimation and neural reconstruction method without prior model information, in step S3.2, the neural network includes an encoder, a classification decoder and a regression decoder, wherein the encoder first uses MLP for feature embedding, then uses the farthest point sampling and grouping strategy proposed by PointNet++ to extract local features, uses the self-attention mechanism to extract global features, and fuses the local features with the global features to obtain the encoded feature F. The classifier performs maximum pooling on the encoded feature F and decodes it into a classification score through MLP. The regressor performs maximum pooling on the feature F and regresses the x and y direction vectors respectively through two MLPs, and then obtains the reference frame correction value through orthogonalization. .

[0014] In the method for real-time pose estimation and neural reconstruction without prior model information, in step S4, steps S1 and S2 are in the same coordinate system. The body reference coordinate system is represented as .

[0015] Compared with the prior art, the present invention has the following advantages: the present invention outputs the pose parameters and reconstruction mesh with the target body coordinate system as the reference system without using a priori model of the target, and can realize real-time pose estimation and three-dimensional reconstruction of the target without prior information and significant domain gap. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description of the preferred specific embodiments below, various other advantages and benefits of the present invention will become clear to those of ordinary skill in the art. The drawings in the specification are only for the purpose of illustrating the preferred embodiments and are not considered to be limitations of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work. Moreover, the same reference numerals are used to represent the same components throughout the drawings.

[0017] In the attached picture: Figure 1 A flow chart of a real-time pose estimation and neural reconstruction method without prior model information proposed by the present invention; Figure 2 Schematic diagram of solving the rough pose in step S1.2; Figure 3 Schematic diagram of optimizing the pose graph to solve the fine pose in step S1.3; Figure 4 is a schematic diagram of the key frame memory pool in step S1.4; Figure 5 is the overall flow chart of step S2; Figure 6 Schematic diagram of the divide and conquer strategy for step S3; Figure 7 The network structure diagram for solving the corrected reference system in step S3.2; Figure 8 Illustration of the aircraft classification defined according to design knowledge and physical meaning, classification labels of the aircraft model point cloud. According to the structural physical meaning and design knowledge of the aircraft, we mainly divide the aircraft into different categories based on symmetry. For different categories, different regression models will be used to predict their reference coordinate systems.

[0018] The present invention is further explained below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0019] The specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the specific embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0020] It should be noted that certain words are used in the specification and claims to refer to specific components. Those skilled in the art should understand that technicians may use different nouns to refer to the same component. This specification and claims do not use the difference in nouns as a way to distinguish components, but use the difference in the functions of the components as the criterion for distinction. As mentioned throughout the specification and claims, "including" or "comprising" is an open term, so it should be interpreted as "including but not limited to". The subsequent description of the specification is a preferred embodiment of the present invention, but the description is based on the general principles of the specification and is not intended to limit the scope of the present invention. The scope of protection of the present invention shall be determined by the attached claims.

[0021] To facilitate understanding of the embodiments of the present invention, further explanation will be given below by taking specific embodiments as examples in conjunction with the accompanying drawings, and each of the accompanying drawings does not constitute a limitation on the embodiments of the present invention. Example 1

[0022] like Figures 1 to 8 As shown, the real-time pose estimation and neural reconstruction method without prior model information includes the following steps: Step S1, collecting aircraft data stream, where: In step S1.1, the LoFTR algorithm is used to obtain matching point pairs between a frame in the aircraft data stream and the previous frame; In step S1.2, the coarse pose parameters of the frame are solved using an iterative closest point algorithm; In step S1.3, the coarse pose parameters and matching relationships of multiple frames are used to construct and optimize the pose graph to obtain fine pose parameters; In step S1.4, the frame with fine pose parameters is put into the memory pool for communication; In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model; Step S3, solving the target body coordinate system, where: In step S3.1, the frames in the multiple memory pools are merged into a complete point cloud according to the corresponding pose parameters; In step S3.2, the target body coordinate system is solved using a neural network; In step S4, the detailed pose parameters obtained in step S1 and the reference coordinate system of the aircraft model reconstructed in step S2 are corrected to the target body coordinate system.

[0023] In a preferred implementation of the real-time pose estimation and neural reconstruction method without prior model information, in step S1, steps S1, S2 and S3 are run in parallel and calculated using CUDA acceleration, wherein step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model at regular intervals, and step S3 solves the target body coordinate system globally only once.

[0024] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S1.2, the matching point pairs are used to reduce the nonlinearity of the iterative closest point algorithm. The formula of the iterative closest point algorithm is as follows: the pose is solved by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}.

[0025] in, is the coarse pose parameter, and Indicates the current point cloud and the previous point cloud. Matching points, and Represents the rotation and translation parts of the pose parameters to be optimized.

[0026] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S1.3, a pose graph is constructed. , where the vertices of the pose graph consist of the fine pose parameters of the frames in the selected K memory pools and the coarse and fine pose parameters of the current frame, ,in , the edges of the pose graph are formed by pairwise matching of vertices, ,in Represents the matching relationship between key points, represents the indices of vertex pairs with matching relationships, Represents a vertex pair The index of the matching relationship between the pose graph and the coarse pose parameters of the current frame As the initial value, the objective function is to minimize the Euclidean distance between point pairs and the normal vector angle error, and then optimize and solve to obtain the fine pose parameters , where the Euclidean distance error between point pairs is defined as follows:

[0027] in, Represents a set of vertices The i-th and j-th frames in Represents the Huber robust kernel function, which measures the 3D feature corresponding points and The distance The normal vector angle error is defined as follows: , in, represents the unit normal vector corresponding to the point, Indicates calculating the angle between two vectors, the coefficient represents a scalar factor, set to To maintain Of the same order of magnitude, this loss measures the angle error of the normal vectors of corresponding points.

[0028] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S2, the geometric function Accepts 3D point coordinates of a point cloud As input, and output the signed distance value , appearance function Accepts geometric network intermediate feature vectors , point normal and the ray direction As input, and output color , minimize the unary loss function , And the objective function, the one-dimensional loss function Measure the point cloud of the current frame Point-wise distance from the neural implicit shape: , represents the L1 error between the predicted Euclidean distance and the ground truth distance to the object surface at each query point: , in, represents the ground truth distance label, represents the L2 error between the predicted RGB color and the labeled color at each query point .

[0029] In a preferred embodiment of the real-time pose estimation and neural reconstruction method without prior model information, in step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training, and in learning the geometric function and appearance function At the same time, the pose parameters of the reference frame are also optimized. The optimized pose parameters assist the subsequent online pose graph optimization.

[0030] In a preferred implementation of the real-time pose estimation and neural reconstruction method without prior model information, in step S3.2, the body reference coordinate system of the target is predicted by a neural network.

[0031] In one embodiment, a real-time pose estimation and neural reconstruction method without prior model information includes: In step S1.1, the LoFTR algorithm is used to obtain the matching point pairs between the current frame and the previous frame. Figure 2 As shown, in step S1.2, the matching result obtained in step S1.1 is used to reduce the nonlinearity of the iterative closest point algorithm, making the coarse pose parameters In theory, there is an analytical solution, which solves the pose by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}. Figure 3 As shown, in step S1.3, the pose parameters and matching relationships of multiple frames are used to construct a pose graph , where the vertices of the graph consist of the fine poses of the selected K memory frames and the coarse pose of the current frame: ,in The edges of the graph are formed by pairwise matching of vertices: ,in Represents the matching relationship between key points, represents the indices of vertex pairs with matching relationships, Represents a vertex pair The pose graph is based on the coarse pose of the current frame. As the initial value, the Euclidean distance between point pairs and the normal vector angle error are minimized as the objective function, and the fine pose is obtained by optimization. .like Figure 4 As shown, in step S1.4, the optimized frame is obtained from step S1.3, put into the memory pool, and communicated with steps S2 and S3 through the memory pool; In step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training. Define the geometric function Accepts 3D point coordinates of a point cloud As input, and output the signed distance value . Define the appearance function Accepts geometric network intermediate eigenvectors , point normal and the ray direction As input, and output color , S2 step process and reconstruction results are as follows Figure 5 shown.

[0032] When enough key frames are collected in the memory pool, step S3 is executed. First, step S3.1 merges the depth data of multiple memory frames into a complete point cloud according to the corresponding pose parameters. Then step S3.2 predicts the target's body reference coordinate system through the neural network. The neural network adopts a divide-and-conquer training strategy. The merged aircraft point cloud is first classified, and then the corresponding expert regression model is used to predict its coordinate system correction value. ,like Figure 6 and Figure 7 As shown. The neural network includes an encoder, a classification decoder, and a regression decoder. The encoder first uses MLP for feature embedding, then uses the farthest point sampling and grouping strategy proposed by PointNet++ to extract local features, uses the self-attention mechanism to extract global features, and fuses local features and global features to obtain the encoded feature F. The classifier performs maximum pooling on the encoded feature F and decodes it into a classification score through MLP. After the regressor performs maximum pooling on the feature F, it regresses the x and y direction vectors respectively through two MLPs, and then obtains the reference frame correction value through orthogonalization. The training set comes from point cloud data of 200 real aircraft models launched in history. The point cloud of each aircraft model is enhanced to 1000 using random rotation, random point perturbation and random point cropping. The classification labels of these aircraft point clouds are defined based on design knowledge and physical meaning, such as Figure 8 Finally, step S4 is used to correct the pose parameters obtained in step S1 and step S2 and the reference coordinate system of the reconstructed model to the target body coordinate system. Example 2

[0033] Reference Figure 5 In step S2, an online neural implicit reconstruction is performed. This step can be implemented independently of steps S1 and S3 as a separate method. and appearance function At the same time, the pose parameters of the reference frame are optimized. Therefore, the pose parameters can be set to random initial values. Specifically, the image and depth data streams of the aircraft are collected by the camera and depth sensor, and a certain number of data frames are gradually used for the reconstruction process. In the reconstruction process, the geometric function is learned Accepts 3D point coordinates of a point cloud As input, and output the signed distance value , learn the appearance function Accepts geometric network intermediate eigenvectors , point normal and the ray direction As input, and output color By minimizing the loss function , and Get the texture model output of the target. Among them, the one-dimensional loss function Measure the point cloud of the current frame Point-wise distance from the neural implicit shape:

[0034] in, Different from Example 1, it uses random pose parameters for initialization instead of the results of S1 optimization. represents the Huber robust kernel function, Indicates the number of data frames selected for this round of training. represents the L1 error between the predicted Euclidean distance and the ground truth distance to the object surface at each query point:

[0035] in, represents the ground truth distance label, Indicates the number of query points sampled. represents the L2 error between the predicted RGB color and the labeled color at each query point.

[0036]

[0037] in, represents the ground truth color label, Indicates the number of rays sampled.

[0038] In this way, without the pose estimation of step S1 and the coordinate system correction of step S3, S2 can independently reconstruct the aircraft texture model with the first frame of the sensor as the reference coordinate system.

[0039] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A real-time pose estimation and neural reconstruction method without prior model information, It is characterized in that The steps include: Step S1, collecting aircraft data stream, including RGB image data and depth data from the camera and depth sensor, where: In step S1.1, the LoFTR algorithm is used to obtain matching point pairs between an image frame and a previous frame of an image in an aircraft data stream; In step S1.2, the coarse pose parameters of the frame are solved using an iterative closest point algorithm; In step S1.3, the coarse pose parameters and matching point pairs of multiple frames are used to construct and optimize the pose graph to obtain fine pose parameters; In step S1.4, the frame with fine pose parameters is put into the memory pool for communication; In step S2, the frame data in the memory pool is used to perform online neural implicit reconstruction of the aircraft model; Step S3, solving the target body coordinate system, where: In step S3.1, the frames in the multiple memory pools are merged into a complete point cloud according to the corresponding pose parameters; In step S3.2, the target body coordinate system is solved using a neural network; In step S4, the detailed pose parameters obtained in step S1 and the reference coordinate system of the aircraft model reconstructed in step S2 are corrected to the target body coordinate system.

2. A method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that Preferably, in step S1, steps S1, S2 and S3 are run in parallel and use CUDA to accelerate the calculation, wherein step S1 processes each frame of the data stream, step S2 performs online neural implicit reconstruction of the aircraft model at regular intervals, and step S3 solves the target body coordinate system globally only once.

3. A method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S1.2, the matching point pairs are used to reduce the nonlinearity of the iterative closest point algorithm. The formula of the iterative closest point algorithm is as follows: the pose is solved by maximizing the alignment between the current point cloud {Pt} and the previous frame point cloud {Ps}. ， in, is the coarse pose parameter, They represent the rotation and translation parts of the coarse pose parameters respectively, and represents the rotation and translation part of the pose parameters to be optimized, and Indicates the current point cloud and the previous point cloud. Matching points.

4. The method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S1.3, construct the pose graph , where the vertices of the pose graph consist of the fine pose parameters of the frames in the selected K memory pools and the coarse and fine pose parameters of the current frame, , where pool∈[1,K], the edges of the pose graph are formed by pairwise matching of vertices. ,in Represents the matching relationship between key points, represents the indices of vertex pairs with matching relationships, Represents a vertex pair The index of the matching relationship between the pose graph and the coarse pose parameters of the current frame As the initial value, the objective function is to minimize the Euclidean distance between point pairs and the normal vector angle error, and then optimize and solve to obtain the fine pose parameters , where the Euclidean distance error between point pairs is defined as follows: ， in, Represents a set of vertices The i-th and j-th frames in Represents the Huber robust kernel function, which measures the 3D feature corresponding points and The distance The normal vector angle error is defined as follows: , in, represents the unit normal vector corresponding to the point, Indicates calculating the angle between two vectors, the coefficient represents a scalar factor, set to To maintain Of the same order of magnitude, this loss measures the angle error of the normal vectors of corresponding points.

5. The method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S2, the geometric function Accepts 3D point coordinates of a point cloud As input, and output the signed distance value , appearance function Accepts geometric network intermediate feature vectors , point normal and the ray direction As input, and output color , minimize the unary loss function , And the objective function, the one-dimensional loss function Measure the point cloud of the current frame Point-wise distance from the neural implicit shape: ， represents the L1 error between the predicted Euclidean distance and the ground truth distance to the object surface at each query point: ， in, represents the ground truth distance label, represents the L2 error between the predicted RGB color and the labeled color at each query point 。 6. A method for real-time pose estimation and neural reconstruction without prior model information according to claim 5, It is characterized in that In step S2, online training of the neural implicit neural network is performed, which uses the most recent part of the memory pool for training, and learns the geometric function and appearance function At the same time, the pose parameters of the reference frame are also optimized. The optimized pose parameters assist the subsequent online pose graph optimization.

7. The method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S3.2, the target's body reference coordinate system is predicted by a neural network, where a divide-and-conquer training strategy is adopted. The merged aircraft point cloud is first classified, and then the corresponding expert regression model is used to predict its coordinate system correction value. The training set comes from point cloud data of 200 real aircraft models launched historically. The point cloud of each aircraft model is enhanced to 1000 using random rotation, random point perturbation and random point cropping. The classification labels of the aircraft point cloud are defined according to design knowledge and physical meaning.

8. The method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S3.2, the neural network includes an encoder, a classification decoder and a regression decoder, wherein the encoder first uses MLP for feature embedding, then uses the farthest point sampling and grouping strategy proposed by PointNet++ to extract local features, uses the self-attention mechanism to extract global features, and fuses the local features with the global features to obtain the encoded feature F. The classifier performs maximum pooling on the encoded feature F and decodes it into a classification score through MLP. The regressor performs maximum pooling on the feature F and regresses the x and y direction vectors respectively through two MLPs, and then obtains the reference frame correction value through orthogonalization. .

9. The method for real-time pose estimation and neural reconstruction without prior model information according to claim 1, It is characterized in that In step S4, steps S1 and S2 are in the same coordinate system The body reference coordinate system is represented as .

Images