Method and device for pose estimation of non-cooperative aerospace targets based on deep feature point matching

Abstract

The present invention relates to a method and device for estimating the pose of aerospace non-cooperative targets based on deep feature point matching, and belongs to the fields of image processing, deep learning, and aerospace. A feature point regression and extraction module is constructed, and an encoder-decoder structure is used to extract sub-pixel key points from RGB images and depth images. A multi-channel feature point pairing network is designed, and three loss functions are used to obtain pairings of key points in the body reference system. A non-iterative mismatch removal method is proposed to further improve the pairing accuracy, and the target rotation matrix is ​​extracted from the matched key points to obtain the pose information of the non-cooperative target. The present invention solves the problem of unnoticeable foreground and background of non-cooperative targets and the problem of low accuracy in feature point extraction of non-cooperative targets, thereby improving the accuracy and robustness of target pose estimation.

Description

Technical Field

[0001] The present invention belongs to the fields of image processing, deep learning, and aerospace, and specifically relates to a method and device for estimating the pose of an aerospace non-cooperative target based on deep feature point matching. Background Art

[0002] The development and application of on-orbit services is a key breakthrough in aerospace technology, crucial for meeting future needs such as space capture, debris removal, spacecraft recovery, and on-orbit maintenance. With the rapid increase in spacecraft launches worldwide in recent years, the demand for on-orbit space services has exploded. During on-orbit servicing, obtaining the relative position and attitude between the servicing and target satellites is crucial for subsequent guidance and control. Therefore, accurate position and attitude estimation of targets in space environments is extremely urgent and of great significance to strengthening space security and developing the national economy. For non-cooperative targets, the relative position and attitude estimation problem is particularly complex due to the lack of information on their motion, surface structure, and inertial parameters.

[0003] In on-orbit services such as space capture, debris removal, and spacecraft maintenance, most space targets are non-cooperative targets, that is, they cannot actively interact with the spacecraft for status information, are not equipped with optical markers, and have unknown geometric models. For such non-cooperative targets, the problem of relative position and attitude estimation becomes particularly complicated due to the lack of information such as the target's motion, surface structure, and inertial parameters. At the same time, the selection of onboard sensors is strictly constrained by mass, power consumption, volume, etc. Monocular visible light cameras are still the preferred means of achieving non-cooperative target pose estimation. At the same time, the surface structures of different non-cooperative targets vary greatly, and target imaging is easily affected by complex space environments, such as local occlusion of space targets caused by backlighting, and local dimming of space targets caused by high light. This poses a major obstacle to image-based non-cooperative target pose estimation. In summary, the following problems need to be solved regarding non-cooperative target pose estimation:

[0004] (1) Aiming at the problem that the foreground and background of non-cooperative targets are not obvious, we fully explore the generalization characteristics of non-cooperative targets in complex spatial environments and improve the accuracy and generalization ability of deep learning target recognition models;

[0005] (2) To address the problem of low accuracy in extracting feature points of non-cooperative targets, sub-pixel key point features of the target are extracted to reduce the error caused by discrete pixels and improve the accuracy and robustness of target pose estimation.

[0006] Existing attitude estimation methods are often based on traditional feature point matching algorithms, which are greatly affected by complex conditions such as changes in illumination in the space environment and target occlusion. The matching results have poor robustness and are difficult to adapt to the needs of satellite-borne attitude estimation. Summary of the Invention

[0007] The technical problems to be solved by the present invention are:

[0008] In order to effectively improve the pose estimation model's ability to estimate the target pose in spatial non-cooperative target images and improve the performance of the model's pose estimation, the present invention proposes a non-cooperative target pose estimation model based on feature point regression, introduces a multi-channel feature point pairing network and a non-iterative mismatch removal module, and improves the model's accuracy and robustness in estimating non-cooperative targets.

[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0010] A method for estimating the pose of an aerospace non-cooperative target based on deep feature point matching, characterized by comprising:

[0011] Obtain video data of non-cooperative targets and decompose them into single-frame images in sequence;

[0012] Detect and locate the single-frame image, intercept the target point, delete the irrelevant background in the image, and make the target in the intercepted image located in the middle of the image;

[0013] Build a target image key point extraction model to extract sub-pixel key points from the RGB image and depth image in the first and subsequent frames respectively;

[0014] Construct a multi-dimensional matching model and use the triple loss function to obtain the pairing of key points in the body reference system;

[0015] A non-iterative mismatch removal method is used to extract the target rotation matrix from the matched key points to obtain the pose information of the non-cooperative target.

[0016] A further technical solution of the present invention also includes performing data enhancement on the obtained single-frame image data to enhance the data diversity of the original data set and expand the data distribution range.

[0017] A further technical solution of the present invention is as follows: the target image key point extraction model includes an encoder-decoder feature extraction module and a feature point coordinate sub-pixelization module, the encoder-decoder feature extraction module decomposes a single-frame image into a matching score map and a feature description map; the feature point coordinate sub-pixelization module converts the pixel-level key points of the matching score map into sub-pixel key points, and uses the sub-pixel key points to perform bilinear interpolation and regularization on the feature description map to obtain a sub-pixel feature description map.

[0018] A further technical solution of the present invention: the encoder-decoder feature extraction module includes four convolution layers, including two downsampling layers and two upsampling layers, the convolution kernel size is 3, and the convolution step size is 1.

[0019] A further technical solution of the present invention is to convert the pixel-level key points of the matching score map into sub-pixel key points:

[0020] The feature scores of the key point and its surrounding points are used as the probability of the feature point, and the local window is integrally regressed. The softargmax method is used to calculate the expected coordinates of the key point:

[0021]

[0022] Among them, i and j represent the pixel offset in the x and y directions respectively. i ,y j ) represents the coordinates of the key points with deviation; s(x i ,y j ) is to perform Softargmax operation on the key point coordinates;

[0023] By obtaining Δx and Δy, the modified sub-pixel keypoint coordinates (x′, y′) are expressed as:

[0024] (x′,y′)=(x0,y0)+(Δx,Δy).

[0025] A further technical solution of the present invention: the multidimensional matching model includes:

[0026] Classify the extracted key points and remove duplicate feature points;

[0027] Use camera intrinsic parameters to unify feature points into the same coordinate system;

[0028] The cross-attention module and self-attention module are used to extract matching descriptors, and the multi-dimensional Sinkhorn iteration algorithm is used to obtain the weight distribution map. The weight distribution map is iteratively updated using the triple loss function to finally output the matching scores and matching pairs, thereby achieving correct matching pairs of feature points.

[0029] A further technical solution of the present invention is: the triple loss function is:

[0030]

[0031] in, represents the allocation matrix, D ap and D an Represents the matching score with positive and negative distances, that is, D ap is the sum of the matching scores of the positive matching pairs, D an is the matching score of the unmatched pairs, and the highest-scoring margin is a non-negative factor.

[0032] A further technical solution of the present invention is as follows: the non-iterative mismatch removal method is specifically:

[0033] The matching key points (M kptsi , M kptsj ) for sampling;

[0034] Generate a Hessian matrix H from two pairs of matching key points, and generate a rotation matrix R through the singular value decomposition algorithm;

[0035] Generate the attitude value P from the rotation matrix R through Rodrigues' rotation formula;

[0036] Predicting image M from image i by using the rotation matrix R kpts'j ; for M kpts'j and M kptsj The key point distances between them are sorted and sorted from M kptsi and M kptsj Remove unmatched keypoints from the dataset; the pose is estimated using the selected keypoint pairs.

[0037] A device for estimating the pose of an aerospace non-cooperative target based on deep feature point matching, characterized by comprising:

[0038] A data acquisition module is used to acquire video data of non-cooperative targets and decompose them into single-frame images in sequence;

[0039] The data processing module is used to detect and locate single-frame images, intercept target points, and delete irrelevant background in the image so that the target in the intercepted image is located in the center of the image; build a target image key point extraction model to extract sub-pixel key points from the RGB image and depth image in the first and subsequent frames; build a multi-dimensional matching model and use a triple loss function to obtain key point pairings in the body reference system;

[0040] The pose estimation module is used to extract the target rotation matrix from the matched key points and obtain the pose information of the non-cooperative target.

[0041] The beneficial effects of the present invention are:

[0042] The present invention provides a method and device for estimating the pose of aerospace non-cooperative targets based on deep feature point matching. It also proposes an end-to-end non-cooperative target pose estimation model based on feature point regression, which can accurately estimate the pose of an object. A feature point regression extraction module is constructed to extract sub-pixel key points from RGB images and depth images. A multi-channel feature point pairing network is designed, and three loss functions are used to obtain pairings of key points in the body reference system. A non-iterative mismatch removal method is proposed to further improve pairing accuracy, and the target rotation matrix is ​​extracted from the matched key points to obtain the pose information of the non-cooperative target.

[0043] The method of the present invention solves the problem that the foreground and background of a non-cooperative target are not significant and the problem that the accuracy of extracting feature points of a non-cooperative target is not high, thereby improving the accuracy and robustness of target posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0045] Figure 1 Schematic diagram of the non-cooperative target pose estimation process designed for the present invention.

[0046] Figure 2 Schematic diagram of the target image key point extraction model designed for the present invention.

[0047] Figure 3 Flowchart of the multidimensional matching model designed for the present invention.

[0048] Figure 4 Flowchart of the non-iterative mismatch screening algorithm designed for the present invention.

[0049] Figure 5 Figure 2. ROC of supervised pose estimation for different methods. (a) Recall ROC of pose error when the pose difference is 0 to 10 degrees; (b) Recall ROC of pose error when the pose difference is 10 to 20 degrees; (c) Recall ROC of pose error when the pose difference is from 20° to 30°. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0051] This paper proposes an end-to-end non-cooperative target pose estimation model based on feature point regression, which can accurately estimate the pose of an object. A feature point regression extraction module is constructed, using an encoder-decoder architecture to extract sub-pixel keypoints from RGB and depth images. A multi-channel feature point pairing network is designed, using three loss functions to obtain keypoint pairs in the body reference frame. A non-iterative mismatch removal method is proposed to further improve pairing accuracy. The target rotation matrix is ​​extracted from the matched keypoints to obtain the pose information of the non-cooperative target.

[0052] The steps include:

[0053] Step 1: The sensor platform equipped with an RGBD camera on the spacecraft obtains video data containing multiple non-cooperative targets such as abandoned satellites and meteorites, and decomposes them into single-frame images in frame order;

[0054] Step 2: Perform data enhancement on the acquired single-frame image data, including randomly changing the image contrast, brightness, rotation, flipping, translation and other operations to enhance the data diversity of the original dataset and expand the data distribution range;

[0055] Step 3: Use a mature target detection model (such as FasterRCNN, YOLO series, Centernet target detection model) to detect and locate the image, intercept the target point, delete the irrelevant background in the image, and make the target in the intercepted image located in the center of the image;

[0056] Step 4: Construct a feature point regression extraction module and use an encoder-decoder structure based on a multi-layer convolutional neural network to extract sub-pixel key points from the RGB image and depth image in the first and subsequent frames respectively.

[0057] Step 5: Design a multi-channel feature point pairing network and use three loss functions to obtain the pairing of key points in the body reference system;

[0058] Step 6: A non-iterative mismatch removal method is proposed to further improve the pairing accuracy, and the target rotation matrix is ​​extracted from the matched key points to obtain the pose information of the non-cooperative target;

[0059] Step 7: Compare and analyze the existing spatial non-cooperative target posture estimation method with the posture estimation results of the present invention.

[0060] Since the number of datasets of non-cooperative target images is limited, in order to make the distribution of data features more comprehensive, the dataset is enhanced using operations such as random contrast, random brightness, random flipping, and random rotation to improve the robustness of the model in various situations. After the dataset is enhanced, it is randomly divided into training set, validation set, and test set.

[0061] Figure 1The proposed method for pose estimation of non-cooperative targets aims to efficiently and accurately estimate the behavior of non-cooperative targets, enabling autonomous machines to replace current manual modeling-based estimations and minimize estimation errors. This work employs a three-part pose estimation process based on deep learning to achieve accurate estimation. To extract sub-pixel keypoints from RGB and depth images, an encoder-decoder feature extraction module and a sub-pixel extraction module are first proposed. A triple-loss multi-channel matching module is then used to obtain matched keypoint pairs in the system. Finally, the successfully matched keypoints are used to extract the target rotation matrix, and a mismatch removal method is then used to further improve matching accuracy.

[0062] Figure 2 The target image key point extraction model designed for the present invention has a single-frame image with a length and width of H and W respectively as its input, and a description image with a shape of H×W×256 as its output. The matching score map describes the probability that each pixel in the original image is a feature key point. By setting a suitable threshold for the score map, feature points can be extracted, but at this time the coordinates of the key points are pixel-level integer coordinates, which limits the accuracy of the model for key point positioning. In order to solve this problem, the present invention designs a feature point coordinate sub-pixelization module to fuse the neighborhood pixel features with the original feature points to achieve sub-pixel accuracy for each feature point. The specific steps are: taking the feature scores of the key point and its surrounding adjacent points as the probability of the feature point, performing integral regression on the local window, and calculating the coordinate expectation of the key point. In addition, in order to maintain the differentiable property, a Softargmax method is introduced to calculate the coordinate expectation. The formula is:

[0063]

[0064] Where i and j represent the pixel offset in the x and y directions respectively. Their values ​​are -2, -1, 1, and 2. (x o ,y j ) represents the keypoint coordinates with deviation. By obtaining Δx and Δy, the modified sub-pixel keypoint coordinates (x′, y′) can be expressed as:

[0065] (x′,y′)=(x0,y0)+(Δx,Δy)

[0066] After that, bilinear interpolation is performed on the generated sub-pixel keypoints. Each keypoint has a 256-dimensional vector as its descriptor. These vectors are then normalized using the L2 function to generate the final 256-dimensional descriptor.

[0067] Figure 3Flowchart showing the multi-dimensional matching model. First, duplicate locations are processed and key points extracted from the RGB image and depth image are classified and processed. Duplicate feature points are removed through the NMS process, and the processed key point set becomes:

[0068] P i =(P o ,P d ,P o,d )

[0069] Among them, P o,d Represents the key points in the RGB and depth feature maps. o and P d Represent the key points from the RGB feature map and depth map respectively. The camera intrinsic parameters are used to unify the feature points into the same coordinate system.

[0070] Afterwards, the keypoint sets are introduced into the matching backbone respectively, and matching descriptors are extracted through the cross-attention and self-attention modules. The score matrix based on the matching descriptors can be used to calculate the distribution map. We design the multi-dimensional matching score as the similarity of the matching descriptors, and the formula is:

[0071]

[0072] where <·> represents the inner product. λ1 and λ2 are hyperparameters controlling the weights of the deep score graph and the comprehensive score graph. To find the correct matching pair from the matching scores, the optimization problem is formulated as an optimal transfer problem involving two discrete distributions a and b with scores S. Its entropy regularized formulation naturally yields the desired soft assignment, so the Sinkhorn algorithm is introduced for correct matching.

[0073] During network training, in order to help the model learn the correct matching score, the present invention designs an integrated loss function that combines triple loss and negative log-likelihood (nll) loss. The loss function is as follows:

[0074]

[0075] in, represents the distribution matrix. Dap and Dan represent matching scores with positive and negative distances. That is, Dap is the sum of the matching scores for positive matching pairs, and Dan is the matching score for negative matching pairs. The highest score, margin, is a non-negative factor. The present invention applies a loss function to increase Dap and decrease Dan, thereby improving matching accuracy.

[0076] Figure 4Flowchart representing the non-iterative mismatch removal method. Due to pixel errors and mismatch points, the pose accuracy usually cannot meet the estimation requirements. Iterative methods such as RANSAC are not well optimized for GPUs. This paper proposes a non-iterative method to speed up the estimation process and improve the pose accuracy. The details of the algorithm are as follows Figure 4 shown.

[0077] like Figure 4 As shown in Algorithm 1, the matching pair (Mi, Mj) is the output of the proposed matching model. And the matching key points (Mj) generated by the farthest point sampling algorithm (FPS) are kptsi , M kptsj ) for sampling. After that, the two pairs of matching key points are used to generate a Hessian matrix H, and the rotation matrix R is generated by the singular value decomposition algorithm. Then, the posture value P is generated from the rotation matrix R by the Rodrigues rotation formula. In order to remove the unmatched points, the M of image j is predicted from image i by the retrieved rotation matrix kpts'j . kpts'j and M kptsj The key point distances between them are sorted and sorted from M kptsi and M kptsj Unmatched keypoints are removed from the dataset. The pose is then estimated using the selected keypoint pairs.

[0078] Figure 5 The method proposed in this invention is verified on the constructed space target dataset. Figure 5 Different ROC curves and AP scores of different image matching models are shown. Figure 5 (a) Figure 5 (b) and Figure 5 (c) shows the matching results at different target rotation rates. Figure 5 As shown in (a), the HardNet+Superglue model achieves the highest AUC score, while the proposed model achieves the second-place AUC score of 0.85. In fact, most of the compared models achieve acceptable performance when the target rotation rate is 0-10°. As for the target rotation rate from 10° to 20°, from 20° to 30°. Figure 5 (b) and Figure 5(c) shows that the proposed method achieves the best performance in the range of 10° to 30°. When the target rotation rate is in the range of 20° to 30°, the AUC score of the proposed method reaches 0.70. The AUC and APE scores in the table. The experiments verify that the proposed method achieves competitive performance, especially in the case of large target rotation rates. The mAPE and mMS of the proposed method are 0.011rad and 0.767rad respectively. Compared with the proposed method, HardNet-superglue and LoFTR methods achieve the second best performance. When the rotation rate is less than 10°. HardNet-superglue and LoFTR methods demonstrate their efficiency in extracting local features from RGB images. When the rotation rate is high, the proposed method has obvious advantages in extracting key points from RGB images and depth images.

[0079] Table 1. Supervised attitude estimation results using different detection models on the test dataset

[0080]

[0081] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. A method for estimating the pose of non-cooperative air and space targets based on deep feature point matching, characterized by include: Obtain video data of non-cooperative targets and decompose them into single-frame images in sequence; Detect and locate the single-frame image, intercept the target point, delete the irrelevant background in the image, and make the target in the intercepted image located in the middle of the image; A target image key point extraction model is constructed to extract sub-pixel key points from the RGB image and depth image in the first frame and subsequent frames respectively; the target image key point extraction model includes an encoder-decoder feature extraction module and a feature point coordinate sub-pixelization module, the encoder-decoder feature extraction module decomposes a single frame image into a matching score map and a feature description map; the feature point coordinate sub-pixelization module converts the pixel-level key points of the matching score map into sub-pixel key points, and uses the sub-pixel key points to perform bilinear interpolation and regularization on the feature description map to obtain a sub-pixel feature description map; The pixel-level key points of the matching score map are converted into sub-pixel key points as follows: The feature scores of the key point and its surrounding points are used as the probability of the feature point, and the local window is subjected to integral regression. The softargmax method is used to calculate the expected coordinates of the key point: in, i and j Respectively x and y The pixel offset in the direction, represents the keypoint coordinates with deviations; It is to perform Softargmax operation on the key point coordinates; By getting and , modified sub-pixel keypoint coordinates Expressed as: Construct a multi-dimensional matching model and use the triple loss function to obtain the pairing of key points in the body reference system; A non-iterative mismatch removal method is used to extract the target rotation matrix from the matched key points to obtain the pose information of the non-cooperative target.

2. The method for estimating the pose of an air-space non-cooperative target based on deep feature point matching according to claim 1, wherein: It also includes data enhancement of the obtained single-frame image data to enhance the data diversity of the original data set and expand the data distribution range.

3. The method for estimating the pose of an air-space non-cooperative target based on deep feature point matching according to claim 1, wherein: The encoder-decoder feature extraction module includes four convolutional layers, including two downsampling layers and two upsampling layers, with a convolution kernel size of 3 and a convolution step size of 1.

4. The method for estimating the pose of an air-space non-cooperative target based on deep feature point matching according to claim 1, wherein: The multidimensional matching model includes: Classify the extracted key points and remove duplicate feature points; Use camera intrinsic parameters to unify feature points into the same coordinate system; The cross-attention module and self-attention module are used to extract matching descriptors, and the multi-dimensional Sinkhorn iteration algorithm is used to obtain the weight distribution map. The weight distribution map is iteratively updated using the triple loss function to finally output the matching scores and matching pairs, thereby achieving correct matching pairs of feature points.

5. The method for estimating the pose of an air-space non-cooperative target based on deep feature point matching according to claim 4, characterized in that: The triple loss function is: in, represents the allocation matrix, and represents the matching score with positive and negative distances, that is, is the sum of the matching scores of the positive matching pairs, is the matching score of the unmatched pairs, and the highest score margin is a non-negative factor.

6. The method for estimating the pose of an air-space non-cooperative target based on deep feature point matching according to claim 1, wherein: The non-iterative mismatch removal method is specifically as follows: The matching key points generated by the farthest point sampling algorithm FPS (M kptsi , M kptsj ) for sampling; Generate a Hessian matrix H from two pairs of matching key points, and generate a rotation matrix R through the singular value decomposition algorithm; Generate the attitude value P from the rotation matrix R through Rodrigues' rotation formula; Predict M of image j from image i by rotation matrix R kpts' j ; for M kpts' j and M kptsj The key point distances between them are sorted and sorted from M kptsi and M kptsj Remove unmatched keypoints from the dataset; the pose is estimated using the selected keypoint pairs.

7. A device for estimating the pose of non-cooperative air and space targets based on deep feature point matching, characterized in that include: A data acquisition module is used to acquire video data of non-cooperative targets and decompose them into single-frame images in sequence; The data processing module is used to detect and locate the single-frame image, intercept the target point, delete the irrelevant background in the image, and make the target in the intercepted image be located in the middle of the image; build a target image key point extraction model, and extract sub-pixel key points from the RGB image and depth image in the first and subsequent frames respectively; A multidimensional matching model is constructed, and a triple loss function is used to obtain pairings of key points in a body reference system. The target image key point extraction model includes an encoder-decoder feature extraction module and a feature point coordinate sub-pixelization module. The encoder-decoder feature extraction module decomposes a single-frame image into a matching score map and a feature description map. The feature point coordinate sub-pixelization module converts pixel-level key points of the matching score map into sub-pixel key points. The sub-pixel key points are used to perform bilinear interpolation and regularization on the feature description map to obtain a sub-pixel feature description map. The pixel-level key points of the matching score map are converted into sub-pixel key points as follows: The feature scores of the key point and its surrounding points are used as the probability of the feature point, and the local window is subjected to integral regression. The softargmax method is used to calculate the expected coordinates of the key point: in, i and j Respectively x and y The pixel offset in the direction, represents the keypoint coordinates with deviations; It is to perform Softargmax operation on the key point coordinates; By getting and , modified sub-pixel keypoint coordinates Expressed as: The pose estimation module is used to extract the target rotation matrix from the matched key points and obtain the pose information of the non-cooperative target.

Images