A high-precision spacecraft pose estimation method based on deep learning

Abstract

The present invention relates to a method for estimating the pose of a spacecraft, and in particular to a method for estimating the pose of a spacecraft with high precision based on deep learning, which solves the technical problems that the network model of the existing method for estimating the pose of a spacecraft has poor interpretability, poor generalization ability, or difficulty in ensuring accuracy. The present invention decomposes the pose estimation task of a spacecraft into three-dimensional view reconstruction of the spacecraft and key point regression, which has strong interpretability and stable generalization ability; the two-dimensional coordinates of the key points are predicted by the key point regression initial model, and then the predicted pose and loss function of the spacecraft are calculated in turn, and the gradient of the loss function with respect to the model parameters is used as a constraint condition to participate in the back propagation of the key point regression initial model, so that the pose solution process of the spacecraft participates in the updating of the model parameters, which can increase the overall fit of the trained key point regression model, avoid the pose estimation from falling into the local optimum, and improve the accuracy of the pose estimation of the spacecraft.

Description

Technical Field

[0001] The present invention relates to a method for estimating a spacecraft's pose, and in particular to a method for estimating a spacecraft's pose based on deep learning with high precision. Background Art

[0002] High-precision pose estimation of spacecraft is a prerequisite for completing major space missions such as on-orbit servicing, rendezvous and docking, and debris removal. However, since spacecraft lack prior information about their own three-dimensional structure, they do not have reliable identification and extraction of cooperative signs, and different spacecraft have different physical structural characteristics, it is impossible to design an overall pose estimation strategy for spacecraft based on a universal model. At the same time, spacecraft are susceptible to uneven lighting, noise interference, and occlusion caused by pose changes in the space environment, and traditional pose estimation algorithms are often not robust.

[0003] At present, the pose estimation methods of spacecraft are generally divided into two types: model feature-based and deep learning-based. The model feature-based method is currently a relatively mature solution in the study of spacecraft pose estimation. The mainstream solutions at home and abroad mainly determine the geometric features on the spacecraft first, and then extract the target features from the image containing the spacecraft through various advanced feature extraction technologies, and match them with the existing three-dimensional features of the spacecraft, and transform them into solving the PnP (Perspective-n-Point, efficient perspective n-point algorithm) problem to obtain the relative pose. However, when faced with complex lighting environments, weak surface textures of spacecraft, or occlusion caused by posture changes, it performs poorly in terms of stability and robustness. In addition, if the pose prior information is not combined, the accuracy of the pose estimation result is poor, and it is difficult to actually apply it to spacecraft detection tasks.

[0004] Deep learning refers to learning more complex nonlinear mapping relationships in a supervised, semi-supervised, or unsupervised manner by deepening the number of neural network layers to extract more advanced features. Deep convolutional neural networks have developed rapidly in computer vision fields such as ground pose estimation due to their powerful feature extraction and expression capabilities, and have gradually attracted the attention of researchers in the field of spacecraft pose estimation.

[0005] Spacecraft pose estimation methods based on deep learning can be mainly divided into two categories, namely direct methods and indirect methods. The direct method refers to the direct regression of translation and rotation components using convolutional neural networks. Sharma et al. proposed a network structure for spacecraft relative pose classification. Using the transfer learning method, the AlexNet model was pre-trained on a large ImageNet dataset and used to train a dataset synthesized using the Tango spacecraft in the Prisma mission. This can reduce the steps of manual preprocessing and feature extraction, so that there is no error accumulation from the original image input to the final pose result of the network model, and the pose estimation method based on model features is more accurate. However, the network structure lacks interpretability, the pose estimation accuracy is heavily dependent on the refinement of the pose classification, and the generalization ability is poor.

[0006] The indirect method combines spacecraft images based on convolutional neural networks with traditional geometric vision. Park and Sharma et al. used the SPN framework (from Sharma et al.’s paper published in the 2019 AIAA / AAS Space Flight Mechanics Conference)<Pose Estimation for Non-Cooperative Spacecraft Rendezvous UsingNeuralNetworks> ) is based on the real-time target detection network and random sample consistency algorithm to solve the PnP problem, and the network model of spacecraft pose estimation is decoupled into the spacecraft target detection network and the key point regression network to respectively determine the bounding box coordinates of the spacecraft and the two-dimensional coordinates of the key points. It has strong interpretability and stable generalization ability, but it only outputs the two-dimensional coordinates of the key points of the spacecraft, which is only partially optimal for pose estimation and it is difficult to guarantee accuracy. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problems that the existing spacecraft pose estimation methods based on deep learning have poor interpretability, poor generalization ability, or difficulty in ensuring accuracy, and to provide a high-precision spacecraft pose estimation method based on deep learning.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] A high-precision spacecraft pose estimation method based on deep learning, which is special in that it includes the following steps:

[0010] Step 1: Select a spacecraft dataset and preprocess it to obtain an initial dataset, wherein the spacecraft dataset consists of spacecraft images with pose labels; then use the pose labels to perform multi-viewing of the spacecraft according to the initial dataset. Figure 3 3D reconstruction to obtain a 3D view of the spacecraft;

[0011] Step 2: Select key points on the spacecraft, then mark the real two-dimensional coordinates of the key points in each spacecraft image, and use them together with the pose label as training labels to obtain a training data set with training labels;

[0012] Step 3: Establish a key point regression initial model based on the high-resolution network, initialize the model parameters, and then input the training data set into the key point regression initial model;

[0013] Step 4: The key point regression initial model processes a spacecraft image in the training data set and outputs the predicted two-dimensional coordinates of the key points in the spacecraft image;

[0014] Step 5: using a PnP solver to solve the predicted position and posture of the spacecraft in the spacecraft image according to the predicted two-dimensional coordinates of the key points in the spacecraft image and the three-dimensional view of the spacecraft;

[0015] Step 6: Calculate the loss function according to the predicted two-dimensional coordinates of the key points in the spacecraft image, the predicted position and training labels of the spacecraft in the spacecraft image. If the loss function converges, the key point regression model is obtained and step 8 is executed; otherwise, step 7 is executed.

[0016] Step 7: Based on the implicit function theorem, according to the predicted position and loss function of the spacecraft in the spacecraft image, the gradient of the loss function with respect to the model parameters is calculated, and then the model parameters are updated using the gradient descent method, and the process returns to step 4 to process the next spacecraft image in the training data set.

[0017] Step 8: Multi-view the target spacecraft Figure 3 dimensional reconstruction to obtain a three-dimensional view of the target spacecraft; then obtain the target spacecraft image and input it into the key point regression model to obtain the predicted two-dimensional coordinates of the key points of the target spacecraft;

[0018] Step 9: Based on the predicted two-dimensional coordinates of the key points of the target spacecraft and the three-dimensional view of the target spacecraft, the predicted position and pose of the target spacecraft is solved using the PnP solver to complete the position and pose estimation of the target spacecraft.

[0019] Furthermore, in step 6, the loss function is calculated by the following formula:

[0020]

[0021]

[0022]

[0023] in, , represents the loss function, The three-dimensional coordinates of the key points of the spacecraft are respectively projected onto the imaging plane through the predicted pose of the spacecraft in the spacecraft image and the pose label of the spacecraft image, is the regularization term, λ is the regularization factor;

[0024] x represents the predicted two-dimensional coordinates of the key points in the spacecraft image, y represents the predicted position and posture of the spacecraft in the spacecraft image;

[0025] The three-dimensional coordinates representing the key points of the spacecraft are projected onto the two-dimensional coordinates on the imaging plane through the predicted position of the spacecraft in the spacecraft image. z Represents the three-dimensional coordinates of key points of a spacecraft, obtained from the three-dimensional view of the spacecraft. K represents the camera internal parameters, represents the square of the two-norm, π ( ) is the projection mapping function;

[0026] The three-dimensional coordinates representing the key points of a spacecraft are projected onto the two-dimensional coordinates on the imaging plane through the pose labels of the spacecraft image. Represents the pose label of the spacecraft image.

[0027] Furthermore, step 7 is specifically as follows:

[0028] Step 7.1, define the objective function of the PnP solver as:

[0029]

[0030] in, is the objective function of the PnP solver, , indicating the first i The predicted two-dimensional coordinates of the key points and the spacecraft i The three-dimensional coordinates of the key points are projected onto the two-dimensional coordinates of the imaging plane through the predicted position of the spacecraft in the spacecraft image. i =1,2,…, n , n is the number of key points;

[0031] Indicates the first i The predicted 2D coordinates of key points;

[0032] , indicating the spacecraft i The three-dimensional coordinates of the key points are projected to the two-dimensional coordinates on the imaging plane through the predicted position of the spacecraft in the spacecraft image. For spacecraft i The three-dimensional coordinates of the key points;

[0033] Step 7.2, the objective function of the PnP solver is used to obtain the first-order derivative of the predicted position of the spacecraft in the spacecraft image, and the stability constraint is obtained as shown in the following formula:

[0034]

[0035] in, is the output of the PnP solver, Indicates the use of PnP solver to solve y , , represents the first predicted pose component of the spacecraft in the spacecraft image, Indicates the first spacecraft in the spacecraft image. m Predicted pose components, m The dimension representing the predicted position and posture of the spacecraft in the spacecraft image;

[0036] Step 7.3: Construct the constraint function of the PnP solver based on the stability constraint:

[0037]

[0038] in:

[0039]

[0040] is the constraint function of the PnP solver, Indicates the first spacecraft in the spacecraft image. j The constraint function of the PnP solver corresponding to the predicted pose component;

[0041] Step 7.4, using the implicit function theorem, according to the constraint function of the PnP solver, obtain the Jacobian matrix of the predicted pose of the spacecraft in the spacecraft image with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image:

[0042]

[0043] in, A Jacobian matrix representing a predicted pose of a spacecraft in the spacecraft image with respect to predicted two-dimensional coordinates of key points in the spacecraft image;

[0044] Step 7.5: According to the loss function, the gradient of the loss function with respect to the predicted position of the spacecraft in the spacecraft image and the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image are calculated by the following formula:

[0045]

[0046]

[0047] in, represents the gradient of the loss function with respect to the predicted pose of the spacecraft in the spacecraft image, represents the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image;

[0048] Step 7.6. Calculate the gradient of the loss function with respect to the model parameters using the following formula:

[0049]

[0050] in, represents the gradient of the loss function with respect to the model parameters, θ are model parameters, Represents the gradient of the predicted two-dimensional coordinates of the key points in the spacecraft image with respect to the model parameters, represents the gradient of the regularization term with respect to the model parameters, and They are all automatically calculated by the key point regression initial model;

[0051] Step 7.7: Use the gradient descent method to update the model parameters using the following formula, then return to step 4 to process the next spacecraft image in the training dataset:

[0052]

[0053] in, are the updated model parameters, is the learning rate.

[0054] Furthermore, in step 5, the predicted position and posture of the spacecraft in the spacecraft image is solved by the following formula:

[0055] .

[0056] Furthermore, in step 3, the specific method of establishing the key point regression initial model based on the high-resolution network is:

[0057] The Basicblock unit of the feature extraction module in each Stage structure of the high-resolution network is replaced with the ShuffleNet-v2-Block unit based on the SE attention mechanism, and the convolution unit of the feature fusion module in each Stage structure of the high-resolution network is replaced with a hollow convolution unit, and then the scSE module is added between the feature extraction module and the feature fusion module of each Stage structure of the high-resolution network;

[0058] The ShuffleNet-v2-Block unit based on the SE attention mechanism includes an equal mapping channel and a depth convolution channel parallel to each other, and the depth convolution channel includes a first SE module, a 3×3 depth convolution module and a second SE module connected in sequence.

[0059] Furthermore, in step 1, the specific method of the pretreatment is:

[0060] The spacecraft dataset is subjected to random Gaussian white noise addition, brightness and contrast adjustment, random flipping and rotation.

[0061] Furthermore, in step 1, the spacecraft dataset is a SPEED+ dataset.

[0062] Compared with the prior art, the present invention has the following beneficial technical effects:

[0063] 1. The present invention provides a high-precision spacecraft pose estimation method based on deep learning. The method predicts the two-dimensional coordinates of key points through a key point regression initial model, and then calculates the predicted pose and loss function of the spacecraft in sequence. The gradient of the loss function with respect to the model parameters is used as a constraint condition to participate in the back propagation of the key point regression initial model to update the model parameters, so that the pose solution process of the spacecraft participates in the update of the key point regression initial model parameters, thereby obtaining a key point regression model, which can increase the overall fit of the key point regression model, avoid the pose estimation from falling into the local optimum, and improve the accuracy of the pose estimation of the spacecraft.

[0064] 2. The present invention provides a high-precision spacecraft pose estimation method based on deep learning, which decomposes the pose estimation task of a spacecraft into three-dimensional view reconstruction and key point regression of the spacecraft. It has strong interpretability and stable generalization ability. Based on the implicit function theorem, the gradient of the loss function with respect to the model parameters is used to update the model parameters, thereby further realizing interpretable end-to-end pose estimation.

[0065] 3. The present invention provides a high-precision spacecraft pose estimation method based on deep learning. The EPnP algorithm is used in the PnP solver to solve the pose of the spacecraft. The calculation complexity is low, the calculation time can be significantly shortened, the operation efficiency is improved, and the accuracy is high and the versatility is strong.

[0066] 4. The present invention provides a high-precision spacecraft pose estimation method based on deep learning. When establishing the initial model of key point regression, a hole convolution unit is used in the feature fusion process to expand the receptive field, which can capture the global information in the spacecraft image, effectively utilize the geometric connection relationship between the key points, and more accurately infer the pixel position of the occluded key points, which can improve the accuracy of key point regression in the occluded scene; an scSE module is added between the feature extraction module and the feature fusion module to introduce a hybrid attention mechanism, which complements the hole convolution. The hole convolution expands the receptive field to obtain global information, and the hybrid attention mechanism focuses on key local and global features, further improving the effectiveness of feature representation; and in the feature extraction process, the ShuffleNet-v2-Block unit is used to introduce a deep separable convolution, which greatly reduces the number of parameters and the amount of calculation. At the same time, the SE attention mechanism is used to retain a high expression ability in feature extraction, further optimize the lightweight of the high-resolution network, and achieve a balance between accuracy and efficiency;

[0067] 5. The present invention provides a high-precision spacecraft pose estimation method based on deep learning, which uses preprocessing methods such as adding random Gaussian white noise, adjusting brightness and contrast, and random flipping and rotation to perform image enhancement on the spacecraft data set, which can improve the sample type richness of the data set and enhance the generalization ability of the key point regression model. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of the principle of an embodiment of a method for high-precision spacecraft pose estimation based on deep learning of the present invention;

[0069] Figure 2 This is the structure diagram of the existing spacecraft key point regression network based on high-resolution network;

[0070] Figure 3 This is the structural diagram of the existing ShuffleNet-v2-Block unit;

[0071] Figure 4 This is a structural diagram of the ShuffleNet-v2-Block unit based on the SE attention mechanism used in step 3 of an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the scSE module operation used in step 3 of an embodiment of the present invention;

[0073] Figure 6 This is a flow chart of step 4 of an embodiment of the present invention;

[0074] Figure 7-10 Schematic diagram of the influence of the regularization term of the model loss function on the pose estimation result and the key point regression result in an embodiment of the present invention, Figure 7 middle h ( I ; θ )= θ , λ =0, Figure 8 middle h ( I ; θ )= θ , λ =1, Fig. 9 middle h ( I ; θ )= vgg ( I ) 、 λ =0, Fig.10 middle h ( I ; θ )= vgg ( I ), λ =1; (a) is the change curve of the loss function, (b) is the schematic diagram of the change of the two-dimensional projection of the key point under the predicted posture, and (c) is the schematic diagram of the change of the predicted two-dimensional coordinates of the key point;

[0075] Fig.11 The figure is a comparison diagram of predicted pose results obtained by using an embodiment of the present invention, wherein (a), (b), (c), (d), (e), (f), (g) and (h) are comparison diagrams of predicted pose results of target spacecraft images under different environments, respectively. DETAILED DESCRIPTION

[0076] The following is a further detailed description of a high-precision spacecraft pose estimation method based on deep learning proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0077] A high-precision spacecraft pose estimation method based on deep learning, such as Figure 1 As shown, the following steps are included:

[0078] Step 1: Select a spacecraft dataset and preprocess it to obtain an initial dataset; the spacecraft dataset consists of spacecraft images with pose labels. Then, use the pose labels to perform multi-viewing of the spacecraft based on the initial dataset. Figure 3 The three-dimensional reconstruction is performed to obtain the three-dimensional view of the spacecraft in the initial data set. The specific method of preprocessing is: adding random Gaussian white noise, adjusting brightness and contrast, and randomly flipping and rotating the spacecraft data set in sequence.

[0079] In this embodiment, the spacecraft dataset selects the SPEED+ dataset with pose labels. SPEED (Spacecraft Pose Estimation Dataset) is the first public spacecraft attitude estimation dataset released by Stanford University's Space Rendezvous and Docking Laboratory and the European Space Agency in 2019. In October 2021, a new generation of SPEED+ dataset was released. The SPEED+ dataset not only has up to 59,960 synthetic images, but also simulates light box reflected lighting and high-intensity direct sunlight through real lighting conditions (referred to as light box images and sun lamp images, respectively), thereby collecting a large number of real images. Compared with synthetic images, light box images and sun lamp images can capture space environments such as extreme lighting, stray light, and shadows that are not easy to synthesize by computer.

[0080] Step 2: Select key points on the spacecraft, then mark the real two-dimensional coordinates of the key points in each spacecraft image, and use them together with the pose labels as training labels to obtain a training data set with training labels.

[0081] Step 3: Establish a key point regression initial model based on the high-resolution network, initialize the model parameters, and then input the training data set into the key point regression initial model. The key point regression initial model is based on the high-resolution network, and the accuracy of key point regression is further improved by adding a hybrid attention mechanism in the feature extraction process and expanding the receptive field in the feature fusion process. The specific method for establishing the key point regression initial model is:

[0082] The Basicblock unit of the feature extraction module in each Stage structure of the high-resolution network is replaced with the ShuffleNet-v2-Block unit based on the SE (Squeeze-and-Excitation) attention mechanism, and the convolution unit of the feature fusion module in each Stage structure of the high-resolution network is replaced with a hole convolution unit, and then the scSE module (Concurrentspatial and channel Squeeze & Excitation, an attention module combining spatial and channel information) is added between the feature extraction module and the feature fusion module of each Stage structure of the high-resolution network. Among them, the ShuffleNet-v2-Block unit based on the SE attention mechanism includes an equal mapping channel and a depth convolution channel parallel to each other, and the depth convolution channel includes a first SE module, a 3×3 depth convolution module, and a second SE module connected in sequence.

[0083] In this embodiment, the key point regression initial model established in step 3 is based on Figure 2 The high-resolution network (HRNet) shown in the figure adopts the feature fusion strategy of parallel subnets to keep the features extracted by the model at high resolution throughout the whole process. The key point regression initial model of this embodiment reduces the number of parameters and the amount of calculation while ensuring the accuracy by the following methods:

[0084] (1) Replace the Basicblock unit of the feature extraction module in each Stage structure with the ShuffleNet-v2-Block unit based on the SE attention mechanism.

[0085] like Figure 3 As shown in Figure 1, the depth convolution channel of the existing ShuffleNet-v2-Block unit includes 1×1 convolution, 3×3 depth convolution and 1×1 convolution connected in sequence. In the 3×3 depth convolution, the depthwise separable convolution (DSC) is used instead of the ordinary convolution, which can greatly reduce the number of parameters and calculations.

[0086] If the input is D F ×D F ×M, the output is D F ×D F ×N, then N D K ×D K The amount of computation for a normal convolution with a convolution kernel of size D × M is K ×D K ×M×D F ×D F×N. And by M D K ×D K The computational cost of the depthwise convolution of a ×1 convolution kernel and the depthwise separable convolution consisting of N 1×1×M convolution kernels is D K ×D K ×M+D F ×D F + M×D F ×D F ×N. The ratio of the computational effort between the two is:

[0087]

[0088] It can be seen that using depth-wise separable convolution instead of ordinary convolution can greatly reduce the number of parameters and calculations.

[0089] If a 1×1 convolution with C channels has quadratic time complexity , 3×3 depthwise convolution has linear time complexity , then the complexity of two 1×1 convolutions is much higher than that of 3×3 depth convolution when the number of channels C>5. Therefore, the network parameters can be further reduced by replacing the 1×1 convolution in the ShuffleNet-v2-Block unit. For the ShuffleNet-v2-Block unit, the shuffle operation and the depth convolution do not exchange information across channels, and the 1×1 convolution plays a key role in exchanging information across channels. Since the role of the 1×1 convolution is to exchange information on the feature map in the channel dimension, the channel weighting method used by the SE attention mechanism is borrowed, such as Figure 4 As shown in the figure, the ShuffleNet-v2-Block unit based on the SE attention mechanism in this embodiment gives different weights to each channel in the feature map according to the feature information to highlight certain features and suppress other features, thereby replacing the role of 1×1 convolution. The computational complexity of channel weighting is linearly related to the number of channels, which is much lower than the complexity of 1×1 convolution.

[0090] (2) Replace the convolutional unit of the feature fusion module in each Stage structure with a dilated convolutional unit.

[0091] Compared with ordinary convolution, the dilated convolution defines the spacing between the values ​​during convolution kernel operation through the parameter of the dilation rate. For the same convolution kernel, the receptive field can be increased by increasing the dilation rate, so that the global information (such as the geometric connection relationship between key points) can be more effectively used to infer the pixel coordinates of the occluded key points. In this embodiment, the fusion of high-resolution features and low-resolution features is achieved by downsampling with a 3×3 convolution with a step size of 2. In order to increase the receptive field while downsampling the parallel subnet, this embodiment uses a dilated convolution with a dilation rate of 2 instead of the ordinary convolution in the downsampling process.

[0092] (3) Add the scSE module between the feature extraction module and the feature fusion module of each Stage structure.

[0093] The scSE module refers to a module with space and channel squeeze excitation, such as Figure 5 As shown in the figure, the spatial squeezing excitation module sSE integrates the multi-channel features back into the original features after dimensionality reduction; the channel squeezing excitation module cSE uses global pooling to reduce dimensionality, and uses channel weighting to give each channel of the original features a weight. The results of the spatial squeezing excitation module are added to the results of the channel squeezing excitation module to obtain the output of the final feature map. The scSE attention mechanism is introduced between the feature extraction stage and the feature fusion stage, which can make the fusion of features of different resolutions more efficient.

[0094] Step 4: The key point regression initial model processes a spacecraft image in the training data set and outputs the predicted two-dimensional coordinates of the key points in the spacecraft image. Figure 6 As shown, specifically:

[0095] Step 4.1: Extract features from a spacecraft image in the training data set to obtain an initial feature map.

[0096] Step 4.2: Convolve the initial feature map, add a scale branch, obtain the initial feature map of the two scale branches, and input it into the feature extraction module of the Stage structure.

[0097] Step 4.3: The feature extraction module uses the ShuffleNet-v2-Block unit based on the SE attention mechanism to extract features from the initial feature map of each scale branch, obtain the deep feature maps of the two scale branches, and input them into the scSE module. Figure 4 As shown, step 4.3 is specifically as follows:

[0098] 4.3.1. The ShuffleNet-v2-Block unit based on the SE attention mechanism performs channel separation on the initial feature maps of the two scale branches, and inputs the initial feature maps of each scale branch into the equal mapping channel and the deep convolution channel respectively. The equal mapping channel performs equal mapping on the initial feature maps of the two scale branches respectively, and obtains the mapping feature maps of the two scale branches;

[0099] 4.3.2. The first SE module of the deep convolution channel performs adaptive average pooling on the initial feature maps of the two scale branches, scales them to the minimum scale, and then splices them. Then, 1×1 convolution, ReLU activation, and Sigmoid activation are performed in sequence to obtain the weights corresponding to the two scale branches. Then, they are upsampled to the corresponding scale branches, and the weight maps corresponding to the initial feature maps of the two scale branches are obtained.

[0100] 4.3.3. According to the weight maps corresponding to the initial feature maps of the two scale branches, the initial weighted feature maps of the two scale branches are calculated by the following formula, and then sent to the 3×3 depth convolution module of the depth convolution channel:

[0101]

[0102] in, represents the initial weighted feature map of the h-th scale branch, 1≤h≤2, Represents the weight map corresponding to the initial feature map of the h-th scale branch, represents the initial feature map of the h-th scale branch, It means element-wise multiplication;

[0103] 4.3.4. The 3×3 deep convolution module of the deep convolution channel performs 3×3 deep convolution operations on the initial weighted feature maps of the two scale branches respectively to obtain the deep convolution feature maps of the two scale branches, and sends them to the second SE module of the deep convolution channel;

[0104] 4.3.5. The second SE module of the deep convolution channel respectively processes the deep convolution feature maps of the two scale branches through global average pooling, fully connected layer, ReLU activation, fully connected layer, Sigmoid activation and Ws in sequence to obtain the weight maps corresponding to the deep convolution feature maps of the two scale branches;

[0105] 4.3.6. According to the weight maps corresponding to the deep convolution feature maps of the two scale branches, the weight maps corresponding to the deep convolution feature maps of the two scale branches are element-wise multiplied with the deep convolution feature maps of the two scale branches to obtain the deep convolution weighted feature maps of the two scale branches;

[0106] 4.3.7. Concatenate the mapping feature maps of the two scale branches obtained in step 4.3.1 and the deep convolution weighted feature maps of the two scale branches obtained in step 4.3.6 respectively, and then perform channel mixing and channel shuffling in turn to obtain the deep feature maps of the two scale branches, and input them into the scSE module.

[0107] Step 4.4, the scSE module performs spatial information calibration and channel information calibration on the depth feature maps of the two scale branches respectively to obtain spatial information calibrated depth feature maps and channel information calibrated depth feature maps, and then adds the two together to obtain the calibrated depth feature maps of the two scale branches, and inputs them into the feature fusion module.

[0108] Step 4.5: The atrous convolution unit of the feature fusion module performs convolution processing on the calibrated depth feature maps of the two scale branches respectively, extracts their global information, obtains all scale branches of the calibrated depth feature maps of the two scale branches, and then fuses the calibrated depth feature maps of the corresponding scale branches respectively to obtain the fused feature maps of the two scale branches.

[0109] Step 4.6: Use the fused feature map of the two scale branches obtained in step 4.5 as the initial feature map, and then return to step 4.2 until the fused feature maps of four different scale branches are obtained.

[0110] Step 4.7: Fuse the fused feature maps of the four branches at different scales to obtain the feature map of the spacecraft image.

[0111] Step 4.8: Generate a key point prediction heat map based on the feature map of the spacecraft image to obtain the predicted two-dimensional coordinates of the key points in the spacecraft image.

[0112] Step 5: Based on the predicted two-dimensional coordinates of the key points in the spacecraft image and the three-dimensional view of the spacecraft, the predicted position and posture of the spacecraft in the spacecraft image are solved by the PnP solver according to the following formula:

[0113]

[0114] in, y represents the predicted position and posture of the spacecraft in the spacecraft image;

[0115] Indicates the first i The predicted 2D coordinates of the key points, i =1,2,…, n , n is the number of key points, n is an integer greater than 5, and in this embodiment, n=8;

[0116] , indicating the spacecraft i The three-dimensional coordinates of the key points are projected to the two-dimensional coordinates on the imaging plane through the predicted position of the spacecraft in the spacecraft image. For spacecraft i The three-dimensional coordinates of the key points are obtained from the three-dimensional view of the spacecraft; K represents the camera internal parameters, represents the square of the two-norm, π ( ) is the projection mapping function.

[0117] In this embodiment, the PnP solver uses the above-mentioned EPnP algorithm (Efficient Perspective-n-Point) to solve the posture, which has low computational complexity, can significantly shorten the calculation time, improve the operation efficiency, and has high accuracy and strong versatility.

[0118] Step 6: Calculate the loss function based on the predicted two-dimensional coordinates of the key points in the spacecraft image, the predicted position and training labels of the spacecraft in the spacecraft image. If the loss function converges, the key point regression model is obtained and step 8 is executed; otherwise, step 7 is executed. The loss function is calculated by the following formula:

[0119]

[0120]

[0121]

[0122] in, , represents the loss function, The three-dimensional coordinates of the key points of the spacecraft are respectively projected onto the imaging plane through the predicted pose of the spacecraft in the spacecraft image and the pose label of the spacecraft image, is the regularization term, λ is the regularization factor;

[0123] x Represents the predicted two-dimensional coordinates of key points in the spacecraft image;

[0124] The three-dimensional coordinates representing the key points of the spacecraft are projected onto the two-dimensional coordinates on the imaging plane through the predicted position of the spacecraft in the spacecraft image. z The three-dimensional coordinates of the key points of the spacecraft are obtained from the three-dimensional view of the spacecraft;

[0125] The three-dimensional coordinates representing the key points of a spacecraft are projected onto the two-dimensional coordinates on the imaging plane through the pose labels of the spacecraft image. Represents the pose label of the spacecraft image.

[0126] In the loss function, It is the sum of square errors of the unnormalized coordinates of the real three-dimensional coordinates of the key points of the spacecraft, which are projected on the imaging plane through the predicted pose of the spacecraft and the pose label of the spacecraft image. It is used to ensure that the predicted pose can converge to the real pose indicated by the pose label through training. The regularization term It ensures that the predicted two-dimensional coordinates of key points of spacecraft converge to the real two-dimensional coordinates.

[0127] Step 7: Based on the implicit function theorem, according to the predicted position and loss function of the spacecraft in the spacecraft image, calculate the gradient of the loss function with respect to the model parameters, and then use the gradient descent method to update the model parameters, return to step 4, and process the next spacecraft image in the training data set. Specifically:

[0128] Step 7.1, define the objective function of the PnP solver as:

[0129]

[0130] in, is the objective function of the PnP solver, , indicating the first i The predicted two-dimensional coordinates of the key points and the spacecraft i The three-dimensional coordinates of the key points are projected onto the difference between the two-dimensional coordinates of the imaging plane through the predicted position of the spacecraft in the spacecraft image.

[0131] Step 7.2, the objective function of the PnP solver is used to obtain the first-order derivative of the predicted position of the spacecraft in the spacecraft image, and the stability constraint is obtained as shown in the following formula:

[0132]

[0133] in, is the output of the PnP solver, Indicates the use of PnP solver to solve y , , represents the first predicted pose component of the spacecraft in the spacecraft image, Indicates the first spacecraft in the spacecraft image. m Predicted pose components, mRepresents the dimension of the predicted position and posture of the spacecraft in the spacecraft image.

[0134] Step 7.3: Construct the constraint function of the PnP solver based on the stability constraint:

[0135]

[0136] in:

[0137]

[0138] is the constraint function of the PnP solver, Indicates the first spacecraft in the spacecraft image. j The constraint function of the PnP solver corresponding to the predicted pose component;

[0139] Step 7.4, using the implicit function theorem, according to the constraint function of the PnP solver, obtain the Jacobian matrix of the predicted pose of the spacecraft in the spacecraft image with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image:

[0140]

[0141] in, A Jacobian matrix representing a predicted pose of a spacecraft in the spacecraft image with respect to predicted two-dimensional coordinates of key points in the spacecraft image.

[0142] Step 7.5: According to the loss function, the gradient of the loss function with respect to the predicted position of the spacecraft in the spacecraft image and the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image are calculated by the following formula:

[0143]

[0144]

[0145] in, represents the gradient of the loss function with respect to the predicted pose of the spacecraft in the spacecraft image, Represents the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image.

[0146] Step 7.6. Calculate the gradient of the loss function with respect to the model parameters using the following formula:

[0147]

[0148] in, represents the gradient of the loss function with respect to the model parameters, θare model parameters, Represents the gradient of the predicted two-dimensional coordinates of the key points in the spacecraft image with respect to the model parameters, represents the gradient of the regularization term with respect to the model parameters, and They are all automatically calculated by the key point regression initial model.

[0149] Step 7.7: Using the gradient descent method, update the model parameters using the following formula, then return to step 4 to process the next spacecraft image in the training dataset:

[0150]

[0151] in, are the updated model parameters, is the learning rate.

[0152] Step 8: Multi-view the target spacecraft Figure 3 dimensional reconstruction to obtain a three-dimensional view of the target spacecraft; then obtain the target spacecraft image and input it into the key point regression model to obtain the predicted two-dimensional coordinates of the key points of the target spacecraft.

[0153] Step 9: Based on the predicted two-dimensional coordinates of the key points of the target spacecraft and the three-dimensional view of the target spacecraft, the predicted position and pose of the target spacecraft is solved using the PnP solver to complete the position and pose estimation of the target spacecraft.

[0154] like Figure 7 , Figure 8 As shown, let the key points regress the output of the initial model , and then make the key point regression initial model output the predicted two-dimensional coordinates of the key points in the spacecraft image, where I is the input spacecraft image; Fig. 9 , Fig.10 As shown, , and then make the key point regression initial model output the predicted two-dimensional coordinates of the key points of the spacecraft in the spacecraft image, where vgg ( I ) refers to the output of the classic convolutional neural network VGG (Visual Geometry Group Network). Figure 7-10In the figure, (a) is the change curve of the loss function, (b) is a schematic diagram of the change of the two-dimensional projection of the key point in the predicted pose, the red square represents the two-dimensional projection of the key point in the actual pose, the black dot represents the two-dimensional projection of the key point in the initial predicted pose, and the green dot represents the two-dimensional projection of the key point in the predicted pose when the loss function converges; (c) is a schematic diagram of the result of the predicted two-dimensional coordinates of the key point, the red square is the actual position of the two-dimensional coordinates of the key point, the black dot represents the initial position of the predicted two-dimensional coordinates of the key point, and the green dot represents the position of the predicted two-dimensional coordinates of the key point when the loss function converges.

[0155] like Figure 7-10 As shown, whether in ,still , the loss function is successfully minimized, and the predicted pose can converge to the true pose, indicating the effectiveness of the loss function. λ =0 and λ =1, indicating that adding regularization terms can ensure the convergence of key point regression.

[0156] like Fig.11 As shown, a comparison chart of the results of target space spacecraft pose estimation for target space spacecraft images in different environments using the method of this embodiment is shown. The red arrow is the real pose, and the green arrow is the predicted pose. It can be seen that the method provided by this embodiment has good accuracy in different environments.

[0157] The high-precision spacecraft pose estimation method based on deep learning provided in this embodiment can evaluate its accuracy by calculating the rotation angle error and translation error of the predicted pose of the target spacecraft.

[0158] The rotation angle error is calculated by the following formula:

[0159]

[0160] in, represents the predicted position and rotation angle error of the target spacecraft, Represents the predicted position quaternion of the target spacecraft, The quaternion representing the true position of the spacecraft in the target space.

[0161] The translation error is calculated by the following formula:

[0162]

[0163] in, represents the predicted position and translation error of the target spacecraft, represents the predicted position translation vector of the target spacecraft, Represents the actual position translation vector of the target spacecraft.

[0164] Then the total error of the predicted position and attitude of the target spacecraft is:

[0165]

[0166] in, Represents the total error of the predicted position and attitude of the spacecraft in the target space.

[0167] In this embodiment, KRN (Keypoints Regression Network, a key point regression network from Tae Ha Park et al. published in the 2019 AAS / AIAA Astrodynamics Expert Conference) is first used.<TowardsRobust Learning-Based Pose Estimation of Non-Cooperative Spacecraft> ), HRNet and the key point regression model of this embodiment, and SPN and the key point regression model of this embodiment, respectively perform pose prediction on the synthetic images in the SPEED+ dataset, and obtain the predicted pose total errors as shown in Tables 1 and 2. It can be seen that the result of this embodiment using HRNet as the backbone for key point regression and assisted by the EPnP algorithm for pose estimation is greatly improved compared with KRN, and by back-propagating the EPnP algorithm, the accuracy of pose estimation is used to guide the update of the parameters of the initial model of key point regression, which can give the model more adjustment space, thereby further improving the accuracy of pose estimation.

[0168] Table 1 Comparison of pose errors predicted by synthetic images Table 1

[0169]

[0170] Table 2 Comparison of pose errors predicted by synthetic images Table 2

[0171]

[0172] Table 3 Comparison of pose errors of light box image prediction

[0173]

[0174] Table 4 Comparison of pose errors of sun lamp image prediction

[0175] Then, KRN, SPN and the key point regression model of this embodiment are used to estimate the pose of the light box image and the sun lamp image in the SPEED+ dataset, respectively, and the total predicted pose error is obtained as shown in Tables 3 and 4. The predicted pose error of this embodiment on the light box image and the sun lamp image is significantly lower than that of other models. It can be seen that in harsh lighting environments, this embodiment has obvious advantages over other models.

Claims

1. A high-precision spacecraft pose estimation method based on deep learning, characterized in that: The following steps are involved: Step 1: Select a spacecraft dataset and preprocess it to obtain an initial dataset, wherein the spacecraft dataset consists of spacecraft images with pose labels; then use the pose labels to perform multi-view 3D reconstruction of the spacecraft according to the initial dataset to obtain a 3D view of the spacecraft; Step 2: Select key points on the spacecraft, then mark the real two-dimensional coordinates of the key points in each spacecraft image, and use them together with the pose label as training labels to obtain a training data set with training labels; Step 3: Establish a key point regression initial model based on a high-resolution network, initialize the model parameters, and then input the training data set into the key point regression initial model; the specific method of establishing the key point regression initial model based on a high-resolution network is: The Basicblock unit of the feature extraction module in each Stage structure of the high-resolution network is replaced with the ShuffleNet-v2-Block unit based on the SE attention mechanism, and the convolution unit of the feature fusion module in each Stage structure of the high-resolution network is replaced with a hollow convolution unit, and then the scSE module is added between the feature extraction module and the feature fusion module of each Stage structure of the high-resolution network; The ShuffleNet-v2-Block unit based on the SE attention mechanism includes an equal mapping channel and a depth convolution channel parallel to each other, and the depth convolution channel includes a first SE module, a 3×3 depth convolution module and a second SE module connected in sequence; Step 4: The key point regression initial model processes a spacecraft image in the training data set and outputs the predicted two-dimensional coordinates of the key points in the spacecraft image; Step 5: using a PnP solver to solve the predicted position and posture of the spacecraft in the spacecraft image according to the predicted two-dimensional coordinates of the key points in the spacecraft image and the three-dimensional view of the spacecraft; Step 6: Calculate the loss function according to the predicted two-dimensional coordinates of the key points in the spacecraft image, the predicted position and training labels of the spacecraft in the spacecraft image. If the loss function converges, the key point regression model is obtained, and step 8 is executed. Otherwise, go to step 7; Step 7: Based on the implicit function theorem, according to the predicted position and loss function of the spacecraft in the spacecraft image, the gradient of the loss function with respect to the model parameters is calculated, and then the model parameters are updated using the gradient descent method, and the process returns to step 4 to process the next spacecraft image in the training data set; specifically: Step 7.1, define the objective function of the PnP solver; Step 7.2, taking the first-order derivative of the objective function of the PnP solver with respect to the predicted position and posture of the spacecraft in the spacecraft image, and obtaining a stationary constraint; Step 7.3, construct the constraint function of the PnP solver according to the stability constraint; Step 7.4, using the implicit function theorem and according to the constraint function of the PnP solver, obtain the Jacobian matrix of the predicted pose of the spacecraft in the spacecraft image with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image; Step 7.5, according to the loss function, calculate the gradient of the loss function with respect to the predicted position of the spacecraft in the spacecraft image and the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image by the following formula; Step 7.6, calculating the gradient of the loss function with respect to the model parameters according to the gradient of the loss function with respect to the predicted position of the spacecraft in the spacecraft image and the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image; Step 7.7, update the model parameters using the gradient descent method according to the gradient of the loss function with respect to the model parameters, and then return to step 4 to process the next spacecraft image in the training data set; Step 8: Perform multi-view 3D reconstruction on the target spacecraft to obtain a 3D view of the target spacecraft; Then, the target spacecraft image is acquired and input into the key point regression model to obtain the predicted two-dimensional coordinates of the key points of the target spacecraft; Step 9: Based on the predicted two-dimensional coordinates of the key points of the target spacecraft and the three-dimensional view of the target spacecraft, the predicted position and pose of the target spacecraft is solved using the PnP solver to complete the position and pose estimation of the target spacecraft.

2. According to the high-precision spacecraft pose estimation method based on deep learning in claim 1, it is characterized in that: In step 6, the loss function is calculated by the following formula: l(x,y)=E(x,y)+λR(x,y) Among them, l(x,y)=l, represents the loss function, E(x,y) represents the error between the three-dimensional coordinates of the key points of the spacecraft and the projection of the predicted pose of the spacecraft in the spacecraft image and the pose label of the spacecraft image onto the imaging plane, R(x,y)=h is the regularization term, and λ is the regularization factor; x represents the predicted two-dimensional coordinates of the key points in the spacecraft image, and y represents the predicted position and posture of the spacecraft in the spacecraft image; π(z|y,K) represents the three-dimensional coordinates of the key points of the spacecraft projected onto the two-dimensional coordinates of the imaging plane through the predicted pose of the spacecraft in the spacecraft image, z represents the three-dimensional coordinates of the key points of the spacecraft, obtained from the three-dimensional view of the spacecraft, K represents the camera internal parameters, represents the square of the two-norm, π() is the projection mapping function; π(z|y * , K) represents the three-dimensional coordinates of the key points of the spacecraft projected onto the two-dimensional coordinates on the imaging plane through the pose label of the spacecraft image, y * Represents the pose label of the spacecraft image.

3. The high-precision spacecraft pose estimation method based on deep learning according to claim 2 is characterized in that: Step 7 is as follows: Step 7.1, define the objective function of the PnP solver as: Among them, o(x,y,z,K) is the objective function of the PnP solver, r i =x i -π i , represents the difference between the predicted two-dimensional coordinates of the i-th key point in the spacecraft image and the three-dimensional coordinates of the i-th key point of the spacecraft projected onto the imaging plane through the predicted posture of the spacecraft in the spacecraft image, i = 1, 2, ..., n, n is the number of key points; x i represents the predicted two-dimensional coordinates of the i-th key point in the spacecraft image; π i =π(z i |y,K), represents the three-dimensional coordinates of the i-th key point of the spacecraft projected onto the two-dimensional coordinates of the imaging plane through the predicted position of the spacecraft in the spacecraft image, z i is the three-dimensional coordinate of the i-th key point of the spacecraft; Step 7.2, the objective function of the PnP solver is used to obtain the first-order derivative of the predicted position of the spacecraft in the spacecraft image, and the stability constraint is obtained as shown in the following formula: Where g(x,z,K) is the output of the PnP solver, y=g(x,z,K) means using the PnP solver to solve y, y=[y1,…,y m ] T , y1 represents the first predicted pose component of the spacecraft in the spacecraft image, y m represents the mth predicted pose component of the spacecraft in the spacecraft image, and m represents the dimension of the predicted pose of the spacecraft in the spacecraft image; Step 7.3: Construct the constraint function of the PnP solver based on the stability constraint: f=f(x,y,z,K)=[f1,…,f m ] T in: f=f(x,y,z,K) is the constraint function of the PnP solver, f j represents the constraint function of the PnP solver corresponding to the j-th predicted pose component of the spacecraft in the spacecraft image; Step 7.4, using the implicit function theorem, according to the constraint function of the PnP solver, obtain the Jacobian matrix of the predicted pose of the spacecraft in the spacecraft image with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image: in, A Jacobian matrix representing a predicted pose of a spacecraft in the spacecraft image with respect to predicted two-dimensional coordinates of key points in the spacecraft image; Step 7.5: According to the loss function, the gradient of the loss function with respect to the predicted position of the spacecraft in the spacecraft image and the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image are calculated by the following formula: in, represents the gradient of the loss function with respect to the predicted pose of the spacecraft in the spacecraft image, represents the gradient of the loss function with respect to the predicted two-dimensional coordinates of the key points in the spacecraft image; Step 7.

6. Calculate the gradient of the loss function with respect to the model parameters using the following formula: in, represents the gradient of the loss function with respect to the model parameters, θ is the model parameter, Represents the gradient of the predicted two-dimensional coordinates of the key points in the spacecraft image with respect to the model parameters, represents the gradient of the regularization term with respect to the model parameters, and They are all automatically calculated by the key point regression initial model; Step 7.7: Use the gradient descent method to update the model parameters using the following formula, then return to step 4 to process the next spacecraft image in the training dataset: Among them, θ0 is the updated model parameter and α is the learning rate.

4. The high-precision spacecraft pose estimation method based on deep learning according to claim 3 is characterized in that: In step 5, the predicted position and posture of the spacecraft in the spacecraft image is solved by the following formula:

5. A high-precision spacecraft pose estimation method based on deep learning according to any one of claims 1 to 4, characterized in that: In step 1, the specific method of the pretreatment is: The spacecraft dataset is subjected to random Gaussian white noise addition, brightness and contrast adjustment, random flipping and rotation.

6. The high-precision spacecraft pose estimation method based on deep learning according to claim 5, characterized in that: In step 1, the spacecraft dataset is a SPEED+ dataset.

Images