Spacecraft monocular six-degree-of-freedom attitude estimation method and device

Abstract

The present invention provides a method and device for estimating the six-degree-of-freedom attitude of a spacecraft monocular, the method comprising: inputting a spacecraft image into a trained MKPNet model to obtain an attitude pseudo-label; smoothing the attitude pseudo-label based on the motion consistency between multiple consecutive frames of images to obtain a smoothed attitude label; projecting the 3D key points of the optimized spacecraft 3D model onto the spacecraft image according to the smoothed attitude label to obtain a 2D key point pseudo-label, a heat map pseudo-label and a bounding box pseudo-label corresponding to the spacecraft image; fine-tuning the trained MKPNet model based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed attitude label corresponding to the spacecraft image; inputting the spacecraft image into the fine-tuned MKPNet model to obtain an attitude estimation corresponding to the spacecraft image. The present invention improves the accuracy of monocular 6-DoF attitude estimation of spacecraft.

Description

Technical Field

[0001] The present invention relates to the technical field of spacecraft attitude estimation, and in particular to a method and device for estimating the attitude of a spacecraft monocular with six degrees of freedom. Background Art

[0002] Monocular six degrees of freedom (6-DoF) attitude estimation is crucial for spacecraft to achieve precise navigation and positioning. It helps to obtain the position and rotation information of the spacecraft in real time, so that the spacecraft can autonomously adjust its attitude in different space environments and improve the efficiency of mission execution.

[0003] At present, most methods for monocular 6-DoF attitude estimation of spacecraft mainly use single-frame images to estimate the attitude information of the spacecraft, and directly migrate the model from synthetic data to real data, which results in insufficient accuracy of attitude estimation. Summary of the invention

[0004] The present invention provides a method and device for estimating the six-degree-of-freedom attitude of a spacecraft monocular, so as to solve the defect that the accuracy of the spacecraft monocular 6-DoF attitude estimation in the prior art is not high enough, and improve the accuracy of the spacecraft monocular 6-DoF attitude estimation.

[0005] The present invention provides a spacecraft monocular six-degree-of-freedom attitude estimation method, comprising:

[0006] Input multiple spacecraft images into the trained MKPNet model to obtain the pose pseudo-label corresponding to each spacecraft image;

[0007] Based on the motion consistency between multiple consecutive frames of images, the attitude pseudo-label corresponding to each of the spacecraft images is smoothed to obtain a smoothed attitude label corresponding to each of the spacecraft images;

[0008] According to the smoothed posture label corresponding to each of the spacecraft images, the 3D key points of the optimized spacecraft 3D model are projected onto each of the spacecraft images to obtain the 2D key point pseudo-labels, heat map pseudo-labels and bounding box pseudo-labels corresponding to each of the spacecraft images;

[0009] Based on the 2D key point pseudo-labels, the heat map pseudo-labels, the bounding box pseudo-labels, and the smoothed posture labels corresponding to each of the spacecraft images, fine-tuning the trained MKPNet model to obtain a fine-tuned MKPNet model;

[0010] The multiple spacecraft images are input into the fine-tuned MKPNet model to obtain a posture estimation corresponding to each of the spacecraft images.

[0011] In some embodiments, based on the motion consistency between the continuous multiple frames of pictures, smoothing the attitude pseudo-label corresponding to each of the spacecraft pictures to obtain the smoothed attitude label corresponding to each of the spacecraft pictures includes:

[0012] Constructing multiple sets of data pairs based on the attitude pseudo-label and time series number corresponding to each of the spacecraft images;

[0013] For each data pair, the following steps are performed:

[0014] Fitting the current group of data pairs using the least squares method to obtain an initial smooth fitting curve corresponding to the current group of data pairs;

[0015] According to the distance threshold and the distance between each data pair in the current group and the initial smooth fitting curve corresponding to the current group data pair, the initial smooth fitting curve corresponding to the current group data pair is corrected to obtain the smooth fitting curve corresponding to the current group data pair;

[0016] Obtaining a current group of smoothed data pairs according to a smoothed fitting curve corresponding to the current group of data pairs;

[0017] According to each set of smoothed data pairs, a smoothed attitude label corresponding to each of the spacecraft images is obtained.

[0018] In some embodiments, the method further comprises:

[0019] The loss function corresponding to the predicted heat map and the true heat map, the loss function corresponding to the predicted bounding box and the true bounding box, and the loss function corresponding to the predicted posture and the true posture are added together as the total loss function for training the MKPNet model.

[0020] In some embodiments, the method further comprises:

[0021] Using the feature extraction network in the MKPNet model to extract features from the spacecraft training images in the training set;

[0022] Using the heat map prediction head in the MKPNet model to perform heat map prediction on the extracted features to obtain the predicted heat map;

[0023] Using the bounding box prediction head in the MKPNet model to perform bounding box prediction on the extracted features to obtain the predicted bounding box;

[0024] The predicted posture is obtained based on the 2D key points corresponding to the filtered predicted heat map; the filtered predicted heat map is obtained by filtering the predicted heat map based on a filtering threshold.

[0025] In some embodiments, the method further comprises:

[0026] Selecting spacecraft training images of spacecraft in multiple different trajectory sequences from the training set as training subsets;

[0027] Based on the manually annotated 2D key points on each spacecraft training image in the training subset and the correspondence between the 2D key points and the 3D key points on the spacecraft, an initial spacecraft 3D model is obtained;

[0028] Based on the real pose information and camera intrinsic parameters, the initial 3D key points of the initial spacecraft 3D model are projected onto each spacecraft training picture in the training set to obtain the real 2D key points of each spacecraft training picture in the training set;

[0029] Based on the real 2D key points of each spacecraft training picture in the training set, the real heat map and the real bounding box of each spacecraft training picture in the training set are obtained.

[0030] In some embodiments, the method further comprises:

[0031] Iterate through the following steps:

[0032] Use the MKPNet model to predict 2D key points of the spacecraft training images in the training set to obtain predicted 2D key points of the spacecraft training images in the training set;

[0033] Based on the predicted 2D key points of the spacecraft training pictures in the training set and the real pose labels of the spacecraft training pictures in the training set, the initial spacecraft 3D model is optimized to obtain the optimized spacecraft 3D model;

[0034] Based on the optimized spacecraft 3D model, obtaining optimized 2D key points of the spacecraft training images in the training set;

[0035] The MKPNet model is retrained based on the optimized 2D key points of the spacecraft training images in the training set.

[0036] The present invention also provides a spacecraft monocular six-degree-of-freedom attitude estimation device, comprising:

[0037] An input module is used to input multiple spacecraft images into the trained MKPNet model to obtain a posture pseudo-label corresponding to each spacecraft image;

[0038] A smoothing module, used for smoothing the attitude pseudo-label corresponding to each of the spacecraft images based on the motion consistency between multiple frames of images, to obtain a smoothed attitude label corresponding to each of the spacecraft images;

[0039] A projection module is used to project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images according to the smoothed posture labels corresponding to each of the spacecraft images, so as to obtain 2D key point pseudo labels, heat map pseudo labels and bounding box pseudo labels corresponding to each of the spacecraft images;

[0040] A fine-tuning module, configured to fine-tune the trained MKPNet model based on the 2D key point pseudo-labels, the heat map pseudo-labels, the bounding box pseudo-labels and the smoothed posture labels corresponding to each of the spacecraft images, to obtain a fine-tuned MKPNet model;

[0041] The estimation module is used to input the multiple spacecraft images into the fine-tuned MKPNet model to obtain the attitude estimation corresponding to each of the spacecraft images.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for estimating a spacecraft monocular six-degree-of-freedom attitude as described above is implemented.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for estimating the spacecraft monocular six-degree-of-freedom attitude as described in any one of the above is implemented.

[0044] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for estimating the monocular six-degree-of-freedom attitude of a spacecraft.

[0045] The present invention provides a method and device for estimating the six-degree-of-freedom attitude of a spacecraft monocular. First, a spacecraft image is input into a trained MKPNet model to obtain an attitude pseudo-label. The attitude pseudo-label is smoothed based on the motion consistency between multiple consecutive frames of images to obtain a smoothed attitude label, thereby improving the accuracy and robustness of the attitude label. Then, according to the smoothed attitude label, 3D key points of an optimized spacecraft 3D model are projected onto the spacecraft image to obtain 2D key point pseudo-labels, heat map pseudo-labels and bounding box pseudo-labels corresponding to the spacecraft image. Based on the 2D key point pseudo-labels, heat map pseudo-labels, bounding box pseudo-labels and smoothed attitude labels corresponding to the spacecraft image, the trained MKPNet model is fine-tuned to improve the adaptability and extensibility of the MKPNet model. Finally, the spacecraft image is input into the fine-tuned MKPNet model to obtain an attitude estimation corresponding to the spacecraft image, thereby improving the accuracy of the spacecraft monocular 6-DoF attitude estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 This is one of the flow charts of the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention;

[0048] Figure 2 This is the second flow chart of the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention;

[0049] Figure 3 It is a structural schematic diagram of a spacecraft monocular six-degree-of-freedom attitude estimation device provided by the present invention;

[0050] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Figure 1FIG. 1 is a flow chart of a method for estimating a spacecraft monocular six-degree-of-freedom attitude provided by the present invention. Figure 1 As shown, the present invention provides a spacecraft monocular six-degree-of-freedom attitude estimation method, comprising the following steps:

[0053] Step 110, multiple spacecraft images are input into the trained MKPNet model to obtain the posture pseudo-label corresponding to each spacecraft image.

[0054] Specifically, the Multi-task Key-point Prediction Network (MKPNet) model is a multi-task model that includes predicting the key points of a spacecraft, predicting the bounding box of a spacecraft, and predicting the attitude estimation of a spacecraft. The MKPNet model includes a multi-scale shared feature encoder and multiple prediction heads that perform different tasks. The MKPNet model is mainly used to learn key points, and improves the accuracy of spacecraft attitude estimation by improving the prediction accuracy of key points.

[0055] Since it is difficult and rare to obtain spacecraft images in real scenes, the MKPNet model is trained using a synthetic simulation data set (i.e., a training set consisting of spacecraft training images). The input of the model application process is real data (i.e., real spacecraft images). There are differences between real data and synthetic simulation data. Directly applying the trained MKPNet model to real data will lead to a decrease in model performance.

[0056] Therefore, inputting multiple spacecraft images in real scenes into the trained MKPNet model cannot obtain accurate pose estimation, and the initial pose estimation is regarded as the pose pseudo-label corresponding to each spacecraft image.

[0057] Step 120 , based on the motion consistency between multiple consecutive frames of images, the attitude pseudo-label corresponding to each spacecraft image is smoothed to obtain a smoothed attitude label corresponding to each spacecraft image.

[0058] Specifically, there is a correlation between sequential spacecraft images. Based on the motion consistency between multiple consecutive frames of images, the time information of the sequential spacecraft images is fully utilized to smooth the attitude pseudo-label corresponding to each spacecraft image. The smoothed attitude label corresponding to each spacecraft image is obtained to improve the attitude accuracy.

[0059] Step 130, based on the smoothed posture label corresponding to each spacecraft image, the 3D key points of the optimized spacecraft 3D model are projected onto each of the spacecraft images to obtain the 2D key point pseudo-labels, heat map pseudo-labels and bounding box pseudo-labels corresponding to each spacecraft image.

[0060] Specifically, after obtaining the smoothed pose label corresponding to each spacecraft image, for any spacecraft image, according to the corresponding smoothed pose label, the 3D key points of the optimized spacecraft 3D model are projected onto each spacecraft image to obtain the 2D key point pseudo label corresponding to each spacecraft image. Then, according to the 2D key point pseudo label corresponding to each spacecraft image, the heat map pseudo label and bounding box pseudo label corresponding to each spacecraft image are obtained.

[0061] Step 140, based on the 2D key point pseudo-labels, heat map pseudo-labels, bounding box pseudo-labels and smoothed posture labels corresponding to each spacecraft image, fine-tune the trained MKPNet model to obtain a fine-tuned MKPNet model.

[0062] Specifically, the 2D key point pseudo-labels, heat map pseudo-labels, bounding box pseudo-labels and smoothed pose labels corresponding to each spacecraft image are input into the trained MKPNet model, and the trained MKPNet model is fine-tuned so that the fine-tuned MKPNet model can adapt to the spacecraft images in the model application process.

[0063] Step 150 , multiple spacecraft images are input into the fine-tuned MKPNet model to obtain a pose estimate corresponding to each spacecraft image.

[0064] Specifically, after obtaining the fine-tuned MKPNet model, the spacecraft image is input into the fine-tuned MKPNet model to obtain the attitude estimation corresponding to each spacecraft image.

[0065] The present invention provides a method for estimating the six-degree-of-freedom attitude of a spacecraft monocular. First, a spacecraft image is input into a trained MKPNet model to obtain an attitude pseudo-label. The attitude pseudo-label is smoothed based on the motion consistency between multiple consecutive frames of images to obtain a smoothed attitude label, thereby improving the accuracy and robustness of the attitude label. Then, according to the smoothed attitude label, 3D key points of an optimized spacecraft 3D model are projected onto each spacecraft image to obtain a 2D key point pseudo-label, a heat map pseudo-label and a bounding box pseudo-label corresponding to the spacecraft image. Based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed attitude label corresponding to the spacecraft image, the trained MKPNet model is fine-tuned to improve the adaptability and extensibility of the MKPNet model. Finally, the spacecraft image is input into the fine-tuned MKPNet model to obtain an attitude estimation corresponding to the spacecraft image, thereby improving the accuracy of the spacecraft monocular 6-DoF attitude estimation.

[0066] In some embodiments, based on the motion consistency between multiple consecutive frames of images, the attitude pseudo-label corresponding to each spacecraft image is smoothed to obtain a smoothed attitude label corresponding to each spacecraft image, including:

[0067] Construct multiple sets of data pairs based on the attitude pseudo-label and time series number corresponding to each spacecraft image;

[0068] For each data pair, the following steps are performed:

[0069] The least square method is used to fit the current group of data pairs to obtain the initial smooth fitting curve corresponding to the current group of data pairs;

[0070] According to the distance threshold and the distance between each data pair in the current group and the initial smooth fitting curve corresponding to the current group data pair, the initial smooth fitting curve corresponding to the current group data pair is corrected to obtain the smooth fitting curve corresponding to the current group data pair;

[0071] According to the smooth fitting curve corresponding to the current group of data pairs, the current group of smooth data pairs is obtained;

[0072] According to each set of smoothed data pairs, the smoothed attitude label corresponding to each spacecraft image is obtained.

[0073] Specifically, the posture information includes the rotation amount R and the translation amount t. In order to better describe the change of the rotation amount R over time, the rotation amount R is converted into Euler angles ,in, represents the roll angle, represents the pitch angle, represents the yaw angle, Represents the Euler angle, and the translation t is converted into the translation component ,in, express The translational component of the direction, express The translational component of the direction, express The translation component of the direction.

[0074] According to the time series number j corresponding to each spacecraft image, the attitude pseudo-label corresponding to each spacecraft image can include ,in, Indicates the pseudo roll angle corresponding to the spacecraft image with time series number j, Indicates the pseudo pitch angle corresponding to the spacecraft image with time series number j, Indicates the pseudo yaw angle corresponding to the spacecraft image with time series number j, Indicates the spacecraft image with time sequence number j corresponding to The pseudo-translation component of the direction, Indicates the spacecraft image with time sequence number j corresponding to The pseudo-translation component of the direction, Indicates the spacecraft image with time sequence number j corresponding to The pseudo-translation component of the direction.

[0075] According to the time series number j and the translation component , can form three sets of data pairs . Due to the rotational component , and There is a strong correlation between them, and two sets of data pairs can be formed according to the rotation component and , that is, a total of five sets of data pairs.

[0076] In order to obtain the smoothed data pairs corresponding to each set of data pairs, for any set of data pairs, the following steps are performed:

[0077] First, the least squares method is used to calculate the current group data pair (in represents the number of data pairs contained in the current group) to obtain the initial smooth fitting curve corresponding to the current group data pair .

[0078] The expression of the smooth fitting function is as follows:

[0079]

[0080] In the formula, represents the output of the smooth fitting function, represents a constant term, represents the fitting coefficients under Fourier sine basis, , represents the fitting coefficients under Fourier cosine basis, , represents the index number of the Fourier basis, represents the input of the smooth fitting function, Indicates that during the fitting process, , and Parameters that need to be estimated.

[0081] Then, for each data pair in the current group , determine the relationship between each data pair in the current group and the initial smooth fitting curve The distance between , detection Whether the distance threshold is exceeded If it exceeds, the data Identify the abnormal data pairs and remove them. Traverse all the data pairs in the current group, remove all the abnormal data pairs contained in the current group, and obtain the smooth fitting curve corresponding to the data pairs in the current group based on the remaining data pairs in the current group. .

[0082] Since there are five groups of data pairs, five smooth fitting curves can be obtained, which are The smooth fitting curve corresponding to the data pair ,and The smooth fitting curve corresponding to the data pair ,and The smooth fitting curve corresponding to the data pair ,and The smooth fitting curve corresponding to the data pair and The smooth fitting curve corresponding to the data pair .

[0083] Finally, the obtained smooth fitting curve is applied to all data pairs in the current group, that is, Replace with , thus obtaining the current group of smoothed data pairs.

[0084] After obtaining the smoothed data pairs corresponding to each set of data pairs, the smoothed attitude labels corresponding to each spacecraft image can be obtained according to each set of smoothed data pairs. The expression of the smoothed attitude label is as follows:

[0085]

[0086] In the formula, Indicates the smoothed Euler angle corresponding to the spacecraft image with time series number j, Indicates the smoothed translation corresponding to the spacecraft image with time series number j, represents a smooth function, Indicates the pseudo-Euler angle corresponding to the spacecraft image with time series number j, Indicates the pseudo translation corresponding to the spacecraft image with time series number j, Indicates the spacecraft image with time sequence number j corresponding to The translational component of the direction, Indicates that when the input is j, the smooth fitting curve The output, Indicates the spacecraft image with time sequence number j corresponding to The translational component of the direction, Indicates that when the input is j, the smooth fitting curve The output, Indicates the spacecraft image with time sequence number j corresponding to The translational component of the direction, Indicates that when the input is j, the smooth fitting curve The output, Indicates the roll angle corresponding to the spacecraft image with time series number j, Indicates the pseudo roll angle corresponding to the spacecraft image with time series number j, Indicates the pitch angle corresponding to the spacecraft image with time series number j, Indicates that the input is When the smooth fitting curve The output, Indicates the yaw angle corresponding to the spacecraft image with time series number j, Indicates that the input is When the smooth fitting curve Output.

[0087] The spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention obtains an initial smooth fitting curve by fitting data pairs, then corrects the initial smooth fitting curve by removing outliers, and finally applies the smooth fitting curve to obtain a smoothed attitude label, thereby further improving the accuracy of the smoothed attitude label.

[0088] In some embodiments, the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention further includes:

[0089] The loss function corresponding to the predicted heat map and the true heat map, the loss function corresponding to the predicted bounding box and the true bounding box, and the loss function corresponding to the predicted pose and the true pose are added together as the total loss function for training the MKPNet model.

[0090] Specifically, during training, the loss between the predicted heatmap and the true heatmap is the mean square error, expressed as follows:

[0091]

[0092] In the formula, represents the real heat map, represents the prediction heat map, represents the loss between the predicted heat map and the true heat map, Indicates The real heat map corresponding to the 2D key points, Indicates The predicted heatmap corresponding to the 2D key points, Indicates the number of 2D keypoints.

[0093] During training, the loss between the predicted bounding box and the true bounding box is calculated by the Intersection over Union (IoU) loss, which is expressed as follows:

[0094]

[0095] In the formula, represents the ground-truth bounding box, represents the predicted bounding box, represents the loss between the predicted bounding box and the true bounding box, represents the intersection-over-union loss function.

[0096] Predicted pose includes predicted rotation and predicted translation , the function Q() can predict the rotation amount Convert to predicted quaternion, i.e. , Represents the predicted quaternion. Therefore, the expression of the loss between the predicted pose and the true pose is as follows:

[0097]

[0098] In the formula, represents the loss between the predicted pose and the true pose, represents a real quaternion, represents the true translation vector, represents the predicted quaternion, represents the predicted translation, Represents dot product.

[0099] Adding the above three loss functions together, we get the total loss function for training the MKPNet model, which is expressed as follows:

[0100]

[0101] In the formula, represents the total loss function for training the MKPNet model, Represents the loss function corresponding to the predicted heat map and the true heat map, represents the loss function corresponding to the predicted bounding box and the true bounding box, Represents the loss function corresponding to the predicted pose and the true pose.

[0102] The spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention realizes the construction of a multi-task MKPNet model by adding the loss function corresponding to the predicted heat map and the real heat map, the loss function corresponding to the predicted bounding box and the real bounding box, and the loss function corresponding to the predicted attitude and the real attitude as the total loss function for training the MKPNet model.

[0103] In some embodiments, the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention further includes:

[0104] The feature extraction network in the MKPNet model is used to extract features from the spacecraft training images in the training set;

[0105] The heat map prediction head in the MKPNet model is used to perform heat map prediction on the extracted features to obtain the predicted heat map;

[0106] The bounding box prediction head in the MKPNet model is used to predict the bounding box of the extracted features to obtain the predicted bounding box;

[0107] Based on the 2D key points corresponding to the filtered prediction heat map, the predicted posture is obtained; the filtered prediction heat map is obtained by filtering the prediction heat map based on the filtering threshold.

[0108] Specifically, a large number of spacecraft training images are obtained to construct a training set for training the MKPNet model.

[0109] Use the feature extraction network F in the MKPNet model to extract features from each spacecraft training image in the training set, and then use the heat map prediction head H in the MKPNet model to perform heat map prediction on the extracted features to obtain the predicted heat map , where x represents the spacecraft training image to predict the 2D key points .

[0110] After using the feature extraction network F to extract features from each spacecraft training image in the training set, the bounding box prediction head B in the MKPNet model can also be used to perform bounding box prediction on the extracted features to obtain the predicted bounding box , where x represents the spacecraft training image.

[0111] First use the filter threshold Prediction heat map Filter by forming an index set To select 2D key points with high confidence. In order to ensure that the EPnP algorithm can accurately estimate the attitude of the spacecraft, the threshold (i.e. ), until at least 6 key points are retained ( ). The spacecraft attitude is estimated by the EPnP algorithm as follows:

[0112]

[0113] In the formula, represents the predicted rotation amount, represents the predicted translation, represents the projection scaling factor corresponding to the spacecraft training image, represents the predicted 2D key points, represents the camera internal parameter, R represents the rotation amount, Indicates 3D key points, Indicates the amount of translation.

[0114] The spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention is beneficial to the training of the MKPNet model by obtaining a predicted heat map, a predicted bounding box and a predicted attitude.

[0115] In some embodiments, the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention further includes:

[0116] Selecting spacecraft training images in which the spacecraft is in multiple different trajectory sequences from the training set as training subsets;

[0117] Based on the manually annotated 2D key points on each spacecraft training image in the training subset, and the correspondence between the 2D key points and 3D key points on the spacecraft, the initial spacecraft 3D model is obtained;

[0118] Based on the real pose information and camera intrinsic parameters, the initial 3D key points of the initial spacecraft 3D model are projected onto each spacecraft training image in the training set to obtain the real 2D key points of each spacecraft training image in the training set.

[0119] Based on the real 2D key points of each spacecraft training image in the training set, the real heat map and real bounding box of each spacecraft training image in the training set are obtained.

[0120] Specifically, a small number of spacecraft training images of spacecraft in multiple different trajectory sequences are selected from the training set as training subsets. Spacecraft training images of different trajectory sequences can provide spacecraft from multiple perspectives. Using spacecraft training images of different trajectory sequences, the initial spacecraft 3D model can be better estimated, thereby improving the estimation accuracy of the initial spacecraft 3D model, and finally improving the accuracy of attitude estimation.

[0121] Manually annotate the spacecraft training images in the training subset with 2D key points. The number of manually annotated 2D key points on each spacecraft training image is at least 6. For example, 11 2D key points are manually annotated. Based on the known attitude information and camera intrinsics , the correspondence between 2D key points and 3D key points is established.

[0122] Based on the manually annotated 2D key points on the spacecraft training images in the training subset and the correspondence between the 2D key points and 3D key points on the spacecraft, the initial 3D key points of the spacecraft are obtained. The initial 3D key points of the spacecraft are connected to obtain the initial spacecraft 3D model. The expression of the initial 3D key points is as follows:

[0123]

[0124] In the formula, Indicates 3D key points, , represents the number of 2D key points, , Indicates A training image of a spacecraft with manually labeled 2D key points, , Represents the number of spacecraft training images with manually annotated 2D key points, Indicates The first of the spacecraft training images with manually annotated 2D key points 2D key points, represents the camera internal parameters, Indicates The rotation corresponding to the spacecraft training image with manually annotated 2D key points, Indicates The translation corresponding to the spacecraft training image with manually annotated 2D key points, Indicates The projection scaling factor corresponding to the spacecraft training images with manually annotated 2D key points, .

[0125] Based on real pose information and camera intrinsics , the initial 3D key points of the initial spacecraft 3D model Project it onto the 2D image (i.e., each spacecraft training image in the training set) to obtain the true 2D key points of each spacecraft training image in the training set , the expression is as follows:

[0126]

[0127] In the formula, Indicates the image projection scaling factor, Indicates real 2D key points, represents the camera internal parameters, represents the amount of rotation, Indicates 3D key points, represents the number of 2D key points, Indicates the amount of translation.

[0128] Generate a real heat map of each spacecraft training image in the training set based on the real 2D key points of each spacecraft training image in the training set . Determine the true bounding box of each spacecraft training image in the training set based on the true 2D key points of each spacecraft training image in the training set :

[0129]

[0130] In the formula, and Represents the x-coordinate and y-coordinate of the 2D key point respectively.

[0131] The spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention is beneficial to the training of the MKPNet model by obtaining a real heat map, a real bounding box and a real attitude.

[0132] In some embodiments, Figure 2 FIG. 2 is a flow chart of the method for estimating the attitude of a spacecraft monocular six degrees of freedom provided by the present invention. Figure 2 As shown, the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention also includes:

[0133] Iterate through the following steps:

[0134] Step 210, using the MKPNet model to predict 2D key points of the spacecraft training images in the training set, to obtain predicted 2D key points of the spacecraft training images in the training set;

[0135] Step 220, optimizing the initial spacecraft 3D model based on the predicted 2D key points and true pose labels of the spacecraft training images in the training set to obtain an optimized spacecraft 3D model;

[0136] Step 230, based on the optimized spacecraft 3D model, obtain optimized 2D key points of the spacecraft training images in the training set;

[0137] Step 240, retrain the MKPNet model based on the optimized 2D key points of the spacecraft training images in the training set.

[0138] Specifically, since the initial spacecraft 3D model is obtained from a limited number of manually labeled spacecraft training images, which will lead to significant errors in the estimation of 3D key points, in order to make full use of the spacecraft training images and reduce the estimation error, the initial spacecraft 3D model is optimized.

[0139] Several spacecraft training images are randomly extracted from the training set, and the MKPNet model is used to predict the 2D key points of each extracted spacecraft training image to obtain the predicted 2D key points of the spacecraft training image.

[0140] From the formula of the spacecraft attitude estimated by the EPnP algorithm, it can be seen that when the predicted 2D key points and the true attitude labels of the spacecraft training images are known, the spacecraft 3D model is used as a variable. Through this set of spacecraft 3D model variables and predicted 2D key points, the predicted attitude of the training set with the spacecraft 3D model as the variable is obtained. The attitude is compared with the true attitude to calculate the attitude loss. Taking minimizing the attitude loss as the optimization goal, the genetic algorithm is used to optimize the initial spacecraft 3D model to obtain the optimized spacecraft 3D model.

[0141] After obtaining the optimized spacecraft 3D model, the optimized 2D key points of the spacecraft training image are obtained using the optimized spacecraft 3D model, and the MKPNet model is retrained using the optimized 2D key points of the spacecraft training image to obtain the optimized MKPNet model.

[0142] Repeat the above steps 210 to 240 to perform alternate iterative optimization on the spacecraft 3D model and the MKPNet model.

[0143] The spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention performs alternating iterative optimization of the spacecraft 3D model and the MKPNet model to continuously enhance the MKPNet model's prediction capability for 2D key points and improve the ability to evaluate the accuracy of the 3D key points of the spacecraft 3D model.

[0144] In order to evaluate the effect of the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the present invention, a comparative experiment was carried out: the Efficient-Pose network was used to directly regress the spacecraft's attitude under different settings, and the performance of the network with batch normalization (Batch Normalization, BN) and group normalization (Group Normalization, GN) structures in spacecraft six-degree-of-freedom attitude estimation was compared.

[0145] Table 1 is a comparison result table. It can be seen from Table 1 that the attitude estimation network with the BN structure has higher performance than the attitude estimation network with the GN structure. Compared with these benchmark regression models, it is observed that the proposed method has achieved significant improvement in the accuracy of attitude estimation. This improvement verifies the effectiveness of the iterative optimization of the spacecraft 3D model and the attitude smoothing optimization technology introduced in the method provided by the present invention.

[0146] Table 1 Comparison results

[0147]

[0148] The spacecraft monocular six-degree-of-freedom attitude estimation device provided by the present invention is described below. The spacecraft monocular six-degree-of-freedom attitude estimation device described below and the spacecraft monocular six-degree-of-freedom attitude estimation method described above can be referenced to each other.

[0149] Figure 3 : is a schematic diagram of the structure of the spacecraft monocular six-degree-of-freedom attitude estimation device provided by the present invention, such as Figure 3 As shown, the spacecraft monocular six-degree-of-freedom attitude estimation device provided by the present invention comprises:

[0150] An input module 310 is used to input multiple spacecraft images into the trained MKPNet model to obtain a posture pseudo label corresponding to each of the spacecraft images;

[0151] A smoothing module 320 is used to smooth the attitude pseudo-label corresponding to each of the spacecraft images based on the motion consistency between the continuous multiple frames of images, so as to obtain a smoothed attitude label corresponding to each of the spacecraft images;

[0152] A projection module 330 is used to project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images according to the smoothed posture labels corresponding to each of the spacecraft images, so as to obtain 2D key point pseudo labels, heat map pseudo labels and bounding box pseudo labels corresponding to each of the spacecraft images;

[0153] A fine-tuning module 340 is used to fine-tune the trained MKPNet model based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed posture label corresponding to each of the spacecraft images to obtain a fine-tuned MKPNet model;

[0154] The estimation module 350 is used to input the multiple spacecraft images into the fine-tuned MKPNet model to obtain a posture estimation corresponding to each of the spacecraft images.

[0155] In some embodiments, the smoothing module 320 is specifically used to:

[0156] Constructing multiple sets of data pairs based on the attitude pseudo-label and time series number corresponding to each of the spacecraft images;

[0157] For each data pair, the following steps are performed:

[0158] Fitting the current group of data pairs using the least squares method to obtain an initial smooth fitting curve corresponding to the current group of data pairs;

[0159] According to the distance threshold and the distance between each data pair in the current group and the initial smooth fitting curve corresponding to the current group data pair, the initial smooth fitting curve corresponding to the current group data pair is corrected to obtain the smooth fitting curve corresponding to the current group data pair;

[0160] Obtaining a current group of smoothed data pairs according to a smoothed fitting curve corresponding to the current group of data pairs;

[0161] According to each set of smoothed data pairs, a smoothed attitude label corresponding to each of the spacecraft images is obtained.

[0162] In some embodiments, the apparatus further comprises:

[0163] The training module is used to add the loss function corresponding to the predicted heat map and the real heat map, the loss function corresponding to the predicted bounding box and the real bounding box, and the loss function corresponding to the predicted posture and the real posture as the total loss function for training the MKPNet model.

[0164] In some embodiments, the apparatus further comprises:

[0165] A feature extraction module, used to extract features from the spacecraft training images in the training set using the feature extraction network in the MKPNet model;

[0166] A first prediction module, configured to use a heat map prediction head in the MKPNet model to perform heat map prediction on the extracted features to obtain the predicted heat map;

[0167] A second prediction module is used to use the bounding box prediction head in the MKPNet model to perform bounding box prediction on the extracted features to obtain the predicted bounding box;

[0168] The third prediction module is used to obtain the predicted posture based on the 2D key points corresponding to the filtered prediction heat map; the filtered prediction heat map is obtained by filtering the prediction heat map based on a filtering threshold.

[0169] In some embodiments, the apparatus further comprises:

[0170] A selection module is used to select spacecraft training images of spacecraft in multiple different trajectory sequences from the training set as training subsets;

[0171] A first obtaining module is used to obtain an initial spacecraft 3D model based on the manually annotated 2D key points on each spacecraft training image in the training subset and the correspondence between the 2D key points and the 3D key points on the spacecraft;

[0172] The second obtaining module is used to project the initial 3D key points of the initial spacecraft 3D model onto each spacecraft training picture in the training set based on the real pose information and the camera internal parameters, so as to obtain the real 2D key points of each spacecraft training picture in the training set;

[0173] The third obtaining module is used to obtain the real heat map and the real bounding box of each spacecraft training picture in the training set based on the real 2D key points of each spacecraft training picture in the training set.

[0174] In some embodiments, the apparatus further comprises:

[0175] Iteration module, used to iteratively execute the following steps:

[0176] Use the MKPNet model to predict 2D key points of the spacecraft training images in the training set to obtain predicted 2D key points of the spacecraft training images in the training set;

[0177] Based on the predicted 2D key points and true pose labels of the spacecraft training images in the training set, the initial spacecraft 3D model is optimized to obtain the optimized spacecraft 3D model;

[0178] Based on the optimized spacecraft 3D model, obtaining optimized 2D key points of the spacecraft training images in the training set;

[0179] The MKPNet model is retrained based on the optimized 2D key points of the spacecraft training images in the training set.

[0180] It should be noted here that the above-mentioned spacecraft monocular six-degree-of-freedom attitude estimation device provided by the present invention can implement all the method steps implemented by the above-mentioned method embodiment, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be specifically described here.

[0181] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the spacecraft monocular six-degree-of-freedom attitude estimation method, which includes:

[0182] Input multiple spacecraft images into the trained MKPNet model to obtain the pose pseudo-label corresponding to each of the spacecraft images; based on the motion consistency between multiple consecutive frames of images, smooth the pose pseudo-label corresponding to each of the spacecraft images to obtain the smoothed pose label corresponding to each of the spacecraft images; according to the smoothed pose label corresponding to each of the spacecraft images, project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images to obtain the 2D key point pseudo-label, heat map pseudo-label and bounding box pseudo-label corresponding to each of the spacecraft images; based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed pose label corresponding to each of the spacecraft images, fine-tune the trained MKPNet model to obtain the fine-tuned MKPNet model; input the multiple spacecraft images into the fine-tuned MKPNet model to obtain the pose estimation corresponding to each of the spacecraft images.

[0183] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0184] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the above methods, the method comprising:

[0185] Input multiple spacecraft images into the trained MKPNet model to obtain the pose pseudo-label corresponding to each of the spacecraft images; based on the motion consistency between multiple consecutive frames of images, smooth the pose pseudo-label corresponding to each of the spacecraft images to obtain the smoothed pose label corresponding to each of the spacecraft images; according to the smoothed pose label corresponding to each of the spacecraft images, project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images to obtain the 2D key point pseudo-label, heat map pseudo-label and bounding box pseudo-label corresponding to each of the spacecraft images; based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed pose label corresponding to each of the spacecraft images, fine-tune the trained MKPNet model to obtain the fine-tuned MKPNet model; input the multiple spacecraft images into the fine-tuned MKPNet model to obtain the pose estimation corresponding to each of the spacecraft images.

[0186] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the spacecraft monocular six-degree-of-freedom attitude estimation method provided by the above methods, the method comprising:

[0187] Input multiple spacecraft images into the trained MKPNet model to obtain the pose pseudo-label corresponding to each of the spacecraft images; based on the motion consistency between multiple consecutive frames of images, smooth the pose pseudo-label corresponding to each of the spacecraft images to obtain the smoothed pose label corresponding to each of the spacecraft images; according to the smoothed pose label corresponding to each of the spacecraft images, project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images to obtain the 2D key point pseudo-label, heat map pseudo-label and bounding box pseudo-label corresponding to each of the spacecraft images; based on the 2D key point pseudo-label, the heat map pseudo-label, the bounding box pseudo-label and the smoothed pose label corresponding to each of the spacecraft images, fine-tune the trained MKPNet model to obtain the fine-tuned MKPNet model; input the multiple spacecraft images into the fine-tuned MKPNet model to obtain the pose estimation corresponding to each of the spacecraft images.

[0188] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spacecraft monocular six-degree-of-freedom attitude estimation method, characterized in that: include: Input multiple spacecraft images into the trained MKPNet model to obtain the pose pseudo-label corresponding to each spacecraft image; Based on the motion consistency between multiple consecutive frames of images, the attitude pseudo-label corresponding to each of the spacecraft images is smoothed to obtain a smoothed attitude label corresponding to each of the spacecraft images; According to the smoothed posture label corresponding to each of the spacecraft images, the 3D key points of the optimized spacecraft 3D model are projected onto each of the spacecraft images to obtain the 2D key point pseudo-labels, heat map pseudo-labels and bounding box pseudo-labels corresponding to each of the spacecraft images; Based on the 2D key point pseudo-labels, the heat map pseudo-labels, the bounding box pseudo-labels, and the smoothed posture labels corresponding to each of the spacecraft images, fine-tuning the trained MKPNet model to obtain a fine-tuned MKPNet model; The multiple spacecraft images are input into the fine-tuned MKPNet model to obtain a posture estimation corresponding to each of the spacecraft images.

2. The spacecraft monocular six-degree-of-freedom attitude estimation method according to claim 1, characterized in that: The method of smoothing the attitude pseudo-label corresponding to each of the spacecraft images based on the motion consistency between the continuous multiple frames of images to obtain the smoothed attitude label corresponding to each of the spacecraft images includes: Constructing multiple sets of data pairs based on the attitude pseudo-label and time series number corresponding to each of the spacecraft images; For each data pair, the following steps are performed: Fitting the current group of data pairs using the least squares method to obtain an initial smooth fitting curve corresponding to the current group of data pairs; According to the distance threshold and the distance between each data pair in the current group and the initial smooth fitting curve corresponding to the current group data pair, the initial smooth fitting curve corresponding to the current group data pair is corrected to obtain the smooth fitting curve corresponding to the current group data pair; Obtaining a current group of smoothed data pairs according to a smoothed fitting curve corresponding to the current group of data pairs; According to each set of smoothed data pairs, a smoothed attitude label corresponding to each of the spacecraft images is obtained.

3. The spacecraft monocular six-degree-of-freedom attitude estimation method according to claim 1, characterized in that: The method further comprises: The loss function corresponding to the predicted heat map and the true heat map, the loss function corresponding to the predicted bounding box and the true bounding box, and the loss function corresponding to the predicted posture and the true posture are added together as the total loss function for training the MKPNet model.

4. The spacecraft monocular six-degree-of-freedom attitude estimation method according to claim 3, characterized in that: The method further comprises: Using the feature extraction network in the MKPNet model to extract features from the spacecraft training images in the training set; Using the heat map prediction head in the MKPNet model to perform heat map prediction on the extracted features to obtain the predicted heat map; Using the bounding box prediction head in the MKPNet model to perform bounding box prediction on the extracted features to obtain the predicted bounding box; The predicted posture is obtained based on the 2D key points corresponding to the filtered predicted heat map; the filtered predicted heat map is obtained by filtering the predicted heat map based on a filtering threshold.

5. The spacecraft monocular six-degree-of-freedom attitude estimation method according to claim 4, characterized in that: The method further comprises: Selecting spacecraft training images of spacecraft in multiple different trajectory sequences from the training set as training subsets; Based on the manually annotated 2D key points on each spacecraft training image in the training subset and the correspondence between the 2D key points and the 3D key points on the spacecraft, an initial spacecraft 3D model is obtained; Based on the real pose information and camera intrinsic parameters, the initial 3D key points of the initial spacecraft 3D model are projected onto each spacecraft training picture in the training set to obtain the real 2D key points of each spacecraft training picture in the training set; Based on the real 2D key points of each spacecraft training picture in the training set, the real heat map and the real bounding box of each spacecraft training picture in the training set are obtained.

6. The spacecraft monocular six-degree-of-freedom attitude estimation method according to claim 5, characterized in that: The method further comprises: Iterate through the following steps: Use the MKPNet model to predict 2D key points of the spacecraft training images in the training set to obtain predicted 2D key points of the spacecraft training images in the training set; Based on the predicted 2D key points and true pose labels of the spacecraft training images in the training set, the initial spacecraft 3D model is optimized to obtain the optimized spacecraft 3D model; Based on the optimized spacecraft 3D model, obtaining optimized 2D key points of the spacecraft training images in the training set; The MKPNet model is retrained based on the optimized 2D key points of the spacecraft training images in the training set.

7. A spacecraft monocular six-degree-of-freedom attitude estimation device, characterized in that: include: An input module is used to input multiple spacecraft images into the trained MKPNet model to obtain a posture pseudo-label corresponding to each spacecraft image; A smoothing module, used for smoothing the attitude pseudo-label corresponding to each of the spacecraft images based on the motion consistency between multiple consecutive frames of images, to obtain a smoothed attitude label corresponding to each of the spacecraft images; A projection module is used to project the 3D key points of the optimized spacecraft 3D model onto each of the spacecraft images according to the smoothed posture labels corresponding to each of the spacecraft images, so as to obtain 2D key point pseudo labels, heat map pseudo labels and bounding box pseudo labels corresponding to each of the spacecraft images; A fine-tuning module, configured to fine-tune the trained MKPNet model based on the 2D key point pseudo-labels, the heat map pseudo-labels, the bounding box pseudo-labels and the smoothed posture labels corresponding to each of the spacecraft images, to obtain a fine-tuned MKPNet model; The estimation module is used to input the multiple spacecraft images into the fine-tuned MKPNet model to obtain a posture estimation corresponding to each of the spacecraft images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the spacecraft monocular six-degree-of-freedom attitude estimation method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for estimating the spacecraft monocular six-degree-of-freedom attitude as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for estimating the spacecraft monocular six-degree-of-freedom attitude as claimed in any one of claims 1 to 6 is implemented.

Images