A deep learning-based adaptive pose estimation system for spatial targets

Abstract

This invention discloses a deep learning-based adaptive pose estimation system for space targets. It mainly includes a non-cooperative space target dataset unit, an image preprocessing unit, a deep learning network model unit, a cross-domain adaptive model unit, and a high-precision pose estimation unit for non-cooperative targets. This invention obtains an image dataset from a 3D satellite model through rendering, performs image preprocessing using techniques such as denoising, data augmentation, correction, and registration, constructs a highly robust pose estimation model using deep learning, and simultaneously performs cross-domain adaptive optimization of the model parameters, thereby achieving cross-domain adaptive pose estimation for non-cooperative targets. This invention also discloses a method for implementing non-cooperative space target pose estimation, achieving high-precision pose estimation for non-cooperative space targets economically and efficiently, making a significant contribution to my country's field of adaptive non-cooperative space target pose estimation.

Description

Technical Field

[0001] This invention relates to a deep learning-based adaptive pose estimation system for spatial targets, belonging to the field of spatial target pose calculation. Background Technology

[0002] With the increasing frequency of human space activities, the importance of space exploration is becoming increasingly prominent. Space exploration aims to comprehensively understand the distribution of satellites, space debris, and potential threat targets. However, image samples of these targets are scarce, making their detection extremely difficult. Existing research on space target pose estimation techniques is limited and severely inadequate in terms of accuracy and cross-domain capability: most existing techniques estimate the pose of ground targets, and due to the scarcity of space image datasets, deep learning training lacks sufficient samples, resulting in low accuracy in estimating space target poses. Furthermore, existing techniques lack pose estimation methods for cross-domain space targets; that is, during testing, estimating the pose of targets similar to the training data performs well, but estimating the pose of targets with significant differences from the training data performs poorly. Therefore, research on adaptive pose estimation systems for space targets is urgently needed. Summary of the Invention

[0003] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a space target adaptive pose estimation system based on deep learning, which improves the accuracy of space target pose estimation and also achieves good results in cross-domain space target pose estimation.

[0004] The technical solution of this invention is: a spatial target adaptive pose estimation system based on deep learning, comprising: a spatial target dataset unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit, and a high-precision pose estimation unit, wherein:

[0005] The spatial target dataset unit renders satellite 3D models based on real materials and environment, generates original image datasets based on satellite 3D models, and divides them into training set, validation set and test set;

[0006] The image preprocessing unit performs image preprocessing on the received raw image dataset to obtain a preprocessed image;

[0007] The deep learning network model unit determines the network model. The network model is trained using the received preprocessed image and the model cross-domain adaptive unit to jointly generate a preliminary domain adaptive keypoint location estimation network model. The generated preliminary domain adaptive keypoint location estimation network model is validated using validation set data to generate a keypoint location estimation network model.

[0008] The cross-domain adaptive unit of the model participates in the training of the deep learning network model unit and tests the key point location estimation network model. During the testing process, the deep learning network model is further optimized to generate the final cross-domain adaptive key point location estimation network model.

[0009] The high-precision pose estimation unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the corresponding three-dimensional coordinates of each key point based on the quaternion and translation matrix, calculates the target pose using the PnP method, and judges the accuracy of the pose estimation.

[0010] After generating the final cross-domain adaptive keypoint location estimation network model, input the spatial image sequence into the final cross-domain adaptive keypoint location estimation network model to obtain the cross-domain adaptive pose estimation result for each image sequence.

[0011] Preferably, the method for rendering a satellite 3D model based on real materials and environment in the space target dataset unit, and generating an original image dataset based on the satellite 3D model, is as follows:

[0012] Import a 3D satellite model into a 3D model rendering software;

[0013] The model is rendered based on the real satellite material and texture features, while lighting and reflection conditions are added, and some images are occluded to generate a rendered 3D satellite model.

[0014] The rendered 3D satellite model is made into a frame animation, and the quaternions, translation matrix and key point information of the target in each frame image are extracted.

[0015] Each frame of the image is labeled with quaternions, translation matrices, and key point location information. The image and its corresponding label information together form the original image dataset.

[0016] Preferably, the obtained original image dataset is divided into a training set, a validation set, and a test set, wherein:

[0017] The images in the training set comprise 80% of the original image dataset and are used to train the network model.

[0018] The images in the validation set comprise 10% of the original image dataset and are used to tune hyperparameters and optimize the network model.

[0019] The test set contains 10% of the images in the original image dataset and is used to evaluate the performance of the network model.

[0020] Preferably, the image preprocessing unit performs image preprocessing on the received raw image dataset using the following method:

[0021] Load the original image dataset and perform various image processing methods, including randomly rotating, flipping, scaling, and cropping each frame of the image;

[0022] Enhancement processing of images after various processing methods includes: grayscale conversion, normalization, and image denoising;

[0023] The image size of the enhanced image is adjusted to fit the input requirements of the deep learning network model units, and then output as a preprocessed image.

[0024] The preferred method for generating a keypoint location estimation network model from deep learning network model units is as follows:

[0025] The network model is trained on the training set, where the network model uses the Hourglass network model.

[0026] During training, the Hourglass network model is used to extract key point features, which includes two stages: forward propagation and back propagation. The BPnP algorithm is added to the back propagation process, and the forward and back propagation processes are iterated continuously. The Adam optimization algorithm is used to dynamically adjust the weights and hyperparameters of the Hourglass network model. At the same time, the model's cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model. The two units together generate a preliminary domain-adaptive key point location estimation network model.

[0027] Based on the preliminary domain-adaptive keypoint location estimation network model, the validation set data is used for further iterative inference during the validation process. The keypoint location coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. The weights and hyperparameters of the preliminary domain-adaptive keypoint location estimation network model are adjusted according to the inference results of the validation set to generate the keypoint location estimation network model.

[0028] Preferably, the method for determining the accuracy of pose estimation is as follows:

[0029] If the pose estimation accuracy of the spatial target meets the requirement of being better than 3°, then the pose estimation is considered correct.

[0030] If the spatial target pose estimation does not meet the requirement of being better than 3°, it is considered an incorrect pose estimation, and its result is discarded. The target's current pose is then re-estimated.

[0031] Preferably, when the cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model, the two units jointly generate a preliminary domain-adaptive keypoint location estimation network model. The specific process is as follows:

[0032] During the training of the deep learning network model unit, when the adjusted weights and hyperparameters of the Hourglass network model are obtained, the cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model through pseudo-label iteration, generates and fine-tunes new pseudo-labels, and after the iteration is completed, it works with the deep learning network model unit to generate a preliminary domain adaptive keypoint location estimation network model.

[0033] Preferably, the method for testing the keypoint location estimation network model using the cross-domain adaptive unit, and further optimizing the deep learning network model during the testing process to generate the final cross-domain adaptive keypoint location estimation network model is as follows:

[0034] During testing, the keypoint location estimation network model generated from the validation set in the deep learning network model unit is loaded. The ODR method is used to optimize only the parameters of the normalization layer. The training continues iteratively to optimize the parameters of the batch normalization layer and generate the final cross-domain adaptive keypoint location estimation network model.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) This invention applies deep learning-based target pose estimation to space. It uses a dataset it created to train deep learning and generates a more accurate key point position estimation network model. Then, it uses the PnP algorithm to calculate a more accurate pose.

[0037] (2) This invention proposes a pseudo-label iterative training and online domain refinement (ODR) method, which can also achieve good results in cross-domain spatial target pose estimation. Attached Figure Description

[0038] Figure 1 This is a block diagram of the spatial target pose estimation system of the present invention. Detailed Implementation

[0039] The technical solution of this invention is:

[0040] A deep learning-based spatial target adaptive pose estimation system includes: a spatial target dataset unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit, and a high-precision pose estimation unit.

[0041] I. Space Target Dataset Unit: Target images in space are difficult to obtain directly. To generate an image dataset, a satellite 3D model needs to be rendered based on the real materials and environment. Quaternions, translation matrices, and key point location information are then labeled onto each image (quaternions, translation matrices, and key point locations are corresponding). The dataset is then divided into a training set (80%), a validation set (10%), and a test set (10%).

[0042] The specific steps of the method for synthesizing images from spatial target dataset units are as follows:

[0043] Step 1-1: Import an accurate 3D satellite model into Blender software;

[0044] Steps 1-2: Render the model based on real satellite materials, making the model's materials and textures infinitely close to those of a real satellite, while adding lighting and reflection conditions, and occluding a portion of the image.

[0045] Steps 1-3: Create frame animations from the rendered model, and extract the quaternions, translation matrices, and key point information of the target in each frame image.

[0046] Steps 1-4: Label each image in the dataset with quaternions, translation matrices, and key point location information. After the labeling information corresponds one-to-one with the image, an image dataset is formed.

[0047] Steps 1-5: Divide the dataset into a training set (80%), a validation set (10%), and a test set (10%). The training set is used to train the network model, the validation set is used to tune hyperparameters and optimize the network model, and the test set is used to evaluate the performance of the network model.

[0048] 2. Image Preprocessing Unit: Preprocesses the received raw images in the dataset, including denoising, enhancement, and correction, to facilitate the subsequent training process of the deep learning network model.

[0049] The specific processing steps of the image preprocessing unit are as follows:

[0050] Step 2-1: Load the dataset images and randomly perform various processing on each frame of the image, including one or more of the following: rotation, flipping, scaling and cropping, to increase data diversity and improve the generalization ability of the network model.

[0051] Step 2-2: Perform data augmentation processing on the diversified processed images, including grayscale conversion, normalization, and noise reduction;

[0052] Steps 2-3: Adjust the image size of the enhanced image to fit the input requirements of the deep learning network model units, and output it as a preprocessed image;

[0053] III. Deep Learning Network Model Unit: This unit receives image information from the image preprocessing unit, extracts key point features using the Hourglass network, and generates a key point location estimation network model after deep learning training, so as to enable subsequent cross-domain adaptive processing and output of key point locations.

[0054] Specifically, the Hourglass network, with its unique hourglass structure, enables keypoint feature processing from coarse to fine and back to coarse. During network model training, the BPnP algorithm is incorporated into the backpropagation process to continuously optimize the weight parameters for keypoint location estimation, allowing the network model to output more accurate keypoint locations. The Adam optimization algorithm is chosen to dynamically adjust the weights and hyperparameters of the Hourglass network model.

[0055] The specific steps of a deep learning network model unit are as follows:

[0056] Step 3-1: Obtain the preprocessed training set images;

[0057] Step 3-2: Create the model object, data loader, and optimization object;

[0058] Step 3-3: Import model parameters (Hourglass keypoint detection model);

[0059] Steps 3-4: Generate the camera intrinsic parameter matrix, distortion coefficients, and 3D coordinates in the world coordinate system, and create a BPnP object.

[0060] Steps 3-5: Train the Hourglass network model on the training set and optimize the weight parameters of the Hourglass network model through continuous forward and backward propagation. At the same time, the cross-domain adaptive unit adjusts the weight parameters of the Hourglass network model through pseudo-label iteration. The two units together generate a preliminary domain adaptive keypoint location estimation network model.

[0061] Steps 3-6: Based on the preliminary domain adaptive keypoint position estimation network model, further iterative inference is performed on the validation set. The keypoint position coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. The hyperparameters of the preliminary domain adaptive keypoint position estimation network model are adjusted according to the inference results of the validation set to generate the keypoint position estimation network model.

[0062] IV. Model Cross-Domain Adaptive Unit: The model cross-domain adaptive unit participates in the training and testing process of the key point location estimation network model in the deep learning network model unit. By further optimizing the network model, the final cross-domain adaptive key point location estimation network model is generated, thereby realizing cross-domain adaptive capability.

[0063] The optimization mainly includes two aspects: First, during the training process of the deep learning network model unit (training the Hourglass network model on the training set and optimizing the weight parameters of the Hourglass network model through continuous forward and backward propagation), a semi-supervised learning approach is adopted. The weight parameters of the Hourglass network model are adjusted through pseudo-label iteration to generate a preliminary domain adaptive keypoint position estimation network model. Second, during the testing process, the unit adopts the online domain refinement (ODR) method to optimize the normalization layer parameters of the keypoint position estimation network model generated by the validation set in the deep learning network model unit, generating the final cross-domain adaptive keypoint position estimation network model, thereby improving the cross-domain adaptive capability of the spatial target adaptive pose estimation system.

[0064] The specific steps of the model's cross-domain adaptive unit are as follows:

[0065] Step 4-1: During the training of the deep learning network model unit, pseudo-labels are generated for training and iterative process. During each iteration, the weights of the Hourglass network model are loaded, new pseudo-labels are generated and fine-tuned. After the iteration is completed, the pseudo-labels are generated together with the deep learning network model unit to generate a preliminary domain adaptive key point location estimation network model.

[0066] Step 4-2: During the testing process, load the key point location estimation network model generated from the validation set in the deep learning network model unit, use the online domain refinement (ODR) method, keep other parameters unchanged, optimize only the parameters of the normalization layer, continue iterative training, optimize the parameters of the batch normalization layer until the set number of iterations is reached, and generate the final cross-domain adaptive key point location estimation network model.

[0067] V. High-precision pose estimation unit: This unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the corresponding 3D coordinates of each key point based on the quaternion and translation matrix, and then calculates the pose of the spatial target using the PnP method, and judges the accuracy of the spatial target pose estimation.

[0068] The specific steps for calculating the pose of a spatial target are as follows:

[0069] Step 5-1: Receive the key point coordinates output by the deep learning network model unit, and obtain the corresponding quaternion and translation matrix based on the key point coordinates;

[0070] Step 5-2: Calculate the three-dimensional coordinates corresponding to the two-dimensional key point based on the key point coordinates, quaternions, and translation matrix;

[0071] Step 5-3: Based on the two-dimensional and three-dimensional coordinates of the key points, estimate the pose of the spatial target using the PnP algorithm and output it.

[0072] The process of determining the accuracy of spatial target pose estimation is as follows:

[0073] If the accuracy of spatial target pose estimation meets the requirement of being better than 3°, then the spatial target pose estimation is considered to be correct.

[0074] If the requirement of being better than 3° is not met, it is considered an incorrect attitude estimate, and the target's current attitude is re-estimated.

[0075] In subsequent use, the spatial image sequence is input into the final cross-domain adaptive keypoint location estimation network model to obtain the cross-domain adaptive pose estimation result for each image sequence.

[0076] This invention can be used for pose estimation of visible non-cooperative targets in space, and efficiently achieves high-precision pose estimation of non-cooperative targets in space. It has the advantages of being economical and highly practical, and can be widely applied to other pose estimation tasks of non-cooperative targets in space.

[0077] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A spatial target adaptive pose estimation system based on deep learning, characterized in that... include: The system comprises a spatial target dataset unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit, and a high-precision pose estimation unit, among which: The spatial target dataset unit renders satellite 3D models based on real materials and environment, generates original image datasets based on satellite 3D models, and divides them into training set, validation set and test set; The image preprocessing unit performs image preprocessing on the received raw image dataset to obtain a preprocessed image; The deep learning network model unit determines the network model. The network model is trained using the received preprocessed image and the model cross-domain adaptive unit to jointly generate a preliminary domain adaptive keypoint location estimation network model. The generated preliminary domain adaptive keypoint location estimation network model is validated using validation set data to generate a keypoint location estimation network model. The cross-domain adaptive unit of the model participates in the training of the deep learning network model unit and tests the key point location estimation network model. During the testing process, the deep learning network model is further optimized to generate the final cross-domain adaptive key point location estimation network model. The high-precision pose estimation unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the corresponding three-dimensional coordinates of each key point based on the quaternion and translation matrix, calculates the target pose using the PnP method, and judges the accuracy of the pose estimation. After generating the final cross-domain adaptive keypoint location estimation network model, input the spatial image sequence into the final cross-domain adaptive keypoint location estimation network model to obtain the cross-domain adaptive pose estimation result for each image sequence. The method for generating a keypoint location estimation network model using deep learning network model units is as follows: The network model is trained on a training set, using the Hourglass network model. During training, the Hourglass network model is used to extract keypoint features, specifically including two stages: forward propagation and backpropagation. The BPnP algorithm is incorporated into the backpropagation process, iterating continuously between forward and backpropagation. The Adam optimization algorithm is used to dynamically adjust the weights and hyperparameters of the Hourglass network model. Simultaneously, a cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model. The cross-domain adaptive unit and the deep learning network model units jointly generate a preliminary domain-adaptive keypoint location estimation network model. When the cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model, the two units jointly generate a preliminary domain-adaptive keypoint location estimation network model. The specific process is as follows: when the adjusted weights and hyperparameters of the Hourglass network model are obtained during the training process of the deep learning network model unit, the cross-domain adaptive unit further adjusts the weights and hyperparameters of the Hourglass network model through the pseudo-label iteration method, generates and fine-tunes new pseudo-labels, and after the iteration is completed, it jointly generates a preliminary domain-adaptive keypoint location estimation network model with the deep learning network model unit. The cross-domain adaptive unit tests the keypoint location estimation network model. During the testing process, the deep learning network model is further optimized to generate the final cross-domain adaptive keypoint location estimation network model. The specific method is as follows: During the testing process, the keypoint location estimation network model generated by the validation set in the deep learning network model unit is loaded. The ODR method is used to optimize only the parameters of the normalization layer. The training continues iteratively to optimize the parameters of the batch normalization layer and generate the final cross-domain adaptive keypoint location estimation network model.

2. The spatial target adaptive pose estimation system based on deep learning according to claim 1, characterized in that: The method for rendering satellite 3D models based on real materials and environment, and generating original image datasets based on satellite 3D models, is as follows: Import a 3D satellite model into a 3D model rendering software; The model is rendered based on the real satellite material and texture features, while lighting and reflection conditions are added, and some images are occluded to generate a rendered 3D satellite model. The rendered 3D satellite model is made into a frame animation, and the quaternions, translation matrix and key point information of the target in each frame image are extracted. Each frame of the image is labeled with quaternions, translation matrices, and key point location information. The image and its corresponding label information together form the original image dataset.

3. The spatial target adaptive pose estimation system based on deep learning according to claim 2, characterized in that: The obtained original image dataset is divided into a training set, a validation set, and a test set, where: The images in the training set comprise 80% of the original image dataset and are used to train the network model. The images in the validation set comprise 10% of the original image dataset and are used to tune hyperparameters and optimize the network model. The test set contains 10% of the images in the original image dataset and is used to evaluate the performance of the network model.

4. The spatial target adaptive pose estimation system based on deep learning according to claim 1, characterized in that: The image preprocessing unit performs image preprocessing on the received raw image dataset as follows: Load the original image dataset and perform various image processing methods, including randomly rotating, flipping, scaling, and cropping each frame of the image; Enhancement processing of images after various processing methods includes: grayscale conversion, normalization, and image denoising; The image size of the enhanced image is adjusted to fit the input requirements of the deep learning network model units, and then output as a preprocessed image.

5. The spatial target adaptive pose estimation system based on deep learning according to claim 1, characterized in that: Methods for generating keypoint location estimation network models from deep learning network model units also include: Based on the preliminary domain-adaptive keypoint location estimation network model, the validation set data is used for further iterative inference during the validation process. The keypoint location coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. The weights and hyperparameters of the preliminary domain-adaptive keypoint location estimation network model are adjusted according to the inference results of the validation set to generate the keypoint location estimation network model.

6. The spatial target adaptive pose estimation system based on deep learning according to claim 1, characterized in that: The specific method for determining the accuracy of pose estimation is as follows: If the pose estimation accuracy of the spatial target meets the requirement of being better than 3°, then the pose estimation is considered correct. If the spatial target pose estimation does not meet the requirement of being better than 3°, it is considered an incorrect pose estimation, and its result is discarded. The target's current pose is then re-estimated.

Images