Space target adaptive pose estimation system based on deep learning

Abstract

The invention discloses a space target adaptive pose estimation system based on deep learning. The system mainly comprises a spatial non-cooperative target data set unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit and a non-cooperative target high-precision pose estimation unit. According to the method, an image data set is obtained after a 3D satellite model is rendered, image preprocessing is performed through technologies such as denoising, data enhancement, correction and registration, a high-robustness pose estimation model is constructed by using deep learning, and meanwhile, cross-domain adaptive optimization is performed on model parameters, so that cross-domain adaptive non-cooperative target pose estimation is realized. Meanwhile, the invention discloses an implementation method of spatial non-cooperative target pose estimation, high-precision pose estimation of the spatial non-cooperative target is economically and efficiently achieved, and a great contribution is made to the field of spatial self-adaptive non-cooperative target pose estimation in China.

Description

Technical Field

[0001] The present invention relates to a space target adaptive posture estimation system based on deep learning, and belongs to the field of space target posture solution. Background Art

[0002] As human space activities become increasingly frequent, the importance of space exploration becomes more and more prominent. Space exploration aims to fully grasp the distribution of satellites, space debris and potential threat targets, but the image samples of these targets are scarce, so their detection is extremely difficult. There is little research on the existing space target pose estimation technology, and it is seriously insufficient in accuracy and cross-domain capability: most of the existing technologies estimate the pose of ground targets. Due to the scarcity of space image data sets, deep learning training does not have enough samples, resulting in low estimation accuracy of space target pose; in addition, the existing technology does not have a pose estimation method for cross-domain space targets, that is, during testing, the estimated pose of targets similar to training is better, but the estimated pose of targets with large differences from training is very poor. Therefore, it is urgent to carry out research on adaptive pose estimation systems for space targets. Summary of the invention

[0003] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, to provide a space target adaptive pose estimation system based on deep learning, to improve the solution accuracy of the space target pose, and to achieve good results in cross-domain space target pose estimation.

[0004] The technical solution of the present invention is: a space target adaptive pose estimation system based on deep learning, comprising: a space target data set unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit and a high-precision pose estimation unit, wherein:

[0005] The space target dataset unit renders the satellite 3D model according to the real material and environment, generates the original image dataset based on the satellite 3D model, and divides it into training set, validation set and test set;

[0006] The image preprocessing unit performs image preprocessing on the received original image data set to obtain a preprocessed image;

[0007] The deep learning network model unit determines the network model, uses the received preprocessed image in combination with the model cross-domain adaptation unit to train the network model, and jointly generates a preliminary domain adaptive key point position estimation network model; uses the validation set data to verify the generated preliminary domain adaptive key point position estimation network model, and generates a key point position estimation network model;

[0008] The model cross-domain adaptive unit participates in the training of the network model by the deep learning network model unit, and tests the key point position estimation network model. During the test, the deep learning network model is further optimized to generate the final cross-domain adaptive key point position estimation network model.

[0009] The high-precision pose estimation unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the three-dimensional coordinates corresponding to each key point coordinate according to the quaternion and translation matrix corresponding to the point, calculates the pose of the target in combination with the PnP method, and determines the accuracy of the pose estimation;

[0010] After the final cross-domain adaptive key point position estimation network model is generated, the spatial image sequence is input into the final cross-domain adaptive key point position estimation network model to obtain the cross-domain adaptive pose estimation result of each sequence image.

[0011] Preferably, the space target data set unit renders the satellite three-dimensional model according to the real material and environment, and generates the original image data set according to the satellite three-dimensional model in the following method:

[0012] Import 3D satellite models into 3D model rendering software;

[0013] Render the model based on the real satellite material and texture features, add lighting and reflection conditions, and block part of the image to generate a rendered 3D satellite model;

[0014] The rendered 3D satellite model is made into a frame animation, and the quaternion, translation matrix and key point information of the target in each frame image are extracted;

[0015] The quaternion, translation matrix and key point position information are annotated for each frame of the image. The image and the one-to-one corresponding annotation information together form the original image data set.

[0016] Preferably, the original image data set is divided into a training set, a validation set, and a test set, where:

[0017] The number of images in the training set accounts for 80% of the original image dataset and is used to train the network model;

[0018] The number of images in the validation set accounts for 10% of the original image dataset and is used to adjust hyperparameters and optimize the network model;

[0019] The number of images in the test set accounts for 10% of the original image dataset and is used to evaluate the performance of the network model.

[0020] Preferably, the image preprocessing unit performs image preprocessing on the received original image data set in the following manner:

[0021] Loading the original image data set, and performing diversified processing on the image, including: randomly rotating, flipping, scaling and shearing each frame of the image;

[0022] Performing enhancement processing on the diversified processed images, including grayscale, normalization and image denoising;

[0023] The image size of the enhanced image is adjusted to adapt to the input requirements of the deep learning network model unit as the preprocessed image output.

[0024] Preferably, the method for the deep learning network model unit to generate a key point position estimation network model is:

[0025] Train the network model on the training set, where the network model uses the Hourglass network model;

[0026] The Hourglass network model is used to extract key point features during the training process, which includes two stages: forward propagation and back propagation. The BPnP algorithm is added to the back propagation process, and the forward propagation and back propagation processes are continuously iterated. The Adam optimization algorithm is used to dynamically adjust the weights and hyperparameters of the Hourglass network model. At the same time, the cross-domain adaptive unit of the model further adjusts the weights and hyperparameters of the Hourglass network model. The two units jointly generate a preliminary domain adaptive key point position estimation network model.

[0027] According to the preliminary domain adaptive key point position estimation network model, the validation set data is used for further iterative reasoning during the verification process. The key point position coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. The weights and hyperparameters of the preliminary domain adaptive key point position estimation network model are adjusted according to the reasoning results of the validation set to generate the key point position estimation network model.

[0028] Preferably, the method for determining the accuracy of pose estimation is specifically:

[0029] If the estimated accuracy of the space target's pose meets the requirement of better than 3°, the pose estimation is considered correct;

[0030] If the spatial target pose estimation does not meet the requirement of being better than 3°, it is regarded as an incorrect pose estimation, and its result is discarded, and the current pose of the target is re-estimated.

[0031] Preferably, when the model cross-domain adaptation unit further adjusts the weights and hyperparameters of the Hourglass network model, the two units jointly generate a preliminary domain adaptive key point position estimation network model, and the specific process is as follows:

[0032] When the adjusted weights and hyperparameters of the Hourglass network model are obtained during the training process of the deep learning network model unit, the model cross-domain adaptation unit further adjusts the weights and hyperparameters of the Hourglass network model through the pseudo-label iteration method, generates and fine-tunes new pseudo-labels, and after the iteration is completed, jointly generates a preliminary domain adaptive key point position estimation network model with the deep learning network model unit.

[0033] Preferably, the model cross-domain adaptive unit tests the key point position estimation network model, further optimizes the deep learning network model during the test, and generates the final cross-domain adaptive key point position estimation network model in the following manner:

[0034] During the test, the key point position estimation network model generated by the validation set in the deep learning network model unit is loaded. The ODR method is used to optimize only the parameters of the normalization layer. The training is continued iteratively to optimize the batch normalization layer parameters and generate the final cross-domain adaptive key point position estimation network model.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) The present invention applies the target pose estimation based on deep learning to the outer space, uses a self-made data set for deep learning training, generates a more accurate key point position estimation network model, and then uses the PnP algorithm to calculate a more accurate pose.

[0037] (2) The present invention proposes a pseudo-label iterative training and online domain refinement (ODR) method, which can also achieve good results in cross-domain spatial target pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a block diagram of the composition of the space target pose estimation system of the present invention. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is:

[0040] A deep learning-based space target adaptive pose estimation system comprises: a space target data set unit, an image preprocessing unit, a deep learning network model unit, a model cross-domain adaptive unit and a high-precision pose estimation unit.

[0041] 1. Space target dataset unit: It is difficult to directly obtain target images in space. It is necessary to render the satellite 3D model according to the real material and environment, generate the image dataset, and annotate the quaternion, translation matrix and key point position information on each image (the quaternion, translation matrix and key point position are corresponding). Then the dataset is divided into training set (80%), validation set (10%) and test set (10%).

[0042] The specific steps of the method for synthesizing images of space target dataset units are as follows:

[0043] Step 1-1: Import the accurate 3D satellite model into Blender software;

[0044] Step 1-2: Render the model based on the real satellite material, making the model's material and texture as close to the real satellite as possible, adding lighting and reflection conditions, and occluding part of the image;

[0045] Step 1-3: Make the rendered model into a frame animation, and extract the quaternion, translation matrix and key point information of the target in each frame image;

[0046] Step 1-4: Annotate each image in the data set with quaternion, translation matrix and key point position information. After the annotated information and the image are matched one by one, an image data set is formed.

[0047] Step 1-5: Divide the dataset into training set (80%), validation set (10%), and test set (10%). The training set is used to train the network model, the validation set is used to adjust hyperparameters and optimize the network model, and the test set is used to evaluate the performance of the network model.

[0048] 2. Image preprocessing unit: preprocess the original images of the received data set, including denoising, enhancement and correction, to facilitate the subsequent deep learning network model training process.

[0049] The specific processing steps of the image preprocessing unit are as follows:

[0050] Step 2-1: Load the dataset images and randomly perform diversified processing on each frame of the image, including one or more of rotation, flipping, scaling and shearing processing, to increase data diversity and improve the generalization ability of the network model;

[0051] Step 2-2: Perform data enhancement processing such as grayscale conversion, normalization, and denoising on the diversified processed images;

[0052] Step 2-3: Adjust the image size of the enhanced image to meet the input requirements of the deep learning network model unit as the preprocessed image output;

[0053] 3. Deep learning network model unit: This unit receives the image information from the image preprocessing unit, uses the hourglass network to extract key point features, and after deep learning training, generates a key point position estimation network model for subsequent cross-domain adaptive processing and output of key point positions;

[0054] Specifically, the Hourglass network is used to achieve key point feature processing from coarse to fine and then back to coarse through its unique hourglass structure. When training the network model, the BPnP algorithm is added to the back propagation process to continuously optimize the weight parameters of the key point position estimation so that the network model can output more accurate key point positions. The Adam optimization algorithm is selected to dynamically adjust the weights and hyperparameters of the Hourglass network model.

[0055] The specific steps of the deep learning network model unit are as follows:

[0056] Step 3-1: Obtain preprocessed training set images;

[0057] Step 3-2: Create model objects, data loaders, and optimizer objects;

[0058] Step 3-3: Import model parameters (Hourglass key point detection model);

[0059] Step 3-4: Generate the camera intrinsic parameter matrix, distortion coefficient, 3D coordinates in the world coordinate system, and create a BPnP object.

[0060] Step 3-5: Train the Hourglass network model on the training set, and optimize the weight parameters of the Hourglass network model through continuous forward propagation and back propagation processes; at the same time, the cross-domain adaptation unit of the model adjusts the weight parameters of the Hourglass network model through the pseudo-label iteration method, and the two units jointly generate a preliminary domain adaptive key point position estimation network model.

[0061] Step 3-6: Based on the preliminary domain adaptive key point position estimation network model, further iterative reasoning is performed on the validation set, and the key point position coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. According to the reasoning results of the validation set, the hyperparameters of the preliminary domain adaptive key point position estimation network model are adjusted to generate the key point position estimation network model.

[0062] 4. Model cross-domain adaptation unit: The model cross-domain adaptation unit participates in the training and testing process of the deep learning network model unit on the key point position estimation network model. By further optimizing the network model, the final cross-domain adaptive key point position estimation network model is generated, thereby realizing cross-domain adaptive capabilities.

[0063] The optimization mainly includes two aspects: first, in the process of training the deep learning network model unit (training the Hourglass network model on the training set, optimizing the weight parameters of the Hourglass network model through continuous forward propagation and back propagation processes), a semi-supervised learning method is used to adjust the weight parameters of the Hourglass network model through the pseudo-label iteration method to generate a preliminary domain adaptive key point position estimation network model; second, in the testing process, the unit uses the online domain refinement (ODR) method to optimize the normalization layer parameters of the key point position estimation network model generated by the validation set in the deep learning network model unit, and generate the final cross-domain adaptive key point position estimation network model, thereby improving the cross-domain adaptive capability of the spatial target adaptive pose estimation system.

[0064] The specific steps of the model cross-domain adaptive unit are as follows:

[0065] Step 4-1: Generate pseudo labels for training and iterate continuously during the training of the deep learning network model unit. During each iteration, load the weights of the Hourglass network model, generate and fine-tune new pseudo labels, and after the iteration is completed, jointly generate a preliminary domain adaptive key point position estimation network model with the deep learning network model unit;

[0066] Step 4-2: During the test, load the key point position estimation network model generated by the validation set in the deep learning network model unit, adopt the online domain refinement (ODR) method, keep other parameters unchanged, only optimize the parameters of the normalization layer, continue iterative training, optimize the batch normalization layer parameters until the set number of iterations is reached, and generate the final cross-domain adaptive key point position estimation network model.

[0067] 5. High-precision pose estimation unit: This unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the three-dimensional coordinates corresponding to each key point coordinate based on the quaternion and translation matrix corresponding to the point, and then calculates the pose of the space target in combination with the PnP method, and determines the accuracy of the pose estimation of the space target.

[0068] The specific steps for calculating the spatial target pose are as follows:

[0069] Step 5-1: Receive the key point position coordinates output by the deep learning network model unit, and obtain the corresponding quaternion and translation matrix according to the key point coordinates;

[0070] Step 5-2: Calculate the three-dimensional coordinates corresponding to the two-dimensional key point according to the key point coordinates, quaternion and translation matrix;

[0071] Step 5-3: Based on the two-dimensional and three-dimensional coordinates of the key points, use the PnP algorithm to estimate the pose of the space target and output it.

[0072] The process of judging the accuracy of space target pose estimation is as follows:

[0073] If the space target pose estimation accuracy meets the requirement of better than 3°, the space target pose estimation is considered correct;

[0074] If the requirement of being better than 3° is not met, it is regarded as an incorrect attitude estimation and the current attitude of the target is re-estimated.

[0075] In subsequent use, the spatial image sequence is input into the final cross-domain adaptive key point position estimation network model to obtain the cross-domain adaptive pose estimation results of each sequence image.

[0076] The present invention can be used for pose estimation tasks of visible non-cooperative targets in space, and efficiently realizes high-precision pose estimation of non-cooperative targets in space, has beneficial effects of economy and high practicality, and can be widely used in other pose estimation tasks of non-cooperative targets in space, etc.

[0077] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A deep learning-based space target adaptive pose estimation system, characterized by include: Space target dataset unit, image preprocessing unit, deep learning network model unit, model cross-domain adaptation unit and high-precision pose estimation unit, among which: The space target dataset unit renders the satellite 3D model according to the real material and environment, generates the original image dataset based on the satellite 3D model, and divides it into training set, validation set and test set; The image preprocessing unit performs image preprocessing on the received original image data set to obtain a preprocessed image; The deep learning network model unit determines the network model, uses the received preprocessed image in combination with the model cross-domain adaptation unit to train the network model, and jointly generates a preliminary domain adaptive key point position estimation network model; uses the validation set data to verify the generated preliminary domain adaptive key point position estimation network model, and generates a key point position estimation network model; The model cross-domain adaptive unit participates in the training of the network model by the deep learning network model unit, and tests the key point position estimation network model. During the test, the deep learning network model is further optimized to generate the final cross-domain adaptive key point position estimation network model. The high-precision pose estimation unit receives the key point coordinates output during the deep learning network model unit verification process, calculates the three-dimensional coordinates corresponding to each key point coordinate according to the quaternion and translation matrix corresponding to the point, calculates the pose of the target in combination with the PnP method, and determines the accuracy of the pose estimation; After the final cross-domain adaptive key point position estimation network model is generated, the spatial image sequence is input into the final cross-domain adaptive key point position estimation network model to obtain the cross-domain adaptive pose estimation result of each sequence image.

2. The deep learning-based space target adaptive pose estimation system according to claim 1, characterized in that: The space target dataset unit renders the satellite 3D model according to the real material and environment, and generates the original image dataset according to the satellite 3D model as follows: Import 3D satellite models into 3D model rendering software; Render the model based on the real satellite material and texture features, add lighting and reflection conditions, and block part of the image to generate a rendered 3D satellite model; The rendered 3D satellite model is made into a frame animation, and the quaternion, translation matrix and key point information of the target in each frame image are extracted; The quaternion, translation matrix and key point position information are annotated for each frame of the image. The image and the one-to-one corresponding annotation information together form the original image data set.

3. The deep learning-based space target adaptive pose estimation system according to claim 2, characterized in that: The original image dataset is divided into training set, validation set and test set, where: The number of images in the training set accounts for 80% of the original image dataset and is used to train the network model; The number of images in the validation set accounts for 10% of the original image dataset and is used to adjust hyperparameters and optimize the network model; The number of images in the test set accounts for 10% of the original image dataset and is used to evaluate the performance of the network model.

4. The deep learning-based space target adaptive pose estimation system according to claim 1, characterized in that: The method for the image preprocessing unit to perform image preprocessing on the received original image data set is as follows: Loading the original image data set, and performing diversified processing on the image, including: randomly rotating, flipping, scaling and shearing each frame of the image; Performing enhancement processing on the diversified processed images, including grayscale, normalization and image denoising; The image size of the enhanced image is adjusted to adapt to the input requirements of the deep learning network model unit as the preprocessed image output.

5. The deep learning-based space target adaptive pose estimation system according to claim 1, characterized in that: The method for the deep learning network model unit to generate a key point position estimation network model is: Train the network model on the training set, where the network model uses the Hourglass network model; The Hourglass network model is used to extract key point features during the training process, which includes two stages: forward propagation and back propagation. The BPnP algorithm is added to the back propagation process, and the forward propagation and back propagation processes are continuously iterated. The Adam optimization algorithm is used to dynamically adjust the weights and hyperparameters of the Hourglass network model. At the same time, the cross-domain adaptive unit of the model further adjusts the weights and hyperparameters of the Hourglass network model. The two units jointly generate a preliminary domain adaptive key point position estimation network model. According to the preliminary domain adaptive key point position estimation network model, the validation set data is used for further iterative reasoning during the verification process. The key point position coordinates obtained in the process are output to the high-precision pose estimation unit to solve the spatial target pose and evaluate the pose accuracy. The weights and hyperparameters of the preliminary domain adaptive key point position estimation network model are adjusted according to the reasoning results of the validation set to generate the key point position estimation network model.

6. The deep learning-based space target adaptive pose estimation system according to claim 1, characterized in that: The method for judging the accuracy of pose estimation is as follows: If the estimated accuracy of the space target's pose meets the requirement of better than 3°, the pose estimation is considered correct; If the spatial target pose estimation does not meet the requirement of being better than 3°, it is regarded as an incorrect pose estimation, and its result is discarded, and the current pose of the target is re-estimated.

7. The deep learning-based space target adaptive pose estimation system according to claim 5, characterized in that: When the cross-domain adaptation unit of the model further adjusts the weights and hyperparameters of the Hourglass network model, the two units jointly generate a preliminary domain adaptive key point position estimation network model. The specific process is as follows: When the adjusted weights and hyperparameters of the Hourglass network model are obtained during the training process of the deep learning network model unit, the model cross-domain adaptation unit further adjusts the weights and hyperparameters of the Hourglass network model through the pseudo-label iteration method, generates and fine-tunes new pseudo-labels, and after the iteration is completed, jointly generates a preliminary domain adaptive key point position estimation network model with the deep learning network model unit.

8. The deep learning-based space target adaptive pose estimation system according to claim 5, characterized in that: The model cross-domain adaptive unit tests the key point position estimation network model, further optimizes the deep learning network model during the test, and generates the final cross-domain adaptive key point position estimation network model as follows: During the test, the key point position estimation network model generated by the validation set in the deep learning network model unit is loaded. The ODR method is used to optimize only the parameters of the normalization layer. The training is continued iteratively to optimize the batch normalization layer parameters and generate the final cross-domain adaptive key point position estimation network model.

Images