A spacecraft pose estimation method based on fusion rendering information

By integrating rendering information and improved SwinTransformer Block, the problem of insufficient accuracy and real-time accuracy of spacecraft pose estimation in harsh environments is solved, more efficient pose estimation and robustness are achieved, and technology for on-orbit service is provided support.

CN117557622BActive Publication Date: 2024-11-19PLA PEOPLES LIBERATION ARMY OF CHINA STRATEGIC SUPPORT FORCE AEROSPACE ENG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311328376.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-13
Publication Date
2024-11-19
Estimated Expiration
2043-10-13

AI Technical Summary

Technical Problem

The existing spacecraft posture estimation methods are insufficient in accuracy and real-time under harsh lighting conditions and high texture backgrounds, and are poorly robust to unpredictable space environments, and deep learning methods are relatively low in recognition efficiency.

Method used

Using the method of fusion rendering information, the spacecraft posture is estimated through visual means, using the improved SwinTransformer Block, the number of channels is increased to reduce model training costs, extract multi-scale features, and through the nonlinear residual self-attention mechanism and post-processing module Perform pose prediction.

Benefits of technology

It improves the accuracy and real-time nature of spacecraft posture estimation, improves the efficiency and robustness of model training, and provides technical support for on-orbit services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117557622B_ABST
    Figure CN117557622B_ABST
Patent Text Reader

Abstract

A method for estimating the pose of a spacecraft that integrates rendering information. The method process includes: constructing an RGB image dataset containing the pose information of the spacecraft, rendering the image to obtain a rendered image, generating a foreground mask for each of the original image and the rendered image, and forming the original image, the rendered image and their respective masks into 8 channels of the input network. After batch partitioning and batch merging, the Swin Transformer Block is lightly improved, and self-attention calculation is performed therein to extract multi-scale feature information, and the translation and rotation are input into the fully connected layer to obtain the estimated pose parameters. In view of the defects of spacecraft pose estimation, this method not only optimizes the input data more, but also improves the efficiency of the Swin Transformer self-attention calculation, while ensuring the accuracy of pose estimation, it improves its rapidity and space on-orbit service capability, and also has practical application significance for space attack and defense.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of spacecraft pose estimation, and in particular to a spacecraft pose estimation method that integrates rendering information. Background Art

[0002] In space missions, image-based spacecraft pose estimation is an emerging research hotspot, such as Surrey Space Center's Remove DEBRIS, NASA's Restore-L, DARPA's Phoenix Project; and some space startups' planned missions, such as Astroscale and Infinite Orbits, are conducting research in this field. Spacecraft pose estimation is crucial for on-orbit missions, especially in harsh lighting conditions and against a high-texture background (i.e., the Earth). The completion of non-cooperative target tracking and positioning tasks, formation flying, rendezvous, docking, repair of faulty satellites, and space debris removal all rely on visual sensors to accurately and robustly estimate the pose of close-range targets.

[0003] Swin Transformer is a hierarchical Vision Transformer based on a moving window. This hierarchical structure not only provides feature information at all scales, but is also very flexible. In addition, because self-attention is calculated within a small window, it is different from the original Transformer network with a time complexity of O(n 2 d)(n is the sequence length, d is the vector dimension), its computational complexity increases linearly with the image size, which greatly reduces the computational complexity and improves the efficiency of downstream task settlement; at the same time, through the shifting operation, the two adjacent windows interact with each other, and a hierarchical connection is formed between the upper and lower layers to achieve global modeling. For pose estimation, multi-scale features are crucial. Therefore, using SwinTransformer as the training backbone network can effectively improve the accuracy of the results.

[0004] The invention patent with application number CN 107103621 A discloses a "non-cooperative spacecraft three-dimensional pose measurement system based on a single light field camera". The spacecraft position, three-dimensional shape and pose are obtained by calibrating a single light field camera, depth estimation algorithm and three-dimensional reconstruction algorithm. The system has a simple working mode and low energy consumption, but the method relies on pre-launch device calibration and has poor robustness to unpredictable space environments. The invention patent with patent number CN 113091754 A discloses a "non-cooperative spacecraft pose integrated estimation and inertial parameter determination method". By establishing a dual quaternion framework to describe pose motion, this method can improve the accuracy of pose estimation, but its recognition efficiency is lower than that of deep learning methods. Summary of the invention

[0005] (I) Purpose of the invention

[0006] The purpose of the present invention is to provide a method for estimating the pose of a spacecraft by integrating rendering information. The method estimates the pose information of the spacecraft by visual means, renders the spacecraft image, and inputs the rendered data into the improved SwinTransformer Block. The number of channels is increased by the rendering part, and the self-attention model solving block of the model is reduced, which effectively reduces the training cost of the model, improves the training efficiency, and improves the real-time and accuracy of the spacecraft pose estimation.

[0007] (II) Technical solution

[0008] The technical solution of the present invention is a method for estimating the position and posture of a spacecraft by integrating rendering information, which is characterized by comprising: constructing a satellite attitude rotation data set, preprocessing satellite attitude image data, extracting local features of satellite images by a convolutional neural network, position encoding the local features by a position encoder, re-extracting features of the encoded image features by a nonlinear residual self-attention mechanism, processing features by a post-processing module, and finally outputting attitude prediction information by a fully connected layer. The implementation steps of the method are as follows:

[0009] A certain type of spacecraft is carried on a robotic arm rotating platform, and data is collected on it in 6 degrees of freedom. A spacecraft attitude dataset is constructed and preprocessed to unify the image size to 224×224×3. To enhance the position information of the spacecraft in the dataset, 3DMax software is used to automatically render the images, and a foreground mask is generated for each input image and rendered image, resulting in a total of four images. An enlarged bounding box is then generated based on the two foreground masks to ensure that the bounding box has the same aspect ratio as the original image, so that the four images can be cropped. Finally, the cropped images are scaled and bilinear interpolation is applied to make the cropped images and the original images have the same size. Among them, the input image and the rendered image each have 3 channels (RGB), and the mask images of the input image and the rendered image each have 1 channel (grayscale image) to form the 8 channels of the subsequent network input. The obtained 8-channel image, i.e., H×W×8, is batch-segmented with a batch size of 4×4 to obtain a tensor of The batches are merged to extract multi-scale feature information. A 1×1 convolution is used to increase the number of channels in the dimension of 128 to obtain a The data is input into Swin Transformer for self-attention calculation, and only three levels of feature extraction are performed, and finally a 7×7×1024 tensor is obtained.

[0010] The mechanism of nonlinear self-attention mechanism is as follows:

[0011] Multiply the two-dimensional feature matrix A encoded by the encoder with the three weight matrices to obtain Qa ,K a ,V a :

[0012]

[0013] W q , W k , W v The three weight matrices composed of query vector q, key vector k, and value vector v are respectively obtained by a ,K a ,V a Use activation functions for nonlinear operations:

[0014]

[0015] Then add Q, K, V to matrix A to get Q', K', V':

[0016]

[0017] Calculate the self-attention value:

[0018]

[0019] where d k is the square root of the dimension of the key vector.

[0020] The output self-attention value is passed to the post-processing module, and the features of the post-processing module are input to the fully connected layer, which is composed of multiple neurons. Two fully connected layers are set for the translation and rotation respectively to output the required parameters, thus completing the position and posture estimation of the spacecraft.

[0021] (III) Main advantages of the present invention

[0022] The above technical solution of the present invention has the following advantages: The present invention is mainly used for vision-based spacecraft pose estimation, providing technical support for space on-orbit services. A spacecraft pose estimation network structure that integrates rendering information is used to improve the accuracy of spacecraft pose estimation, and a lightweight Swin Transformer Block structure is proposed to enhance the model's solution capability, further improving the training efficiency of deep learning network models and the real-time performance of pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a process framework diagram of the present invention;

[0024] Figure 2 It is a schematic diagram of the network architecture of the main method of the present invention;

[0025] Figure 3It is a schematic diagram of picture processing of the present invention; DETAILED DESCRIPTION

[0026] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions of the present invention are further described in conjunction with specific examples.

[0027] Embodiment 1 of the present invention is a method for estimating the position and posture of a spacecraft by integrating rendering information, see Figure 1 , follow the following steps: use a robotic arm rotating platform to carry a certain type of spacecraft, collect data on it in 6 degrees of freedom, and preprocess it to unify the image size to 224×224×3; after obtaining the satellite attitude rotation data set, render the obtained RGB image, and generate a foreground mask for each input image and rendered image, and get a total of four images; generate an enlarged bounding box based on the two foreground masks to crop the image, and scale it to make it the same size as the original image, and the aspect ratio of the image remains unchanged during this process; scale the cropped image and apply bilinear interpolation to make the cropped image and the original image have the same size. Among them, the input image and the rendered image each have 3 channels (RGB), and the mask image of the input image and the rendered image each has 1 channel (grayscale image) to form the 8 channels of the subsequent network input. The input image, the rendered image and their respective masks form the input network, first use a batch size of 4×4 to obtain the 8-channel image, that is, H×W×8, and the image size obtained after batch segmentation In order to extract multi-scale feature information, batch merging is performed. In order to extract multi-scale feature information, a 1×1 convolution is used to increase the number of channels in the dimension of 128 to obtain a The data is input into the Swin Transformer self-attention calculation, and the output self-attention value is passed to the post-processing module, which consists of a global average pooling layer and a Dropout layer. The global average pooling layer regularizes the network to prevent overfitting. The parameter of the Dropout layer is set to 0.5 to improve the generalization ability of the model; the output image size remains unchanged, and the first stage is completed. Similarly, the second and third level feature extraction is performed, and finally a 7×7×1024 tensor is obtained; similarly, the operations of the subsequent two stages are performed to obtain the feature map, which is then input into two fully connected layers set for translation and rotation respectively, where the network learning parameter is set to 0.001, the cross entropy loss function is selected, and the deep learning model is trained using the gradient descent method. The loss function is converged through continuous iterative training, and finally the spacecraft pose estimation parameters are obtained.

[0028] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A method for estimating a spacecraft pose by integrating rendering information, characterized in that: Construct a spacecraft pose dataset, render the original RGB images obtained, and generate a foreground mask for the original image and the rendered image. The traditional 4-module Swin Transformer network is lightweight and improved to a 3-module network. To this end, the input data is processed and the number of channels is expanded. The input of the improved Swin Transformer module is used to perform three-level self-attention calculations in sequence to extract multi-scale feature information, which is then input into two fully connected layers set for translation and rotation to obtain the estimated pose parameters. The specific steps are as follows: 1) Constructing a spacecraft pose dataset In a darkroom environment with a single light source, a robotic arm is used to rotate and move a certain type of spacecraft on a platform to simulate its motion posture. The optical camera position is fixed to collect RGB images of the spacecraft containing posture information, including three translational degrees of freedom along the three rectangular coordinate axes of x, y, and z and three rotational degrees of freedom of pitch, yaw, and roll, a total of six degrees of freedom posture data. According to the input format of the subsequently improved Swin Transformer module, the data is preprocessed and the input data is set to a tensor of size 224×224×3. 2) Get the rendering data to form 8 channels of data In order to enhance the position information of the spacecraft in the dataset, the images were automatically rendered using 3DMax software, and a foreground mask was generated for the original image and the rendered image respectively. An enlarged bounding box was generated based on the two foreground masks to achieve reasonable cropping of the image, ensuring that the bounding box had the same aspect ratio as the input image. Scale the cropped image and apply bilinear interpolation to make the cropped image and the original image have the same size; the original image and the rendered image are RGB images, each occupying 3 channels, and the mask image is a grayscale image, occupying 1 channel, which together constitute the 8 channels of the subsequent network input; 3) Reduce Swin Transformer calculation modules In order to reduce the memory usage of data and improve the efficiency of model solving, the four computing modules of the traditional Swin Transformer are reduced to three modules, and the input data is processed. The obtained 8-channel image, i.e. H×W×8, is obtained by batch segmentation using a batch size of 4×4. The image size is , batched and merged to extract multi-scale feature information, and a 1×1 convolution is used to increase the number of channels in the dimension of 128 to obtain The tensor is used as the input of the improved lightweight Swin Transformer module; 4) Construct hierarchical feature extraction blocks Will The tensor of size is input into the Swin Transformer module for self-attention calculation. The mechanism of the self-attention mechanism is: multiply the two-dimensional feature matrix A encoded by the encoder with the three weight matrices to obtain Q a ,K a ,V a : W q , W k , W v The three weight matrices composed of query vector q, key vector k, and value vector v are respectively obtained by a ,K a ,V a Use activation functions for nonlinear operations: Then add Q, K, V to matrix A to get Q', K', V': according to Calculate the self-attention value, where d k It is the square root of the key vector dimension. For the second and third level feature extraction, a 1×1 convolution is also used to increase the number of channels in the dimension of 128, and finally a 7×7×1024 tensor is obtained, which is then input into the fully connected layer to obtain the estimated pose parameters.