A method, device and equipment for estimating relative posture of non-cooperative spacecraft

Abstract

The present application belongs to the technical field of pose estimation, and relates to a method, device and equipment for estimating the relative pose of a non-cooperative spacecraft. The method comprises: obtaining a training data set, the training data set comprises a plurality of initial images; performing data enhancement on the plurality of initial images to obtain a plurality of enhanced images; using the plurality of enhanced images and the label information corresponding to the enhanced images as the input of the neural network, performing network training to obtain a trained neural network; obtaining a target image and performing data enhancement and then inputting the trained neural network to obtain a corresponding predicted heat map; decoding the predicted heat map to obtain the two-dimensional coordinates of the key points on the predicted heat map; obtaining camera parameters, and constructing a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of the key points; solving the perspective n-point problem to obtain the relative pose of the non-cooperative spacecraft. The present application can improve the generalization accuracy and generalization consistency.

Description

Technical Field

[0001] The present application relates to the technical field of posture estimation, and in particular to a method, device and equipment for estimating relative posture of non-cooperative spacecraft. Background Art

[0002] Relative pose estimation of non-cooperative spacecraft is a basic key technology in active space debris processing, on-orbit servicing and space rendezvous and docking. Monocular vision is gradually becoming the preferred solution for relative pose estimation of non-cooperative spacecraft due to its advantages such as simple system structure, low cost, wide field of view and low power consumption, and has a very important application prospect.

[0003] In response to problems such as complex lighting conditions in space, weak target texture, and complex scenes, the manually designed features that traditional methods rely on lack stability and robustness. Deep learning methods have powerful feature expression and feature extraction capabilities and have gradually developed into the mainstream method in recent years.

[0004] Since the acquisition of real images and the annotation of poses are complex, difficult, time-consuming and labor-intensive, in the existing technology, monocular non-cooperative spacecraft pose estimation based on deep learning is generally based on simulation image training, and then domain adaptation methods or domain generalization methods are used to improve the performance of the model in real images.

[0005] However, domain adaptation methods rely on real images and cannot be estimated when real images are unknown. In most cases, only simulated images can be obtained during the model training phase, and real images can only be obtained after the spacecraft is launched into space. Under this limitation, mainstream domain adaptation methods cannot cope with it because they need to use images of the real target domain during the offline training phase. Domain generalization methods are not accurate and are difficult to adapt to the needs of existing technologies. Summary of the invention

[0006] Based on this, it is necessary to provide a non-cooperative spacecraft relative pose estimation method, device and equipment to address the above-mentioned technical problems, which can improve the generalization accuracy and generalization consistency, and does not require the use of real images in the offline training stage. Through network structure design and data enhancement, the model's ability to extract cross-domain invariant features (such as geometric features) can be improved, which can improve the generalization performance of the model on unseen target domain images.

[0007] A method for estimating relative pose of a non-cooperative spacecraft, comprising:

[0008] A training data set is obtained, wherein the training data set includes a plurality of initial images of a non-cooperative spacecraft; data enhancement is performed on the plurality of initial images to obtain a plurality of enhanced images; the data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation; the random foreground texture transformation includes:

[0009] ;

[0010] in, In the formula, is the random foreground texture transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset; is the first The transparency of the image; for The corresponding color-corrected image; Used to calculate the grayscale mean, the mean calculation range is the foreground mask The foreground area of ​​the logo; Used to calculate the grayscale mean, the mean calculation range is the entire image;

[0011] Using multiple enhanced images and label information corresponding to the enhanced images as inputs of the neural network, network training is performed to obtain a trained neural network;

[0012] Obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting it into the trained neural network to obtain the corresponding predicted thermal map; decode the predicted thermal map to obtain the two-dimensional coordinates of the key points on the predicted thermal map;

[0013] Obtain camera parameters, and construct a perspective n-point problem based on the camera parameters and the two-dimensional coordinates of key points;

[0014] Solve the perspective n-point problem and obtain the relative positions of non-cooperative spacecraft.

[0015] In one embodiment, data enhancement is performed on a plurality of initial images to obtain a plurality of enhanced images, including:

[0016] Performing basic data augmentation and / or advanced data augmentation on the multiple initial images to obtain multiple enhanced images;

[0017] The basic data enhancement includes: random brightness, random contrast, random erasure, solar flare, blur and Gaussian noise in sequence;

[0018] The advanced data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation.

[0019] In one embodiment, when the data enhancement includes basic data enhancement and advanced data enhancement, data enhancement is performed on multiple initial images to obtain multiple enhanced images, including:

[0020] Advanced data enhancement is first performed on multiple initial images, and then basic data enhancement is performed to obtain multiple enhanced images.

[0021] In one embodiment, after obtaining the corresponding predicted heat map, the method further includes: smoothing the predicted heat map to alleviate the peak problem so as to satisfy the Gaussian distribution assumption:

[0022] ;

[0023] In the formula, is the smoothed prediction heat map, is the Gaussian kernel, is the convolution operation, Prediction heat map.

[0024] In one embodiment, after smoothing the predicted heat map, the method further includes: converting the predicted heat map to maintain the magnitude of the predicted heat map:

[0025] ;

[0026] In the formula, The heatmap of the predictions after smoothing and transformation.

[0027] In one embodiment, the predicted heat map is decoded to obtain the two-dimensional coordinates of key points on the predicted heat map, including:

[0028] The initial representation of the predicted heat map distribution function is logarithmically processed to obtain the logarithmic representation of the predicted heat map distribution function:

[0029] ;

[0030] In the formula, for Logarithmic representation of ;

[0031] Let the derivative of the logarithmic representation of the predicted heat map distribution function at the mean of the two-dimensional Gaussian distribution be zero, and use the second-order Taylor expansion to approximate it, and get the final representation of the predicted heat map distribution function:

[0032] ;

[0033] ;

[0034] in, In the formula, The logarithm of the heat map distribution function is expressed in the two-dimensional Gaussian distribution mean The derivative at To predict the final representation of the heat map distribution function; for of Pick hour Logarithmic representation of ; is the point of Taylor expansion, when The closer When , the second-order Taylor expansion is closer to the original function; for exist The first-order derivative at , for exist The second-order derivative at can be solved in the prediction thermal diagram;

[0035] Solve the final representation of the predicted heat map distribution function to obtain the two-dimensional coordinates of the key points on the predicted heat map:

[0036] ;

[0037] in, In the formula, To predict the two-dimensional coordinates of key points on the heat map, is the heat map resolution scaling factor.

[0038] In one embodiment, obtaining camera parameters and constructing a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of key points includes:

[0039] Obtain camera parameters, which include: camera intrinsic parameters, rotation matrix, and translation vector;

[0040] According to the two-dimensional coordinates of the key points, the corresponding three-dimensional coordinates are obtained;

[0041] According to the camera intrinsic parameters, rotation matrix, translation vector, two-dimensional coordinates of key points and corresponding three-dimensional coordinates, construct the perspective n-point problem:

[0042] ;

[0043] In the formula, For the Key points, is the number of key points, is the scale parameter, For the The estimated values ​​of the key point coordinates, are the camera internal parameters, is the rotation matrix, is the translation vector, is the corresponding 3D coordinate in the model.

[0044] In one embodiment, solving the perspective n-point problem to obtain the relative positions and poses of non-cooperative spacecraft includes:

[0045] The EPnP algorithm is used as the solver of the perspective n-point problem. The two-dimensional coordinates of the key points and the corresponding three-dimensional coordinates are used as the input of the solver. The root mean square error between the two-dimensional coordinates of the key points and the two-dimensional coordinates of the reprojected key points is calculated. The RANSAC framework is used to obtain the two-dimensional coordinates of the key points that meet the root mean square error and use them as the output of the solver. The output of the solver is used as the relative pose of the non-cooperative spacecraft.

[0046] A non-cooperative spacecraft relative position and attitude estimation device, comprising:

[0047] The enhancement module is used to obtain a training data set, wherein the training data set includes a plurality of initial images of a non-cooperative spacecraft; perform data enhancement on the plurality of initial images to obtain a plurality of enhanced images; the data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation; the random foreground texture transformation includes:

[0048] ;

[0049] in, In the formula, is the random foreground texture transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset; is the first The transparency of the image; for The corresponding color-corrected image; Used to calculate the grayscale mean, the mean calculation range is the foreground mask The foreground area of ​​the logo; Used to calculate the grayscale mean, the mean calculation range is the entire image;

[0050] A training module, used to use a plurality of enhanced images and label information corresponding to the enhanced images as inputs of a neural network to perform network training and obtain a trained neural network;

[0051] The decoding module is used to obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting it into the trained neural network to obtain the corresponding predicted thermal map; the predicted thermal map is decoded to obtain the two-dimensional coordinates of the key points on the predicted thermal map;

[0052] A construction module, used for obtaining camera parameters, and constructing a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of key points;

[0053] The solving module is used to solve the perspective n-point problem and obtain the relative position and posture of non-cooperative spacecraft.

[0054] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0055] The above-mentioned non-cooperative spacecraft relative pose estimation method, device and equipment, when the real image is unknown, based on the simulation training data, improves the ability of the deep learning model to extract cross-domain invariant features, and optimizes the heat map decoding and pose solution process to improve the generalization accuracy and generalization consistency of the model. Specifically: data enhancement is performed, including random background texture enhancement and random foreground texture enhancement based on components, and random style enhancement of the image as a whole, so as to randomize the background texture, foreground texture and random style respectively, realize random enhancement of training data, and improve the generalization accuracy of the model on multiple test domains; a key point distribution-aware heat map decoding strategy is adopted to reduce the quantization error in the process of converting the estimated heat map into the final coordinates of the key points, and improve the accuracy of key point positioning; a pose solution strategy based on the RANSAC (random sampling consistency) framework and a Refine (pose iterative optimization) strategy are combined to achieve robust pose estimation, thereby achieving high-precision estimation of spacecraft pose estimation; combined with heat map decoding and pose estimation, the accuracy of spacecraft pose estimation is effectively improved. In addition, unlike the domain adaptation method that can train different models for different test domains, the domain generalization involved in this application only trains one model to adapt to different test domains, and does not require the use of real images in the offline training stage, which can improve the generalization performance of the model on the target image (including generalization accuracy and generalization consistency). BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of a flow chart of a method for estimating relative pose of a non-cooperative spacecraft in one embodiment;

[0057] Figure 2 A schematic diagram of a non-cooperative spacecraft relative attitude estimation method applied to an on-orbit relative attitude measurement task in an embodiment;

[0058] Figure 3is a schematic diagram of advanced data enhancement in one embodiment;

[0059] Figure 4 is a structural diagram of an SPNv2 model used in an embodiment;

[0060] Figure 5 is a structural block diagram of a non-cooperative spacecraft relative pose estimation device in one embodiment;

[0061] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0063] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0064] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0065] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0066] This application provides a method for estimating relative pose of non-cooperative spacecraft, such as Figure 1 The flowchart shown, in one embodiment, includes:

[0067] Step 101, obtaining a training data set, the training data set including a plurality of initial images of a non-cooperative spacecraft; performing data enhancement on the plurality of initial images to obtain a plurality of enhanced images.

[0068] Specifically:

[0069] obtaining a training data set, the training data set including a plurality of initial images of a non-cooperative spacecraft;

[0070] Performing basic data augmentation and / or advanced data augmentation on the multiple initial images to obtain multiple enhanced images;

[0071] Basic data augmentation includes, in order: random brightness, random contrast, random erasing, sun flare, blur, and Gaussian noise;

[0072] Advanced data augmentation includes: random background texture transformation, random foreground texture transformation, and random neural style transformation.

[0073] More specifically:

[0074] Define a piecewise function:

[0075] ;

[0076] In the formula, , is the random foreground texture transformation, is a random background texture transformation, is random neural style transfer; For equivalue mapping (such as );

[0077] Advanced data augmentation includes:

[0078] ;

[0079] In the formula, To enhance the image, is the function composite symbol, For execution The trigger probability of the transformation, For execution The trigger probability of the transformation, For execution The trigger probability of the transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset;

[0080] Random background texture transformation Includes: Based on Identify the non-foreground range and convert the image Set as Image background;

[0081] Random foreground texture transformation include:

[0082] ;

[0083] in, ;

[0084] In the formula, is the random foreground texture transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset; is the first The transparency of each image varies randomly between 0.25 and 0.75. for The corresponding color-corrected image; Used to calculate the grayscale mean, the mean calculation range is the foreground mask The foreground area of ​​the logo; Used to calculate the grayscale mean, the mean calculation range is the entire image;

[0085] Random Neural Style Transfer Use existing technology.

[0086] Advanced data enhancement can be applied before or after basic data enhancement. They are independent of each other and there are three enhancement methods: , , The order can be randomly swapped.

[0087] Preferably, when data enhancement includes basic data enhancement and advanced data enhancement, data enhancement is performed on multiple initial images to obtain multiple enhanced images, including: first performing advanced data enhancement on the multiple initial images, and then performing basic data enhancement to obtain multiple enhanced images, so as to improve the random enhancement effect.

[0088] In this step, the training dataset may be a COCO dataset or other public datasets, and random images in the dataset are used to randomly transform the textures of the background and foreground of the training images to achieve random data enhancement.

[0089] The specific process of basic data enhancement belongs to the existing technology.

[0090] In advanced data enhancement, random foreground texture transformation and random background texture transformation (i.e., component-aware domain randomization, which is also domain randomization of image components) can effectively highlight the geometric features of foreground targets and improve adaptability to different backgrounds that appear in practical applications, thereby weakening the model's learning of texture features and enhancing the model's learning of geometric features; random style enhancement can improve the model's fitting of texture style, thereby improving the model's ability to extract geometric features; since the main difference between images in different domains is texture, and geometric features are invariant across domains, the combination of the above two enhancement methods can be seamlessly integrated into model training, significantly improving the model's ability to extract geometric features and improving the model's generalization performance on multiple test domains.

[0091] Step 102, using the multiple enhanced images and the label information corresponding to the enhanced images as inputs of the neural network, performing network training to obtain a trained neural network.

[0092] Specifically:

[0093] Each enhanced image has corresponding label information, which includes: pose, heat map, target mask and detection frame; the image has a pose label; the three-dimensional model of the target is obtained, the mask image is obtained based on the target three-dimensional model and several semantic key points are predefined, and the three-dimensional coordinate values ​​of these points are based on the target body coordinate system; the camera intrinsic parameters are known, and the three-dimensional coordinate values ​​of the semantic key points are reprojected to the two-dimensional image plane through the pose parameters marked by the pose label to obtain the two-dimensional coordinates of the corresponding semantic key points; based on the target mask image or the two-dimensional coordinates of the semantic key points, the detection frame label of the target is obtained; the two-dimensional coordinates of the semantic key points are used to generate the heat map label of the target.

[0094] A plurality of enhanced images and label information corresponding to the enhanced images are used as inputs of the neural network to perform network training and obtain a trained neural network.

[0095] In this step, network training includes: image segmentation (training through detection box labels), pose regression (training through pose labels), and heat map regression (training through heat map labels) to perform multi-task learning to make the output results of the trained neural network more robust. The specific training process belongs to the existing technology.

[0096] The trained neural network has multiple output results, among which the output prediction heat map is decoded.

[0097] Step 103, obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting it into the trained neural network to obtain the corresponding predicted thermal map; decode the predicted thermal map to obtain the two-dimensional coordinates of the key points on the predicted thermal map.

[0098] Specifically:

[0099] Acquire a target image of a non-cooperative spacecraft, and perform data enhancement on the target image to obtain an enhanced target image; input the enhanced target image into the trained neural network as the input of the neural network to obtain a predicted thermal map corresponding to the enhanced target image;

[0100] Performing logarithmic processing on the initial representation of the predicted heat map distribution function to obtain the logarithmic representation of the predicted heat map distribution function;

[0101] The derivative of the logarithmic representation of the predicted heat map distribution function at the mean of the two-dimensional Gaussian distribution is set to zero, and the second-order Taylor expansion is used for approximation to obtain the final representation of the predicted heat map distribution function;

[0102] The final representation of the predicted heat map distribution function is solved to obtain the two-dimensional coordinates of the key points on the predicted heat map.

[0103] Preferably, after obtaining the predicted heat map corresponding to the enhanced target image, the method further includes: smoothing the predicted heat map to alleviate the peak problem so as to satisfy the Gaussian distribution assumption.

[0104] Further preferably, after smoothing the predicted heat map, the method further includes: converting the predicted heat map to maintain the magnitude of the predicted heat map.

[0105] More specifically:

[0106] Acquire a target image of a non-cooperative spacecraft, and perform data enhancement on the target image to obtain an enhanced target image;

[0107] Input the enhanced target image into the trained neural network as the input of the neural network, and obtain the predicted heat map corresponding to the enhanced target image:

[0108] ;

[0109] in, ;

[0110] In the formula, To predict the initial representation of the heat map distribution function, To predict the pixel coordinates in the heat map, is the mean of the two-dimensional Gaussian distribution, is the covariance matrix, for The transpose of is the variance;

[0111] The initial representation of the predicted heat map distribution function is logarithmically processed to obtain the logarithmic representation of the predicted heat map distribution function:

[0112] ;

[0113] In the formula, for Logarithmic representation of ;

[0114] Based on the assumption that the predicted heat map satisfies the two-dimensional Gaussian distribution, the logarithm of the predicted heat map distribution function is expressed in the mean of the two-dimensional Gaussian distribution The derivative at is zero, and the second-order Taylor expansion is used for approximation to obtain the final representation of the predicted heat map distribution function:

[0115] ;

[0116] ;

[0117] in, ;

[0118] In the formula, The logarithm of the heat map distribution function is expressed in the two-dimensional Gaussian distribution mean The derivative at To predict the final representation of the heat map distribution function; for of Pick hour Logarithmic representation of ; is the point of Taylor expansion, when The closer When , the second-order Taylor expansion is closer to the original function; for exist The first-order derivative at , for exist The second-order derivative at can be solved in the prediction thermal diagram;

[0119] Solve the final representation of the predicted heat map distribution function to obtain the two-dimensional coordinates of the key points on the predicted heat map:

[0120] ;

[0121] in, ;

[0122] In the formula, To predict the two-dimensional coordinates of key points on the heat map, is the heat map resolution scaling factor.

[0123] Preferably, the prediction heat map is smoothed to alleviate the problem of multiple peaks in the prediction heat map, so that the prediction heat map better meets the Gaussian distribution assumption:

[0124] ;

[0125] In the formula, is the smoothed prediction heat map, is the Gaussian kernel, is the convolution operation, To predict the heat map;

[0126] Further preferably, the smoothed predicted heat map is transformed to maintain the magnitude of the predicted heat map:

[0127] ;

[0128] In the formula, The heatmap of the predictions after smoothing and transformation.

[0129] In this step, heatmap decoding uses the distribution assumption of the heatmap to estimate the sub-pixel coordinates of the potential maximum activation.

[0130] Step 104, obtaining camera parameters, and constructing a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of the key points.

[0131] Specifically:

[0132] Get the camera parameters, which include: camera intrinsic parameters, rotation matrix and translation vector;

[0133] According to the two-dimensional coordinates of the key points, the corresponding three-dimensional coordinates are obtained;

[0134] Construct a perspective n-point problem based on the camera's intrinsic parameters, rotation matrix, translation vector, two-dimensional coordinates of key points, and corresponding three-dimensional coordinates.

[0135] More specifically:

[0136] Get the camera parameters, which include: camera intrinsic parameters, rotation matrix and translation vector;

[0137] According to the two-dimensional coordinates of the key points, the corresponding three-dimensional coordinates are obtained;

[0138] According to the camera intrinsic parameters, rotation matrix, translation vector, two-dimensional coordinates of key points and corresponding three-dimensional coordinates, construct the perspective n-point problem (PnP problem):

[0139] ;

[0140] In the formula, For the Key points, is the number of key points, is the scale parameter, For the The estimated values ​​of the key point coordinates, are the camera internal parameters, is the rotation matrix, is the translation vector, is the corresponding 3D coordinate in the model.

[0141] In this step, the camera parameters are known quantities and can be directly obtained according to the prior art.

[0142] Step 105, solving the perspective n-point problem to obtain the relative position and posture of the non-cooperative spacecraft.

[0143] Specifically:

[0144] The EPnP algorithm is used as the solver of the perspective n-point problem. The two-dimensional coordinates of the key points and the corresponding three-dimensional coordinates constitute multiple 2D-3D coordinate pairs. The coordinate pairs are randomly sampled using the RANSAC framework and used as the input of the solver to obtain the output of the solver, i.e., the initial pose. The key points are reprojected to obtain the corresponding projection points. The root mean square error between the two-dimensional coordinates of the key points and the two-dimensional coordinates of the projection points is calculated. The key points that meet the root mean square error (less than the root mean square error threshold) are considered as inliers, and those that do not meet the requirement are considered as outliers.

[0145] Perform multiple random sampling and iterations until the number of iterations reaches the preset upper limit or the number of inliers reaches the preset threshold (for example, the inlier ratio reaches 90%), and then obtain the current inlier set; obtain the corresponding 2D-3D coordinate pair based on the current inlier set, and further iterate and solve in combination with the initial pose obtained in RANSAC to optimize the initial pose and obtain the relative pose of the non-cooperative spacecraft.

[0146] The root mean square error is defined as follows:

[0147] ;

[0148] In the formula, RMSE is the two-dimensional coordinate of the key point And the two-dimensional coordinates of the projection point The RMS error between It is The two-dimensional coordinates of the key points or Two-dimensional coordinates of the projection points; It is twice the number of key points because each key point coordinate contains two values ​​on the X-axis and Y-axis.

[0149] In this step, RANSAC is combined with iterative optimization to improve the accuracy and robustness of the final pose solution.

[0150] The above non-cooperative spacecraft relative pose estimation method, based on simulated training data, improves the ability of deep learning models to extract cross-domain invariant features when the real image is unknown, and optimizes the heat map decoding and pose solution process to improve the generalization accuracy and consistency of the model. Specifically: data enhancement is performed, including random background texture enhancement and random foreground texture enhancement based on components, and random style enhancement of the entire image, so as to randomize the background texture, foreground texture and random style respectively, realize random enhancement of training data, and improve the generalization accuracy of the model on multiple test domains; a key point distribution-aware heat map decoding strategy is adopted to reduce the quantization error in the process of converting the estimated heat map into the final coordinates of the key points, and improve the accuracy of key point positioning; a pose solution strategy based on the RANSAC (random sampling consistency) framework and a Refine (pose iterative optimization) strategy are combined to achieve robust pose estimation, thereby achieving high-precision estimation of spacecraft pose estimation; the combination of heat map decoding and pose estimation effectively improves the accuracy of spacecraft pose estimation. In addition, unlike the domain adaptation method that can train different models for different test domains, the domain generalization involved in this application only trains one model to adapt to different test domains, and does not require the use of real images in the offline training stage, which can improve the generalization performance of the model on the target image (including generalization accuracy and generalization consistency).

[0151] In a specific embodiment, Figure 2 As shown, the present application is used to solve the relative posture relationship between the service spacecraft and the target spacecraft in space. After the model is trained on the ground, it is deployed on the service spacecraft. During the use phase, the service spacecraft uses the image of the target spacecraft taken by the optical camera to solve the relative posture between the two ( and ), thereby providing measurement support for the next step of task implementation. This application can be seamlessly integrated into the training and attitude estimation stages of the model, providing concise and efficient data enhancement options, as well as accurate and effective technical processes for heat map decoding and pose solution, which has important research significance and broad application prospects.

[0152] After the model is trained, the spacecraft pose estimation consists of two parts: key point regression and pose calculation. The key point regression is implemented using a deep learning model based on a neural network, and the pose calculation uses the EPnP algorithm as the basic solver and the RANSAC+Refine strategy (i.e., the RANSAC framework is used to remove wild value interference, and iterative optimization is used to obtain the optimal pose). Specifically:

[0153] 1) Select several key points for the target spacecraft.

[0154] 2) Construct training data including simulated images, key point labels, foreground masks and pose labels. Due to the limited availability of real training data, this application only uses simulated images to train the model.

[0155] 3) Three advanced data augmentations, random style, random background and random foreground, are used for simulated images, such as Figure 3 shown. Figure 3 The transformed images shown in the figure (including: random style images after style transformation, random background images after random background transformation, and random foreground images after random foreground transformation) show the effect of using each advanced data enhancement independently. They are not the final enhanced images. The final enhanced image is obtained by superimposing three advanced image enhancements and several basic data enhancements.

[0156] 4) There is no restriction on the deep learning model used. Here, SPNv2 is used as an example to illustrate the specific model training and testing process. Figure 4 The structure of the SPNv2 model shown in the figure, SPNv2 consists of an EfficientNet backbone network for feature extraction, a bidirectional feature pyramid network (BiFPN) for multi-scale feature fusion, and a multi-branch prediction head with shared weights for image segmentation, pose regression (such as and ) and heatmap regression.

[0157] 5) The heat map branch is decoded and the pose is solved to get the pose, which is better than the pose obtained by direct network regression. Under normal circumstances, the pose solved by the heat map is selected as the final pose; when the pose solved by the heat map fails, the pose by direct regression is used as the final pose to ensure the robustness of the pose solution.

[0158] Taking the SPEED+ dataset as an example, after obtaining the trained model, the model is tested on two unknown target domains, Sunlamp and Lightbox, to evaluate the generalization accuracy and generalization consistency of the model.

[0159] Among them, in order to evaluate the comprehensive generalization ability of the method proposed in this application on different test domains, and to combine it with other generalization accuracy indicators to more comprehensively measure the generalization performance of the model, a mean level standard deviation (MLSTD, defined as the ratio of the standard deviation of the model's error on different test domains to the mean) is proposed to quantitatively evaluate the generalization consistency:

[0160] ;

[0161] In the formula, is the error in the estimated spacecraft pose, which can measure the accuracy level of the model in a certain test domain, that is, the generalization accuracy.

[0162] As shown in Table 1, the accuracy comparison of data enhancement by different methods shows that the strategy of random style ratio 25% + random background ratio 25% + random foreground ratio 25% achieves the best performance and significantly improves the accuracy.

[0163] As shown in Table 2, the accuracy comparison of pose estimation by different methods shows that the strategy of heat map decoding and iterative optimization can improve the performance and significantly improve the generalization accuracy.

[0164] Table 1: Comparison of the accuracy of data enhancement by different methods

[0165]

[0166] Table 2: Comparison of the accuracy of pose estimation by different methods

[0167]

[0168] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0169] The present application also provides a non-cooperative spacecraft relative posture estimation device, such as Figure 5 As shown, in one embodiment, it includes: an enhancement module 501, a training module 502, a decoding module 503, a construction module 504 and a solution module 505, wherein:

[0170] The enhancement module 501 is used to obtain a training data set, wherein the training data set includes a plurality of initial images of a non-cooperative spacecraft; perform data enhancement on the plurality of initial images to obtain a plurality of enhanced images;

[0171] A training module 502 is used to use a plurality of enhanced images and label information corresponding to the enhanced images as inputs of a neural network to perform network training to obtain a trained neural network;

[0172] The decoding module 503 is used to obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting the image into the trained neural network to obtain the corresponding predicted thermal map; decode the predicted thermal map to obtain the two-dimensional coordinates of the key points on the predicted thermal map;

[0173] A construction module 504 is used to obtain camera parameters and construct a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of the key points;

[0174] The solving module 505 is used to solve the perspective n-point problem and obtain the relative position and posture of the non-cooperative spacecraft.

[0175] For the specific definition of a non-cooperative spacecraft relative pose estimation device, please refer to the definition of a non-cooperative spacecraft relative pose estimation method mentioned above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0176] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a non-cooperative spacecraft relative posture estimation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0177] Those skilled in the art can understand that Figure 6 The structure shown in Figure 6 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0178] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method in the above embodiment are implemented.

[0179] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0180] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0181] The content not described in detail in this specification belongs to the prior art well known to those skilled in the art.

[0182] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0183] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for estimating relative pose of non-cooperative spacecraft, characterized in that: include: acquiring a training data set, the training data set comprising a plurality of initial images of a non-cooperative spacecraft; Performing data enhancement on multiple initial images to obtain multiple enhanced images; Data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation; random foreground texture transformation includes: ； in, In the formula, is the random foreground texture transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset; is the first The transparency of the image; for The corresponding color-corrected image; Used to calculate the grayscale mean, the mean calculation range is the foreground mask The foreground area of ​​the logo; Used to calculate the grayscale mean, the mean calculation range is the entire image; Using multiple enhanced images and label information corresponding to the enhanced images as inputs of the neural network, network training is performed to obtain a trained neural network; Obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting it into the trained neural network to obtain the corresponding predicted thermal map; decode the predicted thermal map to obtain the two-dimensional coordinates of the key points on the predicted thermal map; Obtain camera parameters, and construct a perspective n-point problem based on the camera parameters and the two-dimensional coordinates of key points; Solve the perspective n-point problem and obtain the relative positions of non-cooperative spacecraft.

2. A non-cooperative spacecraft relative pose estimation method according to claim 1, characterized in that: Data enhancement is performed on multiple initial images to obtain multiple enhanced images, including: Performing basic data augmentation and / or advanced data augmentation on the multiple initial images to obtain multiple enhanced images; The basic data enhancement includes: random brightness, random contrast, random erasure, solar flare, blur and Gaussian noise in sequence; The advanced data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation.

3. A non-cooperative spacecraft relative pose estimation method according to claim 2, characterized in that: When data enhancement includes basic data enhancement and advanced data enhancement, data enhancement is performed on multiple initial images to obtain multiple enhanced images, including: Advanced data enhancement is first performed on multiple initial images, and then basic data enhancement is performed to obtain multiple enhanced images.

4. A non-cooperative spacecraft relative pose estimation method according to any one of claims 1 to 3, characterized in that: After obtaining the corresponding predicted heat map, the following steps are also included: smoothing the predicted heat map to alleviate the peak problem and satisfy the Gaussian distribution assumption: ； In the formula, is the smoothed prediction heat map, is the Gaussian kernel, is the convolution operation, Prediction heat map.

5. A non-cooperative spacecraft relative pose estimation method according to claim 4, characterized in that: After smoothing the predicted heat map, it also includes: transforming the predicted heat map to maintain the magnitude of the predicted heat map: ； In the formula, The heatmap of the predictions after smoothing and transformation.

6. A non-cooperative spacecraft relative pose estimation method according to any one of claims 1 to 3, characterized in that: Decode the predicted heat map to obtain the two-dimensional coordinates of the key points on the predicted heat map, including: The initial representation of the predicted heat map distribution function is logarithmically processed to obtain the logarithmic representation of the predicted heat map distribution function: ； In the formula, for Logarithmic representation of ; Let the derivative of the logarithmic representation of the predicted heat map distribution function at the mean of the two-dimensional Gaussian distribution be zero, and use the second-order Taylor expansion to approximate it, and get the final representation of the predicted heat map distribution function: ； ； in, In the formula, The logarithm of the heat map distribution function is expressed in the two-dimensional Gaussian distribution mean The derivative at To predict the final representation of the heat map distribution function; for of Pick hour Logarithmic representation of ; is the point of Taylor expansion, when The closer When , the second-order Taylor expansion is closer to the original function; for exist The first-order derivative at , for exist The second-order derivative at can be solved in the prediction thermal diagram; Solve the final representation of the predicted heat map distribution function to obtain the two-dimensional coordinates of the key points on the predicted heat map: ； in, In the formula, To predict the two-dimensional coordinates of key points on the heat map, is the heat map resolution scaling factor.

7. A non-cooperative spacecraft relative pose estimation method according to any one of claims 1 to 3, characterized in that: Obtain camera parameters, and construct a perspective n-point problem based on the camera parameters and the two-dimensional coordinates of key points, including: Obtain camera parameters, which include: camera intrinsic parameters, rotation matrix, and translation vector; According to the two-dimensional coordinates of the key points, the corresponding three-dimensional coordinates are obtained; According to the camera intrinsic parameters, rotation matrix, translation vector, two-dimensional coordinates of key points and corresponding three-dimensional coordinates, construct the perspective n-point problem: ； In the formula, For the Key points, is the number of key points, is the scale parameter, For the The estimated values ​​of the key point coordinates, are the camera internal parameters, is the rotation matrix, is the translation vector, is the corresponding 3D coordinate in the model.

8. A non-cooperative spacecraft relative pose estimation method according to any one of claims 1 to 3, characterized in that: Solve the perspective n-point problem to obtain the relative position of the non-cooperative spacecraft, including: The EPnP algorithm is used as the solver of the perspective n-point problem. The two-dimensional coordinates of the key points and the corresponding three-dimensional coordinates are used as the input of the solver. The root mean square error between the two-dimensional coordinates of the key points and the two-dimensional coordinates of the reprojected key points is calculated. The RANSAC framework is used to obtain the two-dimensional coordinates of the key points that meet the root mean square error and use them as the output of the solver. The output of the solver is used as the relative pose of the non-cooperative spacecraft.

9. A non-cooperative spacecraft relative pose estimation device, characterized in that: include: An enhancement module, configured to acquire a training data set, wherein the training data set includes a plurality of initial images of a non-cooperative spacecraft; Performing data enhancement on multiple initial images to obtain multiple enhanced images; Data enhancement includes: random background texture transformation, random foreground texture transformation and random neural style transformation; random foreground texture transformation includes: ； in, In the formula, is the random foreground texture transformation, is the first images, is the first The foreground mask corresponding to the image, is an initial image randomly selected from the training dataset; is the first The transparency of the image; for The corresponding color-corrected image; Used to calculate the grayscale mean, the mean calculation range is the foreground mask The foreground area of ​​the logo; Used to calculate the grayscale mean, the mean calculation range is the entire image; A training module, used to use a plurality of enhanced images and label information corresponding to the enhanced images as inputs of a neural network to perform network training and obtain a trained neural network; The decoding module is used to obtain the target image of the non-cooperative spacecraft and perform data enhancement before inputting it into the trained neural network to obtain the corresponding predicted thermal map; the predicted thermal map is decoded to obtain the two-dimensional coordinates of the key points on the predicted thermal map; A construction module, used for obtaining camera parameters, and constructing a perspective n-point problem according to the camera parameters and the two-dimensional coordinates of key points; The solving module is used to solve the perspective n-point problem and obtain the relative position and posture of non-cooperative spacecraft.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images