Space non-cooperative target 6D attitude estimation method and device, medium and product
Through the improved YOLOv8 key point detection network and attention mechanism, combined with EfficientViT and TripletAttention, the problem of difficulty in taking into account accuracy and efficiency in 6D pose estimation in space non-cooperation targets is solved, and efficient and robust pose estimation effect is achieved.
Patent Information
- Application Number
- CN202411087244.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
AI Technical Summary
In the 6D pose estimation of spatial non-cooperation goals, accuracy and efficiency are difficult to take into account, especially when processing high-resolution images and large amounts of input tokens, traditional convolutional neural networks and attention mechanism methods have inefficiencies. The challenge.
The improved YOLOv8 key point detection network is adopted, combined with the EfficientViT network and the TripletAttention network, and the 2D coordinate prediction model is trained through the A3DKS loss function, and the 6D pose is determined using the perspective n-point algorithm, which improves feature extraction ability and real-time performance, and is under low computing overhead. Enhance robustness.
It achieves higher accuracy and efficiency in spatial non-cooperation target 6D pose estimation, which is suitable for real-time detection on embedded devices with limited computing resources, improving the robustness of the occlusion environment and the accuracy of pose estimation.
Smart Images

Figure CN119068052A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, medium and product for estimating 6D posture of a non-cooperative target in space. Background Art
[0002] Space non-cooperative targets refer to man-made objects in orbit that cannot actively provide their position, attitude or motion information. These objects may include out-of-control satellites, rocket debris, etc. Since these objects are uncontrolled, their existence may pose a certain threat to other spacecraft operating in near-Earth space. At the same time, some spacecraft that were originally of great value may become non-cooperative targets due to the failure of certain key systems. In order to clean up space debris or repair these valuable non-cooperative spacecraft, certain means need to be taken to approach and capture these targets. This requires accurate grasp of the rotation and translation (i.e., 6D attitude) of the target. However, these non-cooperative targets usually lack predefined visual markers, active communication systems or other auxiliary equipment, making it extremely challenging to track, identify and estimate their attitude. With the rapid development of artificial intelligence and computer vision technology, applying deep learning to 6D attitude estimation of space non-cooperative targets has become a mainstream practice. At present, there are many hardware solutions for achieving 6D attitude estimation, including binocular systems, structured light systems, and TOF systems. However, the accuracy of these systems is significantly limited by distance, and the complexity and cost are high. In contrast, the 6D pose estimation method based on monocular images has the advantages of simple hardware equipment and low cost, so it has become the most commonly used solution.
[0003] There are currently two main methods for achieving monocular 6D pose estimation. The first method is to build an end-to-end network to directly regress the 6D pose from a single image. For example, the Spacecraft Pose Network (SPN) extracts features through a multi-layer convolutional neural network, and then regresses the pose probability, pose weight, and target box position through different fully connected layer branches to calculate the 6D pose. The Unreal Rendered Spacecraft on Orbit Network (URSONet) also uses a multi-layer convolutional neural network as the backbone to predict pose and position through different branches. The Localization-oriented Spacecraft Pose Estimation Neural Network (LSPNet) uses two convolutional neural networks for position prediction and secondary optimization, and then predicts the pose through a convolutional neural network to finally obtain the 6D pose. The second method is based on key point detection and perspective n-point algorithm solution, and its accuracy is usually higher than the first method. This type of method usually uses convolutional neural networks to predict key points, but traditional convolutional neural networks have difficulty fitting when facing the diverse 2D projection shape features of spatial three-dimensional targets, resulting in insufficient estimation accuracy. In order to improve accuracy, many studies use heat map methods to generate dense prediction maps of target key points through encoder-decoders. A representative one is the High-Resolution Network (Hourglass), which is a classic encoder-decoder structure similar to an hourglass, which can realize the mapping from the input image to the key point probability distribution map. However, this network cannot maintain the features of high-resolution images and is prone to loss of accuracy. For this reason, a study proposed the HRNet network. In the process of feature extraction, multiple horizontal parallel paths are introduced. The scale is not lost during the transmission of feature maps in the horizontal direction, and the scale of features is lost when the upper branch is passed to the lower branch. In this way, the ability to extract high-level abstract features is guaranteed as well as the ability to extract detailed features. This achieves higher-precision key point prediction. Due to its high prediction density, this type of method has a relatively large number of parameters and requires high computing resources, which is not conducive to deployment on embedded devices.
[0004] In order to reduce background interference, many methods also add target detection as a preliminary stage, using target detectors such as YOLO and Faster-RCNN to extract the target area before performing pose prediction. If the target detection model is pre-placed in the real-time 6D pose estimation model, the required number of parameters and computing power will increase significantly.
[0005] With the widespread application of Transformer networks and other attention mechanisms in the field of computer vision, some studies have incorporated various attention mechanisms into existing networks for key point prediction, and their accuracy can significantly surpass that of methods based on traditional convolutional neural networks. However, when calculating the attention matrix, especially when dealing with high-resolution images and a large number of input tokens, it still faces the challenge of low efficiency.
[0006] To this end, starting from the direct key point regression network with good real-time performance, the YOLOv8 framework was investigated. The framework has achieved SOTA-level accuracy in the real-time target detection network. Its architecture adopts the backbone plus neck and head mode. The backbone completes the main feature extraction to generate 5-level feature maps, and the neck uses the path aggregation network (PANet) structure to realize the feature splicing of feature maps of different levels, and outputs the feature maps of different levels to the head to decouple the relevant parameters. The C2f module and the Spatial Pyramid Pooling-Fast (SPPF) module of YOLOv8 fully consider the parameter quantity and computational efficiency, so that its reasoning speed can meet the real-time requirements. Due to the universality of its architecture, it can be migrated to the field of key point detection. However, when YOLOv8 is migrated to the key point detection task of spatial non-cooperative targets, it still has the problem of limited feature extraction ability and difficult convergence.
[0007] Therefore, it is necessary to propose a lightweight 6D pose estimation method for spatial non-cooperative targets based on monocular image key point detection that takes into account both accuracy and efficiency. Summary of the invention
[0008] The purpose of this application is to provide a method, device, medium and product for 6D posture estimation of space non-cooperative targets, which improves the accuracy and efficiency of 6D posture estimation of space non-cooperative targets.
[0009] To achieve the above objectives, this application provides the following solutions:
[0010] In a first aspect, the present application provides a method for estimating a 6D posture of a space non-cooperative target, comprising:
[0011] Obtain an image of a non-cooperative target in the space to be estimated and 3D coordinates of multiple key points;
[0012] The image of the non-cooperative spatial target to be estimated is input into the 2D coordinate prediction model to obtain the 2D coordinates of each key point of the non-cooperative spatial target to be estimated; the 2D coordinate prediction model is obtained by training an improved YOLOv8 key point detection network using a training set according to an A3DKS loss function; the training set includes multiple images of non-cooperative spatial targets for training and the actual values of the 2D coordinates of multiple corresponding key points, and the improved YOLOv8 key point detection network is obtained by improving the YOLOv8 model using an EfficientViT network and a TripletAttention network;
[0013] The perspective n-point algorithm is used to determine the 6D posture of the non-cooperative spatial target to be estimated based on the 2D coordinates and 3D coordinates of each key point of the non-cooperative spatial target to be estimated; the 6D posture includes a rotation matrix and a translation vector.
[0014] Optionally, the training process of the 2D coordinate prediction model includes:
[0015] Obtaining the training set;
[0016] Construct the improved YOLOv8 key point detection network;
[0017] According to the A3DKS loss function, the images of the non-cooperative targets in the training space are taken as input, and the actual values of the 2D coordinates of the corresponding multiple key points are taken as output, so as to train the improved YOLOv8 key point detection network and obtain the 2D coordinate prediction model.
[0018] Optionally, according to the A3DKS loss function, taking the image of each of the training space non-cooperative targets as input and taking the actual values of the 2D coordinates of the corresponding multiple key points as output, training the improved YOLOv8 key point detection network to obtain the 2D coordinate prediction model, including:
[0019] Initialize the improved YOLOv8 key point detection network;
[0020] Taking the images of the non-cooperative targets in the training space as input and the actual values of the 2D coordinates of the corresponding multiple key points as output, the improved YOLOv8 key point detection network is trained for multiple rounds to obtain the 2D coordinate prediction model; wherein the training process of the improved YOLOv8 key point detection network under any current number of training rounds includes:
[0021] Inputting the images of each of the training spatial non-cooperative targets into the improved YOLOv8 key point detection network under the current training round number, and obtaining the predicted values of the 2D coordinates of multiple key points under the current training round number;
[0022] Calculate the A3DKS loss function value for the current training round based on the predicted values and actual values of the 2D coordinates of multiple key points for the current training round;
[0023] Determine whether a training stop condition is met; the training stop condition is that the A3DKS loss function value under the current number of training rounds is less than a preset value or reaches a preset number of training rounds;
[0024] If yes, the improved YOLOv8 key point detection network under the current training round number is determined as the 2D coordinate prediction model;
[0025] If not, the improved YOLOv8 key point detection network under the current training round is updated to the improved YOLOv8 key point detection network under the next training round, and the method of "inputting the images of each of the training spatial non-cooperative targets into the improved YOLOv8 key point detection network under the current training round to obtain the predicted values of the 2D coordinates of multiple key points under the current training round" is returned until the training stop condition is reached to obtain the 2D coordinate prediction model.
[0026] Optionally, the improved YOLOv8 key point detection network includes: a backbone network, a neck network and a decoupling head;
[0027] The backbone network includes: an overlapping block embedding module, a first EfficientViT module, a first downsampling module, a second EfficientViT module, a second downsampling module, a third EfficientViT module and an SPPF module; the first EfficientViT module includes 1 EfficientViT block, the second EfficientViT module includes 2 EfficientViT blocks, and the third EfficientViT module includes 3 EfficientViT blocks;
[0028] The neck network includes: a path aggregation network and a TripletAttention network; the TripletAttention network includes: three residual transformation modules.
[0029] Optionally, the EfficientViT block includes: a first token interaction layer, a first feedforward network, a second token interaction layer, self-attention calculation, a splicing projection layer, a third token interaction layer and a second feedforward network.
[0030] Optionally, the A3DKS loss function value for the current training round is calculated based on the predicted values and actual values of the 2D coordinates of the multiple key points for the current training round, including:
[0031] The A3DKS loss function is used to calculate the A3DKS loss function value under the current training round according to the predicted values and actual values of the 2D coordinates of multiple key points under the current training round; the A3DKS loss function is:
[0032]
[0033] Among them, L A3DKS is the A3DKS loss function value; N kpts is the total number of key points; d n is the Euclidean distance between the predicted value and the actual value of the 2D coordinate of the nth key point; Z n is the depth of the nth key point; f is the focal length of the camera; s h is the harmonic mean of the scale areas of the non-cooperative targets in each training space in the training set; δ is the visibility of the key point.
[0034] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for estimating a 6D posture of a spatial non-cooperative target.
[0035] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for estimating a 6D posture of a spatial non-cooperative target.
[0036] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for estimating a 6D posture of a spatial non-cooperative target.
[0037] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0038] The present application discloses a method, device, medium and product for estimating the 6D posture of a spatial non-cooperative target. First, an image of the spatial non-cooperative target to be estimated and the 3D coordinates of multiple key points are obtained; then, the image of the spatial non-cooperative target to be estimated is input into a 2D coordinate prediction model to obtain the 2D coordinates of each key point of the spatial non-cooperative target to be estimated; the 2D coordinate prediction model is obtained by training an improved YOLOv8 key point detection network using a training set according to an A3DKS loss function; the training set includes multiple images of spatial non-cooperative targets for training and the actual values of the 2D coordinates of the corresponding multiple key points, and the improved YOLOv8 key point detection network is obtained by improving the YOLOv8 model using an EfficientViT network and a TripletAttention network; finally, the perspective n-point algorithm is used to determine the 6D posture of the spatial non-cooperative target to be estimated based on the 2D coordinates and 3D coordinates of each key point of the spatial non-cooperative target to be estimated; the 6D posture includes a rotation matrix and a translation vector. This application improves the backbone network of the YOLOv8 model by introducing EfficientViT, utilizing its powerful feature extraction capability, and optimizing the real-time performance of the improved YOLOv8 key point detection network; uses the TripletAttention network mechanism in feature fusion to improve the robustness of occluded environments with low computational overhead; designs the A3DKS loss function to comprehensively improve the accuracy and efficiency of 6D pose estimation of spatial non-cooperative targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 A schematic diagram of a flow chart of a method for estimating a 6D posture of a non-cooperative target in space provided in an embodiment of the present application;
[0041] Figure 2 This is a lightweight architecture diagram for 6D pose estimation of spatial non-cooperative targets based on monocular image key point detection;
[0042] Figure 3 This is the EfficientViT block structure diagram;
[0043] Figure 4 This is the structural diagram of the TripletAttention network;
[0044] Figure 5 This is the schematic diagram of the pinhole camera model;
[0045] Figure 6 A flowchart of a method for performing 6D pose estimation of a spatial non-cooperative target using a SPEED dataset provided for a specific embodiment;
[0046] Figure 7 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0048] The purpose of this application is to provide a method, device, medium and product for 6D posture estimation of space non-cooperative targets, aiming to improve the accuracy and efficiency of 6D posture estimation of space non-cooperative targets.
[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0050] In an exemplary embodiment, Figure 1 and Figure 2 As shown, the spatial non-cooperative target 6D posture estimation method in this embodiment includes:
[0051] Step 1: Obtain the image of the non-cooperative target in the space to be estimated and the 3D coordinates of multiple key points.
[0052] Step 2: Input the image of the non-cooperative spatial target to be estimated into the 2D coordinate prediction model to obtain the 2D coordinates of each key point of the non-cooperative spatial target to be estimated.
[0053] Among them, the 2D coordinate prediction model is obtained by training the improved YOLOv8 key point detection network using the training set according to the A3DKS loss function; the training set includes multiple images of training spatial non-cooperative targets and the actual values of the 2D coordinates of multiple corresponding key points. The improved YOLOv8 key point detection network is obtained by improving the YOLOv8 model using the EfficientViT network and the TripletAttention network.
[0054] As an optional implementation, the training process of the 2D coordinate prediction model includes:
[0055] Step 21: Get the training set.
[0056] Step 22: Build an improved YOLOv8 key point detection network.
[0057] As an optional implementation, an improved YOLOv8 key point detection network includes: a backbone network, a neck network and a decoupling head.
[0058] The backbone network includes: an overlapping block embedding module, a first EfficientViT module, a first downsampling module, a second EfficientViT module, a second downsampling module, a third EfficientViT module and an SPPF module; the first EfficientViT module includes 1 EfficientViT block, the second EfficientViT module includes 2 EfficientViT blocks, and the third EfficientViT module includes 3 EfficientViT blocks.
[0059] The neck network includes: a path aggregation network and a TripletAttention network; the TripletAttention network includes: three residual transformation modules.
[0060] As an optional implementation, the EfficientViT block includes: a first token interaction layer, a first feedforward network, a second token interaction layer, self-attention calculation, a splicing projection layer, a third token interaction layer and a second feedforward network.
[0061] Specifically, the EfficientViT network is used as the backbone network of YOLOv8 to replace the original network structure, and a TripletAttention network for high-level features is added to the Path Aggregation Network (PANet) structure of YOLOv8 to obtain an improved YOLOv8 key point detection network.
[0062] like Figure 2 As shown in the figure, the image processing process of the improved YOLOv8 key point detection network includes:
[0063] 1. The input image first passes through the overlapping block embedding module to divide the image into overlapping small areas and map it to the C1-dimensional embedding space. Then, it passes through the area of n1 EfficientViT blocks in series (i.e., the first EfficientViT module), the first downsampling module, and the area of n2 EfficientViT blocks in series (i.e., the second EfficientViT module) to obtain the first feature map. The first feature map passes through the second downsampling module and the area of n3 EfficientViT blocks in series (i.e., the third EfficientViT module) to obtain the second feature map. Finally, the second feature map is subjected to serial multi-scale pooling operation through the SPPF module with a residual structure to obtain the third feature map. Among them, n1=1, n2=2, and n3=3 are selected for lightweight.
[0064] Among them, each EfficientViT module uses the EfficientViT block as the basic module, such as Figure 3 As shown in the figure, the implementation process of each EfficientViT block is as follows:
[0065] (1) The input of the EfficientViT block first passes through a token interaction layer (i.e., the first token interaction layer), which uses depthwise convolution to capture local information.
[0066] (2) Then, feature conversion is performed through a feedforward network (i.e., the first feedforward network) to obtain a converted feature map.
[0067] (3) The converted feature map is divided into multiple parts (i.e., H1, H2, and H i ), each part goes through the following steps: another token interaction layer (i.e., the second token interaction layer), (q, k, v), and self-attention calculation.
[0068] (4) The outputs of multiple self-attention heads are concatenated and passed through a concatenated projection layer.
[0069] (5) The output of the concatenated projection layer passes through a token interaction layer again (i.e., the third token interaction layer).
[0070] (6) The output of the third token interaction layer passes through another feedforward network (i.e., the second feedforward network).
[0071] The structural pattern of feedforward network to multi-head self-attention and then to feedforward network is called sandwich structure. Figure 3 The output of the first feedforward network shown in the figure is divided into multiple feature maps after residual connection, which are respectively passed through H1, H2 and H iThe resulting feature maps are concatenated and projected until the next feedforward network. This model is called Cascaded Group Attention (CGA). i The area of EfficientViT blocks concatenated is EfficientViT blocks × n i (i=1, 2, 3), self-attention is calculated for each head in the cascaded group attention of each EfficientViT block:
[0072]
[0073] in, is the jth attention head H of the i-th EfficientViT block j The output features of; Attn(·) is the self-attention function; X ij is the jth attention head H of the i-th EfficientViT block j Input features of is the jth attention head H of the i-th EfficientViT block j The q projection matrix; is the jth attention head H of the i-th EfficientViT block j The k projection matrix; is the jth attention head H of the i-th EfficientViT block j The v projection matrix.
[0074] The splicing projection module concatenates the outputs of all attention heads and passes them through a linear layer W P The total output of the mapping output dimension is h:
[0075]
[0076] in, is the final output of the splicing projection module; Concat is a cascade operation; is a linear layer that projects the output features of the i-th EfficientViT block back to the same dimension as the input.
[0077] This structure saves computational resources when processing high-dimensional features while maintaining the ability to process information in different subspaces. Each head operates independently on different partitions of the input features, and the final output is fused through a linear layer for further processing. EfficientViT significantly improves the performance of ViT under small samples through a sandwich structure and CGA, and is an efficient visual Transformer backbone network.
[0078] 2. Use PANet to further extract features from the feature maps output by the EfficientViT block ×n2 of EfficientViT, the feature maps output by the EfficientViT block ×n3 (i.e., the first feature map), and the feature maps output by the SPPF module (i.e., the third feature map). Specifically: Path 1 uses a deconvolution block to upsample the third feature map, accompanied by the concatenation operation of the first feature map and the third feature map. Path 2 uses a convolution block to downsample the feature map output by Path 1, accompanied by the feature maps output by the SPPF and the first C2f module of Path 1 in the feature extraction direction, and outputs feature maps of different levels to the decoupling head.
[0079] In the PANet network, in addition to the original structure, this application also introduces an additional TripletAttention path (i.e., TripletAttention network) in the processing of high-level feature maps. TripletAttention consists of three parallel branches, which are responsible for establishing the interaction between the channel dimension and the height dimension, the channel dimension and the width dimension, and the height dimension and the width dimension, to realize the cross-dimensional attention mechanism.
[0080] like Figure 2 As shown in the bold path in , the TripletAttention path takes the third feature map as input.
[0081] like Figure 4 As shown in Figure 1, the implementation process of TripletAttention includes:
[0082] For the first residual transformation module, the three dimensions (C, H, W) of the third feature map are first permuted in the height dimension to make it into a shape of (W, H, C), where C is the channel dimension, H is the height dimension, and W is the width dimension. Then the Z-Pool operation is used to compress the W dimension of the tensor to 2 to obtain a feature map of (2, H, C). Next, a 7×7 convolution kernel is used to perform a convolution operation on the feature map, and the number of output channels is set to 1, and then a (1, H, C) shape attention weight is obtained through the Sigmoid function. Finally, this attention weight is multiplied with the feature map (W, H, C) after the first branch permutation, and the result is permuted back to the shape of (C, H, W). In this way, the attention interaction between the channel dimension and the width and height dimensions is established.
[0083] The second residual transformation module has a similar structure to the first residual transformation module, the only difference is that it permutes the width dimension to establish the interaction between the channel dimension and the height dimension. First, the input feature map is permuted to the shape of (H, C, W), and then the H dimension is compressed to 2 using Z-Pool. After 7×7 convolution and Sigmoid function, an attention weight of the shape of (1, C, W) is obtained. Then the attention weight is applied to the permuted feature map, and finally the result is permuted back to the original (C, H, W) shape.
[0084] The third residual transformation module is relatively simple. It directly applies the Z-Pool operation on the input feature map to compress the channel dimension to 2. Then, after a 7×7 convolution and a Sigmoid function, a (1, H, W) spatial attention map is obtained.
[0085] Compared with the traditional single spatial attention or channel attention mechanism, Triplet Attention cleverly uses dimension permutation. Specifically, the leftmost residual transformation module uses wc channel rotation and cw channel rotation, that is, the width dimension of the feature map is swapped with the channel dimension, and the middle residual transformation module swaps the height dimension with the channel dimension, and also includes the subsequent ch channel rotation to restore. The three residual transformation modules all use the Z-Pool compression technique to achieve attention interaction between different dimensions with a very small parameter amount overhead. While improving the feature expression ability, it also enhances the network's robustness to complex scenes such as target occlusion. Introducing TripletAttention into the neck network of YOLOv8 makes up for the lack of PANet's ability to process advanced features, and can significantly enhance the representation ability of advanced feature maps, thereby improving the performance of the entire target detection network. At the same time, due to its low computational overhead, the introduction of TripletAttention will not significantly increase the number of parameters and computation of the model, which is very suitable for deployment on embedded devices with limited computing resources for real-time detection tasks.
[0086] Step 23: According to the A3DKS loss function, the images of the non-cooperative targets in the training space are used as input, and the actual values of the 2D coordinates of the corresponding multiple key points are used as output to train the improved YOLOv8 key point detection network to obtain a 2D coordinate prediction model.
[0087] As an optional implementation, step 23 includes:
[0088] Step 231: Initialize the improved YOLOv8 key point detection network.
[0089] Step 232: Taking the images of the non-cooperative targets in the training space as input and the actual values of the 2D coordinates of the corresponding multiple key points as output, the improved YOLOv8 key point detection network is trained for multiple rounds to obtain a 2D coordinate prediction model; wherein the training process of the improved YOLOv8 key point detection network under any current number of training rounds includes:
[0090] Step 2321: Input the images of each training spatial non-cooperative target into the improved YOLOv8 key point detection network under the current training round number to obtain the predicted values of the 2D coordinates of multiple key points under the current training round number.
[0091] Step 2322: Calculate the A3DKS loss function value for the current training round based on the predicted values and actual values of the 2D coordinates of multiple key points for the current training round.
[0092] As an optional implementation, step 2322 includes:
[0093] Using the A3DKS loss function, the A3DKS loss function value under the current training round is calculated according to the predicted values and actual values of the 2D coordinates of multiple key points under the current training round; the A3DKS loss function is:
[0094]
[0095] Among them, L A3DKS is the A3DKS loss function value; N kpts is the total number of key points; d n is the Euclidean distance between the predicted value and the actual value of the 2D coordinate of the nth key point; Z n is the depth of the nth key point; f is the focal length of the camera; s h is the harmonic mean of the scale areas of the non-cooperative targets in each training space in the training set; δ is the visibility of the key point, which is 1 if visible and 0 if invisible.
[0096] Specifically, the design process of the A3DKS loss function includes:
[0097] The object key point similarity (OKS) loss function of the YOLOv8 model is:
[0098]
[0099] Among them, L OKS is the OKS loss function value; s is the spatial non-cooperative target scale; k n For custom weights.
[0100] Based on the pinhole camera model, the approximate 3D keypoint similarity (A3DKS) loss function is designed to replace the OKS loss function. Based on the OKS loss function, the A3DKS loss function removes the area scale variable and introduces the key point depth information. Figure 5 As shown in the figure, the cuboid on the right is the target object, the plane on the left is the camera imaging plane, O is the camera optical center, c is the phase plane reference point, and C is the target object coordinate system reference point. For point P on the target object, the imaging point p can be obtained on the left plane according to the pinhole imaging principle. According to the similar triangle principle in the figure, for the real 2D point p and 3D point P, the following equations are obtained:
[0101]
[0102] For the predicted 2D point p' and the corresponding 3D point P', there is an equation:
[0103]
[0104] Subtracting the two equations yields:
[0105]
[0106] Where, f is the focal length of the camera; d err is the Euclidean distance between the predicted and actual values of the 2D coordinates of the key points; D err is the Euclidean distance between the approximate and actual 3D coordinates of the keypoint.
[0107] Finally, D err Replace d err , the harmonic mean constant s of the data target scale h Replace the scale variables s, k n The factor is converted to a constant k by default, which is used to adjust the loss to a reasonable range, and the A3DKS loss function can be obtained.
[0108] Compared with the OKS loss function, the AD3KS loss function makes full use of the depth information. n The approximate 3D scale is restored by the camera focal length f. The model improves the imbalance of key point penalties for the same sample and between different samples by optimizing the loss function of the approximate 3D scale, thereby further improving the convergence accuracy of key point prediction.
[0109] Step 2323: Determine whether the training stop condition is met; the training stop condition is that the A3DKS loss function value under the current training round number is less than a preset value or reaches a preset training round number.
[0110] Step 2324: If yes, the improved YOLOv8 key point detection network under the current training round number is determined as the 2D coordinate prediction model.
[0111] Step 2325: If not, update the improved YOLOv8 key point detection network under the current training round number to the improved YOLOv8 key point detection network under the next training round number, and return to step 2321 until the training stop condition is reached to obtain a 2D coordinate prediction model.
[0112] Specifically, when updating and improving the YOLOv8 key point detection network, the Mosaic data augmentation technology and the Exponential Moving Average (EMA) strategy are used to update the weights. The specific operation is: two weights are set, one is the main weight and the other is the secondary weight. The main weight is obtained by improving the error back propagation of the YOLOv8 key point detection network, and the secondary weight is iterated by the following formula:
[0113] θ′ t1 =αθ′ t1-1 +(1-α)θ t .
[0114] Among them, θ' t1 is the secondary weight after the t1th iteration; θ' t1-1 is the secondary weight after the t1-1th iteration; θ t1 is the main weight after the t1th iteration; α is the attenuation factor, which ranges from 0 to 1. α usually takes a value close to 1, which increases the stability of the model and reduces the impact of noise, thereby achieving stronger generalization ability.
[0115] Step 3: Using the perspective n-point algorithm, the 6D posture of the non-cooperative target to be estimated is determined based on the 2D coordinates and 3D coordinates of each key point of the non-cooperative target to be estimated; the 6D posture includes a rotation matrix and a translation vector.
[0116] Specifically, the Efficient Perspective-n-Points (EPnP) algorithm in the Perspective-n-Points (PnP) algorithm is used. The EPnP algorithm is a method for solving the PnP problem. It is mainly used to calculate the posture and position of the camera when several three-dimensional space points and their corresponding two-dimensional image points are known. The advantage of this algorithm is that it can handle situations with more than four points and use a linear method to estimate the posture of the camera, which greatly improves the calculation efficiency and stability. The EPnP algorithm first represents the three-dimensional point as a linear combination of several virtual control points. The selection of these control points is based on the statistical characteristics of the entire point set.
[0117]
[0118] Among them, P i1 is the 3D coordinate of the i1th key point; α ij P i The j-th control point represents the weight coefficient; C j is the 3D coordinate of the jth control point.
[0119] Through this representation, the key point is that 3DEPnP transforms the complex nonlinear optimization problem into a simple linear problem:
[0120] Mx1=0.
[0121] Among them, M is the matrix after the 3D coordinate projection equations of the 15 key points are sorted; x1 is the matrix composed of the 3D coordinates of the 4 control points. This method simplifies the 15 key points into 4 control points. This achieves a fast solution to the problem, and the parameters R and t related to the posture in M are obtained. In practical applications, considering that the prediction results of the 2D coordinates of the key points are prone to outliers, and the EPnP algorithm can only solve 4 sets of 2D coordinates of the key points and their corresponding 3D coordinates, while this application has 15 sets of key points, the EPnP solution process can still be carried out by appropriately eliminating some points.
[0122] Therefore, EPnP is combined with the RANSAC algorithm to eliminate abnormal points in the matching process. RANSAC selects a subset of these 15 key points by random sampling. For any subset Set, EPnP is used to calculate the corresponding R and t, and the R and t that minimize the reprojection error are selected from all subsets to obtain the final solution. The reprojection error is calculated as follows:
[0123]
[0124] Where e(P) is the reprojection error; (x, y) is the 2D coordinate of the key point; (P x ,P y ,P z ) is the coordinate of the three-dimensional space point; K is the camera intrinsic parameter; R is the rotation matrix; t is the translation vector; [R|t] is the matrix constructed by R and t calculated by EPnP. Specific embodiments
[0126] The specific embodiment uses the SPEED dataset to further verify the method of the present application. The SPEED dataset was proposed by the European Space Agency in 2019. It uses the Tango satellite as a non-cooperative target in space. The SPEED dataset is a dataset that includes images of the Tango satellite and their corresponding 6D pose annotations. Figure 6 As shown, the specific process includes:
[0127] Step S1: Construct training set and test set.
[0128] 1) By observing multiple images of the Tango satellite in the SPEED data set, 15 key points (P0, P1, ..., P 15 ) as the key points of the model.
[0129] 2) For any key point P a (a=1,2,...,15), select g 2D images of non-cooperative targets in the space to be estimated with 6D pose annotations (amg a1 ,...,amg ag ), the poses corresponding to these images are annotated as (q a1 , t a1 ), (q a2 , t a2 ),...,(q ag , t ag ), q ab and t ab Respectively for AMG ab The rotation vector and translation vector of the target 6D posture are calculated by amg ab Observations of the Tango satellite in the manual marking of key points P a The 2D coordinates p ab .
[0130] 3) Using multi-view Figure 3 Corner measurement method, by combining key points P from different perspectives a The 2D coordinates and the corresponding projection matrix are used to construct a linear equation system. The solution of the equation system is the 3D coordinates of each key point, including:
[0131] Let the 3D coordinates P of the key points on the spatial non-cooperative target be a =(X a ,Y a ,Z a ,1), for 2D image amg ab , according to the rotation q of its 6D pose ab With translation t ab , by q ab The corresponding rotation matrix R is derived ab , combined with the camera internal parameters to construct the projection matrix Proj ab =K[R ab |t ab ], combined with P a In AMG ab The manually marked 2D coordinates p ab =(xab ,y ab ), using p ab =Proj ab P a The relationship between , we can get the following equations:
[0132] (x ab Project ab-3 -Proj ab-1 )×[X a ,Y a ,Z a ,1] T =0.
[0133] (y ab Project ab-3 -Proj ab-2 )×[X a ,Y a ,Z a ,1] T =0.
[0134] Among them, Proj ab-1 、Proj ab-2 and Proj ab-3 Proj ab Columns 1, 2, and 3 of .
[0135] For the selected g 2D images (amg a1 ,...,amg ag ):
[0136] (x a1 Project a1-3 -Proj a1-1 )×[X a ,Y a ,Z a ,1] T =0.
[0137] (y a1 Project a1-3 -Proj a1-2 )×[X a ,Y a ,Z a ,1] T =0.
[0138] (x a2 Project a2-3 -Proj a2-1 )×[X a ,Y a ,Z a ,1] T =0.
[0139] (y a2 Project a2-3 -Proj a2-2 )×[X a ,Y a ,Z a ,1] T =0. ...
[0141] (x ag Project ag-3 -Proj ag-1 )×[X a ,Y a ,Z a ,1] T =0.
[0142] (y ag Project ag-3 -Proj ag-2 )×[X a ,Y a ,Z a ,1] T =0.
[0143] [X a ,Y a ,Z a ,1] T Let the left side be A and we can get:
[0144] A×[X a ,Y a ,Z a ,1] T =0.
[0145] Solve using SVD decomposition:
[0146] A=U∑V T .
[0147] The 3D coordinates of the key points are the matrix V T The last column of .
[0148] For all key points, the 3D coordinates P1, P2, ..., P 15 By performing the above operation, we can get the coordinate set Set of the 3D coordinates of these 15 key points. 3D .
[0149] For each 2D image in the SPEED dataset, annotate the rotation matrix and translation vector according to its corresponding 6D posture, calculate its corresponding projection matrix, and convert Set 3DThe 2D coordinates of the key points are obtained by projecting the projection matrix onto the 2D image, and multiple images of the tango satellite and the corresponding 2D coordinates of the key points are obtained.
[0150] 4) Divide the training set and test set into a 4:1 ratio.
[0151] Step S2: Use the training set to train the improved YOLOv8 key point detection network to obtain a 2D coordinate prediction model.
[0152] Step S3: Use the test set and the 2D coordinate prediction model to perform 6D pose estimation. It is found that when performing 6D pose estimation on the test set, the absolute position error is 0.113m and the pose angle error is 1.416°.
[0153] In an exemplary embodiment, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for estimating a 6D posture of a spatial non-cooperative target.
[0154] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a method for estimating a 6D posture of a spatial non-cooperative target is implemented.
[0155] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements a method for estimating a 6D posture of a spatial non-cooperative target when executed by a processor.
[0156] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for estimating 6D posture of a non-cooperative target in space is implemented.
[0157] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0159] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0160] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0162] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A 6D pose estimation method for non-cooperative targets in space. It is characterized in that The space non-cooperative target 6D posture estimation method comprises: Obtain an image of a non-cooperative target in the space to be estimated and 3D coordinates of multiple key points; The image of the non-cooperative spatial target to be estimated is input into the 2D coordinate prediction model to obtain the 2D coordinates of each key point of the non-cooperative spatial target to be estimated; the 2D coordinate prediction model is obtained by training an improved YOLOv8 key point detection network using a training set according to an A3DKS loss function; the training set includes multiple images of non-cooperative spatial targets for training and the actual values of the 2D coordinates of multiple corresponding key points, and the improved YOLOv8 key point detection network is obtained by improving the YOLOv8 model using an EfficientViT network and a TripletAttention network; The perspective n-point algorithm is used to determine the 6D posture of the non-cooperative spatial target to be estimated based on the 2D coordinates and 3D coordinates of each key point of the non-cooperative spatial target to be estimated; the 6D posture includes a rotation matrix and a translation vector.
2. The method for estimating 6D posture of a non-cooperative space target according to claim 1, It is characterized in that The training process of the 2D coordinate prediction model includes: Obtaining the training set; Construct the improved YOLOv8 key point detection network; According to the A3DKS loss function, the images of the non-cooperative targets in the training space are taken as input, and the actual values of the 2D coordinates of the corresponding multiple key points are taken as output, so as to train the improved YOLOv8 key point detection network and obtain the 2D coordinate prediction model.
3. The method for estimating 6D posture of a non-cooperative space target according to claim 2, It is characterized in that According to the A3DKS loss function, the images of the training space non-cooperative targets are used as input, and the actual values of the 2D coordinates of the corresponding multiple key points are used as output, and the improved YOLOv8 key point detection network is trained to obtain the 2D coordinate prediction model, including: Initialize the improved YOLOv8 key point detection network; Taking the images of the non-cooperative targets in the training space as input and the actual values of the 2D coordinates of the corresponding multiple key points as output, the improved YOLOv8 key point detection network is trained for multiple rounds to obtain the 2D coordinate prediction model; wherein the training process of the improved YOLOv8 key point detection network under any current number of training rounds includes: Inputting the images of each of the training spatial non-cooperative targets into the improved YOLOv8 key point detection network under the current training round number, and obtaining the predicted values of the 2D coordinates of multiple key points under the current training round number; Calculate the A3DKS loss function value for the current training round based on the predicted values and actual values of the 2D coordinates of multiple key points for the current training round; Determine whether a training stop condition is met; the training stop condition is that the A3DKS loss function value under the current number of training rounds is less than a preset value or reaches a preset number of training rounds; If yes, the improved YOLOv8 key point detection network under the current training round number is determined as the 2D coordinate prediction model; If not, the improved YOLOv8 key point detection network under the current training round is updated to the improved YOLOv8 key point detection network under the next training round, and return to "input the image of each of the training space non-cooperative targets into the improved YOLOv8 key point detection network under the current training round, and obtain the predicted values of the 2D coordinates of multiple key points under the current training round" until the training stop condition is reached to obtain the 2D coordinate prediction model.
4. The method for estimating 6D posture of a non-cooperative space target according to claim 1, It is characterized in that The improved YOLOv8 key point detection network includes: a backbone network, a neck network and a decoupling head; The backbone network includes: an overlapping block embedding module, a first EfficientViT module, a first downsampling module, a second EfficientViT module, a second downsampling module, a third EfficientViT module and an SPPF module; the first EfficientViT module includes 1 EfficientViT block, the second EfficientViT module includes 2 EfficientViT blocks, and the third EfficientViT module includes 3 EfficientViT blocks; The neck network includes: a path aggregation network and a TripletAttention network; the TripletAttention network includes: three residual transformation modules.
5. The method for estimating 6D posture of a non-cooperative space target according to claim 4, It is characterized in that The EfficientViT block includes: a first token interaction layer, a first feedforward network, a second token interaction layer, self-attention calculation, a splicing projection layer, a third token interaction layer and a second feedforward network.
6. The method for estimating 6D posture of a non-cooperative space target according to claim 3, It is characterized in that The A3DKS loss function value for the current training round is calculated based on the predicted values and actual values of the 2D coordinates of multiple key points for the current training round, including: The A3DKS loss function is used to calculate the A3DKS loss function value under the current training round according to the predicted values and actual values of the 2D coordinates of multiple key points under the current training round; the A3DKS loss function is: Among them, L A3DKS is the A3DKS loss function value; N kpts is the total number of key points; d n is the Euclidean distance between the predicted value and the actual value of the 2D coordinate of the nth key point; Z n is the depth of the nth key point; f is the focal length of the camera; s h is the harmonic mean of the scale areas of the non-cooperative targets in each training space in the training set; δ is the visibility of the key point.
7. A computer device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for estimating a 6D posture of a spatial non-cooperative target as described in any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for estimating 6D posture of a spatial non-cooperative target described in any one of claims 1 to 6 is implemented.
9. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method for estimating 6D posture of a spatial non-cooperative target described in any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Method, device and equipment for estimating relative pose of non-cooperative spacecraft
CN119810202A
A method, device and equipment for estimating relative posture of non-cooperative spacecraft
CN119810202B
Human pelvis pose recognition method and device, electronic equipment and medium
CN120355761A
