A lightweight 6D pose estimation method fusing feature enhancement
By introducing learnable grouped convolution and self-supervised predictive convolutional attention modules, the problems of large parameter quantity and insufficient robustness of 6D pose estimation in complex scenarios are solved, achieving model lightweighting and feature reconstruction, and improving the accuracy and robustness of estimation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIANGTAN UNIV
- Filing Date
- 2026-04-16
- Publication Date
- 2026-05-29
AI Technical Summary
In complex scenes with weak textures, varying brightness, and target occlusion, existing 6D pose estimation methods suffer from large parameter counts, high computational costs, and insufficient robustness, making it difficult to find a balance between efficiency and robustness.
A learnable grouped convolution mechanism is introduced to lighten the model, and a self-supervised predictive convolutional attention module and a parameterless attention mechanism are designed. By dynamically pruning redundant connections and reconstructing occluded region features, a collaboratively enhanced closed-loop feature extraction network is formed.
Without compromising accuracy, it significantly reduces the number of parameters and computational overhead, improving the accuracy and robustness of 6D pose estimation. It is particularly suitable for occluded environments, exhibiting higher real-time performance and stability.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and more particularly to a target 6D pose estimation method based on RGB-D images and deep learning methods. Background Technology
[0002] 6D pose estimation utilizes computer vision methods to determine the 3D translation vector and 3D rotation matrix of a target object in the camera coordinate system. It is a crucial technological link in transforming 2D visual perception into 3D spatial understanding. With the continuous advancement of artificial intelligence technology and the widespread application of deep learning methods, 6D pose estimation has gradually become a research focus in the field of computer vision, demonstrating increasingly significant advantages in many cutting-edge application areas such as industrial automation, aerospace, autonomous driving, and augmented reality.
[0003] Pose estimation in complex scenes with weak textures, varying brightness, and target occlusion is often hampered by insufficient extraction of target feature information. Intending to improve accuracy by using more complex networks would greatly increase the number of parameters, which inevitably leads to problems in the overall efficiency and robustness of the network. Summary of the Invention
[0004] The main objective of this invention is to further optimize the image feature extraction network in terms of both efficiency and robustness, addressing the challenges of extreme occlusion environments. First, a learnable grouped convolution mechanism is introduced into the network to achieve model lightweighting through dynamic pruning of redundant connections, significantly reducing the number of parameters and computational overhead without compromising accuracy. Second, to address the issue of information contamination in occluded regions during point cloud feature processing, a self-supervised predictive convolutional attention module is designed, enabling the network to reconstruct features of occluded regions using contextual information. Simultaneously, a parameter-free attention mechanism is introduced to automatically focus on the most discriminative geometric cues, forming a collaborative enhancement loop of "reconstruction-focusing."
[0005] The technical solution of this invention to solve the above problems is as follows: a lightweight pose estimation algorithm with fused feature enhancement, comprising the following steps:
[0006] S1. Obtain the RGB-D image of the target and perform semantic segmentation to obtain the target mask.
[0007] S2, the RGB image block corresponding to the target mask is input into the lightweight image feature extraction network to extract color features; wherein, the lightweight image feature extraction network adopts a learnable grouped convolution mechanism, and forms structured sparse groups by dynamically pruning redundant convolutional connections during the training phase.
[0008] S3, the depth map corresponding to the target mask is converted into a point cloud and input into the point cloud feature extraction network to extract geometric features; wherein, the point cloud feature extraction network includes a series of self-supervised predictive convolutional attention blocks and a parameterless attention module. The self-supervised predictive convolutional attention block reconstructs the features of the occluded region through a mask convolutional kernel with a hole receptive field. The parameterless attention module calculates the linear separability of each neuron in the feature map based on an energy function and generates attention weights.
[0009] S4, the color features and the geometric features are fused to obtain multimodal fusion features.
[0010] S5. Based on the multimodal fusion feature regression of the target object's rotation and translation parameters, the 6D pose estimation result is obtained and iterative optimization is performed to obtain the final 6D pose.
[0011] The beneficial effects of this invention are:
[0012] This invention addresses the two core requirements of real-time performance and occlusion robustness. Starting from two dimensions—efficient feature extraction and feature enhancement—it introduces learnable grouped convolutions into the image feature extraction network and achieves model lightweighting through dynamic pruning, effectively reducing the number of parameters and improving inference speed. In point cloud feature processing, it constructs a collaborative "reconstruction-focusing" module of SSPCAB and SimAM, effectively utilizing contextual information to compensate for feature loss in occluded regions and automatically focusing on discriminative geometric cues. Compared with current mainstream algorithms, this improved algorithm demonstrates superior accuracy and robustness on multiple datasets, providing a new approach for 6D pose estimation technology. Attached Figure Description
[0013] Figure 1 This is a block diagram of the algorithm of the present invention;
[0014] Figure 2 This is a schematic diagram of the model compression method;
[0015] Figure 3 This is a diagram illustrating the pruning process;
[0016] Figure 4 This is a schematic diagram of a self-supervised predictive convolutional attention block.
[0017] Figure 5 This is a schematic diagram of the SimAM network structure;
[0018] Figure 6 The visualization results of the algorithm on the LineMod dataset;
[0019] Figure 7 The visualization results of the algorithm on the Occluded-LINEMOD dataset;
[0020] Figure 8 This is a comparison chart of ablation experiments. Detailed Implementation
[0021] The present invention will be further illustrated below with specific examples. These examples are only used to explain the present invention and are not intended to limit the scope of the present invention.
[0022] like Figure 1 The lightweight pose estimation algorithm with fusion feature enhancement, as shown, includes the following steps:
[0023] S1. Obtain the RGB-D image of the target and perform semantic segmentation to obtain the target mask.
[0024] S2, the RGB image block corresponding to the target mask is input into the lightweight image feature extraction network to extract color features; wherein, the lightweight image feature extraction network adopts a learnable grouped convolution mechanism, and forms structured sparse groups by dynamically pruning redundant convolutional connections during the training phase.
[0025] S3, the depth map corresponding to the target mask is converted into a point cloud and input into the point cloud feature extraction network to extract geometric features; wherein, the point cloud feature extraction network includes a series of self-supervised predictive convolutional attention blocks and a parameterless attention module. The self-supervised predictive convolutional attention block reconstructs the features of the occluded region through a mask convolutional kernel with a hole receptive field. The parameterless attention module calculates the linear separability of each neuron in the feature map based on an energy function and generates attention weights.
[0026] S4, the color features and the geometric features are fused to obtain multimodal fusion features.
[0027] S5. Based on the multimodal fusion feature regression of the target object's rotation and translation parameters, the 6D pose estimation result is obtained and iterative optimization is performed to obtain the final 6D pose.
[0028] Furthermore, the S1 semantic segmentation framework employs the PoseCNN segmentation framework for semantic segmentation of RGB images. This framework is an encoder-decoder architecture that takes the image as input and generates an N+1-channel semantic segmentation map. High-precision pixel-level classification is achieved through multi-scale feature fusion and deconvolution recovery. The framework uses a VGG16 backbone network as its core, removing the last three fully connected layers and retaining the remaining 13 convolutional layers, which are then combined with four spatial downsampling units to form the encoder. The encoder generates feature maps at two scales, 1 / 8 and 1 / 16 of the original image size, used to focus on object edges and improve classification robustness, respectively. The decoder then upsamples the 1 / 16-sized low-resolution feature map to 1 / 8 size through deconvolutional layers, achieving cross-resolution alignment with shallow features. This is then weighted and fused with the shallow feature map channel-wise, integrating local edge details and contextual semantic information. This design is similar to the feature pyramid network, enhancing the complementarity of features at different levels through cross-layer connections. Finally, the fused feature map is subjected to two deconvolution operations to gradually restore the original image resolution, and the classification probability is output through a Softmax layer to achieve pixel-level semantic segmentation. After the image is cropped by the semantic segmentation network, the depth information located by the segmentation mask is converted into a geometric point cloud using a segmentation mask.
[0029] Furthermore, the learnable grouped convolution mechanism described in S2 employs an efficient model compression method. This method uses learnable dense connections, which can change the connection paths between layers to make the network structure more flexible. The core principle is to use the learnable grouped convolution mechanism added during training to build a classifier model. First, the warm-up image is trained using standard convolution to obtain the weight of each connection point (using the L1 norm). Each convolution kernel in each convolutional layer corresponds to a weight, which is calculated according to formula (1):
[0030] (1)
[0031] in For output / input channel indexes, The height and width of the convolution kernel. These are the weights of the convolution kernel; the larger the absolute value of the weight, the greater its contribution to the final output. Irrelevant and important connections are gradually pruned during the intermediate stages, such as... Figure 3 The output channel is connected to only about 1 / G (usually G=4) of the input channels, forming a structured sparse connection.
[0032] If There are groups, and each group has 10 output channels. The number of input channels is So for the first Group output channel set The set of input channels it retains satisfies equation (2):
[0033] (2)
[0034] Finally, after training, an indexing layer transforms the learned grouping structure into a standard, efficient grouped convolution. This design achieves three model compressions: first, it reduces the number of parameters to 1 / G of the original; second, it eliminates feature redundancy using densely connected feature reuse; and third, it ensures that no additional computational overhead is incurred during inference through the transformation of the indexing layer. Ultimately, this allows the feature extraction network to maintain or improve accuracy while significantly reducing the number of parameters and computational cost, making it particularly suitable for real-time 6D pose estimation tasks. For occluded regions, multi-scale feature reuse can also compensate for information loss.
[0035] Furthermore, the self-supervised predictive convolutional attention block described in S3 converts the depth map corresponding to the target mask into a point cloud, which is then input into the "reconstruction-focusing" point cloud feature extraction network to extract geometric features. The network reconstructs the occluded region features through the self-supervised predictive convolutional attention module and enhances discriminative features through the parameterless attention module SimAM. The introduced self-supervised predictive convolutional attention block (SSPCAB) contains three key parts that work together to achieve powerful feature reconstruction capabilities. First is the mask convolution, whose kernel has a unique hollow receptive field. The core region is completely masked and does not participate in the calculation, while the four corners are the parameters to be learned. Following the mask convolution is a channel attention module that calibrates the prediction results of different channels. Different weights are assigned based on the quality of the channel prediction results, enhancing the model's ability to express important features. Finally, a self-supervised reconstruction loss is constructed, as shown in equation (3):
[0036] (3)
[0037] Output via the calculation module and original input The mean squared error between these errors causes the network to strive to reconstruct the occluded central region at each location, thereby learning the contextual structure and inherent patterns of the data. The improved point cloud feature extraction network embeds the SSPCAB module within the Set Abstraction (SA) layer of PointNet++, making the final center point features more dependent on reliable context rather than potentially contaminated local information, thus enhancing robustness to occlusion. Simultaneously, the module provides different responses to regions with consistent and inconsistent contexts. This difference is further amplified by the channel attention mechanism within the module, resulting in stronger features for normal regions and relatively weaker features for occluded or anomalous regions.
[0038] Furthermore, for each neuron in the feature map, the SimAM (A Simple, Parameter-Free Attention Module) attention mechanism defines an energy function to measure its linear separability with other neurons in the same channel, as shown in Equation (4). The greater the difference between a neuron and its surrounding neurons (i.e., the lower the energy), the more attention it deserves.
[0039] (4)
[0040] The value of the target neuron. The values of other neurons in the same channel. The total number of neurons. This is the mean value of all neurons in this channel. For variance, This is the regularization coefficient. The importance of each neuron is the reciprocal of its energy value. The lower the energy, the higher the importance. Finally, these importance scores are normalized using the Sigmoid function and then multiplied element-wise with the original feature map, thereby enhancing important features and suppressing minor features. The biggest advantage of this mechanism is that it is "parameterless," meaning it does not introduce any additional learnable weights, but only calculates based on the statistical information of the features themselves, resulting in extremely low computational cost. Because neurons at the boundaries are more linearly separable from their surroundings (lower energy), SimAM spontaneously enhances these boundary features, helping the network to more accurately distinguish between objects and occlusions.
[0041] SSPCAB is responsible for supplementing missing features caused by occlusion using contextual information, while SimAM is responsible for precisely "focusing" on the most reliable and discriminative geometric cues from these supplemented features; the two complement each other. Features reconstructed by SSPCAB from context inevitably contain some ambiguity or uncertainty. SimAM amplifies high-confidence features through attention weights while suppressing ambiguous responses, thus making the reconstructed features clearer and sharper. Simultaneously, the SE module in SSPCAB first selects the feature channels most important to the task, and SimAM then further locates the most critical geometric structures (such as edges and corners) within these channels, achieving ultimate feature refinement. From a practical application perspective, after adding only a small amount of computational overhead from SSPCAB, SimAM can significantly improve model performance with almost no additional burden.
[0042] Furthermore, the multimodal feature fusion described in S4 employs a deformable cross-attention mechanism for pixel-level multimodal fusion. Deformable convolution dynamically learns the kernel offset, enabling the kernel to adaptively adjust its position based on the spatial distribution of the input features. In point cloud processing, the fixed receptive field of traditional convolution is ill-suited to irregular geometric structures, while deformable convolution generates position adjustment parameters through an offset network, allowing the kernel to focus on key local regions of the object. For example, when processing occlusion, the offset can guide the kernel to avoid occluded areas, extracting effective local features.
[0043] Furthermore, the iterative optimization described in S5 specifically includes: the input contains only global rotations. Peaceful relocation Without confidence weighting, it directly calculates the average Euclidean distance between the predicted point cloud and the target point cloud, focusing on geometric alignment without confidence weighting. This is possible with an initial pose estimate. In this case, the fused features are pooled through an iterative network to obtain global features, which are then input into a multilayer perceptron to predict the pose residual. After multiple iterations, the rotation and translation corrections from each iteration are multiplied together to obtain the final global attitude matrix, as shown in equation (6):
[0044] (6)
[0045] Example 1:
[0046] The experiment was conducted on a PC running Ubuntu 22.04. The hardware consisted of an AMD Ryzen 97945HX CPU, an NVIDIA GeForce RTX 4060 Laptop GPU, and conda 24.5, python 3.6, cuda 11.1, and pytorch 1.8. For asymmetric objects, the Average Distance of Model Points (ADD) was used for evaluation. This was achieved by calculating the average Euclidean distance between all sampled points on the surface of the 3D model of the object in both the real and predicted poses. Strict spatial alignment of each point was required, as shown in Equation (7).
[0047] (7)
[0048] in, For the first three-dimensional model of the object One sampling point, , and , These are the rotation matrix and translation vector for the true pose and the predicted pose, respectively. This represents the total number of sampling points.
[0049] For objects with symmetrical structures, the Average Closest Point Distances (ADD-S) index is used. By minimizing the average distance between each sample point in the predicted pose and its nearest neighbor in the actual pose, the evaluation ambiguity caused by symmetry is eliminated, as shown in Equation (8):
[0050] (8)
[0051] A correct prediction is defined as one where the ADD / ADD-S ratio is less than 10% of the object's diameter. The accuracy is calculated by summing the number of correct predictions. Experiments were conducted on the LineMOD and Occluded-LINEMOD datasets. The pose estimation network was first trained, and then iteratively trained after reaching a set threshold. The threshold `refine_margin` was set to 0.02, the decay threshold `decay_margin` to 0.023, the batch size to 8, the learning rate to 0.0001, the learning rate decay rate to 0.3, the loss function hyperparameter `w` to 0.01, the decay rate of `w` to 0.3, and the maximum number of training epochs to 500. Specific results are shown in Tables 1 and 2. Figure 6 , Figure 7 As shown:
[0052] Table 1. Pose estimation results on the LineMod dataset.
[0053] DenseFusion NF6D Uni6D Ours ape 92.3 93.5 93.7 96.7 ben. 93.2 95.2 99.8 99.6 cam. 94.4 93.9 95.9 99.5 can 93.1 95.5 99.0 99.1 cat 96.5 98.6 98.1 99.4 drill. 87.0 94.8 99.1 99.2 duck 92.3 96.0 89.9 96.9 egg. 99.8 100 100 100 glue 100 99.7 99.2 100 hole. 92.1 94.5 90.2 98.3 iron 97.0 97.5 99.4 99 lamp 95.3 98.9 99.4 99.7 phone 92.8 95.1 97.4 99.2 MEAN 94.3 96.4 97.0 98.9
[0054] Table 2. Pose estimation results on the Occluded-LINEMOD dataset.
[0055] DenseFusion NF6D Ours ape 73.2 68.4 74.6 can 88.6 92.7 91.7 cat 72.2 78.0 75.2 driller 92.5 95.1 95.3 duck 59.7 62.1 62.5 eggbox 94.2 96.1 95.6 glue 92.6 93.5 94.1 holepuncher 78.8 83.6 85.2 MEAN 81.5 83.7 84.3
[0056] As shown in Example 1, the average accuracy of this algorithm reaches 98.9%, and its overall performance is improved by 4.6%, 2.5%, and 1.9% respectively compared to the other three algorithms, achieving the best overall performance. In terms of stability, this algorithm also achieves the best results. Overall, this algorithm performs best on 11 out of 13 objects.
[0057] Example 2:
[0058] Keeping the overall experimental environment unchanged, Example 2 conducted an ablation experiment to verify the impact of replacing CNN with a learnable grouped convolutional network for image feature extraction and adding a local point cloud processing network on accuracy. Specific results are shown in Table 3 and... Figure 8 As shown.
[0059] Table 3 Evaluation of the Fusion Feature Enhancement Network
[0060] DenseFusion Learnable grouped convolution Point cloud enhancement Both ape 79.5 90.9 92.2 92.5 benchvise 84.2 96.7 97.4 97.6 camera 76.5 97.3 97.8 97.9 can 86.6 96.8 98.6 98.9 cat 88.8 96.2 97.5 97.8 driller 77.7 96.7 97.9 98.3 duck 76.3 91.6 93.3 93.6 eggbox 99.9 99.9 99.9 99.9 glue 99.4 99.9 99.9 99.9 holepuncher 79.0 92.7 94.2 94.4 iron 92.1 97.6 98.4 98.6 lamp 92.3 98.3 98.6 98.8 phone 88.0 97.2 97.8 98.0 MEAN 96.3 97.2 97.4
[0061] As can be seen from Example 2, the improved image feature extraction network and the newly added local point cloud processing network in this algorithm can improve the accuracy of 6D pose estimation. Specifically, replacing the CNN with a learnable grouped convolutional network to extract image features only improves the average accuracy by 0.1%, but makes the overall network more lightweight and more real-time. The newly added local point cloud processing network improves the average accuracy by 1.0%, and when both are combined, the average estimation accuracy improves by 1.2%. Not only do the improvements in each part have positive benefits, but the improved networks also have synergistic gain effects with each other.
Claims
1. A lightweight 6D pose estimation method incorporating feature enhancement, characterized in that, Includes the following steps: S1. Obtain the RGB-D image of the target and perform semantic segmentation to obtain the target mask. 2.S2, Input the RGB image block corresponding to the target mask into the lightweight image feature extraction network to extract color features; wherein, The lightweight image feature extraction network adopts a learnable grouped convolution mechanism, which dynamically prunes redundant convolutional connections during the training phase to form structured sparse groups. 3.S3, the depth map corresponding to the target mask is converted into a point cloud and input into the point cloud feature extraction network to extract geometric features; wherein, the point cloud feature extraction network includes a series of self-supervised predictive convolutional attention blocks and a parameterless attention module. The self-supervised predictive convolutional attention block reconstructs the features of the occluded region through a mask convolutional kernel with a hole receptive field. The parameterless attention module calculates the linear separability of each neuron in the feature map based on an energy function and generates attention weights. 4.S4, The color features and the geometric features are fused to obtain multimodal fusion features. 5.S5, based on the multimodal fusion feature regression of the target object's rotation and translation parameters, obtain the 6D pose estimation result and perform iterative optimization to obtain the final 6D pose.
6. The method according to claim 1, characterized in that, The learnable grouped convolution mechanism described in S2 specifically includes: employing an efficient model compression method, which uses a learnable dense connection method and can change the connection paths between layers to make the network structure more flexible. The core principle is to construct a classifier model by incorporating a learnable grouped convolution mechanism during training. First, standard convolution is used to train the warm-up image to obtain the weights of each connection point (using the L1 norm). Each convolutional kernel in each convolutional layer corresponds to a weight, calculated according to formula (1): （1） in For output / input channel indexes, The height and width of the convolution kernel. The weights are the convolution kernel weights; the larger the absolute value of the weight, the greater its contribution to the final output. Irrelevant and important connections are gradually pruned in the intermediate stages. For example, in Figure 3, the output channel is only connected to about 1 / G (generally G=4) of the input channels, forming a structured sparse connection.
7. If there is There are groups, and each group has 10 output channels. The number of input channels is So for the first Group output channel set The set of input channels it retains satisfies equation (2): （2） Finally, after training, an indexing layer transforms the learned grouping structure into a standard, efficient grouped convolution. This design achieves three model compressions: first, it reduces the number of parameters to 1 / G of the original; second, it eliminates feature redundancy using densely connected feature reuse; and third, it ensures that no additional computational overhead is incurred during inference through the transformation of the indexing layer. Ultimately, this allows the feature extraction network to maintain or improve accuracy while significantly reducing the number of parameters and computational cost, making it particularly suitable for real-time 6D pose estimation tasks. For occluded regions, multi-scale feature reuse can also compensate for information loss.
8. The method according to claim 1, characterized in that, The self-supervised predictive convolutional attention block S3 specifically includes: converting the depth map corresponding to the target mask into a point cloud, inputting it into the "reconstruction-focusing" point cloud feature extraction network, and extracting geometric features; wherein, the network reconstructs the occluded region features through the self-supervised predictive convolutional attention module, and enhances the discriminative features through the parameterless attention module SimAM. The introduced self-supervised predictive convolutional attention block (SSPCAB) contains three key parts, which work together to achieve powerful feature reconstruction capabilities. The first is the mask convolution, whose kernel has a unique hole receptive field. The core region is completely masked and does not participate in the calculation, while the four corners are the parameters to be learned. Immediately following the mask convolution is a channel attention module, which calibrates the prediction results of different channels. Different weights are given according to the quality of the channel prediction results, which enhances the model's ability to express important features. Finally, a self-supervised reconstruction loss is constructed, as shown in Equation (3): （3） This loss function is used as an important criterion for judgment.
9. The method according to claim 1, characterized in that, The parameter-free attention module described in S3 specifically includes: For each neuron in the feature map, the SimAM (A Simple, Parameter-Free Attention Module) attention mechanism defines an energy function to measure its linear separability with other neurons in the same channel, as shown in Equation (4). The greater the difference between a neuron and its surrounding neurons (i.e., the lower the energy), the more attention it deserves.
10. （4） The value of the target neuron. The values of other neurons in the same channel. The total number of neurons. This is the mean value of all neurons in this channel. For variance, This is the regularization coefficient. The importance of each neuron is the reciprocal of its energy value. The lower the energy level, the higher the importance of the decision.
11. The method according to claim 1, characterized in that, The multimodal feature fusion described in S4 specifically includes: the method according to claim 1, wherein the step of fusing color features and geometric features includes: performing pixel-level multimodal fusion using a deformable cross-attention mechanism, wherein the deformable cross-attention adaptively adjusts the receptive field by dynamically learning the convolution kernel offset.
12. The method according to claim 1, characterized in that, The iterative optimization described in S5 specifically includes: the input contains only global rotations. Peaceful relocation Without confidence weighting, it directly calculates the average Euclidean distance between the predicted point cloud and the target point cloud, focusing on geometric alignment without confidence weighting. This is possible with an initial pose estimate. In this case, the fused features are pooled through an iterative network to obtain global features, which are then input into a multilayer perceptron to predict the pose residual. After multiple iterations, the rotation and translation corrections from each iteration are multiplied together to obtain the final global attitude matrix, as shown in equation (5): （5）。