End-to-end lightweight depth fusion 6D pose estimation method

Abstract

The invention provides an end-to-end lightweight deep fusion 6D pose estimation method, and relates to the technical field of computer vision. The method comprises the following steps: step 1, acquiring image data information of a pose estimation object by using an RGB-D sensor, and training the image data information; the image data information comprises an RGB image and a depth image; step 2, constructing an end-to-end 6D pose estimation network (E2pose), and training the E2pose network by using the image data information; the E2pose network comprises a feature extraction layer, a feature fusion layer and a pose estimation layer, and the feature extraction layer comprises a color feature extraction layer and a geometric feature extraction layer; and step 3, inputting to-be-estimated image data information into the trained E2pose network, and performing direct regression to obtain the 6D pose of the object. According to the method, the technical problems of high calculation cost and insufficient robustness in a shielding scene caused by single data mode, feature extraction limitation and estimation process redundancy in the existing 6D pose estimation are solved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to an end-to-end lightweight deep fusion 6D pose estimation method. Background Technology

[0002] 6D pose refers to the position and orientation of an object in three-dimensional space, described by three translational degrees of freedom and three rotational degrees of freedom. Vision-based 6D pose estimation aims to obtain the spatial position information of an object relative to the camera coordinate system through visual features, and has significant application value in the field of artificial intelligence. For example, in autonomous driving, accurate pose estimation can help vehicles grasp the position and orientation of themselves and surrounding objects in real time, enabling effective obstacle avoidance; in robotic arm grasping tasks in complex environments, it can help accurately identify and grasp objects, improving production efficiency; in the field of augmented reality (AR), it can provide a more realistic and natural interactive experience, enhancing the interaction between AR environments and humans.

[0003] However, existing 6D pose estimation methods still suffer from several problems: First, the data modality is limited, relying solely on RGB images makes it difficult to solve the inverse problem of inferring 3D structures from 2D projections. Since many different 3D structures may correspond to the same 2D projection, it becomes difficult to recover 3D information from images. Second, feature extraction is limited. CNN-based feature extraction methods are constrained by local convolutional kernels, focusing more on local information during processing and easily ignoring global relationships. They also struggle to capture global dependencies and are prone to failure in complex occluded scenes. Third, the estimation process is redundant. Traditional 2D-3D keypoint matching methods require complex correspondence and post-processing, resulting in high computational costs. Furthermore, keypoints are susceptible to occlusion interference, leading to poor matching reliability and insufficient estimation accuracy.

[0004] Therefore, there is an urgent need to propose a lightweight end-to-end 6D pose estimation method that can deeply integrate color and geometric multimodal features, effectively capture the correlation between global and local features, simplify the estimation process to reduce computational costs, and maintain high robustness in complex occlusion scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide an end-to-end lightweight deep fusion 6D pose estimation method, which solves the technical problems of existing 6D pose estimation methods, such as single data modality, limited feature extraction, and redundant estimation process leading to high computational cost and insufficient robustness in occluded scenarios.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: This invention provides an end-to-end lightweight deep fusion 6D pose estimation method, characterized by comprising the following steps: Step 1: Use an RGB-D sensor to acquire image data of the pose estimation object and train the image data; the image data includes RGB images and depth images; Step 2: Construct an end-to-end 6D pose estimation network (E2pose) and train the E2pose network using the image data information; the E2pose network includes a feature extraction layer, a feature fusion layer and a pose estimation layer, and the feature extraction layer includes a color feature extraction layer and a geometric feature extraction layer; Step 3: Input the image data information to be estimated into the trained E2pose network to directly regress and obtain the 6D pose of the object.

[0007] Furthermore, during the training of the E2pose network, a loss function is used to calculate the pose estimation loss. The parameters of the 6D target pose estimation network are optimized and adjusted through backpropagation. The training process is repeated until the loss value converges, and finally the trained E2pose pose estimation network is obtained.

[0008] Furthermore, step 3 specifically includes: Step 3.1: Input the RGB image into the color feature extraction layer, and extract color features through encoding and decoding processes; Step 3.2: Convert the depth image into point cloud information and input it into the geometric feature extraction layer to extract geometric features; Step 3.3: Input the color features and geometric features into the feature fusion layer, fuse them through a cross self-attention mechanism, and input the fused features into the pose estimation layer to regress and obtain the 6D pose.

[0009] Furthermore, step 3.1 specifically includes: Step 3.1.1: Pre-segment the input RGB image to obtain a precise mask for the target region; use the mask as input to enter the encoding stage for processing, and output a multi-scale feature map; Step 3.1.2: Input the multi-scale feature map output from the encoding stage into the decoding stage for processing; the decoding stage processes the feature map sequentially through a multilayer perceptron, an upsampling layer, and a convolutional layer; the multilayer perceptron is used to perform nonlinear transformation on the input feature map to enhance its feature representation capability; the upsampling layer is used to restore the feature map size to a suitable size to fuse information from different stages; the convolutional layer is used to further extract and output the final color features.

[0010] Furthermore, the encoding stage includes multiple cascaded processing stages; Each processing stage includes an image patch embedding layer and an EHT module. The EHT module includes multiple Transformer blocks for transforming the input features and outputting feature maps with progressively smaller dimensions.

[0011] Furthermore, in each stage, the non-coding hierarchical Transformer computation layer in the EHT module standardizes the input features in the feature dimension through the layer normalization module. After the layer normalization process, the features enter the EHA module in the non-coding hierarchical Transformer computation layer to perform local and global feature association calculations and generate feature representations containing rich semantic information. The output of the EHA module is added element-wise to the original input, and feature information from different stages is incorporated through residual connections, effectively avoiding the gradient vanishing problem. After another layer normalization operation, it enters the feedforward neural network (FFN) module in the non-coding layered Transformer computation layer. The FFN consists of multiple fully connected layers. The features output by the FFN are added element-wise with the features of the first residual connection to generate the final features and output them.

[0012] Furthermore, the implementation process of the EHA module specifically includes: After layer normalization, the features are first downsampled through a max pooling layer to reduce data dimensionality; this module employs a hierarchical structure of local and global attention, directly inputting local features. and perform space reduction transformation ( After) Remove the embedding of queries, keys, and values; In local attention computation, local features are used as the basis for calculation. As a local feature query, based on the SR transformation As local features, the keys and values ​​are used to calculate local attention weights through the softmax activation function, and then the local attention output is calculated. In global attention computation, the input feature map is used as the basis for the calculation. Queries and global features as global features global features after SR transformation Similarly, global attention weights and outputs are calculated using the keys and values ​​that serve as global features. By weighted fusion of local and global attention outputs, the final hierarchical self-attention output is obtained.

[0013] Furthermore, step 3.2 specifically includes: Step 3.2.1: Divide the 3D point cloud space into portions of size [size missing]. A non-overlapping window is used to divide the point cloud space. For a given specific point cloud data, the geometric features of the point cloud are calculated within the divided window space. ; Step 3.2.2, Calculate geometric features The complexity, that is The degree of change in the geometric properties within the domain is assessed by adaptively adjusting the density of sampling points, increasing the number of sampling points in complex regions at long distances; Step 3.2.3: Perform hierarchical key point sampling for each query point. In window size Perform dense sampling within the core to obtain dense keypoints. : Step 3.2.4, proportionally using farthest point sampling (FPS). Downsample the input point cloud and apply it to a large window. Find sparse key points Duplicate points are counted only once, and dense keypoint sets are created. and sparse key set Find the union of the key points, which will form the final set of key points. .

[0014] Furthermore, step 3.3 specifically includes: Step 3.3.1, define pose estimation loss That is, the distance between the sampling points of the real 3D model of the object in its true pose and the corresponding points after the pose transformation is predicted and estimated; Step 3.3.2: For symmetrical objects, adjust the loss to minimize the minimum distance between corresponding points, that is, calculate the distance between each sampling point in the direction of the object model and the closest point on the real model; Step 3.3.3, based on confidence level With pose estimation loss Calculate the total loss function .

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) Improve the accuracy of pose estimation: This application uses an RGB-D sensor to simultaneously acquire RGB images and depth images, which effectively solves the problem of insufficient 3D information recovery accuracy caused by single-modal feature dependence. Through independent feature extraction of RGB images and depth images and deep fusion of cross self-attention mechanism, the complementary value of color semantic features and geometric space features is fully explored, providing more comprehensive feature support for 6D pose estimation and significantly improving the estimation accuracy.

[0016] (2) Enhanced robustness in complex scenes: This application designs a local and global attention hierarchical structure for the EHA module. Local attention focuses on the detailed features of the occluded area, global attention establishes long-distance spatial relationships, and residual connections ensure the integrity of feature transmission. This can accurately capture local occlusion details and grasp the spatial relationships of the scene globally, greatly enhancing the robustness of pose estimation in complex occlusion and cluttered environments.

[0017] (3) Achieve lightweight and efficient deployment: This invention simplifies the redundant process of traditional 2D-3D key point matching, directly regresses pose results through end-to-end network, and combines the adaptive hierarchical key point sampling strategy with the lightweight design of the EHA module (removing the embedding projection of queries, keys and values). While ensuring performance, it reduces computing costs and memory usage, meets the lightweight deployment requirements of edge devices, and enhances the practical value of the technology and the coverage of application scenarios. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the end-to-end lightweight deep fusion 6D pose estimation network structure of the present invention. Figure 2 Flowchart for color feature extraction from RGB images in this invention; Figure 3 This is a diagram of the geometric feature adaptive hierarchical key point sampling strategy of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] I. Detailed Implementation Instructions This embodiment provides an end-to-end lightweight deep fusion 6D pose estimation method. Please refer to... Figure 1 As shown, it includes the following steps: Step 1: Use an RGB-D sensor to acquire image data of the pose estimation object and train the image data; the image data includes RGB images and depth images.

[0022] Step 1 specifically includes the following: The training image data was obtained using the Linemod dataset and the YCB-Video public dataset. The Linemod dataset contains more than 18,000 real images, providing RGB-D videos and 3D models of 13 objects. 15% of the images were used for training and 85% were used for testing.

[0023] The YCB-Video dataset includes 21 objects and over 130,000 frames of real-world video sequences. Each frame provides RGB-D data and annotations for the objects in the scene. In addition to real-world data, the YCB-Video dataset also includes synthetic data for training. The YCB-Video dataset demonstrates the challenges of pose estimation in cluttered scenes, textureless objects, and varying lighting conditions.

[0024] Step 2, as follows Figure 1 As shown, an end-to-end 6D pose estimation network is constructed. The end-to-end 6D pose estimation network includes a feature extraction layer, a feature fusion layer, and a pose estimation layer. The feature extraction layer includes a color feature extraction layer and a geometric feature extraction layer.

[0025] The end-to-end 6D pose estimation network is trained using the image data information from step 1. During the training process, the pose estimation loss is calculated using a loss function, which is the loss between the predicted result and the true value. The parameters of the 6D pose estimation network are optimized and adjusted through backpropagation. The training process is repeated until the loss value converges, and finally the trained E2pose pose estimation network is obtained.

[0026] Step 3: Input the image data information from Step 1 into the end-to-end 6D pose estimation network in Step 2 to obtain the corresponding pose parameters, and then obtain the 6D pose estimation result.

[0027] Step 3 specifically includes the following steps: Step 3.1: Input the RGB image into the color feature extraction layer, and extract color features through encoding and decoding; the extraction of color features from the RGB image by the color feature extraction layer is divided into an encoding stage and a decoding stage; Step 3.2: Convert the depth image into point cloud information and input it into the geometric feature extraction layer. The point cloud information is then processed by the geometric feature extraction layer to extract geometric features. Step 3.3: The color features output from Step 3.1 and the geometric features output from Step 3.2 are used as inputs to the feature fusion layer for feature fusion. In the feature fusion stage, multimodal information is fused through a cross self-attention mechanism. The integrated multi-scale pose information is used as the fused features and input to the pose estimation layer for direct regression of the object's 6D pose.

[0028] Step 3.1 specifically includes the following steps: Step 3.1.1: Pre-segment the input RGB image to obtain a precise mask of the target region. Use the mask as input to enter the encoding stage and output the feature map. Step 3.1.2: In the decoding stage, the features are output after passing through a multilayer perceptron, upsampling, and convolutional layers; the multilayer perceptron performs nonlinear transformations on the features to enhance their expressive power; the upsampling operation restores the feature map to a suitable size to fuse information from different stages; finally, color features are further extracted through convolutional layers.

[0029] In step 3.1.1, as follows Figure 2 As shown, the encoding stage is divided into four stages, namely stage 1, stage 2, stage 3 and stage 4. Each stage includes an image patch embedding layer and an EHT module. The EHT module includes multiple Transformer blocks. The ratio of the number of Transformer blocks in the four stages is 2:2:2:1. In the four stages, the EHT module based on Transformer blocks is used to directly perform dynamic weight allocation on the feature map. In stage 1, the mask is converted into a feature sequence through an image patch embedding layer. The feature sequence is then processed by a non-coding hierarchical Transformer layer in the EHT module, generating a dimension of... × The feature map of ×C1; where... H Indicates image height, W C1 represents the width of the image, and C1 represents the number of channels in stage 1. In stage 2, the feature map output from stage 1 is passed through an image patch embedding layer and an EHT module to obtain a dimension of × ×C2 feature map; C2 represents the number of channels in stage 2; In stage 3, the feature map output from stage 2 is then passed through an image patch embedding layer and an EHT module to obtain a dimension of × ×C3 feature map; C3 represents the number of channels in stage 3; In stage 4, the feature map output from stage 3 is passed through the image patch embedding layer and the EHT module to obtain a dimension of × The feature map of ×C4; C4 represents the number of channels in stage 4.

[0030] In each stage, the no-coding hierarchical Transformer computation layer in the EHT module standardizes the input features along the feature dimension through the layer normalization module. After layer normalization, the features enter the EHA module in the no-coding hierarchical Transformer computation layer to perform local and global feature association calculations, generating feature representations containing rich semantic information. The output of the EHA module is added element-wise to the original input, incorporating feature information from different stages through residual connections, effectively avoiding the gradient vanishing problem. After another layer normalization operation, the features enter the feedforward neural network (FFN) module in the no-coding hierarchical Transformer computation layer. The FFN consists of multiple fully connected layers. The features output by the FFN are added element-wise to the features from the first residual connection to generate the final features and output them.

[0031] In the EHA module, the features after layer normalization are first entered into the max pooling layer for downsampling, thereby reducing the data dimensionality. The EHA module includes a hierarchical structure of local and global attention. Unlike traditional methods that rely on linear transformation layers, the EHA module directly inputs the extracted local features. and perform space reduction transformation ( After) By removing / omitting the embedding of queries, keys, and values, a lightweight design is achieved while efficiently extracting global context information. , (1) in, Queries representing local features Keys representing local features Values ​​representing local features, This indicates the space reduction achieved through average pooling. Indicates the local reduction ratio; Then the key of local features and the values ​​of local features Weighted calculation of local attention output: (2) (3) in, It is a local attention weight. softmax ( )express softmax Activation function It is the dimension of the key vector. TTo represent the transpose of a matrix, It is the output of local attention; Calculate global attention to obtain contextual information about the scene: (4) in, Queries representing global features Keys representing global features Represents the value of the global feature. It is the input feature map. Represents global features. Indicates the overall reduction ratio; Calculate the global attention weights and output: (5) (6) in, It is the global attention weight. It is the output of global attention. By weighted fusion of local and global attention outputs, the final hierarchical self-attention output is obtained. : (7) in, It is a weighting coefficient; The overall structure of the EHA module is as follows: (8) (9) in, It is the output after processing by the EHA module. Representation layer normalization, It is the output after processing by the EHT module.

[0032] Step 3.2 specifically includes the following steps: Step 3.2.1: Divide the 3D point cloud space into portions of size [size missing]. A non-overlapping window is used to divide the point cloud space. For a given specific point cloud data, the geometric features of the point cloud are calculated within the divided window space. : (10) Among them, Point In the field, This represents the function for calculating curvature. It is the index of a point in a point cloud, used to identify each point in the point cloud. , It is a point field point The index; Step 3.2.2, Calculate geometric features The complexity, that is The degree of change in the geometric properties within the domain is determined by adaptively adjusting the density of sampling points, i.e., the density of representative points in the point cloud data, and increasing sampling points in complex regions at long distances. Step 3.2.3, perform layered key point sampling, such as... Figure 3 As shown, for each query point In window size Perform dense sampling within the core to obtain dense keypoints. : (11) in, Indicates query point In window size The set of neighboring points within, Point Adaptive sampling density, Based on features Dynamically adjusted sampling density function; Step 3.2.4, as follows Figure 3 As shown, the sampling is performed proportionally using the farthest point sampling (FPS). Downsample the input point cloud and apply it to a large window. Find sparse key points Duplicate points are counted only once, and dense keypoint sets are created. and sparse key set Find the union of the key points, which will form the final set of key points. : (12) (13) in, Indicates query point In window size The set of domain points within.

[0033] By adopting an adaptive hierarchical keypoint sampling strategy, different sampling densities are used for near and far keypoints, reducing the computational load of Transformer blocks while improving the targeting of point cloud interactions.

[0034] Step 3.3 specifically includes, Step 3.3.1, define pose estimation loss as the distance between the sampled points of the real 3D model of the object in its true pose and the corresponding points after the pose transformation is predicted, denoted as . Represented as: (14) in, Represents a true 3D model The set of sampling points in Represents the actual rotation matrix. Indicates the estimation of the rotation matrix; Represents the actual translation vector. This represents the estimated translation vector. It is the true pose. It predicts pose; Step 3.3.2: For symmetrical objects, we choose to minimize the minimum distance between corresponding points to estimate the distance between each point in the object model and the nearest point in the real model. The loss function then becomes: (15) in, It is the kth sampling point on the 3D model. k is an iteration variable, representing that the algorithm will traverse from the first point to the mth point in the predicted model point cloud. m refers to the total number of model sampling points; M means "to calculate the average of all points", and m means "to search from the first to the last point when finding the nearest point". Step 3.3.3, based on confidence level With pose estimation loss Calculate the total loss function for: (16) in, It is the number of dense features sampled randomly. It is a balanced hyperparameter.

[0035] II. Specific Evaluation Tests In this embodiment, the end-to-end 6D pose estimation (E2pose) network is evaluated and tested on the Linemod dataset and the YCB-Video dataset.

[0036] For asymmetric objects, the average Euclidean distance (ADD) between the estimated pose and the true pose is used as the evaluation index; for symmetric objects, the average minimum Euclidean distance (ADD-S) is used as the evaluation index.

[0037] For the Linemod dataset, the threshold for the evaluation metric is set to 10% of the estimated object diameter. Values ​​greater than the threshold indicate pose estimation distortion. For the YCB-Video dataset, the evaluation metric is calculated by plotting an accuracy-threshold curve with varying distance thresholds and then calculating the area under the curve (AUC). A higher AUC value indicates higher pose estimation accuracy. The evaluation results are shown in Tables 1 and 2.

[0038] Table 1 shows the comparative experimental data for the Linemod dataset.

[0039] This invention was compared with PVNet, DPOD, PointFusion, DenseFusion, G2L-Net, PVN3D, and FFB6D networks. The results show that the end-to-end 6D pose estimation (E2pose) network of this invention achieves an ADD-0.1d value of 99.8%, which is superior to other models. Furthermore, it achieves good estimation results for the vast majority of estimated objects. This is attributed to E2pose's effective establishment of long-range dependencies based on Transformer, which fully integrates color and geometric features, thereby improving estimation performance and making 6D pose estimation more robust.

[0040] Table 2 shows the comparative experimental data of the YCB-Video dataset.

[0041] This invention was compared with PoseCNN, PointFusion, DCF, DF, PVN3D, PR-GCN, FFB6D, and DFTr networks. The results show that the end-to-end 6D pose estimation network model (E2pose) of this invention achieves an ADD-S value of 96.7%, comparable to DFTr, but higher than the other models. Furthermore, it significantly outperforms the other models in terms of ADD(S) value. This is because E2pose adaptively adjusts the attention distribution, optimizing feature extraction and effectively improving performance while more efficiently aggregating global semantic information.

[0042] The working principle of the end-to-end lightweight deep fusion 6D pose estimation method of this invention: Based on image data acquired from an RGB-D sensor, an end-to-end neural network was built to determine the transformation matrix of an object relative to the camera coordinate system. First, a heterogeneous network architecture was used to process color and geometric features independently. Then, a cross-modal attention module was used to obtain fused features. Finally, a simple MLP layer was used to predict the object's rotation and translation. In E2pose, a lightweight EHT module was designed to address the detail capture problem in occluded scenes, enabling the model to simultaneously acquire fine-grained features of locally occluded regions and global contextual information. The core of the EHT module is the EHA module, which further optimizes computational efficiency during global feature extraction within the Transformer architecture by omitting query, key-value embedding, and linear projection, thereby reducing module complexity and memory usage. An adaptive hierarchical keypoint sampling strategy was employed, using different sampling densities for near and far keypoints. This reduced the computational load of the Transformer block while improving the specificity of point cloud interactions. Furthermore, the global modeling capability of the self-attention mechanism was combined to effectively expand the receptive field, thereby enhancing the model's spatial understanding in occluded scenes and improving accuracy.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end lightweight deep fusion 6D pose estimation method, characterized in that, Includes the following steps: Step 1: Use an RGB-D sensor to acquire image data of the pose estimation object and train the image data; the image data includes RGB images and depth images; Step 2: Construct an end-to-end 6D pose estimation network (E2pose) and train the E2pose network using the image data information; the E2pose network includes a feature extraction layer, a feature fusion layer and a pose estimation layer, and the feature extraction layer includes a color feature extraction layer and a geometric feature extraction layer; Step 3: Input the image data information to be estimated into the trained E2pose network to directly regress and obtain the 6D pose of the object.

2. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 1, characterized in that, During the training of the E2pose network, a loss function is used to calculate the pose estimation loss. The parameters of the 6D target pose estimation network are optimized and adjusted through backpropagation. The training process is repeated until the loss value converges, and finally the trained E2pose pose estimation network is obtained.

3. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 1, characterized in that, Step 3 specifically includes: Step 3.1: Input the RGB image into the color feature extraction layer, and extract color features through encoding and decoding processes; Step 3.2: Convert the depth image into point cloud information and input it into the geometric feature extraction layer to extract geometric features; Step 3.3: Input the color features and geometric features into the feature fusion layer, fuse them through a cross self-attention mechanism, and input the fused features into the pose estimation layer to regress and obtain the 6D pose.

4. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 3, characterized in that, Step 3.1 specifically includes: Step 3.1.1: Pre-segment the input RGB image to obtain a precise mask for the target region; use the mask as input to enter the encoding stage for processing, and output a multi-scale feature map; Step 3.1.2: Input the multi-scale feature map output from the encoding stage into the decoding stage for processing; the decoding stage processes the feature map sequentially through a multilayer perceptron, an upsampling layer, and a convolutional layer; the multilayer perceptron is used to perform nonlinear transformation on the input feature map to enhance its feature representation capability; the upsampling layer is used to restore the feature map size to a suitable size to fuse information from different stages; the convolutional layer is used to further extract and output the final color features.

5. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 4, characterized in that, The encoding stage includes multiple cascaded processing stages; Each processing stage includes an image patch embedding layer and an EHT module. The EHT module includes multiple Transformer blocks for transforming the input features and outputting feature maps with progressively smaller dimensions.

6. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 5, characterized in that, In each stage, the non-coding hierarchical Transformer computation layer in the EHT module standardizes the input features in terms of feature dimension through the layer normalization module. After the layer normalization process, the features enter the EHA module in the non-coding hierarchical Transformer computation layer to perform local and global feature association calculations and generate feature representations containing rich semantic information. The output of the EHA module is added element-wise to the original input, and feature information from different stages is incorporated through residual connections, effectively avoiding the gradient vanishing problem. After another layer normalization operation, it enters the feedforward neural network (FFN) module in the non-coding layered Transformer computation layer. The FFN consists of multiple fully connected layers. The features output by the FFN are added element-wise with the features of the first residual connection to generate the final features and output them.

7. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 6, characterized in that, The implementation process of the EHA module specifically includes: After layer normalization, the features are first downsampled through a max pooling layer to reduce data dimensionality; this module employs a hierarchical structure of local and global attention, directly inputting local features. and perform space reduction transformation ( After) Remove the embedding of queries, keys, and values; In local attention computation, local features are used. As a local feature query, based on the SR transformation As local features, the keys and values ​​are used to calculate local attention weights through the softmax activation function, and then the local attention output is calculated. In global attention computation, the input feature map is used as the basis for the calculation. Queries and global features as global features global features after SR transformation Similarly, global attention weights and outputs are calculated using the keys and values ​​that serve as global features. By weighted fusion of local and global attention outputs, the final hierarchical self-attention output is obtained.

8. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 3, characterized in that, Step 3.2 specifically includes: Step 3.2.1: Divide the 3D point cloud space into segments of size [size missing]. A non-overlapping window is used to divide the point cloud space. For a given specific point cloud data, the geometric features of the point cloud are calculated within the divided window space. ; Step 3.2.2, Calculate geometric features The complexity, that is The degree of change in the geometric properties within the domain is assessed by adaptively adjusting the density of sampling points, increasing the number of sampling points in complex regions at long distances; Step 3.2.3: Perform hierarchical key point sampling for each query point. In window size Perform dense sampling within the core to obtain dense keypoints. : Step 3.2.4, proportionally using farthest point sampling (FPS). Downsample the input point cloud and apply it to a large window. Find sparse key points Duplicate points are counted only once, and dense keypoint sets are created. and sparse key set Find the union of the key points, which will form the final set of key points. .

9. The end-to-end lightweight deep fusion 6D pose estimation method according to claim 3, characterized in that, Step 3.3 specifically includes: Step 3.3.1, define pose estimation loss That is, the distance between the sampling points of the real 3D model of the object in its true pose and the corresponding points after the pose transformation is predicted and estimated; Step 3.3.2: For symmetrical objects, adjust the loss to minimize the minimum distance between corresponding points, that is, calculate the distance between each sampling point in the direction of the object model and the closest point on the real model; Step 3.3.3, based on confidence level With pose estimation loss Calculate the total loss function .

Images