Multi-channel cross fusion method for multi-modal 6D pose estimation based on MCF-Net

Abstract

The invention provides a multi-channel cross fusion method for multi-modal 6D pose estimation based on MCF-Net, and the method is based on a multi-modal cross fusion strategy network which is cooperatively driven by a multi-channel fusion self-attention module and a cross-modal fusion module. The multi-modal cross fusion strategy network comprises a feature extraction module, a multi-modal cross fusion module and a pose estimation module; for feature extraction of point cloud data and a multi-modal feature fusion strategy, a point feature editor P2Net is designed to extract geometric features, the characterization capability of a network on geometric information is improved, a multi-modal cross fusion strategy is constructed, color features and geometric features are reordered by using dynamic range convolution, and through the synergistic effect of an MFSA module and a CMCF module, a multi-modal cross fusion strategy is established. According to the method, effective fusion of multi-modal features is achieved, the method has high pose estimation precision, obvious technical advantages and better performance, the method has remarkable effectiveness in 6D pose estimation in a complex environment, and powerful support is provided for an efficient estimation technology in practical application.

Description

Technical Field

[0001] This invention relates to the field of lighting equipment technology, and specifically to a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net. Background Technology

[0002] In recent years, with the continuous development of artificial intelligence technology, robots are being applied in fields that can significantly improve people's living standards, such as service robots, collaborative robots, and unmanned warehouses. 6D pose estimation of target objects is a key technology for robots to understand a given working environment. 6D pose estimation refers to obtaining the position of a target object in three-dimensional space and its three-dimensional rotation and translation relative to a camera. It is widely used in fields such as robot grasping and manipulation, autonomous driving, and augmented and virtual reality (AR and VR). In ideal environments, the rich environmental information acquired by existing visual sensors can provide relatively accurate pose estimation. However, for complex environments with large changes in lighting, significant texture differences, and severe occlusion, vision-based pose estimation methods still face problems of weak robustness and low accuracy.

[0003] As researchers continue to explore the field of 6D pose estimation, numerous methods have been proposed. Traditional methods such as SIFT, SURF, and ORB rely on extracting a set of invariant feature points (such as corner points and edge points) from an image and estimating the object pose through matching. Methods such as ICP and EPnP utilize known object models and point cloud data for geometric fitting to obtain the pose. However, these methods all rely on manually designed features to establish the correspondence between the 3D model of the target object and the 2D pixel positions. The quality of feature selection and matching is highly sensitive to the environment, directly affecting the accuracy and robustness of pose estimation. In recent years, the rapid development of deep learning has driven the application of convolutional neural networks (CNNs) on RGB images. Some methods classify or regress RGB image features by training deep models to obtain object poses; others detect 2D keypoints as reference centers for pose estimation and combine them with the PnP algorithm to calculate 6D pose. These methods are mostly based on 2D projection. Although the error in 2D projection is small, it is amplified in 3D space, thus limiting their applicability in complex scenes. The emergence of RGB-D sensors provides additional depth information for RGB images and has led to a new research direction: how to effectively fuse features from the two modalities to achieve higher accuracy 6D pose estimation? Most existing methods employ bimodal fusion strategies at the input, feature extraction, or decision layers. For example, some methods first estimate the initial pose based on RGB images and then further optimize it on point cloud data using the ICP algorithm. However, this optimization process is not only cumbersome and time-consuming but also fails to achieve end-to-end pose optimization. Other methods directly concatenate RGB and depth information and input the concatenated data into the model for pose estimation, but ignore the essential differences between the two data sources, thus limiting the estimation accuracy. DenseFusion proposes a more efficient fusion strategy and, similar to some methods, introduces a point cloud network to extract the geometric features required for bimodal fusion. Specifically, this method generates a point cloud based on the pixel coordinates of the RGB image and its corresponding depth values, then extracts color and geometric features from the RGB image and the point cloud respectively. It uses a dense fusion module to replace the simple feature concatenation operation and introduces environmental information during the fusion process, thereby improving the accuracy of 6D pose estimation. However, this method still faces two major challenges: first, it does not fully utilize the rich potential interactive information between RGB and point cloud features when fusing them; second, the robustness and accuracy of the network are still insufficient in complex environments such as insufficient lighting, significant texture differences, and severe occlusion.

[0004] Therefore, there is an urgent need for a multimodal cross-fusion network MCF-Net, which uses a novel multimodal cross-fusion strategy based on information from both RGB and depth modes to fully utilize the complementary information between the two modes, thereby achieving effective fusion of bimodal information and improving the overall 6D pose estimation performance of the network. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net. For feature extraction and multimodal feature fusion strategies for point cloud data, a point feature editor (P2Net) is designed to extract geometric features, enhancing the network's ability to represent geometric information. A multimodal cross-fusion strategy is also constructed, utilizing dynamic range convolution to reorder color and geometric features, enhancing the adaptability of subsequent convolutions under different input dynamic ranges. This allows for more accurate capture of image details. Furthermore, the MFSA and CMCF modules work together to achieve effective fusion of multimodal features, strengthening detail understanding, reducing local feature loss, and achieving efficient fusion of RGB and depth information.

[0006] To achieve the objective of this invention, the technical solution adopted is as follows: A multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is proposed. The multi-channel cross-fusion method is based on a multimodal cross-fusion strategy network driven by a multi-channel fusion self-attention module and a cross-modal fusion module. The multimodal cross-fusion strategy network includes a feature extraction module, a multimodal cross-fusion module, and a pose estimation module. Furthermore, the multimodal cross-fusion strategy network of the multi-channel cross-fusion method includes a dynamic range convolution module, a multi-channel fusion self-attention module, and a cross-modal cross-fusion module. The multi-channel fusion self-attention module includes an information fusion self-attention module and a dual-attention fusion module.

[0007] A multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net, the multi-channel cross-fusion method specifically includes the following steps: S1. The feature extraction module takes an RGB-D image as input, obtains the bounding box of the target object, and crops the RGB and depth images. The depth image is converted into point cloud data using camera intrinsics. Then, two feature extraction branches—CNN and a variant of PointNet++—are used to process the cropped RGB image and the converted point cloud data respectively to capture color features. and geometric features ; S2. The obtained color features are processed through a multimodal cross-fusion module. As complementary color information in point cloud data, and geometric features Both are used as input to this modality. After initial fusion, they are further processed through dynamic range convolution to obtain the reordered fusion feature. Then, through processing, three types of mode features are obtained: cross-modal features. Multi-channel fusion features and global features The three elements are then merged into a final pixel-level fusion feature. ; S3. Pose estimation module for pixel-level fusion features Pixel-by-pixel pose prediction is performed to generate multiple poses. The pose with the highest confidence is selected as the initial pose, and then it is refined through an improved iterative refinement network. The refined pose is then used as the final output multimodal 6D pose.

[0008] As a preferred embodiment, the specific operation of step S1 in the multi-channel cross-fusion method is as follows: S11. First, perform semantic segmentation on the RGB image to obtain the bounding box of the target region. Then, use the bounding box to crop and segment the RGB image and depth image, and input the cropped RGB image and depth image into their respective feature extraction modules to obtain color features. and geometric features ; S12. For RGB images, color feature extraction is achieved using a CNN-based encoder-decoder structure. The encoder employs a ResNet18 architecture, and the decoder uses a PSPNet pyramid with four upsampling layers. The input size is... RGB images mapped to a size of In the space, each pixel is represented by a color feature. 3D vector representation, thus obtaining color features ; S13. For depth images, first convert them into points based on camera intrinsic parameters. Using point cloud data and combining the hierarchical feature learning method in PointNet++, a point feature editor, P2Net, is designed. P2Net consists of two abstraction layers and two point feature propagation layers. Each abstraction layer can be divided into a sampling layer, a grouping layer, and an extraction layer. The sampling layer uses farthest point sampling, primarily for data of size [data missing]. The input point cloud is sampled, and several center points are selected from these points; the grouping layer uses ball query to define partitions, and uses the center points obtained in the previous step to divide the point set into several partitions; the extraction layer uses the mini-PointNet network as a feature extractor to encode each partition, and the obtained features are used as the feature vector of the center point of each partition. First of all Point cloud data is given as input. Points Selected using iterative farthest point sampling Each sampling point is the center point. Next, a ball-based query grouping strategy is used. Centered on the sampling point, within a given radius, the top K points with the largest serial numbers (from smallest to largest) are selected as a group. A total of... Grouping, then using mini-PointNet to extract features from each group, resulting in a size of The point cloud data is used as the input to the next abstraction layer, where 64 refers to the number of new features combined with the local context, which is determined by the number of output channels of the multilayer perceptron (MLP). This is the operation step of an abstraction layer. After processing through two layers of abstraction, the resulting size is... Data, including the number of point clouds The feature count is 512. After processing through two point feature propagation layers, the features of each point are recovered layer by layer from the sampled points, finally resulting in a value of [value missing]. geometric features .

[0009] As a preferred embodiment, the specific operation of step S2 in the multi-channel cross-fusion method is as follows: S21. The dynamic range convolution module converts the initial fused features of the input into... Divided into two segments along the channel dimension and Reorder and reassemble the two into Among them, for Perform reordering operations in the horizontal and vertical directions, and then combine the sorted features with... Connect them:

[0010]

[0011]

[0012] in, Initial fusion features Features of half the channel count after reordering Initial fusion features The characteristics of the unprocessed half of the channel count, This is a splicing operation along the channel dimension. This is an operation that splits features along the channel dimension. and These are vertical and horizontal sorting operations, respectively, and the initial fused features. For input color features and geometric features After two floors The fused features are obtained by convolutional transformation followed by concatenation along the channel dimension. for Pointwise convolution, reordering and fusion of features That is to The input is fed into the dynamic range convolution module, where it is reordered and reassembled along the horizontal and vertical directions. S22. The multi-channel fusion self-attention module combines the self-attention mechanisms of the information fusion self-attention module and the dual-attention fusion module, introducing three functions: query function. Key functions Sum function Applied to connectivity characteristics, respectively for The transformation yields three features: query features. Key features Value characteristics Then and Multiply the transpose of the matrix, then pass the activation function. Obtain adaptive attention mapping This indicates the importance of each feature from the RGB image and the depth image in the entire feature set:

[0013]

[0014] Again and Perform matrix multiplication to obtain the updated cross-modal features. :

[0015] in, For adaptive attention mapping, For activation function, For cross-modal features Size, =500 represents the number of features across modalities. The number of channels for cross-modal features; S23. The cross-modal cross-fusion module will reorder and fuse features processed by the dynamic range convolution module. First, a convolutional transformation is performed to obtain the global features. This fusion feature retains the initial fusion characteristics, participates in the final fusion, and then the resulting global feature is... After being convolved and flattened by the dynamic range convolution module, it is divided into three features: , , The specific expression is as follows:

[0016] , in, To represent the height of a feature, which is the number of pixels or feature units in the vertical direction, The width of the feature is the number of pixels or feature units in the horizontal direction. The number of channels is a feature. Global features Size, All are global features The three are obtained by convolution processing using the dynamic range convolution module, and they are essentially the same in nature. and The main method involves obtaining spatial attention maps and positional attention maps through matrix multiplication, and then... Multiplying these two attention maps by a matrix yields the spatial attention features. Channel attention characteristics ; The obtained feature map and The transpose of the matrix is ​​used for cross matrix multiplication, and then... Obtain spatial attention map Channel attention map :

[0017] in, Spatial attention map Size, Channel attention map Size, spatial attention map For the first The position is the first The influence of each position, channel attention map For the first The first channel is for the first The impact of each channel; Then the obtained spatial attention map Channel attention map With feature map Multiplying the transpose of the matrix yields spatial attention features. Channel attention characteristics :

[0018] Finally and Channel-dimensional concatenation is performed, followed by average pooling to obtain multi-channel fused features. :

[0019] in, Multi-channel fusion features Size, To achieve average pooling operation, both spatial and channel dimensions are combined, and feature representation is optimized from both spatial and semantic dimensions. Finally, cross-modal features will be... Multi-channel fusion features Global features Channel-level concatenation is performed to obtain the final pixel-level fused features. :

[0020] in, Pixel-level fusion features The size.

[0021] As a preferred embodiment, the specific operation of step S3 in the multi-channel cross-fusion method is as follows: S31. Pixel-level fusion features Embedded into the pose estimation module of a residual pose estimation network based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type, features are fused from each pixel level. Predicting a pose yields a set of predicted poses. :

[0022]

[0023]

[0024]

[0025] in, For the summation function, For multilayer perceptrons, The confidence score for regressing the pose for each center point feature. The 3D rotation in the predicted pose generated after feature regression of the i-th pixel is given. The translation relationship in the predicted pose generated after feature regression of the i-th pixel is given. The predicted pose is generated after feature regression of the i-th pixel. It is an intermediate variable in the pose estimation process and is used to simplify the formula for easier understanding. S32. For the obtained set of predicted poses A loss function is set to infer each predicted pose. The confidence score is then used to select the predicted pose with the highest confidence score. The initial pose used for iteration However, the loss function settings differ for asymmetric and symmetric objects; For asymmetric objects:

[0026] For symmetrical objects:

[0027] in, This represents the average distance between the sampled points of the object model in the true pose and the corresponding points in the predicted pose. Indicates the number of features at the center point. This represents the true pose of the j-th point among N randomly selected points in the 3D model of an object. For the true pose, For the three-dimensional rotation in the actual pose, The translation relationship in the actual pose; confidence score With each pixel added to select the most likely correct pose assumption, the total loss function is set as follows:

[0028] in, Let w be the total loss function, and w be a hyperparameter. The confidence score represents the feature regression pose of each center point; The higher the confidence score, the smaller the total loss function; the predicted pose with the highest confidence score is used as the initial pose. ; S33. Optimize the initial pose. The point cloud is processed to obtain a pose-initialized point cloud data, which, along with an RGB image, is used as new input values ​​into a multimodal cross-fusion strategy network to obtain pixel-level fusion features for the first iteration. Subsequently, the pose estimation module based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type residual pose estimation network was used to calculate the pose change in the first iteration. and :

[0029]

[0030] in, This represents the 3D rotational change in the pose during the first iteration. The translation relationship in the pose during the first iteration. The pixel-level fusion features of the first iteration; Finally, the pose change amount and Acting on the initial pose The output pose of the first iteration is obtained. This is one iteration process; the pose output in the previous iteration is used as the initial pose for the next iteration, and so on. After the iteration, the final pose is:

[0031] in, For the number of iterations, For iteration counting, It is in the The pose of the next iteration. For the first The pose correction amount during the next iteration. This is the final pose after iterative refinement. This is the initial pose; Overfitting occurred in the sixth iteration. To prevent overfitting, the final high-precision 6D pose can be obtained in five iterations. This refers to multimodal 6D pose.

[0032] This invention provides a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net, which has the following advantages: This invention presents a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net. It addresses feature extraction and multimodal feature fusion strategies for point cloud data by designing a point feature editor (P2Net) to extract geometric features, enhancing the network's ability to represent geometric information. Furthermore, it constructs a multimodal cross-fusion strategy, utilizing dynamic range convolution to reorder color and geometric features, enhancing the adaptability of subsequent convolutions under different input dynamic ranges. This allows for more accurate capture of image details. The MFSA and CMCF modules work synergistically to achieve effective fusion of multimodal features, strengthening detail understanding, reducing local feature loss, and achieving efficient fusion of RGB and depth information. This method exhibits high pose estimation accuracy, significant technical advantages, and superior performance. It demonstrates significant effectiveness in 6D pose estimation under complex environments, providing strong support for efficient estimation techniques in practical applications. Attached Figure Description

[0033] Figure 1 This is a diagram illustrating the overall structural framework of the multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net, as presented in this invention.

[0034] Figure 2 This is a flowchart of the iterative refinement network of the multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net in this invention.

[0035] Figure 3 This is an iterative network iteration result diagram of the multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net in this invention.

[0036] Figure 4 This is a qualitative result of the 6D pose estimation on the YCB-Video dataset based on the multi-channel cross-fusion method of MCF-Net for multimodal 6D pose estimation in this invention.

[0037] Figure 5This is the qualitative result of the multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net in this invention on the HR-Vision dataset. Detailed Implementation

[0038] The present invention will be further described and illustrated below with reference to specific embodiments and accompanying drawings.

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] In the description of this invention, it should be understood that the terms "upper", "lower", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0041] like Figure 1 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is presented. The multi-channel cross-fusion method is based on a multimodal cross-fusion strategy (MCF) network driven by a multi-channel fusion self-attention (MFSA) module and a cross-modal cross-fusion (CMCF) module. The multimodal cross-fusion strategy network includes a feature extraction module, a multimodal cross-fusion module, and a pose estimation module.

[0042] like Figure 1 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is presented. The multimodal cross-fusion strategy network of the multimodal cross-fusion method includes a dynamic range convolution module (DRC), a multi-channel fusion self-attention module (MFSA), and a cross-modal cross-fusion module (CMCF). The multi-channel fusion self-attention module (MFSA) includes an information fusion self-attention module (IFSA) and a dual attention fusion module (DAF).

[0043] like Figure 1 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net specifically includes the following steps: S1. The feature extraction module takes an RGB-D image as input, obtains the bounding box of the target object, and crops the RGB and depth images. The depth image is converted into point cloud data using camera intrinsics. Then, two feature extraction branches—CNN and a variant of PointNet++—are used to process the cropped RGB image and the converted point cloud data respectively to capture color features. and geometric features ; S2. The obtained color features are processed through a multimodal cross-fusion module. As complementary color information in point cloud data, and geometric features Both are used as input to this modality. After initial fusion, they are further processed through dynamic range convolution to obtain the reordered fusion feature. Then, through processing, three types of mode features are obtained: cross-modal features. Multi-channel fusion features and global features The three elements are then merged into a final pixel-level fusion feature. ; S3. Pose estimation module for pixel-level fusion features Pixel-by-pixel pose prediction is performed to generate multiple poses. The pose with the highest confidence is selected as the initial pose, and then it is refined through an improved iterative refinement network. The refined pose is then used as the final output multimodal 6D pose.

[0044] Furthermore, such as Figure 1 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is described. The specific operation of step S1 in the multi-channel cross-fusion method is as follows: S11. First, perform semantic segmentation on the RGB image to obtain the bounding box of the target region. Then, use the bounding box to crop and segment the RGB image and depth image, and input the cropped RGB image and depth image into their respective feature extraction modules to obtain color features. and geometric features ; S12. For RGB images, color feature extraction is achieved using a CNN-based encoder-decoder structure. The encoder employs a ResNet18 architecture, and the decoder uses a PSPNet pyramid with four upsampling layers. The input size is... RGB images mapped to a size of In the space, each pixel is represented by a color feature. 3D vector representation, thus obtaining color features ; S13. For depth images, first convert them into points based on camera intrinsic parameters. Using point cloud data and combining the hierarchical feature learning method in PointNet++, a point feature editor, P2Net, is designed. P2Net consists of two abstraction layers and two point feature propagation layers. Each abstraction layer can be divided into a sampling layer, a grouping layer, and an extraction layer. The sampling layer uses farthest point sampling, primarily for data of size [data missing]. The input point cloud is sampled, and several center points are selected from these points; the grouping layer uses ball query to define partitions, and uses the center points obtained in the previous step to divide the point set into several partitions; the extraction layer uses the mini-PointNet network as a feature extractor to encode each partition, and the obtained features are used as the feature vector of the center point of each partition. First of all Point cloud data is given as input. Points Selected using iterative farthest point sampling Each sampling point is the center point. Next, a ball-based query grouping strategy is used. Centered on the sampling point, within a given radius, the top K points with the largest serial numbers (from smallest to largest) are selected as a group. A total of... Grouping, then using mini-PointNet to extract features from each group, resulting in a size of The point cloud data is used as the input to the next abstraction layer, where 64 refers to the number of new features combined with the local context, which is determined by the number of output channels of the multilayer perceptron (MLP). This is the operation step of an abstraction layer. After processing through two layers of abstraction, the resulting size is... Data, including the number of point clouds The feature count is 512. After processing through two point feature propagation layers, the features of each point are recovered layer by layer from the sampled points, finally resulting in a value of [value missing]. geometric features .

[0045] Furthermore, such as Figure 1 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is presented. The specific operation of step S2 in the multi-channel cross-fusion method is as follows: S21. The dynamic range convolution module converts the initial fused features of the input into... Divided into two segments along the channel dimension and Reorder and reassemble the two into Among them, for Perform reordering operations in the horizontal and vertical directions, and then combine the sorted features with... Connect them:

[0046]

[0047]

[0048] in, Initial fusion features Features of half the channel count after reordering Initial fusion features The characteristics of the unprocessed half of the channel count, This is a splicing operation along the channel dimension. This is an operation that splits features along the channel dimension. and These are vertical and horizontal sorting operations, respectively, and the initial fused features. For input color features and geometric features After two floors The fused features are obtained by convolutional transformation followed by concatenation along the channel dimension. for Pointwise convolution, reordering and fusion of features That is to The input is fed into the dynamic range convolution module, where it is reordered and reassembled along the horizontal and vertical directions. S22. The multi-channel fusion self-attention module combines the self-attention mechanisms of the information fusion self-attention module and the dual-attention fusion module, introducing three functions: query function. Key functions Sum function Applied to connectivity characteristics, respectively for The transformation yields three features: query features. Key features Value characteristics Then and Multiply the transpose of the matrix, then pass the activation function. Obtain adaptive attention mapping This indicates the importance of each feature from the RGB image and the depth image in the entire feature set:

[0049]

[0050] Again and Perform matrix multiplication to obtain the updated cross-modal features. :

[0051] in, For adaptive attention mapping, For activation function, For cross-modal features Size, =500 represents the number of features across modalities. The number of channels for cross-modal features; S23. The cross-modal cross-fusion module will reorder and fuse features processed by the dynamic range convolution module. First, a convolutional transformation is performed to obtain the global features. This fusion feature retains the initial fusion characteristics, participates in the final fusion, and then the resulting global feature is... After being convolved and flattened by the dynamic range convolution module, it is divided into three features: , , The specific expression is as follows:

[0052]

[0053] in, To represent the height of a feature, which is the number of pixels or feature units in the vertical direction, The width of the feature is the number of pixels or feature units in the horizontal direction. The number of channels is a feature. Global features Size, All are global features The three are obtained by convolution processing using the dynamic range convolution module, and they are essentially the same in nature. and The main method involves obtaining spatial attention maps and positional attention maps through matrix multiplication, and then... Multiplying these two attention maps by a matrix yields the spatial attention features. Channel attention characteristics ; The obtained feature map and The transpose of the matrix is ​​used for cross matrix multiplication, and then... Obtain spatial attention map Channel attention map :

[0054] in, Spatial attention map Size, Channel attention map Size, spatial attention map For the first The position is the first The influence of each position, channel attention map For the first The first channel is for the first The impact of each channel; Then the obtained spatial attention map Channel attention map With feature map Multiplying the transpose of the matrix yields spatial attention features. Channel attention characteristics :

[0055] Finally and Channel-dimensional concatenation is performed, followed by average pooling to obtain multi-channel fused features. :

[0056] in, Multi-channel fusion features Size, To achieve average pooling operation, both spatial and channel dimensions are combined, and feature representation is optimized from both spatial and semantic dimensions. Finally, cross-modal features will be... Multi-channel fusion features Global features Channel-level concatenation is performed to obtain the final pixel-level fused features. :

[0057] in, Pixel-level fusion features The size.

[0058] Furthermore, such as Figure 1 , 2 As shown, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is presented. The specific operation of step S3 in the multi-channel cross-fusion method is as follows: S31. Pixel-level fusion features Embedded into the pose estimation module of a residual pose estimation network based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type, features are fused from each pixel level. Predicting a pose yields a set of predicted poses. :

[0059]

[0060]

[0061]

[0062] in, For the summation function, For multilayer perceptrons, The confidence score for regressing the pose for each center point feature. The 3D rotation in the predicted pose generated after feature regression of the i-th pixel is given. The translation relationship in the predicted pose generated after feature regression of the i-th pixel is given. The predicted pose is generated after feature regression of the i-th pixel. It is an intermediate variable in the pose estimation process and is used to simplify the formula for easier understanding. S32. For the obtained set of predicted poses A loss function is set to infer each predicted pose. The confidence score is then used to select the predicted pose with the highest confidence score. The initial pose used for iteration However, the loss function settings differ for asymmetric and symmetric objects; For asymmetric objects:

[0063] For symmetrical objects:

[0064] in, This represents the average distance between the sampled points of the object model in the true pose and the corresponding points in the predicted pose. Indicates the number of features at the center point. This represents the true pose of the j-th point among N randomly selected points in the 3D model of an object. For the true pose, For the three-dimensional rotation in the actual pose, The translation relationship in the actual pose; confidence score With each pixel added to select the most likely correct pose assumption, the total loss function is set as follows:

[0065] in, Let w be the total loss function, and w be a hyperparameter. The confidence score represents the feature regression pose of each center point; The higher the confidence score, the smaller the total loss function; the predicted pose with the highest confidence score is used as the initial pose. ; S33. Optimize the initial pose. The point cloud is processed to obtain a pose-initialized point cloud data, which, along with an RGB image, is used as new input values ​​into a multimodal cross-fusion strategy network to obtain pixel-level fusion features for the first iteration. Subsequently, the pose estimation module based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type residual pose estimation network was used to calculate the pose change in the first iteration. and :

[0066]

[0067] in, This represents the 3D rotational change in the pose during the first iteration. The translation relationship in the pose during the first iteration. The pixel-level fusion features of the first iteration; Finally, the pose change amount and Acting on the initial pose The output pose of the first iteration is obtained. This is one iteration process; the pose output in the previous iteration is used as the initial pose for the next iteration, and so on. After the iteration, the final pose is:

[0068] in, For the number of iterations, For iteration counting, It is in the The pose of the next iteration. For the first The pose correction amount during the next iteration. This is the final pose after iterative refinement. This is the initial pose; Overfitting occurred in the sixth iteration. To prevent overfitting, the final high-precision 6D pose can be obtained in five iterations. This refers to multimodal 6D pose.

[0069] This invention employs a three-iteration process, ultimately achieving a pose estimation result with high accuracy, demonstrating the adaptability of the new iterative network to MCF-Net. The results of each iteration are as follows: Figure 3 As shown.

[0070] Experimental environment The experiment was conducted on a computer configured with an RTX A6000 GPU, and the network was trained 200 times. The training environment was built in PyTorch, using the following hyperparameters: learning rate 0.0009, batch size 8, iteration threshold 0.01, and pose iterations 5. In the field of pose estimation, there are several authoritative public datasets, such as YCB-Video, LineMOD, and OcclusionLindMOD. This invention will evaluate our method on these datasets as well as the custom dataset HB-Vision. YCB-Video dataset: Contains 21 different object categories, each with its own shape and texture. It consists of 92 RGB-D videos, each showing a subset of 21 objects in a different indoor scene. 16,189 frames were selected from 80 videos and synthesized with other 80,000 frames for training, while 2,949 keyframes were suspended from the remaining 12 videos for testing. LineMOD dataset: It is a dataset consisting of 13 low-texture objects from 13 videos, with a total of 15,780 frames. Each frame is a group, and each group contains corresponding RGB images, depth maps, 3D models of objects, and 6D pose annotations. 2,373 groups are the test set and 13,407 groups are the training set. Occlousion LineMOD dataset: This dataset was created by adding annotations to each scene of the LineMOD dataset. It mainly introduces complex situations where multiple objects occlude each other. Each image has different degrees of occlusion, and target pose estimation with severe occlusion is a challenge faced by this dataset. HB-Vision Dataset: This dataset is customized to solve the problem of grasping objects in home environments. It uses a Realsense D435i camera to capture approximately 40 seconds of video of home objects at 30 frames per second (fps) with a resolution of 1280×720. Image information is saved frame by frame. The dataset contains 7050 sets of 6 types of home objects. Each set includes corresponding RGB images, depth maps, 3D object models, and 6DOF pose annotations. Sets with multiples of 5 were extracted as the training set (1410 sets), and the remainder were used as the test set (5640 sets). To simulate a real home environment, two modes were used: selecting 4 and 5 types of home objects respectively, placing them under different lighting intensities, and adjusting the angle during filming to simulate object occlusion (30%~60%) and lighting changes, thus verifying the performance of the proposed network in practical applications.

[0071] Evaluation indicators Because pose estimation differs significantly between asymmetric and symmetric objects, we use ADD and ADD-S to evaluate our network. First, for asymmetric objects, we use the average distance between model points (ADD), which is the distance between a sampled point in the model after the predicted pose transformation and the corresponding point in the true pose model. The definition of ADD is as follows:

[0072] Secondly, for symmetrical objects, the average distance between nearest points (ADD-S) is used, which minimizes the distance between each point in the estimated model direction and the nearest point on the ground model; the definition of ADD-S is as follows:

[0073] in, Indicates random sampling from the 3D CAD model of the object. The 3D point of the th ? One point, This represents the actual pose on the ground. To predict the pose after pose transformation using a network. Indicates the first The predicted pose is obtained by transforming the pose of each pixel.

[0074] The evaluation metrics used vary slightly depending on the dataset. In the YCB-Video dataset, the average distance between the nearest points (ADD-S) is used to measure pose error, and two metrics are reported: the area under the curve (AUC) of ADD-S with a maximum threshold of 0.1 m; and the percentage of ADD-S values ​​less than 2 cm (<2 cm). In the LineMOD, Occlusion LineMOD, and HB-Vision datasets, the percentage of distances less than 10% of the object diameter (ADD-0.1d) is reported.

[0075] Evaluation results on the YCB-Video dataset: The method of this invention, along with some advanced methods, evaluates 21 objects on the YCB-Video dataset based on ADD(-S) as an evaluation metric.

[0076] Table 1 Results on the YCB-Video dataset

[0077] As shown in Table 1, MCF achieved 97.0% on the ADD-S (AUC) evaluation metric. Compared to other methods, MCF outperformed PoseCNN (21.2%), DenseFusion (3.9%), CMA (3.7%), PVN3D (1.5%), and FFB6D (0.4%). MCF also outperformed the aforementioned methods on the ADD-S (<2 cm) scale. Furthermore, we also... Figure 4 The figure presents qualitative results for 6D pose estimation on the YCB-Video dataset and compares these results with DenseFusion and FFB6D. The objects in the figure that clearly outperform other state-of-the-art methods are outlined with bounding boxes. The figure shows that the three methods are similar in their ability to estimate the 6D pose of a single geometrically unoccluded object, but DenseFusion and FFB6D fail to accurately estimate the pose of occluded objects, while our method can achieve this.

[0078] Evaluation results on the LineMOD dataset: The method of this invention uses ADD-0.1d as an evaluation metric to evaluate the network performance of 13 types of objects on the LineMOD dataset, and compares it with some 6D position estimation methods such as PoseCNN, DenseFusion, CMA, G2L-Net, PVN3D and FFB6D.

[0079] Table 2 Results on the LineMOD dataset

[0080] As shown in Table 2, the method of this invention (MCF) achieves 99.3% of the evaluation value on this dataset, which is about 6% higher than DenseFusion and also outperforms other methods. In particular, it shows significant performance improvements for single objects with low texture, such as ape (12.4%), duck (11.3%), driller (9%), and holepuncher (8.6%).

[0081] In addition, ablation experiments were designed on this dataset to analyze the impact and effect of the proposed model components. There are three types of proposed model components: P2Net, a point feature editor for the feature extraction module based on CNN and PointNet++ variant structures; DRC, a dynamic range convolution module; and MCF, a multi-channel cross-fusion strategy module. In order to verify the improvement effect of each component on the network pose estimation performance, we designed multiple sets of comparative experiments, as shown in Table 3.

[0082] Table 3 Comparison results of ablation experiments on the LineMOD dataset

[0083] As can be seen, when using the multi-channel cross-fusion strategy alone, the performance on ADD-0.1d is improved by 5.3% compared to the original network, which is much better than P2Net (2.3%) and DRC (0.5%), demonstrating the advantages of our proposed MCF dual-modal feature fusion strategy. In addition, the last three sets of comparative experiments in the table are ablation experiments conducted by combining P2Net and DRC with MCF alone, and by combining all three together. The experimental results in Table 3 clearly show that the other two components we proposed can improve the dual-modal feature fusion effect of MCF, and the combination of the three can achieve the best pose estimation performance of the network.

[0084] After verifying the effectiveness of the multimodal cross-fusion strategy, this invention also analyzes the influence and role of each component structure in MCF, and designs four ablation experiments on the LineMOD dataset: (1) neither module is used, and the fusion method in DenseFusion is adopted; (2) only the information fusion self-attention module is used; (3) only the dual attention fusion module is used; (4) both modules are used together. To ensure the fairness of the experiment, we used P2Net and DRC in all four experiments, and the experimental results are shown in Table 4.

[0085] Table 4 shows the ablation experiments of each MCF component on the LineMOD dataset. Each experiment used P2Net and DRC.

[0086] As can be seen from the table, using IFSA or DAF alone improved the performance by 2.5% and 2% respectively compared to the original fusion strategy; and from the results of the fourth set of experiments, the evaluation index was further improved to 99.3% when the two modules worked together. This confirms that the interaction between the two modules in MCF can achieve better pose estimation results.

[0087] Evaluation results on more challenging datasets: For the two datasets mentioned above, objects in the YCB-Video and LineMOD datasets generally do not exhibit significant occlusion, and most labeled objects are clearly visible in the field of view, which still differs from real-world complex scenes. The main research task of the method in this invention is to achieve accurate object grasping by the robot in complex environments. In other words, verifying the robustness of the method in challenging complex environments (such as severe occlusion and large lighting variations) in accurately estimating 6D pose is essential. Therefore, based on ADD(-S) as the evaluation metric, the method is evaluated on two more challenging datasets: the Occluded LineMOD and HR-Vision datasets. The results are compared with some mainstream 6D pose estimation methods, and the evaluation results are shown in Tables 4 and 5.

[0088] Table 4 Results on the Occluded LineMOD dataset

[0089] Table 5 Results on the HR-Vision dataset

[0090] As can be seen from the two tables above, even on more challenging datasets, our method still outperforms other methods in pose estimation accuracy, further confirming its robustness to occlusion; meanwhile, the qualitative results on the HR-Vision dataset are as follows: Figure 5 As shown, the pose estimation effect of the object can be more intuitively represented by the model projection in the figure. It can be seen that, compared with our method, the model projection of FFB6D is more affected by the changes in lighting. This means that our method has a more stable pose estimation effect in environments with large changes in lighting, which confirms the robustness of our method to changes in lighting.

[0091] In this invention, a multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net is designed to extract geometric features from point cloud data and improve the network's ability to represent geometric information. A multimodal cross-fusion strategy is also constructed, utilizing dynamic range convolution to reorder color and geometric features, enhancing the adaptability of subsequent convolutions under different input dynamic ranges. This allows for more accurate capture of image details. Furthermore, the MFSA and CMCF modules work together to achieve effective fusion of multimodal features, strengthening detail understanding, reducing local feature loss, and achieving efficient fusion of RGB and depth information. This method exhibits high pose estimation accuracy, significant technical advantages, and superior performance. It demonstrates significant effectiveness in 6D pose estimation under complex environments, providing strong support for efficient estimation techniques in practical applications.

[0092] The technical solutions disclosed in the embodiments of the present invention have been described in detail above. Specific embodiments have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The description of the above embodiments is only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net, characterized in that: The multi-channel cross-fusion method is based on a multi-modal cross-fusion strategy network driven by a multi-channel fusion self-attention module and a cross-modal fusion module. The multi-modal cross-fusion strategy network includes a feature extraction module, a multi-modal cross-fusion module, and a pose estimation module. The multi-channel cross-fusion method specifically includes the following steps: S1. The feature extraction module takes an RGB-D image as input, obtains the bounding box of the target object, and crops the RGB and depth images. The depth image is converted into point cloud data using camera intrinsics. Then, two feature extraction branches—CNN and a variant of PointNet++—are used to process the cropped RGB image and the converted point cloud data respectively to capture color features. and geometric features ; S2. The obtained color features are processed through a multimodal cross-fusion module. As complementary color information in point cloud data, and geometric features Both are used as input to this modality. After initial fusion, they are further processed through dynamic range convolution to obtain the reordered fusion feature. Then, through processing, three types of mode features are obtained: cross-modal features. Multi-channel fusion features and global features The three elements are then merged into a final pixel-level fusion feature. ; S3. Pose estimation module for pixel-level fusion features Pixel-by-pixel pose prediction is performed to generate multiple poses. The pose with the highest confidence is selected as the initial pose, and then it is refined through an improved iterative refinement network. The refined pose is then used as the final output multimodal 6D pose.

2. The multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net according to claim 1, characterized in that: The specific operation of step S1 in the multi-channel cross-fusion method is as follows: S11. First, perform semantic segmentation on the RGB image to obtain the bounding box of the target region. Then, use the bounding box to crop and segment the RGB image and depth image, and input the cropped RGB image and depth image into their respective feature extraction modules to obtain color features. and geometric features ; S12. For RGB images, color feature extraction is achieved using a CNN-based encoder-decoder structure. The encoder employs a ResNet18 architecture, and the decoder uses a PSPNet pyramid with four upsampling layers. The input size is... RGB images mapped to a size of In the space, each pixel is represented by a color feature. 3D vector representation, thus obtaining color features ; S13. For depth images, first convert them into points based on camera intrinsic parameters. Using point cloud data and combining the hierarchical feature learning method in PointNet++, a point feature editor, P2Net, is designed. P2Net consists of two abstraction layers and two point feature propagation layers. Each abstraction layer can be divided into a sampling layer, a grouping layer, and an extraction layer. The sampling layer uses farthest point sampling, primarily for data of size [data missing]. The input point cloud is sampled, and several center points are selected from these points; the grouping layer uses ball query to define partitions, and uses the center points obtained in the previous step to divide the point set into several partitions; the extraction layer uses the mini-PointNet network as a feature extractor to encode each partition, and the obtained features are used as the feature vector of the center point of each partition. First of all Point cloud data is given as input. Points Selected using iterative farthest point sampling Each sampling point is the center point. Next, a ball-based query grouping strategy is used. Centered on the sampling point, within a given radius, the top K points with the largest serial numbers (from smallest to largest) are selected as a group. A total of... Grouping, then using mini-PointNet to extract features from each group, resulting in a size of The point cloud data is used as the input to the next abstraction layer, where 64 refers to the number of new features combined with the local context, which is determined by the number of output channels of the multilayer perceptron (MLP). This is the operation step of an abstraction layer. After processing through two layers of abstraction, the resulting size is... Data, including the number of point clouds The feature count is 512. After processing through two point feature propagation layers, the features of each point are recovered layer by layer from the sampled points, finally resulting in a value of [value missing]. geometric features .

3. The multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net according to claim 1, characterized in that: The multimodal cross-fusion strategy network of the multi-channel cross-fusion method includes a dynamic range convolution module, a multi-channel fusion self-attention module, and a cross-modal cross-fusion module. The multi-channel fusion self-attention module includes an information fusion self-attention module and a dual-attention fusion module.

4. The multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net according to claim 3, characterized in that: The specific operation of step S2 in the multi-channel cross-fusion method is as follows: S21. The dynamic range convolution module converts the initial fused features of the input into... Divided into two segments along the channel dimension and Reorder and reassemble the two into Among them, for Perform reordering operations in the horizontal and vertical directions, and then combine the sorted features with... Connect them: ， in, Initial fusion features Features of half the channel count after reordering Initial fusion features The characteristics of the unprocessed half of the channel count, This is a splicing operation along the channel dimension. This is an operation that splits features along the channel dimension. and These are vertical and horizontal sorting operations, respectively, and the initial fused features. For input color features and geometric features After two floors The fused features are obtained by convolutional transformation followed by concatenation along the channel dimension. for Pointwise convolution, reordering and fusion of features That is to The input is fed into the dynamic range convolution module, where it is reordered and reassembled along the horizontal and vertical directions. S22. The multi-channel fusion self-attention module combines the self-attention mechanisms of the information fusion self-attention module and the dual-attention fusion module, introducing three functions: query function. Key functions Sum function Applied to connectivity characteristics, respectively for The transformation yields three features: query features. Key features Value characteristics Then and Multiply the transpose of the matrix, then pass the activation function. Obtain adaptive attention mapping This indicates the importance of each feature from the RGB image and the depth image in the entire feature set: ， ， Again and Perform matrix multiplication to obtain the updated cross-modal features. : , in, For adaptive attention mapping, For activation function, For cross-modal features Size, =500 represents the number of features across modalities. The number of channels for cross-modal features; S23. The cross-modal cross-fusion module will reorder and fuse features processed by the dynamic range convolution module. First, a convolutional transformation is performed to obtain the global features. This fusion feature retains the initial fusion characteristics, participates in the final fusion, and then the resulting global feature is... After being convolved and flattened by the dynamic range convolution module, it is divided into three features: , , The specific expression is as follows: ， in, To represent the height of a feature, which is the number of pixels or feature units in the vertical direction, The width of the feature is the number of pixels or feature units in the horizontal direction. The number of channels is a feature. Global features Size, All are global features The three are obtained by convolution processing using the dynamic range convolution module, and they are essentially the same in nature. and The main method involves obtaining spatial attention maps and positional attention maps through matrix multiplication, and then... Multiplying these two attention maps by a matrix yields the spatial attention features. Channel attention characteristics ; The obtained feature map and The transpose of the matrix is ​​used for cross matrix multiplication, and then... Obtain spatial attention map Channel attention map : ， in, Spatial attention map Size, Channel attention map The size, spatial attention diagram is the first The position is the first The influence of each position, channel attention map For the first The first channel is for the first The impact of each channel; Then the obtained spatial attention map Channel attention map With feature map Multiplying the transpose of the matrix yields spatial attention features. Channel attention characteristics : , Finally and Channel-dimensional concatenation is performed, followed by average pooling to obtain multi-channel fused features. : ， in, Multi-channel fusion features Size, To achieve average pooling operation, both spatial and channel dimensions are combined, and feature representation is optimized from both spatial and semantic dimensions. Finally, cross-modal features will be... Multi-channel fusion features Global features Channel-level concatenation is performed to obtain the final pixel-level fused features. : ， in, Pixel-level fusion features The size.

5. The multi-channel cross-fusion method for multimodal 6D pose estimation based on MCF-Net according to claim 1, characterized in that: The specific operation of step S3 in the multi-channel cross-fusion method is as follows: S31. Pixel-level fusion features Embedded into the pose estimation module of a residual pose estimation network based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type, features are fused from each pixel level. Predicting a pose yields a set of predicted poses. : ， ， ， ， in, For the summation function, For multilayer perceptrons, The confidence score for regressing the pose for each center point feature. The 3D rotation in the predicted pose generated after feature regression of the i-th pixel is given. The translation relationship in the predicted pose generated after feature regression of the i-th pixel is given. The predicted pose is generated after feature regression of the i-th pixel. It is an intermediate variable in the pose estimation process and is used to simplify the formula for easier understanding. S32. For the obtained set of predicted poses A loss function is set to infer each predicted pose. The confidence score is then used to select the predicted pose with the highest confidence score. The initial pose used for iteration However, the loss function settings differ for asymmetric and symmetric objects; For asymmetric objects: ， For symmetrical objects: ， in, This represents the average distance between the sampled points of the object model in the true pose and the corresponding points in the predicted pose. Indicates the number of features at the center point. This represents the true pose of the j-th point among N randomly selected points in the 3D model of an object. For the true pose, For the three-dimensional rotation in the actual pose, The translation relationship in the actual pose; confidence score With each pixel added to select the most likely correct pose assumption, the total loss function is set as follows: ， in, Let w be the total loss function, and w be a hyperparameter. The confidence score represents the feature regression pose of each center point; The higher the confidence score, the smaller the total loss function; the predicted pose with the highest confidence score is used as the initial pose. ; S33. Optimize the initial pose. The point cloud is processed to obtain a pose-initialized point cloud data, which, along with an RGB image, is used as new input values ​​into a multimodal cross-fusion strategy network to obtain pixel-level fusion features for the first iteration. Subsequently, the pose estimation module based on a multimodal cross-fusion strategy and a multilayer perceptron (MLP) type residual pose estimation network was used to calculate the pose change in the first iteration. and : ， ， in, This represents the 3D rotational change in the pose during the first iteration. The translation relationship in the pose during the first iteration. The pixel-level fusion features of the first iteration; Finally, the pose change amount and Acting on the initial pose The output pose of the first iteration is obtained. This is one iteration process; the pose output in the previous iteration is used as the initial pose for the next iteration, and so on. After the iteration, the final pose is: ， in, For the number of iterations, For iteration counting, It is in the The pose of the next iteration. For the first The pose correction amount during the next iteration. This is the final pose after iterative refinement. This is the initial pose; Overfitting occurred in the sixth iteration. To prevent overfitting, the final high-precision 6D pose can be obtained in five iterations. This refers to multimodal 6D pose.

Images