A 6D pose estimation method based on an iterative attention fusion network

Abstract

To address the issue of poor performance of current 6D pose estimation methods in cluttered scenes, this invention aims to provide a 6D pose estimation method based on an iterative attention mechanism. This method constructs an iterative attention fusion network, including a semantic segmentation network, a convolutional neural network, a point cloud network, an iterative attention module, a pose prediction module, and a pose iterative optimization module. Using RGB and depth images as inputs, semantic, appearance, and geometric features are extracted from the images through the convolutional neural network and point cloud network. These features are then densely fused, processed by the iterative attention module, and input into the pose prediction module to output the predicted 6D pose. The predicted 6D pose information is further input into the pose iterative optimization module to obtain the final 6D pose. This effectively improves the efficiency and accuracy of target object recognition in cluttered scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a 6D pose estimation method based on an iterative attention fusion network. Background Technology

[0002] 6D pose estimation is a crucial research area in computer vision, playing a vital role in many real-world applications such as robot grasping, autonomous driving, and augmented reality. It aims to estimate the position and pose of an object in three-dimensional space using information such as its RGB and depth images. In recent years, with the rapid development of deep learning technology, deep neural networks have been adopted to perform 6D pose estimation. However, due to sensor noise, scene clutter, and object occlusion, this task still faces considerable challenges.

[0003] Noise from the sensor within a depth camera can cause the loss of depth information for some objects in the depth image. Scene clutter and object occlusion can lead to inaccurate object segmentation in pose estimation algorithms, causing the model to fail to distinguish each distinct object instance, thus severely impacting the accuracy of pose estimation. Currently, many deep learning algorithms are applied to pose estimation tasks. PoseCNN estimates the object's translation by locating the object at the center of the image and predicting its distance from the camera, and then estimates the object's rotation by regressing features to quaternions. Other methods use deep neural networks to detect 2D keypoints of objects and then calculate 6D pose using a perspective n-point algorithm based on the 2D-3D correspondence. However, most of these methods are based on 2D projection, and the errors in projection can be significant in real 3D space, limiting their performance in complex scenes. DenseFusion is an end-to-end network model that performs pixel-level fusion of features from RGB and depth images and inputs the fused features into a pose prediction module to obtain 6D pose information, which significantly improves the pose estimation performance of objects. However, due to the lack of sufficient expressive power and semantic information in the fused features, the method performs poorly in cluttered scenes and when objects are occluded. Furthermore, simply regressing the aggregated features directly through stacking multilayer perceptrons to obtain pose information does not make good use of all the information in the features.

[0004] In summary, this invention addresses the aforementioned problems by designing a 6D pose estimation method based on an iterative attention fusion network. Summary of the Invention

[0005] To address the above technical problems, this invention provides a 6D pose estimation method based on an iterative attention fusion network.

[0006] The technical solution adopted by this invention to solve its technical problem is:

[0007] A 6D pose estimation method based on an iterative attention fusion network, the method comprising the following steps:

[0008] S100: Construct an iterative attention fusion network, including a semantic segmentation network, a convolutional neural network, a point cloud network, an iterative attention module, a pose prediction module, and a pose iterative optimization module;

[0009] S200: Input RGB and depth images from the dataset. The semantic segmentation network performs semantic segmentation on the RGB images to obtain the 2D bounding box and segmentation mask for each object in the RGB images. The segmentation mask and 2D bounding box of the target object are used to crop out the corresponding region of the target object in the RGB and depth images.

[0010] S300: Converts the depth information of the cropped depth image into point cloud data using the camera's intrinsic parameter matrix;

[0011] S400: Features are extracted from the segmentation mask and the cropped RGB image using a convolutional neural network to obtain semantic and appearance features; geometric features are extracted from the point cloud data using a point cloud network.

[0012] S500: Dense feature fusion of semantic features, appearance features and geometric features to obtain fused features;

[0013] S600: The fused features are processed by an iterative attention module, and attention is allocated in both the channel and spatial dimensions of the features to obtain the processed fused feature representation.

[0014] S700: The fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, and the predicted 6D pose is input into the pose iterative optimization module to obtain the final 6D pose.

[0015] Preferably, in S200, the semantic segmentation network performs semantic segmentation on the RGB image to obtain the 2D bounding box of each object in the RGB image and the segmentation mask of each object, specifically as follows:

[0016] Input an RGB image with dimensions H×W×3, where H and W are the height and width of the RGB image, respectively, and 3 is the number of channels. Rotate, crop, and transform the RGB image.

[0017] The target object is segmented and its location is detected by a semantic segmentation network. The output is a 2D bounding box of the target object and a semantic segmentation map with N+1 channels. Each channel of the semantic segmentation map is a segmentation mask image, and N is the number of object instances in the image.

[0018] Preferably, in S200, the target object's segmentation mask and 2D bounding box are used to crop the corresponding regions of the target object in the RGB image and depth image, specifically:

[0019] The code is written using OpenCV function library. It takes a segmentation mask and a 2D bounding box as input to obtain the pixel coordinates of the 2D bounding box of the object. After converting the three channels of the RGB image, the region where the target object is located in the RGB image is cropped using the pixel coordinate values ​​of the 2D bounding box to obtain an RGB image patch.

[0020] The pixel values ​​of the segmentation mask image are only 0 or 255. The image area with a pixel value of 255 is the area where the target object is segmented. The positions with a pixel value of 255 in the segmentation mask image are assigned True, and the rest are False. The positions with a pixel value of 0 in the depth image are assigned False, and the rest are True. This marks the pixel positions in the depth image where the depth value is not equal to 0.

[0021] The segmentation mask image and the depth image are multiplied together. When the pixel position of both the segmentation mask image and the depth image is True, the pixel value of the new image at that position is True. Otherwise, the value of the pixel position at that position is False. Finally, a preprocessed image is obtained. The position of the pixel value True in the preprocessed image represents the region where the target object is located and has a depth value.

[0022] The preprocessed image is cropped using the pixel coordinates of the object's 2D bounding box to obtain the pixel index of the region where the object is located and has a depth value; the obtained pixel index is used to index the cropped depth image to obtain a depth image patch that belongs only to the target object and has a depth value.

[0023] Preferably, the conversion relationship of S300 is as follows:

[0024]

[0025] Where f x f y c is the focal length of the camera. x c y The origin of the camera is u and v, which are the coordinates of the point in the two-dimensional pixel coordinate system, and X, Y, and Z are the coordinates of the point in the three-dimensional camera coordinate system.

[0026] In a depth image, the depth value Z of each pixel is known. Based on the transformation relationship, the formula can be further transformed into the following formula to calculate the X and Y coordinates:

[0027]

[0028] Preferably, S400 specifically comprises:

[0029] The PSPNet network is used to extract appearance features from RGB image patches. The input size of this network is H. * ×W * A 3x3 RGB image is output as H. * ×W * The feature map of ×D, where H * and W * Here, represents the height and width of the RGB image patch, respectively, and D is the dimension of each pixel in the feature space.

[0030] The PSPNet network is used to extract semantic features from the segmentation mask of the object;

[0031] The PointNet network is used to extract geometric features from 3D point cloud data; assuming the input point cloud of this network is P = (p i If |i=1,2,...,N), then the size of the output geometric feature is N×C, where C represents the dimension of each point.

[0032] Preferably, S500 specifically refers to:

[0033] After extracting appearance features and semantic features from RGB image blocks and segmentation masks respectively using the PSPNet network, the two-dimensional appearance features and semantic features are converted into one-dimensional features.

[0034] Three one-dimensional convolutional layers are used to perform convolution operations on appearance features, semantic features and geometric features respectively to obtain three feature maps. The three feature maps are then fused in the channel dimension to obtain the first fused feature map.

[0035] Then, three one-dimensional convolutional layers are used to perform convolution operations on the three feature maps obtained from the first convolution to obtain three feature maps. The three feature maps are then fused a second time in the channel dimension to obtain the second fused feature map.

[0036] The second fused feature map is processed using two one-dimensional convolutional layers and one average pooling layer to obtain the global feature map;

[0037] The feature maps from the first fusion, the second fusion, and the global feature map are fused along the channel dimension to obtain the fused features.

[0038] Preferably, the first fusion process is as follows:

[0039] F′ RGB =ReLU(Conv1(F RGB ))

[0040] F′ mask =ReLU(Conv1(Fmask ))

[0041] F′ geometry =ReLU(Conv1(F geometry ))

[0042] F feat1 =Concat(F′ RGB F′ mask F′ geometry )

[0043] Where ReLU(.) is the activation function of the network model, Conv1(.) is a 1*1 one-dimensional convolution, and F... RGB F mask F geometry These are the appearance features, semantic features, and geometric features before the first feature fusion, F′ RGB F′ mask F′ geometry These represent the appearance features, semantic features, and geometric features extracted in the first step. `Concat(.)` is the feature concatenation and fusion function, used to fuse the appearance, semantic, and geometric features at the pixel level. feat1 These are characteristics of the first fusion;

[0044] The second fusion process is as follows:

[0045] F″ RGB =ReLU(Conv1(F′) RGB ))

[0046] F″ mask =ReLU(Conv1(F′) mask ))

[0047] F″ geometry =ReLU(Conv1(F′) geometry ))

[0048] F feat2 =Concat(F″ RGB , F″ mask , F″ geometry )

[0049] Where F′ RGB F′ mask F′ geometry These are the appearance features, semantic features, and geometric features before the second feature fusion, F″. RGB F” mask 、F″ geometry These are the appearance features, semantic features, and geometric features after the second feature extraction, F feat2This is a characteristic of the second fusion;

[0050] The features obtained after the second fusion are processed by two one-dimensional convolution operations and then by a global average pooling layer to obtain the global features:

[0051] F global =AvgPool(2*Conv1(F feat2 ))

[0052] Where AvgPool(.) is the global average pooling operation, 2*Conv1(.) represents performing two one-dimensional convolution operations on the features, and F global It is a global feature;

[0053] The feature F after the first fusion feat1 The characteristics F after the second fusion feat2 Global features F global Then, feature splicing and fusion are performed to obtain the final fused feature:

[0054] F feat =Concat(F feat1 F feat2 F global )

[0055] Preferably, S600 includes:

[0056] Max pooling and average pooling operations are performed on the fused feature map respectively. Then, through a fully connected layer and element-wise addition, the channel attention map M is obtained. C Specifically:

[0057] M C (F feat )=σ(FC((AvgPool(F feat )); FC(MaxPool(F feat )))

[0058] Where AvgPool(.) represents the average pooling operation, MaxPool(.) represents the max pooling operation, FC(.) represents the fully connected operation, and σ(.) is the sigmoid function;

[0059] Spatial attention mechanisms determine the location with the most information in a channel by calculating spatial dimensions; spatial attention graph M S The calculation formula is:

[0060] M S (F feat )=σ(Conv1D([AvgPool(F feat MaxPool(F) feat )]))

[0061] Conv1D(.) represents a one-dimensional convolution operation.

[0062] Preferably, the pose prediction module includes three independent multilayer perceptrons, which respectively regress and predict translation, rotation, and confidence. In S700, the fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, specifically:

[0063] The fused features processed by the iterative attention module are input into the pose prediction module to predict the rotation, translation, and confidence of the target object. The pose of the target object corresponding to the maximum confidence is taken as the predicted 6D pose.

[0064] Preferably, in S700, the predicted 6D pose is input into the pose iteration optimization module to obtain the final 6D pose, including:

[0065] Predict 6D pose [R] k |t k The transformation relation is applied to the current point cloud data to obtain transformed point cloud data. This transformed point cloud data is then input into the point cloud network to obtain the transformed geometric features. Specifically, the transformation relation is as follows:

[0066]

[0067] Where X, Y, and Z are the current 3D coordinates of the point cloud, and X′, Y′, and Z′ are the transformed 3D coordinates of the point cloud, R k , t k The obtained predicted pose;

[0068] The original semantic and appearance features are densely fused with the transformed geometric features. This process is repeated K times, and the pose estimates from each iteration are concatenated to obtain the final pose:

[0069]

[0070] in Represents the final pose, R k and t k These are the rotation and translation amounts predicted for the kth time, respectively.

[0071] Compared with the prior art, the advantages of the present invention are:

[0072] (1) The technical solution of the present invention proposes a cross-modal feature fusion method, which adds the object segmentation mask image and RGB image to the depth image for feature extraction and fusion, increases the semantic information of the fused features, and enables the network to accurately detect and identify different objects in cluttered scenes.

[0073] (2) The technical solution of the present invention proposes and adopts an iterative attention mechanism, which aims to enhance the expressive power of fused features, obtain more robust feature representations, and thus improve the performance of the network in situations such as cluttered scenes and object occlusion.

[0074] (3) The technical solution of the present invention incorporates the iterative attention mechanism into the pose iterative optimization module. During the pose optimization stage, the attention module will continuously iteratively process the fused features, thereby continuously enhancing the expressive power of the features and enabling the network to learn more robust feature representations in complex scenarios. This significantly improves the performance of the model while ensuring the real-time performance of inference. Attached Figure Description

[0075] Figure 1 This is a flowchart of a 6D pose estimation method based on an iterative attention fusion network in one embodiment of the present invention;

[0076] Figure 2 This is a schematic diagram of the network model structure in one embodiment of the present invention;

[0077] Figure 3 This is a schematic diagram of pixel-level feature fusion in one embodiment of the present invention;

[0078] Figure 4 This is a schematic diagram of the network structure of the iterative attention mechanism module in one embodiment of the present invention;

[0079] Figure 5 This is a schematic diagram of a pose prediction module in one embodiment of the present invention;

[0080] Figure 6 This is a schematic diagram of a pose iteration optimization module in one embodiment of the present invention;

[0081] Figure 7 This is a visualization of the evaluation results of an embodiment of the present invention on the LineMOD dataset. Detailed Implementation

[0082] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0083] In one embodiment, such as Figure 1 As shown, a 6D pose estimation method based on an iterative attention fusion network is presented, the method comprising the following steps:

[0084] S100: Construct an iterative attention fusion network, including a semantic segmentation network, a convolutional neural network, a point cloud network, an iterative attention module, a pose prediction module, and a pose iterative optimization module.

[0085] Specifically, the network model in this embodiment is as follows: Figure 2 As shown, the network model includes a semantic segmentation network, a convolutional neural network, a point cloud network, an iterative attention module, a pose prediction module, and a pose iterative optimization module. The network model takes RGB images and depth images as input and achieves object pose estimation through the cooperation between different networks and modules.

[0086] S200: Input RGB and depth images from the dataset. The semantic segmentation network performs semantic segmentation on the RGB images to obtain the 2D bounding box and segmentation mask for each object in the RGB images. The segmentation mask and 2D bounding box of the target object are used to crop out the corresponding region of the object in the RGB and depth images.

[0087] Specifically, in this embodiment, the classic open-source dataset LineMOD for 6D pose estimation is selected. The dataset contains RGB images and their corresponding depth images. In this embodiment, the basic targets for pose estimation include bottles, cameras, drills, and various model toys, totaling 13 object categories, all of which are included in the images of the dataset.

[0088] In one embodiment, in S200, the semantic segmentation network performs semantic segmentation on the RGB image to obtain the 2D bounding box and segmentation mask of the target object in the RGB image, specifically:

[0089] Input an RGB image with dimensions H×W×3, where H and W are the height and width of the RGB image, respectively, and 3 is the number of channels. Rotate, crop, and transform the RGB image.

[0090] The target object is segmented and its location is detected by a semantic segmentation network. The output is a 2D bounding box of the target object and a semantic segmentation map with N+1 channels. Each channel of the semantic segmentation map is a segmentation mask image, and N is the number of object instances in the image.

[0091] Specifically, an FCN network model is used for semantic segmentation of images. Other image generation network models can also be used as needed. The FCN network model is trained using RGB images from the dataset, allowing it to perform semantic segmentation of objects in the images and generate segmentation masks. Before inputting RGB images into the semantic segmentation network, certain preprocessing should be performed, including image rotation, cropping, and deformation, to enhance the network's generalization ability.

[0092] In one embodiment, in S200, the corresponding regions of the target object in the RGB image and depth image are cropped using a segmentation mask and a 2D bounding box. Specifically:

[0093] The code is written using OpenCV function library. It takes a segmentation mask and a 2D bounding box as input to obtain the pixel coordinates of the 2D bounding box of the object. After converting the three channels of the RGB image, the region where the target object is located in the RGB image is cropped using the pixel coordinate values ​​of the 2D bounding box to obtain an RGB image patch.

[0094] The pixel values ​​of the segmentation mask image are only 0 or 255. The image area with a pixel value of 255 is the area where the target object is segmented. The positions with a pixel value of 255 in the segmentation mask image are assigned True, and the rest are False. The positions with a pixel value of 0 in the depth image are assigned False, and the rest are True. This marks the pixel positions in the depth image where the depth value is not equal to 0.

[0095] The segmentation mask image and the depth image are multiplied together. When the pixel position of both the segmentation mask image and the depth image is True, the pixel value of the new image at that position is True. Otherwise, the value of the pixel position at that position is False. Finally, a preprocessed image is obtained. The position of the pixel value True in the preprocessed image represents the region where the target object is located and has a depth value.

[0096] The preprocessed image is cropped using the pixel coordinates of the object's 2D bounding box to obtain the pixel index of the region where the object is located and has a depth value; the obtained pixel index is used to index the cropped depth image to obtain a depth image patch that belongs only to the target object and has a depth value.

[0097] S300: Converts the depth information of the cropped depth image into point cloud data using the camera's intrinsic parameter matrix.

[0098] Specifically, many cameras come with their intrinsic parameter information at the factory. In a depth image, based on the two-dimensional pixel coordinates of a pixel and its depth value, we can use the two-dimensional to three-dimensional transformation relationship to obtain the three-dimensional spatial coordinates of the two-dimensional pixel coordinates in the camera coordinate system, and then convert it into point cloud data.

[0099] In one embodiment, the conversion relationship of S300 is as follows:

[0100]

[0101] Where f x f y c is the focal length of the camera. x c y The origin of the camera is u and v, which are the coordinates of the point in the two-dimensional pixel coordinate system, and X, Y, and Z are the three-dimensional coordinates of the point in the camera coordinate system.

[0102] In a depth image, the depth value Z of each pixel is known. Based on the transformation relationship, the formula can be further transformed into the following formula to calculate the X and Y coordinates:

[0103]

[0104] S400: Features are extracted from the segmentation mask and cropped RGB images using a convolutional neural network to obtain semantic and appearance features; geometric features are extracted from the point cloud data using a point cloud network.

[0105] In one embodiment, S400 specifically refers to:

[0106] The PSPNet network is used to extract appearance features from RGB image patches. The input size of this network is H. * ×W * A 3x3 RGB image is output as H. * ×W * The feature map of ×D, where H * and W * Here, represents the height and width of the RGB image patch, respectively, and D is the dimension of each pixel in the feature space.

[0107] The PSPNet network is used to extract semantic features from the segmentation mask of the object;

[0108] The PointNet network is used to extract geometric features from 3D point cloud data; assuming the input point cloud of this network is P = (p i If |i=1,2,...,N), then the size of the output geometric feature is N×C, where C represents the dimension of each point.

[0109] Specifically, the feature extraction network for RGB image patches and segmentation masks is the PSPNet convolutional network model. The PSPNet convolutional network model is a pyramid scene analysis network that can aggregate the context of different regions, enabling the model to understand global contextual information. It is an encoder-decoder network structure. Other feature extraction network models can also be used as needed. ResNet-18 is used as the encoder network of the PSPNet network model, and four upsampling layers are selected as the decoder network layers. The PSPNet network is used as the feature extractor for RGB image patches and segmentation masks, obtaining appearance features and semantic features respectively. PointNet is used as the feature extractor for 3D point cloud data. Other point cloud feature extraction network models can also be used as needed. PointNet is a classic network for processing 3D point cloud data.

[0110] S500: Dense feature fusion of semantic features, appearance features and geometric features to obtain fused features.

[0111] In one embodiment, S500 specifically refers to:

[0112] After extracting appearance features and semantic features from RGB image blocks and segmentation masks respectively using the PSPNet network, the two-dimensional appearance features and semantic features are converted into one-dimensional features.

[0113] Three one-dimensional convolutional layers are used to perform convolution operations on appearance features, semantic features and geometric features respectively to obtain three feature maps. The three feature maps are then fused in the channel dimension to obtain the first fused feature map.

[0114] Then, three one-dimensional convolutional layers are used to perform convolution operations on the three feature maps obtained from the first convolution to obtain three feature maps. The three feature maps are then fused a second time in the channel dimension to obtain the second fused feature map.

[0115] The second fused feature map is processed using two one-dimensional convolutional layers and one average pooling layer to obtain the global feature map;

[0116] The feature maps from the first fusion, the second fusion, and the global feature map are fused along the channel dimension to obtain the fused features.

[0117] In one embodiment, the first fusion process is as follows:

[0118] F′ RGB =ReLU(Conv1(F RGB ))

[0119] F′ mask =ReLU(Conv1(F mask ))

[0120] F′ geometry =ReLU(Conv1(F geometry ))

[0121] F feat1 =Concat(F′ RGB F′ mask F′ geometry )

[0122] Where ReLU(.) is the activation function of the network model, Conv1(.) is a 1*1 one-dimensional convolution, and F... RGB F mask F geometry These are the appearance features, semantic features, and geometric features before the first feature fusion, F′ RGB F′ mask F′ geometryThese represent the appearance features, semantic features, and geometric features extracted in the first step. `Concat(.)` is the feature concatenation and fusion function, used to fuse the appearance, semantic, and geometric features at the pixel level. feat1 These are characteristics of the first fusion;

[0123] The second fusion process is as follows:

[0124] F″ RGB =ReLU(Conv1(F′) RGB ))

[0125] F″ mask =ReLU(Conv1(F′) mask ))

[0126] F′ geometry =ReLU(Conv1(F′) geometry ))

[0127] F feat2 =Concat(F″ RGB , F″ mask , F″ geometry )

[0128] Where F′ RGB F′ mask F′ geometry These are the appearance features, semantic features, and geometric features before the second feature fusion, F″. RGB F” mask 、F″ geometry These are the appearance features, semantic features, and geometric features after the second feature extraction, F feat2 This is a characteristic of the second fusion;

[0129] The features obtained after the second fusion are processed by two one-dimensional convolution operations and then by a global average pooling layer to obtain the global features:

[0130] F global =AvgPool(2*Conv1(F feat2 ))

[0131] Where AvgPool(.) is the global average pooling operation, 2*Conv1(.) represents performing two one-dimensional convolution operations on the features, and F global It is a global feature;

[0132] The feature F after the first fusion feat1 The characteristics F after the second fusion feat2 Global features F global Then, feature splicing and fusion are performed to obtain the final fused feature:

[0133] F feat =Concat(F feat1 F feat2 F global ).

[0134] Specifically, a schematic diagram of dense feature fusion is shown below. Figure 3 As shown, feature fusion was performed three times. First, 1*1 convolution and ReLU activation functions were applied to appearance features, semantic features, and geometric features respectively, outputting 64-dimensional features. Then, the three extracted features were fused in the depth dimension to form the first fused feature with a dimension of 192 (64*3). Next, 1*1 convolution and ReLU activation functions were applied to RGB image patches, segmentation masks, and point clouds, outputting three 128-dimensional features. Then, the three extracted features were fused in the depth dimension to form the second fused feature with a dimension of 384 (128*3). Then, 1*1 convolution, ReLU activation function, 1*1 convolution, ReLU activation function were applied to the second fused feature in sequence, outputting 1024-dimensional features. Then, global average pooling was used to process the 1024-dimensional features to obtain global features. Finally, the first fused feature, the second fused feature, and the global feature were fused in the channel dimension to output the final dense fused feature with a dimension of 1600 (192+384+1024).

[0135] S600: The fused features are processed by an iterative attention module, and attention is allocated in both the channel and spatial dimensions of the features to obtain the processed fused feature representation.

[0136] Specifically, due to situations where objects severely occlude or segmentation errors occur, the resulting RGB image patches and point clouds may contain pixels belonging to other objects or the background, which could reduce the generalization ability and robustness of the fused features. To address these issues, an iterative attention mechanism is proposed to optimize the fused features, such as... Figure 4 As shown, this attention mechanism includes channel attention and spatial attention, which enhance the expressive power of features from both channel and spatial dimensions.

[0137] In one embodiment, S600 includes:

[0138] Max pooling and average pooling operations were performed on the fused feature map, and the channel attention map M was obtained through a fully connected layer and element-wise addition. C Specifically:

[0139] M C (F feat )=σ(FC((AvgPool(F feat )); FC(MaxPool(Ffeat )))

[0140] Where AvgPool(.) represents the average pooling operation, MaxPool(.) represents the max pooling operation, FC(.) represents the fully connected operation, and σ(.) is the sigmoid function;

[0141] Spatial attention mechanisms determine the location with the most information in a channel by calculating spatial dimensions; spatial attention graph M S The calculation formula is:

[0142] M S (F feat )=σ(Conv1D([AvgPool(F feat MaxPool(F) feat )]))

[0143] Conv1D(.) represents a one-dimensional convolution operation.

[0144] Specifically, the channel attention mechanism determines the importance of each channel by calculating its weight in the input image. By paying more attention to channels containing important information, it improves feature representation capabilities.

[0145] S700: The fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, and the predicted 6D pose is input into the pose iterative optimization module to obtain the final 6D pose.

[0146] In one embodiment, the pose prediction module includes three independent multilayer perceptrons, which respectively regress and predict translation, rotation, and confidence. In S700, the fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, specifically:

[0147] The fused features processed by the iterative attention module are input into the pose prediction module to predict the rotation, translation, and confidence of the target object. The pose of the target object corresponding to the maximum confidence is taken as the predicted 6D pose.

[0148] In one embodiment, S700 inputs the predicted 6D pose into the pose iteration optimization module to obtain the final 6D pose, including:

[0149] Predict 6D pose [R] k |t k The transformation relationship is applied to the current point cloud data to obtain the transformed point cloud data. The transformed point cloud data is then input into the point cloud network to obtain the transformed geometric features.

[0150] The original semantic and appearance features are densely fused with the transformed geometric features. This process is repeated K times, and the pose estimates from each iteration are concatenated to obtain the final pose:

[0151]

[0152] in Represents the final pose, R k and t k These are the rotation and translation amounts predicted for the kth time, respectively.

[0153] Specifically, the pose estimation method mainly consists of an initial stage and a pose optimization stage. In the initial stage, images from the training set are input to train the pose estimation network model. The purpose of this invention is to stop training the network model once it reaches a certain pose estimation accuracy. An iterative attention module is involved in both the initial and pose optimization stages. In the pose optimization stage, the attention module iteratively processes the fused features, enhancing their expressive power and thus improving pose prediction accuracy. The fused features processed by the iterative attention mechanism are input into the pose prediction module, which outputs the object's pose information with corresponding confidence levels. The higher the confidence level, the more likely the predicted pose is the object's pose. The pose prediction module consists of three independent multilayer perceptrons, which respectively regress and predict translation, rotation, and confidence levels. Each multilayer perceptron includes four fully connected layers. Figure 5 As shown, the pose prediction module is essentially composed of fully connected layers. The pose iterative optimization module is a neural network-based iterative optimization module, such as... Figure 6 As shown, it mainly consists of dense fusion processing, an iterative attention module, fully connected layers, and a PointNet network. The key to this method is training the network to improve previous predicted poses. The previously estimated 6D pose result is used as input for the next iteration to obtain a new predicted pose. This process can be applied iteratively, generating a more refined pose estimate in each iteration.

[0154] Furthermore, the workflow of the pose iterative optimization module is as follows: first, the extracted semantic features, appearance features, and geometric features are densely fused; then, the fused features are input into an iterative attention mechanism for processing to enhance their expressive power. The processed fused features are fed into a fully connected layer to output the pose [R] related to the previous pose prediction. k |t kThe pose is then applied to the current point cloud data through a transformation relation to obtain a new transformed point cloud, which is fed back into the next iteration. The original semantic and appearance features are then densely fused with the geometric features obtained from the transformed point cloud, and the previous steps are repeated. After k iterations, the pose estimates from each iteration are concatenated to obtain the final pose:

[0155]

[0156] in Represents the final pose, R k and t k These are the rotation and translation amounts predicted for the kth time, respectively.

[0157] The LineMod dataset used in this embodiment is a widely used 6D pose benchmark dataset. This dataset contains over 18,000 real-world images with labeled ground-truth 6D poses. It includes 15 low-texture objects; this invention uses only 13 objects, with the training and testing ratio remaining the same as previous work. These objects in the images are placed against cluttered backgrounds, and each object has varying degrees of occlusion. Furthermore, each RGB image in this dataset has its corresponding depth image.

[0158] During the pose iteration optimization process, the pose threshold for starting the optimization process was set to 0.013. The network was trained using the Adam optimizer with an initial learning rate of 0.0001 and a batch size of 2. After the loss of the pose predictor network dropped below 0.016, a decay rate of 0.5 was used to further train the pose iteration optimization module.

[0159] The following loss function is used as the optimization objective of the network.

[0160]

[0161] Where x j Let R represent the j-th point among M randomly selected 3D points from the 3D model of the object, where R and t are the true pose information. and The predicted pose is generated from the fused features of the i-th dense pixel.

[0162] For symmetrical objects, they do not have only one accurate pose. Therefore, this invention uses minimizing the distance between each point in the prediction model and the nearest point in the real model as the training objective. Thus, the loss function becomes:

[0163]

[0164] For comparison, all models in this embodiment use the PyTorch framework, are accelerated with CUDA 11.1, and are trained, validated, and tested on a GTX 3080 GPU. The following experiments were conducted to verify the network model of the method in this embodiment.

[0165] (1) Comparison experiment between the improved network model and other models:

[0166] To verify the effectiveness of the improved model compared to other algorithms, comparative experiments were conducted with other algorithms in the same environment and on the same foreign object detection dataset. Six state-of-the-art (SOTA) network models were tested based on the foreign object detection dataset, including PoseCNN, SSD-6D, PVNet, DenseFusion, and MaskedFusion.

[0167] The training and testing datasets contained 13 object types: ape model, fitter's bench, camera, kettle, cat model, electric drill, duck model, egg carton, glue, hole punch, iron, desk lamp, and telephone. The egg carton and glue were symmetrical objects. The network model was trained using these 13 objects and then tested on the test set. The test results are shown in Table 1.

[0168] Table 1. Comparison of the network model of this invention with other models on the LineMod dataset.

[0169]

[0170] As shown in Table 1, when sampling with different numbers of instances, the network model (Ours) of this invention significantly outperforms other methods. When trained with 15 objects and tested on the test set using their weights, its average pose estimation accuracy reaches 98.2%, effectively verifying the superiority of this invention's method for pose estimation. For the prediction accuracy of a single object, this invention achieves 100% pose prediction accuracy for glue. The method of this invention also leads in pose prediction accuracy for many other weakly textured objects, such as the ape model (96.9%), kettle (98.6%), drill (96.8%), duck model (97.2%), and iron (99.5%), showing a significant improvement in pose prediction accuracy compared to the basic network.

[0171] exist Figure 7 The image shows some visualized RGB images after performing pose estimation using the method of this invention in conjunction with other methods. All methods were tested under the same conditions. After obtaining the predicted pose of an object through a pose estimation network, the point cloud of the object can be transformed into the camera coordinate system using the predicted pose, and then the 3D point cloud can be projected onto a 2D RGB image using the camera's intrinsic parameter matrix. Figure 7 As shown in the figures, these visualizations demonstrate that the method of the present invention has higher accuracy and better robustness.

[0172] (2) Ablation experiments of the improved network model:

[0173] To demonstrate the effectiveness of the proposed iterative attention module and the semantic features extracted from the segmentation mask, ablation experiments were conducted on the LineMOD dataset. The trained weights were applied to the dataset multiple times, and the final test results are shown in the table below.

[0174] Table 2 Ablation Experiment Results of the Network Model

[0175]

[0176] As shown in Table 2, compared to the basic method, fusing semantic features and using an iterative attention module both improve the accuracy of pose prediction. By combining these two approaches, the accuracy can reach a maximum of 98.2%.

[0177] Referring to Table 2: Compared to the original DenseFusion network model, the method in this embodiment, which integrates semantic features, improves pose prediction accuracy by 2.5%, demonstrating the feasibility of semantic features for pose estimation. In cluttered scenes, it effectively addresses the difficulty in accurate segmentation caused by severe object occlusion, thereby improving pose estimation accuracy. Using an iterative attention module, compared to the original DenseFusion network model, improves pose prediction accuracy by 2.8%, demonstrating the feasibility of enhancing feature representation for pose estimation, resulting in more robust feature representations, and thus improving network performance in situations with object occlusion. By combining these two methods, the accuracy reaches a maximum of 98.2%, further demonstrating the effectiveness of both approaches.

[0178] This embodiment employs a network that estimates the 6D pose of an object from RGB-D input. Furthermore, it inputs the object's mask along with the RGB-D input to extract and fully utilize semantic features. Instead of simply concatenating features, the network densely fuses appearance, geometric, and semantic features. To enhance the expressive power of the fused features, an iterative attention module is used to strengthen them in both the channel and spatial domains. During the pose iteration optimization phase, this module continuously processes the fused features to generate a more robust feature representation. Evaluation results on the LineMod dataset demonstrate that this method exhibits better accuracy and robustness compared to existing methods.

[0179] The method provided by this invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A 6D pose estimation method based on an iterative attention fusion network, characterized in that, The method includes the following steps: S100: Construct an iterative attention fusion network, including a semantic segmentation network, a convolutional neural network, a point cloud network, an iterative attention module, a pose prediction module, and a pose iterative optimization module; S200: Input RGB and depth images from the dataset. The semantic segmentation network performs semantic segmentation on the RGB images to obtain the 2D bounding box and segmentation mask for each object in the RGB images. The segmentation mask and 2D bounding box of the target object are used to crop out the corresponding region of the target object in the RGB and depth images. S300: Converts the depth information of the cropped depth image into point cloud data using the camera's intrinsic parameter matrix; S400: Features are extracted from the segmentation mask and the cropped RGB image using a convolutional neural network to obtain semantic and appearance features; geometric features are extracted from the point cloud data using a point cloud network. S500: Dense feature fusion of semantic features, appearance features and geometric features to obtain fused features; S600: The fused features are processed by an iterative attention module, and attention is allocated in both the channel and spatial dimensions of the features to obtain the processed fused feature representation. S700: The fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, and the predicted 6D pose is input into the pose iterative optimization module to obtain the final 6D pose.

2. The method according to claim 1, characterized in that, In S200, the semantic segmentation network performs semantic segmentation on RGB images, obtaining the 2D bounding box of each object in the RGB image and the segmentation mask of each object, specifically: Input an RGB image with dimensions H×W×3, where H and W are the height and width of the RGB image, respectively, and 3 is the number of channels. Rotate, crop, and transform the RGB image. The target object is segmented and its location is detected by a semantic segmentation network. The output is a 2D bounding box of the target object and a semantic segmentation map with N+1 channels. Each channel of the semantic segmentation map is a segmentation mask image, and N is the number of object instances in the image.

3. The method according to claim 2, characterized in that, In S200, the target object's segmentation mask and 2D bounding box are used to crop the corresponding region of the target object in the RGB image and depth image, specifically: The code is written using OpenCV function library. It takes a segmentation mask and a 2D bounding box as input to obtain the pixel coordinates of the 2D bounding box of the object. After converting the three channels of the RGB image, the region where the target object is located in the RGB image is cropped using the pixel coordinate values ​​of the 2D bounding box to obtain an RGB image patch. The pixel values ​​of the segmentation mask image are only 0 or 255. The image area with a pixel value of 255 is the area where the target object is segmented. The positions with a pixel value of 255 in the segmentation mask image are assigned True, and the rest are False. The positions with a pixel value of 0 in the depth image are assigned False, and the rest are True. This marks the pixel positions in the depth image where the depth value is not equal to 0. The segmentation mask image and the depth image are multiplied together. When the pixel position of both the segmentation mask image and the depth image is True, the pixel value of the new image at that position is True. Otherwise, the value of the pixel position at that position is False. Finally, a preprocessed image is obtained. The position of the pixel value True in the preprocessed image represents the region where the target object is located and has a depth value. The preprocessed image is cropped using the pixel coordinates of the object's 2D bounding box to obtain the pixel index of the region where the object is located and has a depth value. The obtained pixel indexes are used to index the cropped depth image to obtain a depth image patch that belongs only to the target object and has a depth value.

4. The method according to claim 3, characterized in that, The conversion relationship of S300 is as follows: Where f x f y c is the focal length of the camera. x c y The origin of the camera is u and v, which are the coordinates of the point in the two-dimensional pixel coordinate system, and X, Y, and Z are the three-dimensional coordinates of the point in the camera coordinate system. In a depth image, the depth value Z of each pixel is known. Based on the transformation relationship, the formula can be further transformed into the following formula to calculate the X and Y coordinates:

5. The method according to claim 4, characterized in that, The S400 specifically refers to: The PSPNet network is used to extract appearance features from RGB image patches. The input size of this network is H. * ×W * A 3x3 RGB image is output as H. * ×W * The feature map of ×D, where H * and W * Here, represents the height and width of the RGB image patch, respectively, and D is the dimension of each pixel in the feature space. The PSPNet network is used to extract semantic features from the segmentation mask of the object; The PointNet network is used to extract geometric features from 3D point cloud data; assuming the input point cloud of this network is P = (p i If |i=1,2,...,N), then the size of the output geometric feature is N×C, where C represents the dimension of each point.

6. The method according to claim 5, characterized in that, The S500 specifically refers to: After extracting appearance features and semantic features from RGB image blocks and segmentation masks respectively using the PSPNet network, the two-dimensional appearance features and semantic features are converted into one-dimensional features. Three one-dimensional convolutional layers are used to perform convolution operations on appearance features, semantic features and geometric features respectively to obtain three feature maps. The three feature maps are then fused in the channel dimension to obtain the first fused feature map. Then, three one-dimensional convolutional layers are used to perform convolution operations on the three feature maps obtained from the first convolution to obtain three feature maps. The three feature maps are then fused a second time in the channel dimension to obtain the second fused feature map. The second fused feature map is processed using two one-dimensional convolutional layers and one average pooling layer to obtain the global feature map; The feature maps from the first fusion, the second fusion, and the global feature map are fused along the channel dimension to obtain the fused features.

7. The method according to claim 6, characterized in that, The first fusion process is as follows: F′ RGB =ReLU(Conv1(F RGB )) F′ mask =ReLU(Conv1(F mask )) F′ geometry =ReLU(Conv1(F′ geometry )) F feat1 =Concat(F′ RGB ,F′ mask ,F′ geometry ) Where ReLU(.) is the activation function of the network model, Conv1(.) is a 1*1 one-dimensional convolution, and F... RGB F mask F geometry These are the appearance features, semantic features, and geometric features before the first feature fusion, F′ RGB F′ mask F′ geometry These represent the appearance features, semantic features, and geometric features extracted in the first step. `Concat(.)` is the feature concatenation and fusion function, used to fuse the appearance, semantic, and geometric features at the pixel level. feat1 These are characteristics of the first fusion; The second fusion process is as follows: F″ RGB =ReLU(Conv1(F′ RGB )) F″ mask =ReLU(Conv1(F′ mask )) F″ geometry =ReLU(Conv1(F′ geometry )) F feat2 =Concat(F″ RGB ,F″ mask ,F″ geometry ) Where F′ RGB F′ mask F′ geometry These are the appearance features, semantic features, and geometric features before the second feature fusion, F″. RGB 、F″ mask 、F″ geometry These are the appearance features, semantic features, and geometric features after the second feature extraction, F feat2 This is a characteristic of the second fusion; The features obtained after the second fusion are processed by two one-dimensional convolution operations and then by a global average pooling layer to obtain the global features: F global =AvgPool(2*Conv1(F feat2 )) Where AvgPool(.) is the global average pooling operation, 2*Conv1(.) represents performing two one-dimensional convolution operations on the features, and F global It is a global feature; The feature F after the first fusion feat1 The characteristics F after the second fusion feat2 Global features F global Then, feature splicing and fusion are performed to obtain the final fused feature: F feat =Concat(F feat1 ,F feat2 ,F global )。 8. The method according to claim 7, characterized in that, The S600 includes: Max pooling and average pooling operations are performed on the fused feature map respectively. Then, through a fully connected layer and element-wise addition, the channel attention map M is obtained. C Specifically: M C (F feat )<σ(FC((AvgPool(F feat ));FC(MaxPool(F feat ))) Where AvgPool(.) represents the average pooling operation, MaxPool(.) represents the maximum pooling operation, FC(.) represents the fully connected operation, and σ(.) is the sigmoid function; Spatial attention mechanisms determine the location with the most information in a channel by calculating spatial dimensions; spatial attention graph M s The calculation formula is: M S (F feat )=σ(Conv1D([AvgPool(F feat );MaxPool(F feat )])) Conv1D(.) represents a one-dimensional convolution operation.

9. The method according to claim 8, characterized in that, The pose prediction module includes three independent multilayer perceptrons, which respectively regress and predict translation, rotation, and confidence. In the S700, the fused features processed by the iterative attention module are input into the pose prediction module to obtain the predicted 6D pose, specifically: The fused features processed by the iterative attention module are input into the pose prediction module to predict the rotation, translation, and confidence of the target object. The pose of the target object corresponding to the maximum confidence is taken as the predicted 6D pose.

10. The method according to claim 9, characterized in that, In the S700, the predicted 6D pose is input into the pose iteration optimization module to obtain the final 6D pose, including: Predict 6D pose [R] k |t k The transformation relation is applied to the current point cloud data to obtain transformed point cloud data. This transformed point cloud data is then input into the point cloud network to obtain the transformed geometric features. Specifically, the transformation relation is as follows: Where X, Y, and Z are the current 3D coordinates of the point cloud, and X′, Y′, and Z′ are the transformed 3D coordinates of the point cloud, R k , t k The obtained predicted pose; The original semantic and appearance features are densely fused with the transformed geometric features. This process is repeated K times, and the pose estimates from each iteration are concatenated to obtain the final pose: in Represents the final pose, R k and t k These are the rotation and translation amounts predicted for the kth time, respectively.

Images