6d pose estimation method and apparatus with attention mechanism, device, and medium

Abstract

The application discloses a 6D pose estimation method and device fusing an attention mechanism, equipment and a medium, and relates to the technical field of object space pose. The method comprises the following steps: processing an RGB image and a depth image of a target object through a preset multi-modal feature fusion model, wherein the model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. A target mask point cloud is obtained through semantic segmentation, feature fusion is performed by using a cross-attention mechanism and a deformable convolution, and finally, a final pose is obtained through pose estimation and iterative optimization. The method solves the problems of insufficient feature extraction and weak multi-modal feature correlation in complex scenes such as weak texture and occlusion, and improves the accuracy and robustness of pose estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of object space pose, and in particular to a 6D pose estimation method and device fusing attention mechanism, equipment and medium. BACKGROUND

[0002] Six degrees of freedom (6D) pose estimation is one of the core technologies in the field of computer vision and robotics, and is widely used in logistics and warehousing, manufacturing, autonomous driving, virtual reality and augmented reality, and medical fields. Current 6D pose estimation methods are mainly divided into three categories: instance-level, class-level and unseen object pose estimation. Instance-level methods include traditional geometric methods such as Linemod based on template matching and depth normal vector features, and also include deep learning methods such as GDR-Net that fuse deep learning features. Class-level methods include NOCS that unifies object representation through normalized coordinate space. Unseen object pose estimation methods include MAV6De that proposes self-training unsupervised domain adaptation and enhances the robustness of outdoor scenes through pseudo-labels.

[0003] However, in complex scenes such as weak texture and occlusion, the feature extraction capability of existing methods is insufficient, resulting in insufficient target feature information and affecting the accuracy of pose estimation. Traditional geometric methods are prone to failure when dealing with weak texture objects, while deep learning methods, although improving the feature extraction capability to some extent, still struggle to cope with complex lighting changes and occlusions, and in multi-modal feature fusion, simple concatenation or weighted summation is usually used, ignoring the semantic association between different modal features. This simple fusion method cannot fully utilize the complementary information in RGB images and depth images, resulting in insufficient accuracy of feature fusion. Therefore, there is an urgent need for a 6D pose estimation method fusing attention mechanism to improve the accuracy and robustness of pose estimation. SUMMARY

[0004] The main purpose of the present application is to provide a 6D pose estimation method fusing attention mechanism, device, equipment and medium, aiming to solve the technical problem of how to improve the accuracy and robustness of pose estimation.

[0005] To achieve the above purpose, the present application provides a 6D pose estimation method fusing attention mechanism, comprising:

[0006] obtaining a depth RGB image of a target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object;

[0007] inputting the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module;

[0008] The step of inputting the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result further comprises:

[0009] performing semantic segmentation on the RGB image by the semantic segmentation module to obtain a segmentation mask of a target object;

[0010] cropping the segmentation mask from point cloud converted from the depth image by the semantic segmentation module to obtain a target mask point cloud;

[0011] performing color feature extraction and geometric feature extraction on the target mask point cloud by the feature extraction module to obtain color features and geometric features;

[0012] interacting the color features and the geometric features by the feature fusion module using a cross-attention mechanism to generate cross-modal cross features;

[0013] fusing the cross-modal cross features, the color features, and the geometric features by the feature fusion module using a cross-modal deformable attention mechanism to obtain fused features;

[0014] performing pose calculation on the fused features by the pose estimation module to obtain an initial 6D pose;

[0015] performing iterative optimization on the initial 6D pose by a pose iterative optimization module to obtain a target 6D pose result.

[0016] In an embodiment, the step of performing semantic segmentation on the RGB image by the semantic segmentation module to obtain a segmentation mask of a target object comprises:

[0017] obtaining an original resolution according to the RGB image;

[0018] performing feature extraction on the RGB image by an encoder of the semantic segmentation module to generate a first feature map and a second feature map, wherein the scale of the first feature map is 1 / 16 of the original resolution, and the scale of the second feature map is 1 / 8 of the original resolution;

[0019] up-sampling the scale of the first feature map to the scale of the second feature map by a decoder of the semantic segmentation module to obtain a third feature map;

[0020] perform channel-by-channel weighted fusion on the third feature map and the second feature map to obtain a fused feature map;

[0021] restore the fused feature map to the original resolution through a deconvolution operation to obtain a fourth feature map;

[0022] output a pixel-level classification probability through a Softmax layer for the fourth feature map to obtain a segmentation mask of the target object.

[0023] In an embodiment, the step of cropping the segmentation mask from the point cloud converted from the depth image by the semantic segmentation module to obtain a target mask point cloud comprises:

[0024] obtain camera intrinsic parameters;

[0025] convert the depth image into an unordered point cloud through a preset point cloud conversion algorithm, wherein the unordered point cloud comprises a background point cloud and a target object corresponding point cloud;

[0026] convert the unordered point cloud into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates;

[0027] determine a pixel point coordinate range corresponding to the target object in the unordered point cloud according to a pixel region range of the segmentation mask through the semantic segmentation module;

[0028] screen point cloud data in which the pixel coordinates are within the pixel point coordinate range to obtain screened point cloud data;

[0029] perform denoising processing on the screened point cloud data to obtain a target mask point cloud.

[0030] In an embodiment, the step of performing color feature extraction and geometric feature extraction on the target mask point cloud through the feature extraction module to obtain color features and geometric features comprises a color feature extraction submodule and a geometric feature extraction submodule in the feature extraction module, wherein the color feature extraction submodule comprises a ResNet18 network and a compression and excitation attention layer;

[0031] perform preliminary feature extraction on a cropped RGB image corresponding to the target mask point cloud through the ResNet18 network to obtain initial image features;

[0032] compress the channel dimension of the initial image features through the compression and excitation attention layer to obtain global information of each channel;

[0033] perform dimension reduction processing on the global information to obtain reduced information;

[0034] Dimension recovery is performed on the reduced dimension information to obtain dimension-restored information;

[0035] The dimension-restored information is activated by a Sigmoid function to generate attention weights corresponding to each channel;

[0036] The attention weights are multiplied by the initial image features channel by channel to obtain color features;

[0037] The three-dimensional coordinates of each point in the target mask point cloud are nonlinearly transformed by the geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension;

[0038] The point-by-point geometric features are globally aggregated to obtain geometric features, wherein the geometric features represent the overall geometric properties of the target mask point cloud.

[0039] In an embodiment, the step of generating a cross-modal cross-feature by the feature fusion module using a cross-attention mechanism to interact the color features and the geometric features includes:

[0040] The color features and the geometric features are loaded by the feature fusion module to determine the mapping rule of the cross-attention mechanism;

[0041] According to the mapping rule, the color features are mapped to a Query vector through a preset linear transformation, wherein the Query vector is used to encode the category semantic information of the target object;

[0042] According to the mapping rule, the point cloud coordinate information in the geometric features is respectively mapped to a Key vector and a Value vector through a preset linear transformation, wherein the Key vector is used to encode the geometric position correlation information, and the Value vector is used to encode the geometric position feature information;

[0043] The Query vector and the Key vector are calculated to obtain a similarity;

[0044] The similarity is normalized by a Softmax function to generate an attention weight map;

[0045] The attention weight map and the Value vector are weighted and summed to obtain a weighted result;

[0046] The weighted result is linearly transformed to generate a cross-modal cross-feature.

[0047] In an embodiment, the step of fusing the cross-modal cross-feature, the color feature and the geometry feature by the feature fusion module to obtain a fused feature, the feature fusion module comprises a deformable convolution unit, an offset prediction network and an optimization submodule:

[0048] inputting the cross-modal cross-feature, the color feature and the geometry feature into the offset prediction network to generate an offset parameter;

[0049] adjusting a spatial position of a convolution kernel in the deformable convolution unit according to the offset parameter to obtain an adjusted deformable convolution unit;

[0050] performing convolution operation on the cross-modal cross-feature, the color feature and the geometry feature respectively by using the adjusted deformable convolution unit to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross-feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometry feature;

[0051] performing pixel-by-pixel splicing on the first convolution processing result, the second convolution processing result and the third convolution processing result to form an initial fused feature;

[0052] performing feature optimization on the initial fused feature by the optimization submodule to obtain a fused feature.

[0053] In an embodiment, the step of performing iterative optimization on the initial 6D pose by the pose iterative optimization module to obtain a target 6D pose result comprises:

[0054] decomposing the initial 6D pose to obtain an initial pose matrix;

[0055] performing global feature extraction on the fused feature by a global average pooling operation to obtain a global fused feature;

[0056] predicting a first pose residual by a multi-layer perception machine from the global fused feature;

[0057] performing matrix multiplication operation on the first pose residual and the initial pose matrix to obtain a corrected pose matrix;

[0058] calculating a pose error according to the corrected pose matrix;

[0059] when the pose error exceeds a preset error threshold, inputting the corrected pose matrix as a new initial 6D pose;

[0060] Until the pose error is less than a preset error threshold, a pose corresponding to a current correction pose matrix is determined as a target 6D pose result.

[0061] In addition, to achieve the above object, the application further provides a 6D pose estimation device fusing an attention mechanism, which comprises:

[0062] An acquisition module is configured to acquire a depth RGB image of a target object, wherein the depth RGB image comprises an RGB image and a depth image, and the depth image records an actual distance between a sensor and a surface of the target object.

[0063] A result module is configured to input the RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module; the result module is further configured to perform semantic segmentation on the RGB image by using the semantic segmentation module to obtain a segmentation mask of the target object; the result module is further configured to perform cutting on a point cloud converted from the depth image by using the semantic segmentation module on the segmentation mask to obtain a target mask point cloud; the result module is further configured to perform color feature extraction and geometric feature extraction on the target mask point cloud by using the feature extraction module to obtain color features and geometric features; the result module is further configured to perform interactive processing on the color features and the geometric features by using a cross-attention mechanism of the feature fusion module to generate cross-modal cross features; the result module is further configured to perform fusion on the cross-modal cross features, the color features and the geometric features by using a cross-modal deformable attention mechanism of the feature fusion module to obtain fusion features; the result module is further configured to perform pose calculation on the fusion features by using the pose estimation module to obtain an initial 6D pose; and the result module is further configured to perform iterative optimization on the initial 6D pose by using a pose iterative optimization module to obtain the target 6D pose result.

[0064] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the 6D pose estimation method fusing an attention mechanism as described above.

[0065] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the 6D pose estimation method fusing an attention mechanism as described above.

[0066] The application processes the RGB image and the depth image of the target object through a preset multi-modal feature fusion model, and the model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. The target mask point cloud is obtained through semantic segmentation, the feature fusion is performed by using the cross attention mechanism and the deformable convolution, and finally the pose is obtained through the pose estimation and the iterative optimization, so as to solve the problems of insufficient feature extraction and weak multi-modal feature correlation in a complex scene such as weak texture and occlusion, and improve the accuracy and robustness of the pose estimation. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required to be used in the embodiments or the prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0068] Figure 1 The flowchart of the first embodiment of the 6D pose estimation method of the application fusing the attention mechanism is shown.

[0069] Figure 2 The structure block diagram of the preset multi-modal feature fusion model of the first embodiment of the 6D pose estimation method of the application fusing the attention mechanism is shown.

[0070] Figure 3 The flowchart of the second embodiment of the 6D pose estimation method of the application fusing the attention mechanism is shown.

[0071] Figure 4 The module structure diagram of the 6D pose estimation device of the application fusing the attention mechanism is shown.

[0072] Figure 5 The device structure diagram of the hardware running environment involved in the 6D pose estimation method of the application fusing the attention mechanism is shown.

[0073] The purpose of the application, the functional characteristics and the advantages will be further described with reference to the embodiments and the drawings. DETAILED DESCRIPTION

[0074] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0075] In order to better understand the technical solutions of the present application, the drawings in the specification and the specific embodiments will be described in detail.

[0076] Current 6D pose estimation methods are mainly divided into three categories: instance-level, category-level and unseen object pose estimation. Instance-level methods include traditional geometric methods such as Linemod based on template matching and depth normal vector features, and deep learning methods such as GDR-Net that fuse deep learning features. Category-level methods include NOCS that unifies object representation through normalized coordinate space. Unseen object pose estimation methods include MAV6De that proposes self-training unsupervised domain adaptation and enhances the robustness of outdoor scenes through pseudo-labels. However, in complex scenes such as weak texture and occlusion, the feature extraction capability of existing methods is insufficient, resulting in insufficient target feature information and affecting the accuracy of pose estimation. Traditional geometric methods are prone to failure when dealing with weak texture objects, while deep learning methods, although improving the feature extraction capability to some extent, still struggle with complex lighting changes and occlusions. Moreover, in multi-modal feature fusion, simple concatenation or weighted summation is often used, ignoring the semantic association between different modal features. This simple fusion method fails to fully utilize the complementary information in RGB images and depth images, resulting in insufficient feature fusion accuracy.

[0077] Therefore, the present application proposes a 6D pose estimation method that fuses attention mechanisms to solve the above problems. The main solution of the embodiments of the present application is: obtaining a depth RGB image of a target object; inputting the depth RGB image into a pre-set multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the pre-set multi-modal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module; the step of inputting the depth RGB image into the pre-set multi-modal feature fusion model for processing to obtain the target 6D pose result further includes: performing semantic segmentation on the RGB image through the semantic segmentation module to obtain a segmentation mask of the target object; cropping the target mask point cloud from the point cloud converted from the depth image through the semantic segmentation module; performing color feature extraction and geometric feature extraction on the target mask point cloud through the feature extraction module to obtain color features and geometric features; using a cross-attention mechanism to interactively process the color features and geometric features through the feature fusion module to generate cross-modal cross features; using a cross-modal deformable attention mechanism to fuse the cross-modal cross features, color features, and geometric features through the feature fusion module to obtain fused features; performing pose solving on the fused features through the pose estimation module to obtain an initial 6D pose; performing iterative optimization on the initial 6D pose through the pose iterative optimization module to obtain the target 6D pose result.

[0078] Based on the above, the embodiments of the present application also provide a 6D pose estimation method that fuses attention mechanisms, which is described with reference to Figure 1 , Figure 1This is a flowchart illustrating the first embodiment of the 6D pose estimation method incorporating an attention mechanism according to this application. In this embodiment, the 6D pose estimation method incorporating an attention mechanism includes steps S10-S20:

[0079] Step S10: Obtain the depth RGB image of the target object.

[0080] It should be noted that depth RGB images are a special type of image data. They not only contain traditional RGB image information but also their corresponding depth image information. They provide color and spatial distance information of the target object, enabling computer vision systems to more comprehensively understand the object's position and shape in three-dimensional space.

[0081] Specifically, RGB images, or red, green, and blue three-channel images, provide rich color information, which is crucial for subsequent target recognition and feature extraction. High-resolution RGB cameras can capture the appearance features of target objects under different lighting conditions, including texture, color distribution, and shape details. This detailed information helps distinguish the target object from the background, especially in complex scenes where the target object may be partially occluded by other objects or be similar in color to the background. High-quality RGB images provide sufficient visual cues to help algorithms more accurately locate and identify target objects. Furthermore, acquiring RGB images requires camera calibration and standardization to ensure the accuracy of the image's geometric and color information. The calibration of the camera's intrinsic and extrinsic parameters ensures the correspondence between image data and the actual physical space, which directly affects the accuracy of subsequent pose estimation. In practical applications, such as robot grasping tasks, accurate RGB images help robots better understand the appearance of target objects, thus planning grasping paths more precisely. In autonomous driving scenarios, RGB images help vehicles identify obstacles and traffic signs on the road, providing important information for safe driving. Depth images, on the other hand, record the actual distance between the surface of a target object and a sensor. Unlike RGB images, each pixel value in a depth image represents the distance between that pixel and the sensor, rather than color information. Depth images are typically represented as grayscale images, where smaller pixel values ​​indicate closer distances and larger pixel values ​​indicate farther distances. Depth images provide information about the 3D shape and spatial layout of a target object, which is crucial for understanding the object's geometry and spatial location. Common sensors for acquiring depth images include Kinect and RealSense. These sensors measure the signal strength and time delay reflected from the target object's surface by emitting and receiving infrared light or other types of signals, thereby calculating the actual distance between the target object's surface and the sensor. The accuracy and resolution of the sensor directly affect the quality of the depth image, and consequently, the accuracy of 6D pose estimation.

[0082] Step S20, input the depth RGB image into the preset multi-modal feature fusion model for processing to obtain the target 6D pose result.

[0083] It should be noted that, as Figure 2 shown in the preset multi-modal feature fusion model structure block diagram, the model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module. The preset multi-modal feature fusion model is a model that comprehensively processes multiple data sources (such as RGB images and depth images), aiming to improve the performance of the task by fusing data of different modalities.

[0084] Specifically, the RGB image is the main source of visual information, containing rich color and texture information, which is crucial for the recognition and localization of target objects. Before inputting the model, the RGB image usually needs to be preprocessed, including adjusting the image size to match the input requirements of the model, normalizing the pixel values to improve the generalization ability of the model, and other operations. The depth image records the actual distance from the sensor to the surface of the target object. This composite data can provide more comprehensive visual information, which helps to more accurately estimate the 6D pose of the target object.

[0085] The role of the semantic segmentation module is to segment the target object region in the RGB image, providing accurate regional information for subsequent feature extraction. This process is usually based on convolutional neural networks (CNN) in deep learning, through learning a large amount of labeled data, the model can automatically identify and segment different objects in the image.

[0086] The feature extraction module is responsible for extracting useful features from the segmented target object region. These features can be color features, texture features, or geometric features, etc. In this embodiment, a deep learning network such as ResNet18 is used, combined with an attention mechanism, to extract color features from the RGB image, while extracting geometric features from the point cloud converted from the depth image. The attention mechanism can dynamically adjust the weights of the feature channels, enhancing the representation ability of the key features of the target object, which is particularly important for pose estimation in complex scenes. The task of the feature fusion module is to effectively fuse the extracted color features and geometric features. This fusion is not a simple superposition, but is carried out through cross-attention mechanisms and deformable convolutions, etc. The cross-attention mechanism enables the two kinds of features to complement each other, generating more semantically meaningful cross-modal features. Deformable convolution further enhances the adaptability to complex geometric structures, ensuring that the feature fusion process can fully utilize the complementary information in the RGB image and the point cloud.

[0087] The pose estimation module utilizes the fused features to perform an initial estimation of the 6D pose. This step typically involves predicting the rotation and translation parameters of the object using the model. Since the fused features already contain rich semantic and geometric information, they can provide more accurate pose estimates. The output of the pose estimation module is an initial 6D pose result, which serves as the basis for subsequent optimization. The 6D pose refers to the position and orientation of an object in a three-dimensional space, where "6D" stands for six degrees of freedom. Specifically, it includes three degrees of freedom for translation (movement along X, Y, and Z axes) and three degrees of freedom for rotation (rotation around X, Y, and Z axes). The 6D pose accurately describes the spatial position and direction of an object and is crucial for precise target positioning and interaction in fields such as robotics, autonomous driving, and virtual reality. To further improve the accuracy of the pose estimation, the pose iterative optimization module optimizes the initial 6D pose. This step typically involves iteratively adjusting the pose parameters to minimize the error of the target object under the estimated pose. Through multiple iterations, the model can gradually correct errors in the initial pose and ultimately obtain a more accurate 6D pose result. Through this multi-modal information fusion approach, the model can maintain high pose estimation accuracy even in complex scenarios, such as when the target is occluded or the background is complex.

[0088] Further, from the pose estimation accuracy results table on the LineMOD dataset shown in Table 1, it can be seen that the table lists 13 different types of objects, with each object corresponding to a row of data. The accuracy of each method on different objects is given in percentage form.

[0089] Table 1: Pose estimation accuracy results table on the LineMOD dataset

[0090]

[0091] As can be seen from the table, the method of the embodiment achieves an accuracy of 96.2% on the "ape" type, and an accuracy of 100% on the "glue" type. As can be seen from the table content, the 6D pose estimation method of the fusion attention mechanism of the embodiment achieves an average accuracy of 98.6% on the LineMOD dataset. Specifically, the method of the embodiment achieves the optimal in the pose estimation of 7 out of 13 targets, which is improved by 10%, 12.3% and 8.7% compared with the traditional RGB method, and is improved by 4.3%, 2.2%, 1.6%, 1.3% and 1.1% compared with the RGB-D methods DenseFusion, NF6D, Uni6D, MaskedFusion and CA6D. Compared with the original network DenseFusion method, the average accuracy is improved by 4.3%, the highest single object accuracy is 100%, and the lowest single object accuracy is improved by 9.2%. The experimental results show that the method of the embodiment has obvious improvement in processing weak texture objects (such as ape, drill, duck, hole), and significantly improves the average accuracy. This shows that the method of the embodiment has significant advantages in complex scenes, and can effectively improve the accuracy of 6D pose estimation. In addition, the method of the embodiment is also optimized in efficiency, reaching 12FPS, which can meet the requirements of real-time tasks.

[0092] In summary, the method of the embodiment performs well on the LineMOD dataset, not only surpassing several other RGB-D methods in accuracy, but also meeting the real-time processing requirements in efficiency.

[0093] The embodiment processes the RGB image and the depth image of the target object through a preset multi-modal feature fusion model, which includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. The target mask point cloud is obtained by semantic segmentation, the feature fusion is performed by using the cross attention mechanism and the deformable convolution, and the final pose is obtained through the pose estimation and the iterative optimization, which solves the problems of insufficient feature extraction and weak multi-modal feature correlation in complex scenes such as weak texture and occlusion, and improves the accuracy and robustness of pose estimation.

[0094] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 3 , the 6D pose estimation method of the fusion attention mechanism further includes steps S201-S207:

[0095] Step S201, performing semantic segmentation on the RGB image through the semantic segmentation module to obtain the segmentation mask of the target object.

[0096] It should be noted that when performing semantic segmentation, the semantic segmentation module usually employs deep learning models, especially convolutional neural networks (CNNs), to learn the boundaries and features of different objects in the image. These models can automatically learn complex patterns that distinguish different objects, enabling pixel-level classification, which is typically accomplished by a neural network with an encoder-decoder architecture that can process RGB images and output a segmentation mask. The segmentation mask is the direct result of semantic segmentation and is a binary image of the same size as the input image, where each pixel value indicates whether the pixel belongs to the target object.

[0097] Further, step S201 includes: first, obtaining the original resolution from the RGB image, which is the initial size of the image, usually expressed in pixels, such as 1024x1024 or 2048x2048, etc. The original resolution is crucial for subsequent feature extraction and image processing steps, as it determines the size of the feature map and the accuracy of the final segmentation mask.

[0098] Next, the RGB image is feature-extracted by the encoder of the semantic segmentation module to generate a first feature map and a second feature map. Specifically, the encoder is usually composed of multiple convolutional layers and pooling layers that can extract useful features from the image and gradually reduce the spatial resolution of the feature map. In this process, two feature maps of different scales are generated: the first feature map has a scale of 1 / 16 of the original resolution, meaning its width and height are both 1 / 16 of the original image; the second feature map has a scale of 1 / 8 of the original resolution, with a spatial resolution twice that of the first feature map.

[0099] Next, the first feature map is upsampled to the scale of the second feature map by the decoder of the semantic segmentation module to obtain a third feature map. Specifically, the decoder starts working, and its role is to upsample the low-resolution feature map to a high-resolution. The upsample process usually involves deconvolution or transposed convolution operations that can increase the size of the feature map while preserving the feature information.

[0100] Subsequently, the third feature map and the second feature map are channel-wise weighted fused to obtain a fused feature map. Specifically, the purpose of this step is to integrate feature information from different scales to obtain a more comprehensive feature representation. Channel-wise weighted fusion means that each feature channel is given different weights according to its importance, so that the fused feature map can more accurately represent the object edges and texture details in the image.

[0101] Then, the fused feature maps are passed through a deconvolution operation to recover to the original resolution, obtaining a fourth feature map. Specifically, by further refining the feature maps, they are matched with the size of the original image, providing a basis for the final pixel-level classification.

[0102] Finally, the fourth feature map is output through a Softmax layer to obtain the pixel-level classification probability, obtaining the segmentation mask of the target object. Specifically, the Softmax layer is a commonly used activation function, which can convert the input non-normalized values into normalized probability distribution, so that each pixel has a probability value corresponding to each class. In this way, it can be determined which object category each pixel belongs to, thereby obtaining the segmentation mask of the target object. In 6D pose estimation, this segmentation mask can be used to accurately crop the point cloud of the target object from the depth image, which is crucial for subsequent feature extraction and pose estimation.

[0103] Step S202, the target mask point cloud is obtained by cropping the point cloud converted from the depth image through the segmentation mask by the semantic segmentation module.

[0104] It should be noted that the depth image provides the distance information of each pixel in the scene to the camera, which is obtained through depth sensors such as Kinect or RealSense devices. Converting the depth image into a point cloud means combining these distance information with the two-dimensional coordinates of the image to generate a set of points in a three-dimensional space, each point containing its position information on the X, Y, Z axes. The cropping process involves applying the segmentation mask to the point cloud data to filter out points belonging to the target object. This step is usually achieved by corresponding the two-dimensional coordinates of the mask to the projected coordinates of the point cloud. Each pixel in the mask corresponds to a set of three-dimensional points in the point cloud, through this correspondence, it can be determined which points belong to the target object and separate them from the entire point cloud. The target mask point cloud obtained is a set of points that accurately represent the three-dimensional shape of the target object, and these point cloud data will be used for subsequent feature extraction and pose estimation. For example, these point cloud data can be used to extract the geometric features of the object, or combined with the color features in the RGB image for multi-modal feature fusion to improve the accuracy and robustness of pose estimation.

[0105] Further, step S202 includes: first, obtaining the camera intrinsic parameters, including focal length, principal point coordinates, and radial and tangential distortion coefficients, etc. These parameters describe the imaging characteristics of the camera. Camera intrinsic parameters are crucial for subsequent point cloud conversion and coordinate transformation, as they ensure accurate conversion of image coordinates to world coordinates.

[0106] Then, the depth image is converted into an unordered point cloud by a preset point cloud conversion algorithm, where the unordered point cloud includes a background point cloud and a target object corresponding point cloud. Specifically, each pixel value in the depth image represents the distance from the surface of the corresponding position object to the camera, and the preset point cloud conversion algorithm traverses each pixel of the depth image according to the mapping logic of “pixel coordinates-depth value-three-dimensional coordinates”. Taking the row and column numbers of the pixel as a two-dimensional index, combining the obtained camera intrinsic parameters, the depth value corresponding to the pixel is converted into an initial coordinate in the three-dimensional space through the perspective projection formula, and finally an unordered point cloud containing a large number of three-dimensional points is generated. The unordered point cloud is not screened and contains point cloud data of the surface of the target object (target object corresponding point cloud) and point cloud data of background objects in the scene (such as a desktop and a device support) (background point cloud), which need to be further processed to separate the target point cloud.

[0107] Next, the unordered point cloud is converted into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates. Specifically, since the initially generated unordered point cloud may be based on an image pixel coordinate system, coordinate transformation is performed through the rotation matrix and translation vector in the camera intrinsic parameters: taking the camera optical center as the origin, a camera coordinate system (X axis along the horizontal direction of the camera, Y axis along the vertical direction of the camera, and Z axis along the optical axis direction of the camera) is established, the initial coordinates of each point in the unordered point cloud are substituted into the coordinate conversion formula, the image distortion is eliminated and the coordinate offset is corrected, and the accurate three-dimensional coordinates of each point in the camera coordinate system are obtained. The coordinates can directly reflect the position relationship of the point cloud in the real three-dimensional space and provide a spatial position basis for subsequent target point cloud screening.

[0108] Subsequently, the pixel point coordinate range corresponding to the target object in the unordered point cloud is determined by the semantic segmentation module according to the pixel area range of the segmentation mask. Specifically, the segmentation mask is a binary image generated by the semantic segmentation module after processing the RGB image, where the region with a pixel value of 1 corresponds to the target object, and the region with a pixel value of 0 corresponds to the background. The semantic segmentation module first reads the segmentation mask, traverses all pixels with a pixel value of 1 in the mask, records the row and column number range of these pixels (i.e., the pixel area boundary of the target object in the image), and then reversely deduces the pixel point coordinate range belonging to the target object in the unordered point cloud according to the mapping relationship of “image pixel row and column number-point cloud coordinate in camera coordinate system”, to clearly define the spatial boundary of subsequent point cloud screening.

[0109] Then, the point cloud data whose pixel coordinates are within the pixel point coordinate range is screened to obtain screened point cloud data. Specifically, each point in the unordered point cloud is traversed, the three-dimensional coordinates of each point in the camera coordinate system are extracted, and it is judged whether the corresponding image pixel coordinates fall within the pixel point coordinate range determined above: if they fall within the range, the point is determined to be a target object related point cloud, and the three-dimensional coordinate information of the point is retained; if they exceed the range, they are determined to be background point cloud and are eliminated. Through the screening process, the point cloud data corresponding to the target object can be preliminarily separated from the unordered point cloud, greatly reducing the data amount of subsequent processing, and avoiding the interference of background point cloud on the extraction of target features.

[0110] Finally, the screened point cloud data is denoised to obtain a target mask point cloud. Specifically, the screened point cloud may have isolated noise points (such as discrete points caused by depth measurement errors, false points generated by environmental light interference), and a statistical filtering algorithm needs to be used for denoising: a threshold of the number of neighboring points (such as 50) and a distance threshold (such as 0.01 meters) are set, for each point cloud data after screening, the average distance between it and its surrounding neighboring points is calculated, if the average distance exceeds the distance threshold, it is determined to be a noise point and is deleted; if it is within the threshold, the point is retained. After denoising, the target mask point cloud containing only the effective point cloud of the surface of the target object is obtained, which accurately reflects the three-dimensional geometric shape of the target object and provides high-quality geometric information input for the subsequent feature extraction module.

[0111] Step S203, color feature extraction and geometric feature extraction are performed on the target mask point cloud by the feature extraction module to obtain color features and geometric features.

[0112] It should be noted that the feature extraction module includes a color feature extraction submodule and a geometric feature extraction submodule, wherein the color feature extraction submodule includes a ResNet18 network and a compression and excitation attention layer, which cooperatively extract color features and geometric features from the target mask point cloud. Color feature extraction focuses on capturing color information on the surface of the target object. This is usually achieved by analyzing the pixel points in the RGB image corresponding to the point cloud. Color features can provide visual texture and color distribution of the object surface, and play an important role in object recognition and differentiation. In the feature extraction process, deep learning models such as convolutional neural networks may be used to automatically learn and extract color features. These models can identify color patterns and texture features that are useful for classification and recognition. Geometric feature extraction focuses on geometric information in point cloud data, such as the spatial relationship between points, the shape and size of the object, etc. Geometric features are usually obtained by analyzing the three-dimensional coordinates of points in the point cloud. These features are crucial for understanding the three-dimensional structure of the object. In the feature extraction process, local geometric descriptors of the point cloud (such as the normal of the point cloud, the curvature, etc.) may be used to capture these geometric information.

[0113] Further, step S203 includes: First, preliminary feature extraction is performed on the cropped RGB image corresponding to the target mask point cloud using the ResNet18 network to obtain initial image features. Specifically, the cropped RGB image only contains the target object region, which can reduce background interference. The ResNet18 network avoids the gradient vanishing problem with its residual connection structure. It first performs downsampling and feature extraction on the cropped RGB image through convolutional layers: shallow convolutional layers capture detailed features such as edges and textures, and deep convolutional layers integrate local features to generate abstract semantic features. Finally, it outputs initial image features with uniform dimensions. These features cover key semantic information such as the color distribution and texture pattern of the target object, laying the foundation for subsequent feature optimization.

[0114] Next, the channel dimension of the initial image features is compressed by compressing and activating the attention layer to obtain global information for each channel. Specifically, each channel of the initial image features represents different semantics (e.g., some channels correspond to the surface color of an object, while others correspond to texture details). The compression operation is achieved through global average pooling: for each channel of the initial image features, the average value of all pixels in that channel is calculated, transforming the two-dimensional feature map into a one-dimensional vector. This vector represents the global information of each channel, reflecting the importance of the corresponding channel features in the entire target region and providing a basis for subsequent weight allocation.

[0115] Next, the global information is subjected to dimensionality reduction to obtain the reduced information. Specifically, dimensionality reduction is achieved through a fully connected layer, compressing the dimension of the global information to 1 / 16 of the original dimension (a preset ratio), reducing computational cost and enhancing the model's generalization ability. The fully connected layer uses linear transformations and activation functions to filter key features from the global information, eliminating redundant data, so that subsequent processing focuses more on information effective for pose estimation.

[0116] Subsequently, dimensionality restoration is performed on the dimensionality-reduced information to obtain the dimensionality-restored information. Specifically, another fully connected layer restores the dimensionality of the dimensionality-reduced information to the same dimension as the number of feature channels in the initial image, ensuring that the subsequently generated attention weights can match the initial image features channel by channel. This process reconstructs the channel dimension through linear transformation while retaining the key information selected during dimensionality reduction, maintaining the integrity and effectiveness of the features.

[0117] Then, the dimensionality-restored information is activated using the Sigmoid function to generate attention weights for each channel. Specifically, the Sigmoid function maps the dimensionality-restored information to the 0-1 range, with larger values ​​indicating a greater contribution of the corresponding channel's features to pose estimation. For example, channels with clear textures on the target object have higher weights, while channels with background residue or noise have lower weights, thus dynamically enhancing effective feature channels.

[0118] The attention weight is then multiplied by the initial image features channel by channel to obtain color features. Specifically, during the channel-by-channel multiplication process, channel features with high weights are amplified, and channel features with low weights are suppressed, thereby highlighting the key color and texture information of the target object, weakening noise interference, and generating color features with higher recognition, providing high-quality semantic input for multi-modal fusion.

[0119] Then, the three-dimensional coordinates of each point in the target mask point cloud are subjected to a nonlinear transformation by a geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension. Specifically, the geometric feature extraction submodule adopts a multi-layer perception architecture, first normalizes the three-dimensional coordinates of the target mask point cloud to eliminate scale differences in different scenes, and then gradually maps the three-dimensional coordinates to a high-dimensional space through a multi-layer nonlinear activation function (such as a ReLU function) to extract the local geometric properties (such as adjacent point distance and curvature variation) of each point and generate point-by-point geometric features to capture the local structural details of the target object.

[0120] Finally, the point-by-point geometric features are globally aggregated to obtain geometric features, which represent the overall geometric properties of the target mask point cloud. Specifically, global aggregation is achieved through a max-pooling operation: taking the maximum value in each feature dimension for all point-by-point geometric features to integrate the global spatial information of the target mask point cloud and generate geometric features that can represent the overall geometric form (such as contour, size, and symmetric structure) of the target. The geometric features are complementary to the color features and jointly provide semantic and spatial dual information support for subsequent multi-modal feature fusion.

[0121] Step S204: The color features and the geometric features are interactively processed by a feature fusion module using a cross-attention mechanism to generate cross-modal cross features.

[0122] It should be noted that the feature fusion module first loads the color features and the geometric features extracted earlier, clearly defines the core mapping rules of the cross-attention mechanism, takes the semantic information carried by the color features as the core guide, and takes the spatial position information contained in the geometric features as the associated basis to establish a deep correlation between the two modal features through a Query-Key-Value interaction mode.

[0123] Further, step S204 further comprises: first, loading the color feature and the geometric feature by a feature fusion module to determine a mapping rule of the cross-attention mechanism. Specifically, the feature fusion module first performs dimension consistency checking on the two modal features to ensure that the feature dimensions of the color feature (high-dimensional semantic vector) and the geometric feature (high-dimensional spatial vector) are matched, avoiding dimension conflicts in subsequent interaction calculations; then, based on the task requirements (6D pose estimation needs to strengthen the "semantic-spatial" correlation), the mapping rule is determined: the color feature is used as the semantic guide source, responsible for locating the key semantic area of the target object, and the geometric feature is used as the spatial support source, responsible for providing accurate position information, and the corresponding relationship between the Query vector generated from the color feature and the Key and Value vectors generated from the geometric feature is determined, laying a logical foundation for cross-modal interaction.

[0124] Next, the color feature is mapped into a Query vector according to the mapping rule through a preset linear transformation, wherein the Query vector is used to encode the category semantic information of the target object. Specifically, the preset linear transformation is realized by a trained fully connected layer, and the weight matrix of the fully connected layer is obtained by iterative optimization of a large number of 6D pose estimation samples, which can retain and convert the effective information related to the object category and texture attribute in the color feature into a fixed-dimensional vector structure; in the transformation process, the color feature is normalized simultaneously to eliminate the feature scale differences between different samples, and the finally generated Query vector not only contains the category semantics of "what is the target object" (such as "mechanical parts" and "electronic components"), but also implies the spatial correlation requirement of the semantic area, providing a semantic anchor point for the subsequent matching of the geometric feature.

[0125] Next, the point cloud coordinate information in the geometric feature is mapped into a Key vector and a Value vector according to the mapping rule through a preset linear transformation, wherein the Key vector is used to encode the geometric position correlation information, and the Value vector is used to encode the geometric position feature information. Specifically, for the linear transformation of the Key vector, the relative position relationship (such as the distance and angle between adjacent points) of the point cloud coordinates in the geometric feature is extracted, and a vector reflecting the "position correlation" is generated to facilitate the matching with the semantic requirement of the Query vector; for the linear transformation of the Value vector, the absolute position details (such as three-dimensional coordinate values and local curvature) of the point cloud coordinates are focused on, and the spatial precision of the geometric feature is preserved; the two linear transformation processes are performed independently, and both adopt the same dimension standard as the Query vector transformation to ensure that the three can be interactively calculated.

[0126] Subsequently, the similarity between the Query vector and the Key vector is calculated. Specifically, the similarity between the two is calculated by point multiplication: the Query vector is multiplied element by element with the Key vector, and the sum is taken to obtain the similarity value of a single Query-Key pair. The larger the value, the more closely the semantic requirements of the Query vector match the position association of the Key vector. All combinations of Query vectors and Key vectors are traversed to generate a similarity matrix, where each element corresponds to the matching degree of a set of "semantic requirements-position association", providing a quantitative basis for subsequent weight allocation.

[0127] Then, the similarity is normalized by the Softmax function to generate an attention weight map. Specifically, the Softmax function normalizes each row of the similarity matrix (corresponding to a single Query vector) so that the sum of the elements in each row is 1, converting the similarity value to a weight value in the 0-1 interval. After normalization, the weight value corresponding to the Key vector that matches the semantic requirements of the Query vector is close to 1, and the weight value corresponding to the Key vector with low relevance is close to 0. The attention weight map accurately identifies the key regions in the geometric features that are related to color semantics, achieving the core goal of semantic-guided space selection.

[0128] The attention weight map and the Value vector are then weighted and summed to obtain a weighted result. Specifically, each Value vector is assigned a corresponding attention weight in the manner of "weight value x corresponding Value vector", and all weighted Value vectors are then summed. In this process, Value vectors with high weights (corresponding to geometric regions with strong semantic associations) have a higher proportion in the sum, while Value vectors with low weights (corresponding to geometric regions with weak semantic associations) have a lower proportion. This process integrates color semantic information into the spatial details of geometric features, generating intermediate features that preliminarily associate the two modal information.

[0129] Finally, the weighted result is linearly transformed to generate a cross-modal cross-feature. Specifically, the linear transformation layer trained in the feature fusion module is used to adjust the dimensions and integrate the features of the weighted result, eliminating any feature redundancy that may have been introduced during the weighting process and strengthening the collaborative representation capabilities of semantics and space. The final cross-modal cross-feature contains both the class semantic information of the color feature and the precise position information of the geometric feature, effectively addressing the information deficiencies of single-modal features and providing high-quality interactive feature support for the subsequent deep fusion of multi-modal features.

[0130] In step S205, the feature fusion module uses the cross-modal deformable attention mechanism to fuse the cross-modal cross-feature, the color feature, and the geometric feature to obtain a fused feature.

[0131] It should be noted that the feature fusion module first loads cross-modal features, color features, and geometric features to construct a cross-modal deformable attention fusion framework. The core of this framework is to achieve pixel-level accurate fusion of the three features by dynamically adjusting the receptive field of the convolution. The feature fusion module includes deformable convolutional units, offset prediction networks, and optimization sub-modules.

[0132] Further, step S205 includes: First, inputting the cross-modal cross features, the color features, and the geometric features into the offset prediction network to generate offset parameters. Specifically, the offset prediction network is the core control unit of the cross-modal deformable attention mechanism, constructed using a lightweight multilayer perceptron. It first performs dimensionality unification processing on the three input features, mapping them to a feature space of the same dimension through 1×1 convolutions to avoid prediction bias caused by differences in feature dimensions. Then, it extracts local spatial correlation information of the features through 3×3 convolutions, capturing key spatial clues such as the outline of the target object and occluded areas. Finally, it outputs offset parameters matching the size of the deformable convolutional unit's kernel through the output layer. These parameters include position adjustment values ​​in both horizontal and vertical directions, accurately guiding the spatial position correction of the convolutional kernel and ensuring that the convolutional kernel can adaptively focus on the effective target area.

[0133] Next, the spatial position of the convolutional kernel in the deformable convolutional unit is adjusted according to the offset parameter to obtain the adjusted deformable convolutional unit. Specifically, the initial convolutional kernel of the deformable convolutional unit adopts a fixed grid distribution. After receiving the offset parameter, the offset parameter is converted into the actual position offset of the convolutional kernel sampling point through a coordinate mapping algorithm. For example, when the target object is partially occluded, the offset parameter will cause the convolutional kernel sampling point in the corresponding area to shift to the unoccluded area, avoiding the invalid feature area caused by occlusion. At the same time, boundary checks and interpolation processing are performed on the adjusted sampling point position to ensure that the sampling point always falls within the feature map range, avoiding feature loss caused by coordinate out-of-bounds, and finally forming an adjusted deformable convolutional unit adapted to the spatial distribution of the target object.

[0134] Then, the adjusted deformable convolution unit is used to perform convolution operations on the cross-modal cross feature, the color feature and the geometric feature respectively to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometric feature. Specifically, for the cross-modal cross feature, the adjusted convolution kernel focuses on the region with close semantic-spatial correlation, strengthens the interactive representation of the information of the two modalities, and generates the first convolution processing result rich in associated information; for the color feature, the convolution kernel focuses on capturing the key color and texture details of the target object and suppressing background noise interference, and outputs the second convolution processing result highlighting the semantic attributes; for the geometric feature, the convolution kernel accurately locates the local structure and global contour of the target object, retains the spatial position accuracy, and generates the third convolution processing result with spatial integrity. The three convolution processing results not only retain the core information of each modal feature, but also realize the focusing on the effective region of the target through the dynamic convolution kernel.

[0135] Then, the first convolution processing result, the second convolution processing result and the third convolution processing result are pixel-by-pixel spliced to form an initial fusion feature. Specifically, the pixel-by-pixel splicing stacks the three convolution processing results in the channel dimension based on the spatial coordinates of the feature map: for each pixel position in the feature map, the pixel feature vector of the first convolution processing result, the pixel feature vector of the second convolution processing result and the pixel feature vector of the third convolution processing result are spliced in order to form a pixel feature vector with a dimension being the sum of the dimensions of the three features; through this operation, each pixel contains cross-modal associated information, semantic color information and spatial geometric information at the same time, and the initial fusion feature with coexistence of multi-modal information is constructed, providing a comprehensive feature basis for subsequent optimization.

[0136] Finally, the initial fusion feature is optimized by the optimization sub-module to obtain a fusion feature. Specifically, the optimization sub-module includes a batch normalization layer, a ReLU activation layer and a 1x1 convolution layer: the batch normalization layer normalizes each channel of the initial fusion feature to eliminate training deviation caused by feature value distribution difference and accelerate feature convergence; the ReLU activation layer suppresses negative invalid features in the initial fusion feature through nonlinear transformation to enhance the expression ability of positive effective features; and the 1x1 convolution layer performs dimension compression and information integration on the optimized feature to eliminate redundant information between channels, adjusts the feature dimension to adapt to the dimension of the subsequent pose estimation network, and finally generates the fusion feature with semantic recognition and spatial accuracy, providing high-quality feature input for pose calculation.

[0137] In step S206, the fusion feature is calculated by the pose estimation module to obtain an initial 6D pose.

[0138] It should be noted that the pose estimation module takes the fused features after multi-modal fusion as input, and completes the calculation of the initial 6D pose through the progressive process of feature mapping-parameter regression-coordinate transformation-error calculation.

[0139] Specifically, the pose estimation module inputs the fused features into a preset pose regression network. The network takes a multi-layer perceptron as the core architecture, first converts the dimensions of the fused features through a fully connected layer, mapping the high-dimensional fused features to a feature space suitable for pose parameter regression; then strengthens the non-linear expression ability of the features through a non-linear activation function (such as the ReLU function), highlighting the key features related to the pose (such as the symmetric structure and local contour key points of the target object); finally, outputs three groups of core parameters through the output layer: the rotation quaternion , the translation vector and the confidence , wherein the rotation quaternion and the translation vector together constitute the initial prediction result of the 6D pose, and the confidence is used to quantify the reliability of the prediction result. The higher the confidence value, the stronger the accuracy of the corresponding pose prediction.

[0140] Next, the rotation quaternion obtained by regression is converted in format to a rotation matrix to adapt to subsequent coordinate transformation. The standard formula for converting a quaternion to a rotation matrix is used to map the four-dimensional quaternion to a three-dimensional rotation matrix, ensuring that the rotation information can be directly used for point cloud coordinate transformation. The specific formula is:

[0141] wherein, represents the converted 3x3 rotation matrix, which is used to describe the spatial rotation state of the target object, respectively represent the four components of the rotation quaternion , which satisfy the constraint condition to ensure that the object scale remains unchanged during rotation.

[0142] Subsequently, the predicted point cloud is calculated through coordinate transformation to establish the association between the fused features and the three-dimensional morphology of the target object. The pose estimation module first loads the three-dimensional model sampling point cloud of the target object (the point cloud is the standard morphology point cloud of the object in the world coordinate system , containing three-dimensional sampling points), and then uses the converted rotation matrix and the regression obtained translation vector to convert the standard point cloud to the three-dimensional coordinates under the predicted pose to obtain the predicted point cloud. The specific formula is as follows:

[0143]

[0144] wherein, This represents the target point cloud under the predicted pose, where each element is the three-dimensional coordinate of the predicted point. ), This represents a sampled point cloud of a 3D target model in the world coordinate system, where each element represents a 3D coordinate in the world coordinate system. ), Represents the transpose of a rotation matrix. This represents the translation vector, specifically the positional offset of the target object in three directions within the camera coordinate system. ).

[0145] Then, for symmetrical and asymmetrical target objects, the pose loss is calculated separately to verify the reasonableness of the prediction results. For asymmetrical objects, the point-by-point Euclidean distance between the predicted point cloud and the true pose point cloud is directly calculated and averaged to obtain the pose loss. The specific formula is as follows:

[0146]

[0147] in, This indicates the operation of calculating the average. Represents the L2 norm. This represents the target point cloud in its true pose, where each element is the three-dimensional coordinate of a real point. For symmetrical objects, since different rotation angles may correspond to the same shape, the K-nearest neighbor algorithm needs to be used to find the nearest neighbor correspondence between the predicted point cloud and the real point cloud. After real point cloud is realigned, the average Euclidean distance is calculated to avoid calculation errors caused by symmetrical structures.

[0148] Finally, combining the confidence level and preset balance parameters Calculate the weighted loss The initial 6D poses that meet the accuracy requirements are selected based on the weighted loss. The specific formula for calculating the weighted loss is as follows:

[0149]

[0150] in, This indicates the number of fused features involved in the loss calculation. Indicates the first The pose loss corresponding to each fused feature Indicates the first The confidence level corresponding to each fusion feature This represents the balance parameter, which is set to 0.01 in this embodiment to prevent distortion in loss calculation caused by excessively low confidence levels. ln represents the natural logarithm operation. When the weighted loss is less than a preset loss threshold (such as 0.05), it is determined that the rotation matrix and the translation vector obtained by the current regression are valid results, which together constitute an initial 6D pose and are output to a subsequent iterative optimization module.

[0151] In step S207, the initial 6D pose is iteratively optimized by the pose iterative optimization module to obtain a target 6D pose result.

[0152] It should be noted that the pose iterative optimization module is responsible for fine adjustment of the initial estimated 6D pose to improve the accuracy of the pose estimation. The module usually adopts an iterative method to gradually reduce the pose estimation error through multiple loop optimization processes.

[0153] Further, step S207 includes: first, the initial 6D pose is disassembled to obtain an initial pose matrix, the initial 6D pose includes a rotation matrix and a translation vector , and then combined to obtain the initial pose matrix , which is specifically expressed as:

[0154]

[0155] wherein, represents the initial rotation matrix, describing the rotation state of the target object around the origin of the camera coordinate system, represents the initial translation vector, including the position offset of the target object in direction, 0 and 1 are homogeneous coordinate identifiers to ensure that the matrix can be used for three-dimensional coordinate transformation.

[0156] Next, global feature extraction is performed on the fused features by a global average pooling operation to obtain global fused features. Specifically, the global average pooling calculates the pixel average value of each channel of the fused feature map, and compresses the two-dimensional feature map into a one-dimensional vector. The specific formula for calculating the global fused features is as follows:

[0157]

[0158] wherein, represents the global fused features, represents the pixel value of the i-th channel, the j-th channel, and the k-th channel of the fused feature map, represents the average operation.

[0159] ​​​Then, the global fusion features are predicted by a multi-layer perceptron to obtain a first pose residual. Specifically, the multi-layer perceptron includes two fully connected layers and a ReLU activation function, the global fusion features are first mapped to a 6-dimensional space (3-dimensional rotation residual + 3-dimensional translation residual), and then an orthogonalization process is performed to ensure the effectiveness of the rotation residual, and the first pose residual is output .

[0160] Then, the first pose residual and the initial pose matrix are subjected to matrix multiplication to obtain a corrected pose matrix , and the specific formula is:

[0161] .

[0162] Then, the corrected pose matrix is calculated to obtain a pose error. Specifically, the target object three-dimensional model point cloud is converted into a predicted point cloud in the camera coordinate system by the corrected pose matrix . Each point in the predicted point cloud is , and a real scene point cloud is obtained , and the pose error is calculated, and the specific formula is:

[0163]

[0164] wherein, is the first point coordinate, is the second point coordinate, is the pose error, and the total number of points in the point cloud is represented by N.

[0165] Finally, when the pose error exceeds a preset error threshold, the corrected pose matrix is input as a new initial 6D pose, and the residual prediction and correction process is repeated; until the pose error is less than the preset error threshold, the pose corresponding to the current corrected pose matrix is determined as the target 6D pose result.

[0166] The embodiment first obtains a segmentation mask of the target object by using a semantic segmentation module, and then cuts out the target mask point cloud. Then, color features and geometric features are obtained by a feature extraction module, and fusion features are generated by a feature fusion module. Finally, an accurate 6D pose result is obtained by a pose estimation and iterative optimization module. By processing the RGB image and the depth image through the multi-modal feature fusion model, accurate pose estimation of the target object is realized, and the accuracy and robustness of the pose estimation are improved.

[0167] Based on the first embodiment of the present application, the present application also provides a 6D pose estimation device fusing an attention mechanism, please refer to Figure 4The device comprises:

[0168] The acquisition module 10 is configured to acquire a depth RGB image of a target object, wherein the depth RGB image comprises an RGB image and a depth image recording an actual distance between a sensor and a surface of the target object.

[0169] The result module 20 is configured to input the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module; the result module 20 is further configured to perform semantic segmentation on the RGB image by using the semantic segmentation module to obtain a segmentation mask of the target object; the result module 20 is further configured to perform cropping on a point cloud converted from the depth image by using the semantic segmentation module on the segmentation mask to obtain a target mask point cloud; the result module 20 is further configured to perform color feature extraction and geometric feature extraction on the target mask point cloud by using the feature extraction module to obtain color features and geometric features; the result module 20 is further configured to perform interactive processing on the color features and the geometric features by using the feature fusion module to generate cross-modal cross features by using a cross attention mechanism; the result module 20 is further configured to perform fusion on the cross-modal cross features, the color features and the geometric features by using the feature fusion module to obtain fusion features by using a cross-modal deformable attention mechanism; the result module 20 is further configured to perform pose calculation on the fusion features by using the pose estimation module to obtain an initial 6D pose; and the result module 20 is further configured to perform iterative optimization on the initial 6D pose by using the pose iterative optimization module to obtain the target 6D pose result.

[0170] The 6D pose estimation device with a fusion attention mechanism provided in the application adopts the 6D pose estimation method with a fusion attention mechanism in the above embodiments, and can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the 6D pose estimation device with a fusion attention mechanism provided in the application has the same beneficial effects as the 6D pose estimation method with a fusion attention mechanism provided in the above embodiments, and other technical features in the 6D pose estimation device with a fusion attention mechanism are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0171] In an embodiment, the result module 20 is further configured to obtain an original resolution according to the RGB image.

[0172] The RGB image is feature-extracted by an encoder of the semantic segmentation module to generate a first feature map and a second feature map, wherein a scale of the first feature map is 1 / 16 of an original resolution, and a scale of the second feature map is 1 / 8 of the original resolution; the scale of the first feature map is up-sampled to the scale of the second feature map by a decoder of the semantic segmentation module to obtain a third feature map; the third feature map and the second feature map are channel-by-channel weighted fused to obtain a fused feature map; the fused feature map is subjected to a deconvolution operation to be restored to the original resolution to obtain a fourth feature map; the fourth feature map is output by a Softmax layer to obtain a pixel-level classification probability, and a segmentation mask of a target object is obtained.

[0173] In an embodiment, the result module 20 is further configured to acquire camera intrinsic parameters; convert the depth image into an unordered point cloud by a preset point cloud conversion algorithm, wherein the unordered point cloud comprises a background point cloud and a target object corresponding point cloud; convert the unordered point cloud into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates; determine a pixel point coordinate range corresponding to the target object in the unordered point cloud according to a pixel region range of the segmentation mask by the semantic segmentation module; filter point cloud data in the pixel coordinate range to obtain filtered point cloud data; and perform denoising processing on the filtered point cloud data to obtain a target mask point cloud.

[0174] In an embodiment, the result module 20 is further configured to perform preliminary feature extraction on the cropped RGB image corresponding to the target mask point cloud by the ResNet18 network to obtain initial image features; compress a channel dimension of the initial image features by the compression and excitation attention layer to obtain global information of each channel; perform dimension reduction processing on the global information to obtain reduced information; perform dimension recovery on the reduced information to obtain dimension-recovered information; activate the dimension-recovered information by a Sigmoid function to generate attention weights corresponding to each channel; multiply the attention weights and the initial image features channel by channel to obtain color features; perform nonlinear transformation on three-dimensional coordinates of each point in the target mask point cloud by the geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension; and perform global aggregation on the point-by-point geometric features to obtain geometric features, wherein the geometric features represent overall geometric properties of the target mask point cloud.

[0175] In an embodiment, the result module 20 is further configured to load the color feature and the geometry feature by the feature fusion module, determine a mapping rule of a cross-attention mechanism, map the color feature to a Query vector through a preset linear transformation according to the mapping rule, wherein the Query vector is used to encode category semantic information of a target object, map point cloud coordinate information in the geometry feature to a Key vector and a Value vector through a preset linear transformation according to the mapping rule, wherein the Key vector is used to encode geometry position correlation information, and the Value vector is used to encode geometry position feature information, perform calculation on the Query vector and the Key vector to obtain a similarity, perform normalization processing on the similarity through a Softmax function to generate an attention weight map, perform weighted summation operation on the attention weight map and the Value vector to obtain a weighted result, and perform linear transformation on the weighted result to generate a cross-modal cross feature.

[0176] In an embodiment, the result module 20 is further configured to input the cross-modal cross feature, the color feature and the geometry feature into the offset prediction network to generate an offset parameter, adjust a spatial position of a convolution kernel in the deformable convolution unit according to the offset parameter to obtain an adjusted deformable convolution unit, perform convolution operation on the cross-modal cross feature, the color feature and the geometry feature respectively by using the adjusted deformable convolution unit to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometry feature, perform pixel-by-pixel splicing on the first convolution processing result, the second convolution processing result and the third convolution processing result to form an initial fusion feature, and perform feature optimization on the initial fusion feature by the optimization submodule to obtain a fusion feature.

[0177] In an embodiment, the result module 20 is further configured to disassemble the initial 6D pose to obtain an initial pose matrix, perform global feature extraction on the fusion feature through a global average pooling operation to obtain a global fusion feature, predict a first attitude residual by the global fusion feature through a multi-layer perception, perform matrix multiplication operation on the first attitude residual and the initial pose matrix to obtain a corrected pose matrix, calculate a pose error according to the corrected pose matrix, input the corrected pose matrix as a new initial 6D pose when the pose error exceeds a preset error threshold, and determine a pose corresponding to a current corrected pose matrix as a target 6D pose result until the pose error is less than the preset error threshold.

[0178] The application provides a 6D pose estimation device fusing an attention mechanism, the 6D pose estimation device fusing the attention mechanism comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the 6D pose estimation method fusing the attention mechanism in the above-mentioned embodiment one.

[0179] Reference will now be made to the drawings, in which Figure 5 which shows a structural schematic diagram of the 6D pose estimation device fusing the attention mechanism suitable for being used to implement the embodiments of the application. The 6D pose estimation device fusing the attention mechanism in the embodiments of the application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 5 The 6D pose estimation device fusing the attention mechanism shown is only an example, and should not bring any limitation to the functions and use ranges of the embodiments of the application.

[0180] As Figure 5As shown, the fusion attention mechanism-based 6D pose estimation device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the fusion attention mechanism-based 6D pose estimation device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the fusion attention mechanism-based 6D pose estimation device to communicate wirelessly or wired with other devices to exchange data. Although the fusion attention mechanism-based 6D pose estimation device with various is shown in the figure, it should be understood that all the shown is not required to be implemented or possessed. More or less can be alternatively implemented or possessed.

[0181] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable storage medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0182] The fusion attention mechanism-based 6D pose estimation device provided by the present application adopts the fusion attention mechanism-based 6D pose estimation method in the above-mentioned embodiments, which can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the fusion attention mechanism-based 6D pose estimation device provided by the present application has the same beneficial effects as the fusion attention mechanism-based 6D pose estimation method provided by the above-mentioned embodiments, and other technical features in the fusion attention mechanism-based 6D pose estimation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0183] It is to be understood that the various parts of the disclosure can be implemented in hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0184] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes or modifications that can be made to the application in light of the teachings described herein would be encompassed by the scope of the application. Accordingly, the scope of the application should be determined not with reference to the above description but with reference to the appended claims.

[0185] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., computer programs) for performing the 6D pose estimation method with fusion attention mechanism in the above embodiments.

[0186] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible storage medium containing or storing a program that can be executed by an instruction execution device or used in conjunction with the same. The program code contained on the computer readable storage medium can be transmitted in any suitable storage medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), etc., or any suitable combination of the above.

[0187] The above computer readable storage medium can be included in the 6D pose estimation device with fusion attention mechanism; or can exist separately and not be assembled into the 6D pose estimation device with fusion attention mechanism.

[0188] The computer-readable storage medium described above can bear one or more programs, when the one or more programs are executed by the 6D pose estimation device fusing attention mechanism, the 6D pose estimation device fusing attention mechanism can be written in one or more programming languages or combinations thereof for computer program codes for executing the operations of the present application, the programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, as an independent software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case involving a remote computer, the remote computer can be connected to the user computer through any kind of network including a local area network (LAN: Local Area Network) or a wide area network (WAN: Wide Area Network), or can be connected to an external computer (for example, through the Internet by using an Internet service provider).

[0189] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than those noted in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0190] The modules involved in the embodiments described in the present application can be implemented in software or in hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0191] The readable storage medium provided by the application is a computer readable storage medium, and the computer readable storage medium stores computer readable program instructions (namely, a computer program) for executing the 6D pose estimation method of the fusion attention mechanism, and can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the computer readable storage medium provided by the application has the same beneficial effects as the 6D pose estimation method of the fusion attention mechanism provided by the above-mentioned embodiments, and will not be repeated here.

[0192] The application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the 6D pose estimation method of the fusion attention mechanism as described above.

[0193] The computer program product provided by the application can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the computer program product provided by the application has the same beneficial effects as the 6D pose estimation method of the fusion attention mechanism provided by the above-mentioned embodiments, and will not be repeated here.

[0194] The above-mentioned is only part of the embodiments of the application, and does not limit the patent scope of the application, and any equivalent structural transformation, direct / indirect application in other related technical fields made by using the content of the application specification and drawings under the technical concept of the application are included in the patent protection scope of the application.

Claims

1. A 6D pose estimation method incorporating an attention mechanism, characterized in that, include: Acquire a depth RGB image of the target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object; The depth RGB image is input into a preset multimodal feature fusion model for processing to obtain the target 6D pose result. The preset multimodal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iteration optimization module. The step of inputting the depth RGB image into a preset multimodal feature fusion model for processing to obtain the target 6D pose result further includes: The semantic segmentation module performs semantic segmentation on the RGB image to obtain a segmentation mask for the target object. The semantic segmentation module crops the segmentation mask from the point cloud converted from the depth image to obtain the target mask point cloud; The feature extraction module extracts color and geometric features from the target mask point cloud to obtain color and geometric features. The feature fusion module uses a cross-attention mechanism to interact with the color features and the geometric features to generate cross-modal cross features. The feature fusion module utilizes a cross-modal deformable attention mechanism to fuse the cross-modal cross features, the color features, and the geometric features to obtain fused features. The pose estimation module performs pose calculation on the fused features to obtain the initial 6D pose. The initial 6D pose is iteratively optimized by the pose iteration optimization module to obtain the target 6D pose result. The step of fusing the cross-modal cross features, the color features, and the geometric features using the feature fusion module with a cross-modal deformable attention mechanism to obtain fused features, wherein the feature fusion module includes deformable convolutional units, an offset prediction network, and an optimization submodule: The cross-modal cross features, the color features, and the geometric features are input into the offset prediction network to generate offset parameters; The spatial position of the convolution kernel in the deformable convolutional unit is adjusted according to the offset parameter to obtain the adjusted deformable convolutional unit. The adjusted deformable convolutional unit is used to perform convolution operations on the cross-modal cross feature, the color feature, and the geometric feature respectively to obtain a first convolution processing result, a second convolution processing result, and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometric feature; The results of the first, second, and third convolutional processing are stitched together pixel by pixel to form an initial fusion feature; The initial fusion features are optimized by the optimization submodule to obtain the fusion features.

2. The method as described in claim 1, characterized in that, The step of performing semantic segmentation on the RGB image using the semantic segmentation module to obtain a segmentation mask for the target object includes: The original resolution is obtained from the RGB image; The encoder of the semantic segmentation module extracts features from the RGB image to generate a first feature map and a second feature map, wherein the scale of the first feature map is 1 / 16 of the original resolution and the scale of the second feature map is 1 / 8 of the original resolution. The semantic segmentation module's decoder upsamples the scale of the first feature map to the scale of the second feature map to obtain the third feature map; The third feature map and the second feature map are fused channel by channel to obtain a fused feature map; The fused feature map is then deconvolved to restore it to its original resolution, resulting in a fourth feature map. The fourth feature map is passed through a Softmax layer to output pixel-level classification probabilities, thus obtaining a segmentation mask for the target object.

3. The method as described in claim 1, characterized in that, The step of cropping the segmentation mask from the point cloud converted from the depth image using the semantic segmentation module to obtain the target mask point cloud includes: Obtain camera intrinsic parameters; The depth image is converted into a disordered point cloud by a preset point cloud conversion algorithm, wherein the disordered point cloud includes a background point cloud and a point cloud corresponding to the target object. The disordered point cloud is transformed into the camera coordinate system based on the camera intrinsic parameters to obtain pixel coordinates; The semantic segmentation module determines the range of pixel coordinates corresponding to the target object in the disordered point cloud based on the pixel region range of the segmentation mask. The point cloud data within the range of the pixel coordinates are filtered to obtain filtered point cloud data. The filtered point cloud data is then denoised to obtain the target mask point cloud.

4. The method as described in claim 1, characterized in that, The step of extracting color features and geometric features from the target mask point cloud through the feature extraction module to obtain color features and geometric features, wherein the feature extraction module includes a color feature extraction submodule and a geometric feature extraction submodule, wherein the color feature extraction submodule includes a ResNet18 network and a compression and activation attention layer; The ResNet18 network is used to perform preliminary feature extraction on the cropped RGB image corresponding to the target mask point cloud to obtain initial image features. The channel dimension of the initial image features is compressed by the compression and activation attention layer to obtain global information for each channel; The global information is then subjected to dimensionality reduction processing to obtain the dimensionality-reduced information; The dimensionality-reduced information is then subjected to dimensionality restoration to obtain the dimensionality-restored information; The information after the dimension is restored is activated by the Sigmoid function to generate attention weights for each channel; The attention weights are multiplied channel by channel by channel of the initial image features to obtain the color features; The geometric feature extraction submodule performs a nonlinear transformation on the three-dimensional coordinates of each point in the target mask point cloud to obtain point-by-point geometric features from low dimension to high dimension. The point-by-point geometric features are globally aggregated to obtain geometric features, wherein the geometric features represent the overall geometric properties of the target mask point cloud.

5. The method as described in claim 1, characterized in that, The step of generating cross-modal cross features by using the cross-attention mechanism through the feature fusion module to interactively process the color features and the geometric features includes: The color features and geometric features are loaded through the feature fusion module to determine the mapping rules of the cross-attention mechanism; According to the mapping rule, the color features are mapped into a Query vector through a preset linear transformation, wherein the Query vector is used to encode the category semantic information of the target object; According to the mapping rule, the point cloud coordinate information in the geometric features is mapped into a Key vector and a Value vector respectively through a preset linear transformation, wherein the Key vector is used to encode geometric position association information and the Value vector is used to encode geometric position feature information. The similarity is calculated between the Query vector and the Key vector. The similarity is normalized using the Softmax function to generate an attention weight map; The attention weight map and the value vector are weighted and summed to obtain a weighted result. A linear transformation is performed on the weighted result to generate cross-modal cross features.

6. The method as described in claim 1, characterized in that, The step of iteratively optimizing the initial 6D pose using the pose iteration optimization module to obtain the target 6D pose result includes: The initial pose matrix is ​​obtained by decomposing the initial 6D pose. The fused features are obtained by performing global feature extraction through global average pooling. The first pose residual is obtained by predicting the global fusion features using a multilayer perceptron. Perform matrix multiplication on the first attitude residual and the initial pose matrix to obtain the corrected pose matrix; The pose error is calculated based on the corrected pose matrix. When the pose error exceeds a preset error threshold, the corrected pose matrix is ​​used as the new initial 6D pose for input. Until the pose error is less than a preset error threshold, the pose corresponding to the current corrected pose matrix is ​​determined as the target 6D pose result.

7. A 6D pose estimation device incorporating an attention mechanism, characterized in that, The device includes: The acquisition module is used to acquire a depth RGB image of a target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object; The result module is used to input the depth RGB image into a preset multimodal feature fusion model for processing to obtain the target 6D pose result. The preset multimodal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iteration optimization module. It is also used to perform semantic segmentation on the RGB image through the semantic segmentation module to obtain a segmentation mask of the target object; to crop the segmentation mask from the point cloud converted from the depth image through the semantic segmentation module to obtain a target mask point cloud; to extract color and geometric features from the target mask point cloud through the feature extraction module to obtain color features and geometric features; to use a cross-attention mechanism to interactively process the color features and geometric features through the feature fusion module to generate cross-modal cross features; to use a cross-modal deformable attention mechanism to fuse the cross-modal cross features, the color features, and the geometric features through the feature fusion module to obtain fused features; and to use the pose estimation module to perform pose calculation on the fused features to obtain an initial 6D pose. The pose iteration optimization module iteratively optimizes the initial 6D pose to obtain the target 6D pose result. The feature fusion module includes deformable convolutional units, an offset prediction network, and an optimization submodule: The cross-modal cross features, color features, and geometric features are input into the offset prediction network to generate offset parameters; the spatial position of the convolution kernels in the deformable convolutional units is adjusted according to the offset parameters to obtain adjusted deformable convolutional units; the adjusted deformable convolutional units are used to perform convolution operations on the cross-modal cross features, color features, and geometric features respectively to obtain a first convolution processing result, a second convolution processing result, and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross features, the second convolution processing result corresponds to the color features, and the third convolution processing result corresponds to the geometric features; the first convolution processing result, the second convolution processing result, and the third convolution processing result are concatenated pixel-by-pixel to form an initial fused feature; the initial fused feature is then optimized by the optimization submodule to obtain the fused feature.

8. A 6D pose estimation device incorporating an attention mechanism, characterized in that, The device includes: a memory, a processor, and a 6D pose estimation program of a fusion attention mechanism stored in the memory and running on the processor, the 6D pose estimation program of the fusion attention mechanism being configured to implement the steps of the 6D pose estimation method of the fusion attention mechanism as described in any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a 6D pose estimation program with a fusion attention mechanism, which, when executed by a processor, implements the steps of the 6D pose estimation method with a fusion attention mechanism as described in any one of claims 1-6.

Images