6D pose estimation method and device fusing attention mechanism, equipment and medium

Abstract

The invention discloses a 6D pose estimation method and device fusing an attention mechanism, equipment and a medium, and relates to the technical field of object space poses, and the method comprises the steps: processing an RGB image and a depth image of a target object through a preset multi-modal feature fusion model, the model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, an attitude estimation module and an attitude iterative optimization module. A target mask point cloud is obtained through semantic segmentation, feature fusion is performed by using a cross attention mechanism and deformable convolution, and a final pose is obtained through pose estimation and iterative optimization, so that the problems of insufficient feature extraction and weak multi-modal feature association in complex scenes such as weak texture and shielding are solved, and the accuracy and robustness of pose estimation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of object space pose, and in particular to a 6D pose estimation method, device and equipment fusing attention mechanism and a medium. BACKGROUND

[0002] Six degrees of freedom (6D) pose estimation is one of the core technologies in the field of computer vision and robotics, and is widely used in logistics and warehousing, manufacturing, autonomous driving, virtual reality and augmented reality, and medical fields. Current 6D pose estimation methods are mainly divided into three categories: instance-level, class-level and unseen object pose estimation. Instance-level methods include traditional geometric methods such as Linemod based on template matching and depth normal vector features, and also include deep learning methods such as GDR-Net that fuse deep learning features. Class-level methods include NOCS that unifies object representation through normalized coordinate space. Unseen object pose estimation methods include MAV6De that proposes self-training unsupervised domain adaptation and enhances the robustness of outdoor scenes through pseudo-labels.

[0003] However, in complex scenes such as weak texture and occlusion, the feature extraction capability of existing methods is insufficient, resulting in insufficient target feature information and affecting the accuracy of pose estimation. Traditional geometric methods are prone to failure when dealing with weak texture objects, while deep learning methods, although improving the feature extraction capability to some extent, still struggle to cope with complex lighting changes and occlusions. Furthermore, in multi-modal feature fusion, simple concatenation or weighted summation is often used, ignoring the semantic association between different modal features. This simple fusion method fails to fully utilize the complementary information in RGB images and depth images, resulting in insufficient accuracy in feature fusion. Therefore, there is an urgent need for a 6D pose estimation method fusing attention mechanism to improve the accuracy and robustness of pose estimation. SUMMARY

[0004] The main purpose of the present application is to provide a 6D pose estimation method fusing attention mechanism, device and equipment and a medium, aiming to solve the technical problem of how to improve the accuracy and robustness of pose estimation.

[0005] To achieve the above purpose, the present application provides a 6D pose estimation method fusing attention mechanism, comprising: obtaining a depth RGB image of a target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object; inputting the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. The step of inputting the depth RGB image into the preset multi-modal feature fusion model for processing to obtain a target 6D pose result further includes: performing semantic segmentation on the RGB image by the semantic segmentation module to obtain a segmentation mask of a target object; cropping the segmentation mask from the point cloud converted from the depth image by the semantic segmentation module to obtain a target mask point cloud; performing color feature extraction and geometric feature extraction on the target mask point cloud by the feature extraction module to obtain color features and geometric features; interacting the color features and the geometric features by the feature fusion module using a cross-attention mechanism to generate cross-modal cross features; fusing the cross-modal cross features, the color features, and the geometric features by the feature fusion module using a cross-modal deformable attention mechanism to obtain fused features; performing pose calculation on the fused features by the pose estimation module to obtain an initial 6D pose; performing iterative optimization on the initial 6D pose by the pose iterative optimization module to obtain a target 6D pose result.

[0006] In an embodiment, the step of performing semantic segmentation on the RGB image by the semantic segmentation module to obtain a segmentation mask of a target object includes: obtaining an original resolution from the RGB image; extracting features from the RGB image by an encoder of the semantic segmentation module to generate a first feature map and a second feature map, wherein the scale of the first feature map is 1 / 16 of the original resolution, and the scale of the second feature map is 1 / 8 of the original resolution; up-sampling the scale of the first feature map to the scale of the second feature map by a decoder of the semantic segmentation module to obtain a third feature map; performing channel-by-channel weighted fusion on the third feature map and the second feature map to obtain a fused feature map; restoring the fused feature map to the original resolution through a deconvolution operation to obtain a fourth feature map; outputting pixel-level classification probabilities of the fourth feature map through a Softmax layer to obtain a segmentation mask of a target object.

[0007] In an embodiment, the step of cropping the segmentation mask from the point cloud converted from the depth image by the semantic segmentation module to obtain a target mask point cloud includes: obtaining camera intrinsic parameters; convert the depth image into an unordered point cloud by a preset point cloud conversion algorithm, wherein the unordered point cloud comprises a background point cloud and a target object corresponding point cloud; convert the unordered point cloud into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates; determine a pixel point coordinate range of the target object corresponding point cloud in the unordered point cloud according to a pixel region range of the segmentation mask by the semantic segmentation module; screen the point cloud data of the pixel coordinates in the pixel point coordinate range to obtain screened point cloud data; perform denoising processing on the screened point cloud data to obtain a target mask point cloud.

[0008] In an embodiment, the step of performing color feature extraction and geometric feature extraction on the target mask point cloud by the feature extraction module to obtain color features and geometric features, the feature extraction module comprises a color feature extraction submodule and a geometric feature extraction submodule, wherein the color feature extraction submodule comprises a ResNet18 network and a compression and excitation attention layer; perform preliminary feature extraction on the cropped RGB image corresponding to the target mask point cloud by the ResNet18 network to obtain initial image features; compress the channel dimension of the initial image features by the compression and excitation attention layer to obtain global information of each channel; perform dimension reduction processing on the global information to obtain reduced dimension information; restore the dimension of the reduced dimension information to obtain dimension restored information; activate the dimension restored information by a Sigmoid function to generate attention weights corresponding to each channel; multiply the attention weights and the initial image features channel by channel to obtain color features; perform nonlinear transformation on the three-dimensional coordinates of each point in the target mask point cloud by the geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension; globally aggregate the point-by-point geometric features to obtain geometric features, wherein the geometric features represent the overall geometric properties of the target mask point cloud.

[0009] In an embodiment, the step of generating cross-modal cross features by the feature fusion module using a cross-attention mechanism to interact the color features and the geometric features comprises: load the color features and the geometric features by the feature fusion module to determine a mapping rule of the cross-attention mechanism; mapping the color feature into a Query vector through a preset linear transformation according to the mapping rule, wherein the Query vector is used to encode category semantic information of the target object; mapping point cloud coordinate information in the geometry feature into a Key vector and a Value vector through a preset linear transformation according to the mapping rule, wherein the Key vector is used to encode geometry position correlation information, and the Value vector is used to encode geometry position feature information; calculating the Query vector and the Key vector to obtain a similarity; normalizing the similarity through a Softmax function to generate an attention weight map; performing weighted summation operation on the attention weight map and the Value vector to obtain a weighted result; performing linear transformation on the weighted result to generate a cross-modal cross feature.

[0010] In an embodiment, the step of fusing the cross-modal cross feature, the color feature and the geometry feature through the feature fusion module using the cross-modal deformable attention mechanism to obtain a fused feature, the feature fusion module includes a deformable convolution unit, an offset prediction network and an optimization sub-module: inputting the cross-modal cross feature, the color feature and the geometry feature into the offset prediction network to generate offset parameters; adjusting the spatial position of a convolution kernel in the deformable convolution unit according to the offset parameters to obtain an adjusted deformable convolution unit; performing convolution operation on the cross-modal cross feature, the color feature and the geometry feature respectively using the adjusted deformable convolution unit to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometry feature; performing pixel-by-pixel splicing on the first convolution processing result, the second convolution processing result and the third convolution processing result to form an initial fused feature; performing feature optimization on the initial fused feature through the optimization sub-module to obtain a fused feature.

[0011] In an embodiment, the step of performing iterative optimization on the initial 6D pose through the pose iterative optimization module to obtain a target 6D pose result includes: disassembling the initial 6D pose to obtain an initial pose matrix; Global feature extraction is performed on the fusion feature through a global average pooling operation to obtain a global fusion feature; The global fusion feature is predicted by a multi-layer perception to obtain a first pose residual; The first pose residual and the initial pose matrix are subjected to matrix multiplication to obtain a corrected pose matrix; The pose error is calculated according to the corrected pose matrix; When the pose error exceeds a preset error threshold, the corrected pose matrix is input as a new initial 6D pose; Until the pose error is less than the preset error threshold, the pose corresponding to the current corrected pose matrix is determined as a target 6D pose result.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a 6D pose estimation device fusing attention mechanism, which comprises: An acquisition module is configured to acquire a depth RGB image of a target object, wherein the depth RGB image comprises an RGB image and a depth image, and the depth image records the actual distance between a sensor and the surface of the target object; A result module is configured to input the RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module; the result module is further configured to perform semantic segmentation on the RGB image by the semantic segmentation module to obtain a segmentation mask of the target object; to crop the segmentation mask from point clouds converted from the depth image by the semantic segmentation module to obtain target mask point clouds; to perform color feature extraction and geometric feature extraction on the target mask point clouds by the feature extraction module to obtain color features and geometric features; to interactively process the color features and the geometric features by the feature fusion module using a cross-attention mechanism to generate cross-modal cross features; to fuse the cross-modal cross features, the color features, and the geometric features by the feature fusion module using a cross-modal deformable attention mechanism to obtain fusion features; to perform pose calculation on the fusion features by the pose estimation module to obtain an initial 6D pose; and to perform iterative optimization on the initial 6D pose by the pose iterative optimization module to obtain a target 6D pose result.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the 6D pose estimation method fusing attention mechanism as described above.

[0014] In addition, to achieve the above object, the application also provides a computer program product, which comprises a computer program, and the computer program realizes the steps of the 6D pose estimation method with the fusion attention mechanism when executed by a processor.

[0015] The application processes the RGB image and the depth image of the target object through a preset multi-modal feature fusion model, which comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. The target mask point cloud is obtained through semantic segmentation, the feature fusion is performed by using the cross attention mechanism and the deformable convolution, and the final pose is obtained through the pose estimation and the iterative optimization, so as to solve the problems of insufficient feature extraction and weak multi-modal feature correlation in complex scenes such as weak texture and occlusion, and improve the accuracy and robustness of the pose estimation. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0017] Figure 1 The flowchart of the first embodiment of the 6D pose estimation method with the fusion attention mechanism of the application is shown in the figure. Figure 2 The structure block diagram of the preset multi-modal feature fusion model of the first embodiment of the 6D pose estimation method with the fusion attention mechanism of the application is shown in the figure. Figure 3 The flowchart of the second embodiment of the 6D pose estimation method with the fusion attention mechanism of the application is shown in the figure. Figure 4 The module structure diagram of the 6D pose estimation device with the fusion attention mechanism of the application is shown in the figure. Figure 5 The device structure diagram of the hardware running environment involved in the 6D pose estimation method with the fusion attention mechanism in the embodiment of the application is shown in the figure.

[0018] The object realization, functional characteristics and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the application, and are not used to limit the application.

[0020] In order to better understand the technical solutions of the application, the following will be described in detail with reference to the drawings and specific embodiments in the specification.

[0021] Current 6D pose estimation methods are mainly divided into three categories: instance-level, class-level and unseen object pose estimation. Instance-level methods include traditional geometric methods such as Linemod based on template matching and depth normal vector features, and also include deep learning methods such as GDR-Net which fuses deep learning features. Class-level methods include NOCS which unifies object representation through normalized coordinate space. Unseen object pose estimation methods include MAV6De which proposes self-training unsupervised domain adaptation and enhances the robustness of outdoor scenes through pseudo-labels. However, in complex scenes such as weak texture and occlusion, the feature extraction capability of existing methods is insufficient, resulting in insufficient target feature information and affecting the accuracy of pose estimation. Traditional geometric methods are prone to failure when dealing with weak texture objects, while deep learning methods, although to some extent, improve the feature extraction capability, but still have difficulty in dealing with complex lighting changes and occlusion conditions, and in multi-modal feature fusion, a simple concatenation or weighted sum is usually used, ignoring the semantic association between different modal features. This simple fusion method cannot fully utilize the complementary information in RGB images and depth images, resulting in insufficient accuracy of feature fusion.

[0022] Therefore, the present application proposes a 6D pose estimation method fusing attention mechanism to solve the above problems. The main solution of the embodiment of the present application is: obtaining a depth RGB image of a target object; inputting the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module; the step of inputting the depth RGB image into the preset multi-modal feature fusion model for processing to obtain the target 6D pose result further includes: performing semantic segmentation on the RGB image through the semantic segmentation module to obtain a segmentation mask of the target object; cropping the point cloud converted from the depth image through the semantic segmentation module to obtain a target mask point cloud; performing color feature extraction and geometric feature extraction on the target mask point cloud through the feature extraction module to obtain color features and geometric features; using a cross-attention mechanism to interactively process the color features and the geometric features through the feature fusion module to generate cross-modal cross features; fusing the cross-modal cross features, the color features and the geometric features through a cross-modal deformable attention mechanism to obtain fused features through the feature fusion module; performing pose solving on the fused features through the pose estimation module to obtain an initial 6D pose; performing iterative optimization on the initial 6D pose through the pose iterative optimization module to obtain the target 6D pose result.

[0023] Based on the above, the embodiment of the present application also provides a 6D pose estimation method fusing attention mechanism, which is described with reference to Figure 1 , Figure 1Fig. 1 is a flowchart of a first embodiment of the 6D pose estimation method with fusion attention mechanism according to the present application. In this embodiment, the 6D pose estimation method with fusion attention mechanism comprises steps S10-S20: Step S10, obtaining a depth RGB image of a target object.

[0024] It should be noted that the depth RGB image is a special kind of image data, which not only contains traditional RGB image information, but also contains corresponding depth image information. It provides color information and spatial distance information of the target object, so that the computer vision system can more comprehensively understand the position and shape of the target object in three-dimensional space.

[0025] Specifically, the RGB image, i.e. the red-green-blue three-channel image, can provide rich color information, which is crucial for subsequent target recognition and feature extraction. Through a high-resolution RGB camera, the appearance features of the target object under different lighting conditions can be captured, including texture, color distribution, and shape details. These detailed information helps to distinguish the target object from the background, especially in complex scenes where the target object may be partially occluded by other objects or similar to the background color. High-quality RGB images can provide sufficient visual clues to help the algorithm more accurately locate and identify the target object. In addition, the acquisition of RGB images also needs to consider the calibration and calibration of the camera to ensure the accuracy of the geometric and color information of the image. The calibration of the camera's internal and external parameters can ensure the correspondence between the image data and the actual physical space, which has a direct impact on the accuracy of subsequent pose estimation. In practical applications, for example, in robot grasping tasks, accurate RGB images can help the robot better understand the appearance of the target object, so as to plan the grasping path more accurately; in the automatic driving scene, RGB images help vehicles to identify obstacles and traffic signs on the road, providing an important basis for safe driving. While the depth image is an image that records the actual distance between the target object surface and the sensor. Unlike the RGB image, each pixel value in the depth image represents the distance between the pixel point and the sensor, rather than color information. Depth images are usually represented in grayscale, where smaller pixel values indicate closer distances, and larger pixel values indicate farther distances. Depth images can provide three-dimensional shape and spatial layout information of the target object, which is crucial for understanding the geometric structure and spatial position of the object. Common sensors for acquiring depth images include Kinect, RealSense, etc. These sensors emit and receive infrared light or other types of signals, measure the signal strength and time delay reflected by the target object surface, and then calculate the actual distance between the target object surface and the sensor. The accuracy and resolution of the sensor directly affect the quality of the depth image, and in turn affect the accuracy of the 6D pose estimation.

[0026] Step S20, input the depth RGB image into the preset multi-modal feature fusion model for processing to obtain a target 6D pose result.

[0027] It should be noted that, as Figure 2 shown in the preset multi-modal feature fusion model structure block diagram, the model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module. The preset multi-modal feature fusion model is a model that comprehensively processes multiple data sources (such as RGB images and depth images), aiming to improve the performance of the task by fusing data of different modalities.

[0028] Specifically, the RGB image is the main source of visual information, containing rich color and texture information, which is crucial for the recognition and localization of target objects. Before inputting the model, the RGB image usually needs to be preprocessed, including adjusting the image size to match the input requirements of the model, normalizing the pixel values to improve the generalization ability of the model, and other operations. The depth image records the actual distance from the sensor to the surface of the target object. This composite data can provide more comprehensive visual information, which helps to more accurately estimate the 6D pose of the target object.

[0029] The role of the semantic segmentation module is to segment the target object region in the RGB image, providing accurate regional information for subsequent feature extraction. This process is usually based on convolutional neural networks (CNN) in deep learning, through learning a large amount of labeled data, the model can automatically identify and segment different objects in the image.

[0030] The feature extraction module is responsible for extracting useful features from the segmented target object region. These features can be color features, texture features, or geometric features, etc. In this embodiment, a deep learning network such as ResNet18 is used, combined with an attention mechanism, to extract color features from the RGB image, while extracting geometric features from the point cloud converted from the depth image. The attention mechanism can dynamically adjust the weights of the feature channels, enhancing the representation ability of the key features of the target object, which is particularly important for pose estimation in complex scenes. The task of the feature fusion module is to effectively fuse the extracted color features and geometric features. This fusion is not a simple superposition, but is carried out through cross-attention mechanisms and deformable convolutions, etc. The cross-attention mechanism allows the two types of features to complement each other, generating more semantically meaningful cross-modal features. Deformable convolution further enhances the adaptability to complex geometric structures, ensuring that the feature fusion process can fully utilize the complementary information in the RGB image and the point cloud.

[0031] The pose estimation module utilizes the fused features to perform a preliminary estimation of the 6D pose. This step typically involves predicting the rotation and translation parameters of the object using the model. Since the fused features already contain rich semantic and geometric information, they can provide more accurate pose estimation. The output of the pose estimation module is an initial 6D pose result, which serves as the basis for subsequent optimization. The 6D pose refers to the position and orientation of an object in a three-dimensional space, where "6D" stands for six degrees of freedom. Specifically, it includes three degrees of freedom for translation (movement along X, Y, and Z axes) and three degrees of freedom for rotation (rotation around X, Y, and Z axes). The 6D pose can accurately describe the spatial position and direction of an object, and is a key piece of information in fields such as robotics, autonomous driving, and virtual reality, enabling precise target positioning and interaction. To further improve the accuracy of pose estimation, the pose iterative optimization module optimizes the initial 6D pose. This step typically involves iteratively adjusting the pose parameters to minimize the error of the target object under the estimated pose. Through multiple iterations, the model can gradually correct errors in the initial pose, ultimately obtaining a more accurate 6D pose result. Through this multi-modal information fusion approach, the model can maintain high pose estimation accuracy in complex scenarios, such as when the target is occluded or the background is complex.

[0032] Further, from the pose estimation accuracy results table on the LineMOD dataset shown in Table 1, it can be seen that the table lists 13 different types of objects, with each object corresponding to a row of data. The accuracy of each method on different objects is given in percentage form.

[0033] Table 1: Pose estimation accuracy results table on the LineMOD dataset As can be seen from the table, the method of the embodiment achieves an accuracy of 96.2% on the "ape" type, and an accuracy of 100% on the "glue" type. As can be seen from the table content, the 6D pose estimation method of the fusion attention mechanism of the embodiment achieves an average accuracy of 98.6% on the LineMOD dataset. Specifically, the method of the embodiment achieves the optimal in the pose estimation of 7 out of 13 targets, which is improved by 10%, 12.3% and 8.7% compared with the traditional RGB method, and is improved by 4.3%, 2.2%, 1.6%, 1.3% and 1.1% compared with the RGB-D methods DenseFusion, NF6D, Uni6D, MaskedFusion and CA6D. Compared with the original network DenseFusion method, the average accuracy is improved by 4.3%, the highest single object accuracy is 100%, and the lowest single object accuracy is improved by 9.2%. The experimental results show that the method of the embodiment has obvious improvement in processing weak texture objects (such as ape, drill, duck, hole), and significantly improves the average accuracy. This shows that the method of the embodiment has significant advantages in complex scenes, and can effectively improve the accuracy of 6D pose estimation. In addition, the method of the embodiment is also optimized in efficiency, reaching 12FPS, which can meet the requirements of real-time tasks.

[0034] In summary, the method of the embodiment performs well on the LineMOD dataset, not only surpassing several other RGB-D methods in accuracy, but also meeting the real-time processing requirements in efficiency.

[0035] The embodiment processes the RGB image and the depth image of the target object through a preset multi-modal feature fusion model, which includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module and a pose iterative optimization module. The target mask point cloud is obtained by semantic segmentation, the feature fusion is performed by using the cross attention mechanism and the deformable convolution, and the final pose is obtained through the pose estimation and the iterative optimization, which solves the problems of insufficient feature extraction and weak multi-modal feature correlation in complex scenes such as weak texture and occlusion, and improves the accuracy and robustness of pose estimation.

[0036] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 3 , the step S40 of the 6D pose estimation method of the fusion attention mechanism further includes steps S201-S207: Step S201, performing semantic segmentation on the RGB image through the semantic segmentation module to obtain the segmentation mask of the target object.

[0037] It should be noted that when performing semantic segmentation, the semantic segmentation module usually employs deep learning models, especially convolutional neural networks (CNN), to learn the boundaries and features of different objects in the image. These models can automatically learn complex patterns to distinguish different objects, enabling pixel-level classification, which is usually accomplished by an encoder-decoder architecture neural network that can process RGB images and output a segmentation mask. The segmentation mask is the direct result of semantic segmentation, which is a binary image of the same size as the input image, where each pixel value indicates whether the pixel belongs to the target object.

[0038] Further, step S201 includes: first, obtaining the original resolution from the RGB image, which is the initial size of the image, usually expressed in pixels, such as 1024x1024 or 2048x2048, etc. The original resolution is crucial for subsequent feature extraction and image processing steps, as it determines the size of the feature map and the accuracy of the final segmentation mask.

[0039] Next, the RGB image is feature-extracted by the encoder of the semantic segmentation module to generate a first feature map and a second feature map. Specifically, the encoder is usually composed of multiple convolutional layers and pooling layers, which can extract useful features from the image and gradually reduce the spatial resolution of the feature map. In this process, two feature maps of different scales are generated: the first feature map has a scale of 1 / 16 of the original resolution, meaning its width and height are 1 / 16 of the original image; the second feature map has a scale of 1 / 8 of the original resolution, with a spatial resolution twice that of the first feature map.

[0040] Then, the first feature map is upsampled to the scale of the second feature map by the decoder of the semantic segmentation module to obtain a third feature map. Specifically, the decoder starts to work, and its role is to upsample the low-resolution feature map to high-resolution. The upsample process usually involves deconvolution or transposed convolution operations, which can increase the size of the feature map while maintaining the feature information.

[0041] Subsequently, the third feature map and the second feature map are channel-by-channel weighted fused to obtain a fused feature map. Specifically, the purpose of this step is to integrate feature information from different scales to obtain a more comprehensive feature representation. Channel-by-channel weighted fusion means that each feature channel is given different weights according to its importance, so that the fused feature map can more accurately represent the object edges and texture details in the image.

[0042] Then, the fused feature map is restored to the original resolution through the deconvolution operation to obtain a fourth feature map. Specifically, the feature map is further refined to match the size of the original image, providing a basis for the final pixel-level classification.

[0043] Finally, the fourth feature map is passed through a Softmax layer to output pixel-level classification probabilities, resulting in a segmentation mask of the target object. Specifically, the Softmax layer is a commonly used activation function that can convert the input non-normalized values into normalized probability distributions, so that each pixel has a probability value corresponding to each class. In this way, it can be determined which object category each pixel belongs to according to these probability values, thereby obtaining the segmentation mask of the target object. In 6D pose estimation, this segmentation mask can be used to accurately crop the point cloud of the target object from the depth image, which is crucial for subsequent feature extraction and pose estimation.

[0044] Step S202, the target mask point cloud is obtained by cropping the segmentation mask from the point cloud converted from the depth image through the semantic segmentation module.

[0045] It should be noted that the depth image provides the distance information from each pixel in the scene to the camera, which is obtained through depth sensors such as Kinect or RealSense devices. Converting the depth image to a point cloud means combining this distance information with the two-dimensional coordinates of the image to generate a set of points in a three-dimensional space, each point containing its position information on the X, Y, Z axes. The cropping process involves applying the segmentation mask to the point cloud data to filter out points belonging to the target object. This step is usually achieved by corresponding the two-dimensional coordinates of the mask to the projected coordinates of the point cloud. Each pixel in the mask corresponds to a set of three-dimensional points in the point cloud, and through this correspondence, it can be determined which points belong to the target object and separate them from the entire point cloud. The resulting target mask point cloud is a set of points that accurately represent the three-dimensional shape of the target object, and these point cloud data will be used for subsequent feature extraction and pose estimation. For example, these point cloud data can be used to extract the geometric features of the object, or combined with the color features in the RGB image for multi-modal feature fusion to improve the accuracy and robustness of pose estimation.

[0046] Further, step S202 includes: first, obtaining the camera intrinsic parameters, including focal length, principal point coordinates, and radial and tangential distortion coefficients, etc. These parameters describe the imaging characteristics of the camera. Camera intrinsic parameters are crucial for subsequent point cloud conversion and coordinate transformation, as they ensure accurate conversion of image coordinates to world coordinates.

[0047] Then, the depth image is converted into an unordered point cloud by a preset point cloud conversion algorithm, where the unordered point cloud includes a background point cloud and a target object corresponding point cloud. Specifically, each pixel value in the depth image represents the distance from the surface of the corresponding position object to the camera, and the preset point cloud conversion algorithm traverses each pixel of the depth image according to the mapping logic of “pixel coordinates-depth value-three-dimensional coordinates”. Taking the row and column numbers of the pixel as a two-dimensional index, combining the obtained camera intrinsic parameters, the depth value corresponding to the pixel is converted into an initial coordinate in the three-dimensional space through the perspective projection formula, and finally an unordered point cloud containing a large number of three-dimensional points is generated. The unordered point cloud is not screened and contains point cloud data of the surface of the target object (target object corresponding point cloud) and point cloud data of background objects in the scene (such as a desktop and a device support) (background point cloud), which need to be further processed to separate the target point cloud.

[0048] Next, the unordered point cloud is converted into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates. Specifically, since the initially generated unordered point cloud may be based on an image pixel coordinate system, coordinate transformation is performed through the rotation matrix and translation vector in the camera intrinsic parameters: taking the camera optical center as the origin, a camera coordinate system (X axis along the horizontal direction of the camera, Y axis along the vertical direction of the camera, and Z axis along the optical axis direction of the camera) is established, the initial coordinates of each point in the unordered point cloud are substituted into the coordinate conversion formula, the image distortion is eliminated and the coordinate offset is corrected, and the accurate three-dimensional coordinates of each point in the camera coordinate system are obtained. The coordinates can directly reflect the position relationship of the point cloud in the real three-dimensional space and provide a spatial position basis for subsequent target point cloud screening.

[0049] Subsequently, the pixel point coordinate range corresponding to the target object in the unordered point cloud is determined by the semantic segmentation module according to the pixel area range of the segmentation mask. Specifically, the segmentation mask is a binary image generated by the semantic segmentation module after processing the RGB image, where the region with a pixel value of 1 corresponds to the target object, and the region with a pixel value of 0 corresponds to the background. The semantic segmentation module first reads the segmentation mask, traverses all pixels with a pixel value of 1 in the mask, records the row and column number range of these pixels (i.e., the pixel area boundary of the target object in the image), and then reversely deduces the pixel point coordinate range belonging to the target object in the unordered point cloud according to the mapping relationship of “image pixel row and column number-point cloud coordinate in camera coordinate system”, to clearly define the spatial boundary of subsequent point cloud screening.

[0050] Then, the point cloud data whose pixel coordinates are within the pixel point coordinate range is screened to obtain screened point cloud data. Specifically, each point in the unordered point cloud is traversed, the three-dimensional coordinates of each point in the camera coordinate system are extracted, and it is judged whether the corresponding image pixel coordinates fall within the pixel point coordinate range determined above: if they fall within the range, the point is determined to be a target object related point cloud, and the three-dimensional coordinate information of the point is retained; if they exceed the range, they are determined to be background point cloud and are eliminated. Through the screening process, the point cloud data corresponding to the target object can be preliminarily separated from the unordered point cloud, greatly reducing the data amount of subsequent processing, and avoiding the interference of background point cloud on the extraction of target features.

[0051] Finally, the screened point cloud data is denoised to obtain a target mask point cloud. Specifically, the screened point cloud may have isolated noise points (such as discrete points caused by depth measurement errors, false points generated by environmental light interference), and a statistical filtering algorithm needs to be used for denoising: a threshold of the number of neighboring points (such as 50) and a distance threshold (such as 0.01 meters) are set, for each point cloud data after screening, the average distance between it and its surrounding neighboring points is calculated, if the average distance exceeds the distance threshold, it is determined to be a noise point and is deleted; if it is within the threshold, the point is retained. After denoising, the target mask point cloud containing only the effective point cloud of the surface of the target object is obtained, which accurately reflects the three-dimensional geometric shape of the target object and provides high-quality geometric information input for the subsequent feature extraction module.

[0052] In step S203, the color feature and the geometric feature are obtained by the feature extraction module through color feature extraction and geometric feature extraction on the target mask point cloud.

[0053] It should be noted that the feature extraction module includes a color feature extraction submodule and a geometric feature extraction submodule, wherein the color feature extraction submodule includes a ResNet18 network and a compression and excitation attention layer, which cooperatively extract color features and geometric features from the target mask point cloud. Color feature extraction focuses on capturing color information on the surface of the target object. This is usually achieved by analyzing the pixel points in the RGB image corresponding to the point cloud. Color features can provide visual texture and color distribution of the object surface, and play an important role in object recognition and differentiation. In the feature extraction process, deep learning models such as convolutional neural networks may be used to automatically learn and extract color features. These models can identify color patterns and texture features that are useful for classification and recognition. Geometric feature extraction focuses on geometric information in point cloud data, such as the spatial relationship between points, the shape and size of the object, etc. Geometric features are usually obtained by analyzing the three-dimensional coordinates of points in the point cloud. These features are crucial for understanding the three-dimensional structure of the object. In the feature extraction process, local geometric descriptors of the point cloud (such as the normal of the point cloud, the curvature, etc.) may be used to capture these geometric information.

[0054] Further, step S203 comprises: first, performing preliminary feature extraction on the cropped RGB image corresponding to the target mask point cloud by the ResNet18 network to obtain initial image features. Specifically, the cropped RGB image only contains the target object region, which can reduce background interference. The ResNet18 network avoids the gradient vanishing problem with its residual connection structure. The cropped RGB image is first down-sampled and feature-extracted by a convolutional layer: the shallow convolutional layer captures detailed features such as edges and textures, and the deep convolutional layer integrates local features to generate abstract semantic features. Finally, the initial image features with uniform output dimensions are output, which cover key semantic information such as color distribution and texture pattern of the target object, laying a foundation for subsequent feature optimization.

[0055] Then, the channel dimension of the initial image features is compressed by the compression and excitation attention layer to obtain global information of each channel. Specifically, different semantics are represented by different channels of the initial image features (e.g., some channels correspond to the surface color of the object, and some correspond to the texture details), and the compression operation is realized by global average pooling: for each channel of the initial image features, the average value of all pixel values in the channel is calculated, and the two-dimensional feature map is converted into a one-dimensional vector, which is the global information of each channel and can reflect the importance of the corresponding channel feature in the entire target region, providing a basis for subsequent weight allocation.

[0056] Subsequently, the global information is processed by dimension reduction to obtain reduced dimension information. Specifically, dimension reduction is realized by a fully connected layer, which compresses the dimension of the global information to 1 / 16 (preset ratio) of the original dimension, reducing the amount of calculation and enhancing the generalization ability of the model. The fully connected layer filters the key features in the global information through linear transformation and activation function, and eliminates redundant data, so that the subsequent processing is more focused on the information effective for pose estimation.

[0057] Then, the reduced dimension information is restored to the original dimension to obtain dimension-restored information. Specifically, another fully connected layer is used to restore the reduced dimension information to a dimension consistent with the channel number of the initial image features, ensuring that the attention weights generated subsequently can be matched with the initial image features channel by channel. This process reconstructs the channel dimension through linear transformation while retaining the key information filtered in the dimension reduction process, maintaining the integrity and effectiveness of the features.

[0058] Then, the dimension-restored information is activated by a Sigmoid function to generate attention weights corresponding to each channel. Specifically, the Sigmoid function maps the dimension-restored information to the 0-1 interval, and the larger the value, the greater the contribution of the corresponding channel feature to the pose estimation. For example, the channel weight of the target object texture clear area is higher, and the channel weight of the background residual or noise is lower, achieving dynamic enhancement of effective feature channels.

[0059] The attention weight is then multiplied by the initial image features channel by channel to obtain color features. Specifically, during the channel-by-channel multiplication process, channel features with high weights are amplified, and channel features with low weights are suppressed, thereby highlighting the key color and texture information of the target object, weakening noise interference, and generating color features with higher recognition, providing high-quality semantic input for multi-modal fusion.

[0060] Then, the three-dimensional coordinates of each point in the target mask point cloud are subjected to a nonlinear transformation by a geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension. Specifically, the geometric feature extraction submodule adopts a multi-layer perception architecture, first normalizes the three-dimensional coordinates of the target mask point cloud to eliminate scale differences in different scenes, and then gradually maps the three-dimensional coordinates to a high-dimensional space through a multi-layer nonlinear activation function (such as a ReLU function) to extract the local geometric properties (such as adjacent point distance and curvature variation) of each point and generate point-by-point geometric features to capture the local structural details of the target object.

[0061] Finally, the point-by-point geometric features are globally aggregated to obtain geometric features, which represent the overall geometric properties of the target mask point cloud. Specifically, global aggregation is achieved through a max-pooling operation: taking the maximum value in each feature dimension for all point-by-point geometric features to integrate the global spatial information of the target mask point cloud and generate geometric features that can represent the overall geometric form (such as contour, size, and symmetric structure) of the target. The geometric features are complementary to the color features and jointly provide semantic and spatial dual information support for subsequent multi-modal feature fusion.

[0062] In step S204, the color features and the geometric features are interactively processed by a feature fusion module using a cross-attention mechanism to generate cross-modal cross features.

[0063] It should be noted that the feature fusion module first loads the color features and the geometric features extracted earlier, clearly defines the core mapping rules of the cross-attention mechanism, takes the semantic information carried by the color features as the core guide, and takes the spatial position information contained in the geometric features as the associated basis to establish a deep correlation between the two modal features through a Query-Key-Value interaction mode.

[0064] Further, step S204 further comprises: first, loading the color feature and the geometric feature by a feature fusion module to determine a mapping rule of the cross-attention mechanism. Specifically, the feature fusion module first performs dimension consistency checking on the two modal features to ensure that the feature dimensions of the color feature (high-dimensional semantic vector) and the geometric feature (high-dimensional spatial vector) are matched, avoiding dimension conflicts in subsequent interaction calculations; then, based on the task requirements (6D pose estimation needs to strengthen the "semantic-spatial" correlation), the mapping rule is determined: the color feature is used as the semantic guide source, responsible for locating the key semantic area of the target object, and the geometric feature is used as the spatial support source, responsible for providing accurate position information, and the corresponding relationship between the Query vector generated from the color feature and the Key and Value vectors generated from the geometric feature is determined, laying a logical foundation for cross-modal interaction.

[0065] Next, the color feature is mapped into a Query vector according to the mapping rule through a preset linear transformation, wherein the Query vector is used to encode the category semantic information of the target object. Specifically, the preset linear transformation is realized by a trained fully connected layer, and the weight matrix of the fully connected layer is obtained by iterative optimization of a large number of 6D pose estimation samples, which can retain and convert the effective information related to the object category and texture attribute in the color feature into a fixed-dimensional vector structure; in the transformation process, the color feature is normalized to eliminate the feature scale difference between different samples, and the finally generated Query vector not only contains the category semantics of "what is the target object" (such as "mechanical parts" and "electronic components"), but also implies the spatial correlation requirement of the semantic area, providing a semantic anchor point for the subsequent matching of the geometric feature.

[0066] Next, the point cloud coordinate information in the geometric feature is mapped into a Key vector and a Value vector according to the mapping rule through a preset linear transformation, wherein the Key vector is used to encode the geometric position correlation information, and the Value vector is used to encode the geometric position feature information. Specifically, for the linear transformation of the Key vector, the relative position relationship (such as the distance and angle between adjacent points) of the point cloud coordinates in the geometric feature is extracted to generate a vector that can reflect the "position correlation", which is convenient for matching the semantic requirement of the Query vector; for the linear transformation of the Value vector, the absolute position details (such as three-dimensional coordinate values and local curvature) of the point cloud coordinates are focused on to retain the spatial accuracy of the geometric feature; the two linear transformation processes are independent, and both adopt the same dimension standard as the Query vector transformation to ensure that the three can be interactively calculated.

[0067] Subsequently, the similarity between the Query vector and the Key vector is calculated. Specifically, the similarity between the two is calculated by point multiplication: the Query vector is multiplied element by element with the Key vector, and the sum is taken to obtain the similarity value of a single Query-Key pair. The larger the value, the more closely the semantic requirements of the Query vector match the position association of the Key vector. All combinations of Query vectors and Key vectors are traversed to generate a similarity matrix, where each element corresponds to the matching degree of a set of "semantic requirements-position association", providing a quantitative basis for subsequent weight allocation.

[0068] Then, the similarity is normalized by the Softmax function to generate an attention weight map. Specifically, the Softmax function normalizes each row of the similarity matrix (corresponding to a single Query vector) so that the sum of the elements in each row is 1, converting the similarity value to a weight value in the 0-1 interval. After normalization, the weight value corresponding to the Key vector that matches the semantic requirements of the Query vector is close to 1, and the weight value corresponding to the Key vector with low relevance is close to 0. The attention weight map accurately identifies the key regions in the geometric features that are related to color semantics, achieving the core goal of semantic-guided space selection.

[0069] The attention weight map and the Value vector are then weighted and summed to obtain a weighted result. Specifically, each Value vector is assigned a corresponding attention weight in the manner of "weight value x corresponding Value vector", and all weighted Value vectors are then summed. In this process, Value vectors with high weights (corresponding to geometric regions with strong semantic associations) have a higher proportion in the sum, while Value vectors with low weights (corresponding to geometric regions with weak semantic associations) have a lower proportion. This process integrates color semantic information into the spatial details of geometric features, generating intermediate features that preliminarily associate the two modal information.

[0070] Finally, the weighted result is linearly transformed to generate a cross-modal cross-feature. Specifically, the linear transformation layer trained in the feature fusion module is used to adjust the dimensions and integrate the features of the weighted result, eliminating any feature redundancy that may have been introduced during the weighting process and strengthening the collaborative representation capabilities of semantics and space. The final cross-modal cross-feature contains both the class semantic information of the color feature and the precise position information of the geometric feature, effectively addressing the information deficiencies of single-modal features and providing high-quality interactive feature support for the subsequent deep fusion of multi-modal features.

[0071] In step S205, the feature fusion module uses the cross-modal deformable attention mechanism to fuse the cross-modal cross-feature, the color feature, and the geometric feature to obtain a fused feature.

[0072] It should be noted that the feature fusion module first loads the cross-modal cross feature, the color feature and the geometric feature, constructs a cross-modal deformable attention fusion framework, the core of the framework is to realize the pixel-level accurate fusion of the three features by dynamically adjusting the convolution receptive field, the feature fusion module includes a deformable convolution unit, an offset prediction network and an optimization submodule.

[0073] Further, step S205 includes: first, inputting the cross-modal cross feature, the color feature and the geometric feature into the offset prediction network to generate offset parameters. Specifically, the offset prediction network is the core control unit of the cross-modal deformable attention mechanism, which is built by using a lightweight multilayer perception mechanism. First, the three input features are dimensionally unified, and the three features are mapped to the same dimensional feature space through 1x1 convolution to avoid prediction bias caused by feature dimension difference; then, 3x3 convolution is used to extract local spatial correlation information of the features to capture key spatial clues such as target object contour and occlusion area; finally, the output layer outputs offset parameters matching the convolution kernel size of the deformable convolution unit, which contains horizontal and vertical position adjustment values, can accurately guide the spatial position correction of the convolution kernel, and ensures that the convolution kernel can adaptively focus on the target effective area.

[0074] Then, the spatial position of the convolution kernel in the deformable convolution unit is adjusted according to the offset parameters to obtain an adjusted deformable convolution unit. Specifically, the initial convolution kernel of the deformable convolution unit adopts a fixed grid distribution, and after receiving the offset parameters, the offset parameters are converted into the actual position offset of the convolution kernel sampling points through a coordinate mapping algorithm; for example, when there is local occlusion of the target object, the offset parameters will make the convolution kernel sampling points of the corresponding area offset to the unoccluded area to avoid the invalid feature area caused by the occlusion; at the same time, boundary check and interpolation processing are performed on the adjusted sampling point position to ensure that the sampling points always fall within the feature map range, avoiding feature missing caused by coordinate out-of-bounds, and finally forming an adjusted deformable convolution unit adapted to the spatial distribution of the target object.

[0075] Then, the adjusted deformable convolution unit is used to perform convolution operations on the cross-modal cross feature, the color feature and the geometric feature respectively to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometric feature. Specifically, for the cross-modal cross feature, the adjusted convolution kernel focuses on the region with close semantic-spatial correlation, strengthens the interactive representation of the information of the two modalities, and generates the first convolution processing result rich in associated information; for the color feature, the convolution kernel focuses on capturing the key color and texture details of the target object and suppressing background noise interference, and outputs the second convolution processing result highlighting the semantic attributes; for the geometric feature, the convolution kernel accurately locates the local structure and global contour of the target object, retains the spatial position accuracy, and generates the third convolution processing result with spatial integrity. The three convolution processing results not only retain the core information of each modal feature, but also realize the focusing on the effective region of the target through the dynamic convolution kernel.

[0076] Then, the first convolution processing result, the second convolution processing result and the third convolution processing result are pixel-by-pixel spliced to form an initial fusion feature. Specifically, the pixel-by-pixel splicing stacks the three convolution processing results in the channel dimension based on the spatial coordinates of the feature map: for each pixel position in the feature map, the pixel feature vector of the first convolution processing result, the pixel feature vector of the second convolution processing result and the pixel feature vector of the third convolution processing result are spliced in order to form a pixel feature vector with a dimension being the sum of the dimensions of the three features; through this operation, each pixel contains cross-modal associated information, semantic color information and spatial geometric information at the same time, and the initial fusion feature with coexistence of multi-modal information is constructed, providing a comprehensive feature basis for subsequent optimization.

[0077] Finally, the initial fusion feature is optimized by the optimization sub-module to obtain a fusion feature. Specifically, the optimization sub-module includes a batch normalization layer, a ReLU activation layer and a 1x1 convolution layer: the batch normalization layer normalizes each channel of the initial fusion feature to eliminate training deviation caused by feature value distribution difference and accelerate feature convergence; the ReLU activation layer suppresses negative invalid features in the initial fusion feature through nonlinear transformation to enhance the expression ability of positive effective features; and the 1x1 convolution layer performs dimension compression and information integration on the optimized feature to eliminate redundant information between channels, adjusts the feature dimension to adapt to the dimension of the subsequent pose estimation network, and finally generates the fusion feature with semantic recognition and spatial accuracy, providing high-quality feature input for pose calculation.

[0078] In step S206, the fusion feature is calculated by the pose estimation module to obtain an initial 6D pose.

[0079] It should be noted that the pose estimation module takes the fused features after multimodal fusion as input and completes the initial 6D pose calculation through a progressive process of feature mapping, parameter regression, coordinate transformation and error calculation.

[0080] Specifically, the pose estimation module inputs the fused features into a pre-defined pose regression network. This network uses a multilayer perceptron as its core architecture. First, a fully connected layer transforms the dimensionality of the fused features, mapping the high-dimensional fused features to a feature space adapted for pose parameter regression. Then, a non-linear activation function (such as ReLU) is used to enhance the non-linear expressive power of the features, highlighting key pose-related features (such as the symmetrical structure of the target object and key points of the local contour). Finally, the output layer outputs three sets of core parameters: rotation quaternions. Translation vector and confidence level The rotation quaternion and translation vector together constitute the initial prediction result of the 6D pose. The confidence level is used to quantify the reliability of the prediction result. The higher the confidence level, the stronger the accuracy of the corresponding pose prediction.

[0081] Next, the rotation quaternions obtained from the regression are formatted and converted into rotation matrices to adapt to subsequent coordinate transformations. The standard formula for quaternion-to-rotation-matrix conversion is used to achieve the mapping from four-dimensional quaternions to three-dimensional rotation matrices, ensuring that the rotation information can be directly used for point cloud coordinate transformations. The specific formula is as follows: in, This represents the transformed 3×3 rotation matrix, used to describe the spatial rotation state of the target object. Representing rotation quaternions respectively The four components satisfy The constraints ensure that the object's dimensions remain unchanged during the rotation matrix.

[0082] Subsequently, the predicted point cloud is calculated through coordinate transformation to establish the association between the fused features and the 3D shape of the target object. The pose estimation module first loads the 3D model of the target object and samples the point cloud (this point cloud is the standard shape point cloud of the object in the world coordinate system). ,Include (3D sampling points), and then using the transformed rotation matrix Translation vector obtained from regression The standard point cloud is transformed into three-dimensional coordinates under the predicted pose to obtain the predicted point cloud. The specific formula is as follows: in, This represents the target point cloud under the predicted pose, where each element is the three-dimensional coordinate of the predicted point. ), represents a target three-dimensional model sampling point cloud in a world coordinate system, each element being a three-dimensional coordinate in the world coordinate system represents a transpose matrix of a rotation matrix, represents a translation vector, specifically, a position offset of the target object in three directions in the camera coordinate system .

[0083] Then, for symmetric and asymmetric target objects, pose loss is calculated respectively to verify the rationality of the prediction results. For asymmetric objects, the point-by-point Euclidean distance between the predicted point cloud and the real pose point cloud is directly calculated and averaged to obtain the pose loss , the specific formula is: wherein, represents an average value operation, represents an L2 norm, represents a target point cloud in a real pose, each element being a three-dimensional coordinate of a real point . For symmetric objects, because different rotation angles may correspond to the same shape, the K-nearest neighbor algorithm is used to find the nearest neighbor correspondence between the predicted point cloud and the real point cloud, and the average Euclidean distance is calculated after re-aligning the real point cloud to avoid loss calculation deviation caused by symmetric structure.

[0084] Finally, the weighted loss is calculated by combining the confidence and the preset balance parameter , and the initial 6D pose meeting the accuracy requirement is selected according to the weighted loss, and the specific weighted loss calculation formula is: wherein, represents the number of fusion features participating in loss calculation, represents the pose loss corresponding to the th fusion feature, represents the confidence corresponding to the th fusion feature, represents a balance parameter, which is 0.01 in this embodiment, and is used to prevent loss calculation distortion caused by too low confidence, represents a natural logarithm operation. When the weighted loss is less than a preset loss threshold (such as 0.05), it is determined that the rotation matrix and the translation vector obtained by the regression are effective results, and the two together constitute an initial 6D pose and are output to a subsequent iterative optimization module.

[0085] Step S207, iteratively optimizing the initial 6D pose through a pose iterative optimization module to obtain a target 6D pose result. ​

[0086] It should be noted that the pose iterative optimization module is responsible for fine adjustment of the initial estimated 6D pose to improve the accuracy of pose estimation. This module usually adopts an iterative method to gradually reduce the pose estimation error through multiple loop optimization processes.

[0087] Further, step S207 includes: first, disassembling according to the initial 6D pose to obtain the initial pose matrix, the initial 6D pose including the rotation matrix and the translation vector , disassembling and then combining to obtain the initial pose matrix , which is specifically expressed as: wherein, represents the initial rotation matrix, describing the rotation state of the target object around the origin of the camera coordinate system, represents the initial translation vector, including the position offset of the target object in direction, 0 and 1 are homogeneous coordinate identifiers, ensuring that the matrix can be used for three-dimensional coordinate transformation.

[0088] Next, global feature extraction is performed on the fusion features through a global average pooling operation to obtain global fusion features. Specifically, the global average pooling calculates the pixel average value of each channel of the fusion feature map, compressing the two-dimensional feature map into a one-dimensional vector. The specific formula for calculating the global fusion features is as follows: wherein, represents the global fusion features, represents the pixel value of the th channel, the th channel, and the th channel in the fusion feature map, represents the average operation.

[0089] Next, the global fusion features are predicted through a multi-layer perception to obtain the first pose residual. Specifically, the multi-layer perception includes 2 fully connected layers and ReLU activation functions. First, the global fusion features are mapped to a 6-dimensional space (3-dimensional rotation residual + 3-dimensional translation residual), and then an orthogonalization process is performed to ensure the effectiveness of the rotation residual, outputting the first pose residual .

[0090] Then, the first pose residual and the initial pose matrix are subjected to matrix multiplication to obtain the corrected pose matrix , and the specific formula is: .

[0091] According to the modified pose matrix, the pose error is calculated. Specifically, the point cloud of the target object three-dimensional model is converted into the predicted point cloud in the camera coordinate system through the modified pose matrix . Each point in the point cloud is At the same time, the real scene point cloud is obtained The pose error is calculated, and the specific formula is: Wherein, is the coordinate of the first point, is the coordinate of the first point, is the coordinate of the first point, is the coordinate of the first point, is the coordinate of the first point, is the pose error, represents the total number of points in the point cloud.

[0092] Finally, when the pose error exceeds the preset error threshold, the modified pose matrix is input as a new initial 6D pose, and the residual prediction and correction process is repeated; until the pose error is less than the preset error threshold, the pose corresponding to the current modified pose matrix is determined as the target 6D pose result.

[0093] The embodiment first acquires the segmentation mask of the target object using the semantic segmentation module, and then crops the target mask point cloud. Next, the color feature and the geometric feature are obtained through the feature extraction module, and the fused feature is generated through the feature fusion module. Finally, the accurate 6D pose result is obtained through the pose estimation and iterative optimization module. The RGB image and the depth image are processed through the multi-modal feature fusion model, and the accurate pose estimation of the target object is realized, which improves the accuracy and robustness of the pose estimation.

[0094] Based on the first embodiment of the present application, the present application also provides a 6D pose estimation device fusing attention mechanism, please refer to Figure 4 The device comprises: An acquisition module 10 is configured to acquire a depth RGB image of a target object, wherein the depth RGB image comprises an RGB image and a depth image, and the depth image records the actual distance between a sensor and the surface of the target object. ​The result module 20 is configured to input the depth RGB image into a preset multi-modal feature fusion model for processing to obtain a target 6D pose result, wherein the preset multi-modal feature fusion model comprises a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iterative optimization module; is further configured to perform semantic segmentation on the RGB image by using the semantic segmentation module to obtain a segmentation mask of a target object; perform cropping on the segmentation mask from point cloud converted from the depth image by using the semantic segmentation module to obtain a target mask point cloud; perform color feature extraction and geometric feature extraction on the target mask point cloud by using the feature extraction module to obtain color features and geometric features; perform interactive processing on the color features and the geometric features by using a cross-attention mechanism of the feature fusion module to generate cross-modal cross features; perform fusion on the cross-modal cross features, the color features, and the geometric features by using a cross-modal deformable attention mechanism of the feature fusion module to obtain fused features; perform pose calculation on the fused features by using the pose estimation module to obtain an initial 6D pose; and perform iterative optimization on the initial 6D pose by using a pose iterative optimization module to obtain the target 6D pose result.

[0095] The 6D pose estimation device with the fusion attention mechanism provided in the application adopts the 6D pose estimation method with the fusion attention mechanism in the above embodiments, and can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the 6D pose estimation device with the fusion attention mechanism provided in the application has the same beneficial effects as the 6D pose estimation method with the fusion attention mechanism provided in the above embodiments, and other technical features of the 6D pose estimation device with the fusion attention mechanism are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0096] In an embodiment, the result module 20 is further configured to obtain an original resolution according to the RGB image. The encoder of the semantic segmentation module is configured to perform feature extraction on the RGB image to generate a first feature map and a second feature map, wherein the scale of the first feature map is 1 / 16 of the original resolution, and the scale of the second feature map is 1 / 8 of the original resolution; the decoder of the semantic segmentation module is configured to upsample the scale of the first feature map to the scale of the second feature map to obtain a third feature map; the third feature map and the second feature map are configured to be fused by channel-by-channel weighting to obtain a fused feature map; the fused feature map is configured to be restored to the original resolution by deconvolution operation to obtain a fourth feature map; and the fourth feature map is configured to output pixel-level classification probability by a Softmax layer to obtain a segmentation mask of a target object.

[0097] In an embodiment, the result module 20 is further configured to obtain camera intrinsic parameters; convert the depth image into an unordered point cloud by a preset point cloud conversion algorithm, wherein the unordered point cloud comprises a background point cloud and a target object corresponding point cloud; convert the unordered point cloud into a camera coordinate system according to the camera intrinsic parameters to obtain pixel coordinates; determine a pixel point coordinate range corresponding to the target object in the unordered point cloud according to a pixel region range of the segmentation mask by the semantic segmentation module; screen point cloud data in the pixel coordinate range to obtain screened point cloud data; and perform denoising processing on the screened point cloud data to obtain a target mask point cloud.

[0098] In an embodiment, the result module 20 is further configured to perform preliminary feature extraction on the target mask point cloud corresponding cropped RGB image by the ResNet18 network to obtain initial image features; compress the channel dimension of the initial image features by the compression and excitation attention layer to obtain global information of each channel; perform dimension reduction processing on the global information to obtain reduced information; perform dimension recovery on the reduced information to obtain dimension-recovered information; activate the dimension-recovered information by a Sigmoid function to generate attention weights corresponding to each channel; multiply the attention weights and the initial image features channel by channel to obtain color features; perform nonlinear transformation on three-dimensional coordinates of each point in the target mask point cloud by the geometric feature extraction submodule to obtain point-by-point geometric features from low dimension to high dimension; and perform global aggregation on the point-by-point geometric features to obtain geometric features, wherein the geometric features represent the overall geometric properties of the target mask point cloud.

[0099] In an embodiment, the result module 20 is further configured to load the color features and the geometric features by the feature fusion module to determine a mapping rule of cross-attention mechanism; map the color features into a Query vector by a preset linear transformation according to the mapping rule, wherein the Query vector is used to encode category semantic information of the target object; map point cloud coordinate information in the geometric features into a Key vector and a Value vector by a preset linear transformation according to the mapping rule, wherein the Key vector is used to encode geometric position correlation information and the Value vector is used to encode geometric position feature information; calculate the Query vector and the Key vector to obtain a similarity; perform normalization processing on the similarity by a Softmax function to generate an attention weight map; perform weighted summation operation on the attention weight map and the Value vector to obtain a weighted result; and perform linear transformation on the weighted result to generate a cross-modal cross-feature.

[0100] In an embodiment, the result module 20 is further configured to input the cross-modal cross-feature, the color feature and the geometry feature into the offset prediction network to generate an offset parameter; adjust a spatial position of a convolution kernel in the deformable convolution unit according to the offset parameter to obtain an adjusted deformable convolution unit; perform convolution operation on the cross-modal cross-feature, the color feature and the geometry feature respectively by using the adjusted deformable convolution unit to obtain a first convolution processing result, a second convolution processing result and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross-feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometry feature; perform pixel-by-pixel splicing on the first convolution processing result, the second convolution processing result and the third convolution processing result to form an initial fusion feature; and perform feature optimization on the initial fusion feature by using the optimization submodule to obtain a fusion feature.

[0101] In an embodiment, the result module 20 is further configured to disassemble the initial 6D pose to obtain an initial pose matrix; perform global feature extraction on the fusion feature by using a global average pooling operation to obtain a global fusion feature; predict a first attitude residual by using a multi-layer perception machine; perform matrix multiplication operation on the first attitude residual and the initial pose matrix to obtain a corrected pose matrix; calculate a pose error according to the corrected pose matrix; when the pose error exceeds a preset error threshold, input the corrected pose matrix as a new initial 6D pose; and until the pose error is less than the preset error threshold, determine a pose corresponding to the current corrected pose matrix as a target 6D pose result.

[0102] The present application provides a 6D pose estimation device with a fusion attention mechanism, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the 6D pose estimation method with a fusion attention mechanism in the above-mentioned embodiment one.

[0103] Reference will be made to the following description of the embodiments of the present application Figure 5This document illustrates a schematic diagram of a 6D pose estimation device suitable for implementing the fusion attention mechanism in the embodiments of this application. The 6D pose estimation device with fusion attention mechanism in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The 6D pose estimation device with fusion attention mechanism shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0104] like Figure 5 As shown, a 6D pose estimation device with an attention-integrated mechanism may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the 6D pose estimation device with the attention-integrated mechanism. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the 6D pose estimation device with fused attention mechanisms to wirelessly or wiredly communicate with other devices to exchange data. Although various 6D pose estimation devices with fused attention mechanisms are shown in the figures, it should be understood that implementation or possession of all of them is not required. More or fewer of them may be implemented alternatively.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0106] The fusion attention mechanism 6D pose estimation device provided by the present application adopts the fusion attention mechanism 6D pose estimation method in the above-mentioned embodiments, and can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the fusion attention mechanism 6D pose estimation device provided by the present application has the same beneficial effects as the fusion attention mechanism 6D pose estimation method provided by the above-mentioned embodiments, and other technical features in the fusion attention mechanism 6D pose estimation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0107] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0108] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0109] The present application provides a computer-readable storage medium having stored thereon computer-readable program instructions (i.e. computer program) for executing the fusion attention mechanism 6D pose estimation method in the above-mentioned embodiments.

[0110] The computer readable storage medium provided in the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor, or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible storage medium containing or storing a program that can be executed by or used in conjunction with an instruction execution device. The program code contained on the computer readable storage medium can be transmitted in any suitable storage medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0111] The above computer readable storage medium can be included in the 6D pose estimation device fusing attention mechanism, or can exist separately without being assembled into the 6D pose estimation device fusing attention mechanism.

[0112] The above computer readable storage medium carries one or more programs, which, when executed by the 6D pose estimation device fusing attention mechanism, can cause the 6D pose estimation device fusing attention mechanism to execute computer program code in one or more programming languages or combinations thereof for performing the operations of the present application. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on a user computer, partially on a user computer, as a separate software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the methods and computer program products according to the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or can sometimes be executed in reverse order, depending on the functionality involved. It will also be noted that each block of the block diagrams and / or flowcharts, and combinations thereof, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or combinations of hardware and software.

[0114] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0115] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the 6D pose estimation method of the fusion attention mechanism described above, and can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the 6D pose estimation method of the fusion attention mechanism provided by the above-mentioned embodiments, and will not be repeated here.

[0116] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the 6D pose estimation method of the fusion attention mechanism as described above.

[0117] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy and robustness of pose estimation. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the 6D pose estimation method of the fusion attention mechanism provided by the above-mentioned embodiments, and will not be repeated here.

[0118] The above only describes some embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A 6D pose estimation method incorporating an attention mechanism, characterized in that, include: Acquire a depth RGB image of the target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object; The depth RGB image is input into a preset multimodal feature fusion model for processing to obtain the target 6D pose result. The preset multimodal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iteration optimization module. The step of inputting the depth RGB image into a preset multimodal feature fusion model for processing to obtain the target 6D pose result further includes: The semantic segmentation module performs semantic segmentation on the RGB image to obtain a segmentation mask for the target object. The semantic segmentation module crops the segmentation mask from the point cloud converted from the depth image to obtain the target mask point cloud; The feature extraction module extracts color and geometric features from the target mask point cloud to obtain color and geometric features. The feature fusion module uses a cross-attention mechanism to interact with the color features and the geometric features to generate cross-modal cross features. The feature fusion module utilizes a cross-modal deformable attention mechanism to fuse the cross-modal cross features, the color features, and the geometric features to obtain fused features. The pose estimation module performs pose calculation on the fused features to obtain the initial 6D pose. The initial 6D pose is iteratively optimized using the pose iteration optimization module to obtain the target 6D pose result.

2. The method as described in claim 1, characterized in that, The step of performing semantic segmentation on the RGB image using the semantic segmentation module to obtain a segmentation mask for the target object includes: The original resolution is obtained from the RGB image; The semantic segmentation module extracts features from the RGB image using its encoder to generate a first feature map and a second feature map, wherein the scale of the first feature map is 1 / 16 of the original resolution and the scale of the second feature map is 1 / 8 of the original resolution. The semantic segmentation module's decoder upsamples the scale of the first feature map to the scale of the second feature map to obtain the third feature map; The third feature map and the second feature map are fused channel by channel to obtain a fused feature map; The fused feature map is then deconvolved to restore it to its original resolution, resulting in a fourth feature map. The fourth feature map is passed through a Softmax layer to output pixel-level classification probabilities, thus obtaining a segmentation mask for the target object.

3. The method as described in claim 1, characterized in that, The step of cropping the segmentation mask from the point cloud converted from the depth image using the semantic segmentation module to obtain the target mask point cloud includes: Obtain camera intrinsic parameters; The depth image is converted into a disordered point cloud by a preset point cloud conversion algorithm, wherein the disordered point cloud includes a background point cloud and a point cloud corresponding to the target object. The disordered point cloud is transformed into the camera coordinate system based on the camera intrinsic parameters to obtain pixel coordinates; The semantic segmentation module determines the range of pixel coordinates corresponding to the target object in the disordered point cloud based on the pixel region range of the segmentation mask. The point cloud data within the range of the pixel coordinates are filtered to obtain filtered point cloud data. The filtered point cloud data is then denoised to obtain the target mask point cloud.

4. The method as described in claim 1, characterized in that, The step of extracting color features and geometric features from the target mask point cloud through the feature extraction module to obtain color features and geometric features, wherein the feature extraction module includes a color feature extraction submodule and a geometric feature extraction submodule, wherein the color feature extraction submodule includes a ResNet18 network and a compression and activation attention layer; The ResNet18 network is used to perform preliminary feature extraction on the cropped RGB image corresponding to the target mask point cloud to obtain initial image features. The channel dimension of the initial image features is compressed by the compression and activation attention layer to obtain global information for each channel; The global information is then subjected to dimensionality reduction processing to obtain the dimensionality-reduced information; The dimensionality-reduced information is then subjected to dimensionality restoration to obtain the dimensionality-restored information; The information after the dimension is restored is activated by the Sigmoid function to generate attention weights for each channel; The attention weights are multiplied channel by channel by channel of the initial image features to obtain the color features; The geometric feature extraction submodule performs a nonlinear transformation on the three-dimensional coordinates of each point in the target mask point cloud to obtain point-by-point geometric features from low dimension to high dimension. The point-by-point geometric features are globally aggregated to obtain geometric features, wherein the geometric features represent the overall geometric properties of the target mask point cloud.

5. The method as described in claim 1, characterized in that, The step of generating cross-modal cross features by using the cross-attention mechanism through the feature fusion module to interactively process the color features and the geometric features includes: The color features and geometric features are loaded through the feature fusion module to determine the mapping rules of the cross-attention mechanism; According to the mapping rule, the color features are mapped into a Query vector through a preset linear transformation, wherein the Query vector is used to encode the category semantic information of the target object; According to the mapping rule, the point cloud coordinate information in the geometric features is mapped into a Key vector and a Value vector respectively through a preset linear transformation, wherein the Key vector is used to encode geometric position association information and the Value vector is used to encode geometric position feature information. The similarity is calculated by analyzing the Query vector and the Key vector. The similarity is normalized using the Softmax function to generate an attention weight map; The attention weight map and the value vector are weighted and summed to obtain a weighted result. A linear transformation is performed on the weighted result to generate cross-modal cross features.

6. The method as described in claim 1, characterized in that, The step of fusing the cross-modal cross features, the color features, and the geometric features using the feature fusion module with a cross-modal deformable attention mechanism to obtain fused features, wherein the feature fusion module includes deformable convolutional units, an offset prediction network, and an optimization submodule: The cross-modal cross features, the color features, and the geometric features are input into the offset prediction network to generate offset parameters; The spatial position of the convolution kernel in the deformable convolutional unit is adjusted according to the offset parameter to obtain the adjusted deformable convolutional unit. The adjusted deformable convolutional unit is used to perform convolution operations on the cross-modal cross feature, the color feature, and the geometric feature respectively to obtain a first convolution processing result, a second convolution processing result, and a third convolution processing result, wherein the first convolution processing result corresponds to the cross-modal cross feature, the second convolution processing result corresponds to the color feature, and the third convolution processing result corresponds to the geometric feature; The results of the first, second, and third convolutional processing are stitched together pixel by pixel to form an initial fusion feature; The initial fusion features are optimized by the optimization submodule to obtain the fusion features.

7. The method as described in claim 1, characterized in that, The step of iteratively optimizing the initial 6D pose using the pose iteration optimization module to obtain the target 6D pose result includes: The initial pose matrix is ​​obtained by decomposing the initial 6D pose. The fused features are extracted globally by performing a global average pooling operation to obtain the global fused features; The first pose residual is obtained by predicting the global fused features using a multilayer perceptron. Perform matrix multiplication on the first attitude residual and the initial pose matrix to obtain the corrected pose matrix; The pose error is calculated based on the corrected pose matrix. When the pose error exceeds a preset error threshold, the corrected pose matrix is ​​used as the new initial 6D pose for input. Until the pose error is less than a preset error threshold, the pose corresponding to the current corrected pose matrix is ​​determined as the target 6D pose result.

8. A 6D pose estimation device incorporating an attention mechanism, characterized in that, The device includes: The acquisition module is used to acquire a depth RGB image of a target object, wherein the depth RGB image includes an RGB image and a depth image, and the depth image records the actual distance between the sensor and the surface of the target object; The result module is used to input the depth RGB image into a preset multimodal feature fusion model for processing to obtain the target 6D pose result. The preset multimodal feature fusion model includes a semantic segmentation module, a feature extraction module, a feature fusion module, a pose estimation module, and a pose iteration optimization module. It is also used to perform semantic segmentation on the RGB image through the semantic segmentation module to obtain a segmentation mask of the target object; to crop the segmentation mask from the point cloud converted from the depth image through the semantic segmentation module to obtain a target mask point cloud; to extract color and geometric features from the target mask point cloud through the feature extraction module to obtain color features and geometric features; to use a cross-attention mechanism to interactively process the color features and geometric features through the feature fusion module to generate cross-modal cross features; to fuse the cross-modal cross features, color features, and geometric features through a cross-modal deformable attention mechanism through the feature fusion module to obtain fused features; to calculate the pose of the fused features through the pose estimation module to obtain an initial 6D pose; and to iteratively optimize the initial 6D pose through the pose iteration optimization module to obtain the target 6D pose result.

9. A 6D pose estimation device incorporating an attention mechanism, characterized in that, The device includes: a memory, a processor, and a 6D pose estimation program of a fusion attention mechanism stored in the memory and running on the processor, the 6D pose estimation program of the fusion attention mechanism being configured to implement the steps of the 6D pose estimation method of the fusion attention mechanism as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a 6D pose estimation program with a fusion attention mechanism, which, when executed by a processor, implements the steps of the 6D pose estimation method with a fusion attention mechanism as described in any one of claims 1-7.

Images