Three-dimensional object pose estimation method based on component segmentation

Abstract

The invention relates to the field of computer vision, and discloses a three-dimensional object pose estimation method based on component segmentation, which comprises the following steps: preprocessing an RGB-D image to obtain a color image and depth information; the method comprises the following steps: performing multi-level component segmentation on a three-dimensional object to obtain component-level representation of the object; the segmented component information is fused with the RGB-D image, and multi-scale RGB-D image features of the object are obtained; enhancing relation features among the components, and extracting global point cloud features of the object; fusing the multi-scale RGB-D image features and the global point cloud features through a bidirectional information flow network to obtain comprehensive feature representation; using a component adaptive regression module to predict pose information of an object and key components thereof; and a final object 6D pose is obtained through an accurate point pair optimization algorithm and a trajectory correction strategy. According to the method, more accurate and more robust pose estimation of the object is realized, and the method is particularly excellent in performance under the conditions of complex scenes and partial shielding.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to a method for estimating the pose of a 3D object based on component segmentation. Background Technology

[0002] In practical applications, acquiring the 3D information of the target object is crucial. For example, in robotic arm grasping tasks, identifying the object's 6D pose (3D localization and 3D orientation) provides useful information for grasping and motion planning. The challenge in workpiece pose estimation lies in the fact that workpieces are weakly textured objects, easily affected by lighting and reflection, meaning the texture reflected in a 2D image may not be the true texture of the 3D object's surface. Furthermore, changes in image resolution can lead to significant deviations in the calculated texture, making it difficult for feature extraction algorithms to recognize. In addition, the actual pose estimation process is also affected by objective factors such as occlusion and multiple targets in complex scenes.

[0003] Existing methods for 6D pose estimation mainly address the following: First, by matching local features extracted from the image with features in the object's 3D model, the 6D pose of the object can be recovered using the 2D-3D correspondence. However, this approach does not handle textureless objects well. Second, by discretizing the pose space, the 6D pose estimation problem is transformed into a pose classification or pose regression problem. However, these methods struggle to achieve high-precision pose estimation.

[0004] Traditional object segmentation methods typically focus only on segmentation accuracy without considering how the segmentation results affect subsequent pose estimation. Existing partial segmentation methods, such as SAMPart3D, can segment object parts at multiple granularity levels but do not directly provide pose information. Meanwhile, existing pose estimation methods, such as CPF6D, can utilize part-level features to improve pose estimation accuracy but lack the ability to precisely segment objects.

[0005] Therefore, there is an urgent need for a technical solution that can combine accurate segmentation of 3D objects with high-precision pose estimation to improve the ability to recognize and manipulate objects in complex scenes. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a 3D object pose estimation method based on component segmentation, solving the issues that existing methods cannot effectively handle weakly textured objects such as workpieces, cannot accurately estimate workpiece pose, and have low adaptability. The technical solution is as follows:

[0007] A method for 3D object pose estimation based on component segmentation includes the following steps:

[0008] Step 1: Preprocess the input RGB-D image to obtain the color image and depth information of the object;

[0009] Step 2: Perform multi-level component segmentation of the 3D object based on a multi-granularity segmentation network, and obtain the component-level representation of the RGB and depth maps of the segmented object through the preprocessed color image and depth information;

[0010] Step 3: Fuse the segmented part-level representation with the RGB-D image, and obtain the multi-scale RGB-D image features of the object through a feature extraction network;

[0011] Step 4: Use the point cloud feature enhancement module to enhance the relationship features between components and extract the global point cloud features of the object; the point cloud feature enhancement module adopts an improved structure based on the self-attention algorithm, which aggregates local features and updates geometric features through edge convolution operation of the input point cloud features, and then uses the query weight matrix to interact with the key value features containing normalization operation to obtain the global point cloud features.

[0012] Step 5: Fuse multi-scale RGB-D image features and global point cloud features through a bidirectional information flow network to obtain a comprehensive feature representation;

[0013] Step 6: Use the component adaptive regression module to predict the pose information of the object and its key components;

[0014] Step 7: Obtain the final 6D pose of the object through precise point-to-point optimization algorithm and trajectory correction strategy.

[0015] The beneficial effects of this invention are:

[0016] This invention achieves more accurate pose estimation for objects by combining precise component segmentation and high-precision pose estimation capabilities, especially for objects with weak textures and complex scenes with partial occlusion. Component-based segmentation enhances the model's understanding of local object features, making pose estimation more robust, particularly when objects are partially occluded. Multi-granularity segmentation provides different levels of component segmentation, enabling the system to adapt to application scenarios with varying accuracy requirements. The bidirectional information flow network effectively integrates complementary information from RGB images and point cloud data, improving the richness of feature representation. The pose optimization strategy for conveyor belt scenarios significantly improves the robot's grasping success rate, making the system more suitable for practical industrial applications. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention.

[0018] Figure 2 This is a diagram of the fusion architecture for object component segmentation and pose estimation.

[0019] Figure 3 This is a schematic diagram of the point cloud feature encoding and enhancement module.

[0020] Figure 4(a) is a schematic diagram of a two-way information flow network -- point cloud features synthesized from RGB and point cloud.

[0021] Figure 4(b) is a schematic diagram of a bidirectional information flow network -- RGB and point cloud composite RGB features.

[0022] Figure 5 Flowchart for optimizing pose estimation strategy. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown, this invention proposes a 3D object pose estimation method based on component segmentation, the overall process of which includes seven main steps. First, the input RGB-D data is preprocessed. Then, a multi-granularity segmentation network is used to segment the 3D object into components. Next, a feature encoding network is used to obtain multi-scale features. Then, a component association enhancement module and a bidirectional information flow network are applied to enhance the feature representation. Finally, an accurate 6D pose estimation result is obtained through component adaptive regression and trajectory correction strategies.

[0025] Step 1: Preprocess the input RGB-D image to obtain the object's color image and depth information. The color image is used to extract two-dimensional appearance features, and the depth information is converted into a three-dimensional point cloud for extracting geometric structure features. The two are then fused at the feature level in subsequent steps.

[0026] Step 1.1: Normalize the RGB-D image:

[0027] ;

[0028] in, The normalized RGB-D image, For the original RGB-D image, and These are the mean and standard deviation of the image, respectively.

[0029] Step 1.2: Perform bilateral filtering on the normalized RGB-D image:

[0030] ;

[0031] in, This is the filtered depth map. As the normalization factor, For spatial domain Gaussian functions, It is a Gaussian function with a range. Let x be the window region; x and y be the coordinates of the center pixel and the neighboring pixel in the image, respectively; D(x) and D(y) be the depth values ​​corresponding to pixel x and pixel y in the original depth map, respectively.

[0032] Step 2: Perform multi-level component segmentation on the 3D object based on a multi-granularity segmentation network to obtain the component-level representation of the object.

[0033] The 3D object segmentation process based on multi-granularity segmentation networks consists of three stages:

[0034] (a) Large-scale pre-training phase: learning a 3D feature encoding backbone network from a large number of unlabeled 3D objects;

[0035] (b) In the target-specific optimization stage, a lightweight neural network is trained to achieve scale-conditional grouping;

[0036] (c) In the semantic understanding stage, a multimodal language model is used to assign semantic labels to each component.

[0037] In the large-scale pre-training phase, over 800,000 3D assets from the 3D model dataset were used as training data. Rich 3D prior knowledge was learned by distilling visual features from the visual feature network into the 3D backbone network. During training, point clouds were randomly sampled from the mesh surface of 3D objects as network input. Visual features were enhanced using feature augmentation techniques, serving as point-level supervision for 3D feature extraction. The pre-training loss function is defined as:

[0038] ;

[0039] in, Features output by the 3D backbone network The average value of two-dimensional visual features:

[0040] ;

[0041] in, The total number of virtual camera views used when rendering 3D objects. For a 3D point in the th... Two-dimensional visual features corresponding to the projection position from each viewpoint.

[0042] In the target-specific optimization phase, a basis set mechanism is introduced to control the segmentation granularity by calculating the 3D scale from the mask generated from the 2D segmentation. The scale... The data is fed into a lightweight neural network and combined with long-range connections within the network to adjust and capture low-level features of the point cloud at different segmentation granularities. The scale is determined by the long-range connections of the neural network to capture these low-level features. The scale-conditional segmentation calculation formula is designed as follows:

[0043] ;

[0044] in, , , The standard deviations of the X, Y, and Z coordinates are: This is the scaling factor.

[0045] A lightweight neural network is trained using a contrastive learning method to bring point pairs from the same mask closer together and to push point pairs from different masks further apart.

[0046] In the semantic understanding stage, the corresponding 2D region of each 3D component in the multi-view rendering is identified through 3D-to-2D mapping. A multimodal language model is used to perform semantic queries on the highlighted components to obtain semantic tags for the components.

[0047] like Figure 2 As shown, the fusion architecture for object component segmentation and pose estimation mainly includes a multi-granularity segmentation network module, a feature encoding network, a component association enhancement module, a bidirectional information flow network, a component adaptive regression module, and a trajectory correction module. The multi-granularity segmentation network is responsible for accurate component segmentation of the object. The feature encoding network uses an improved backbone network to extract multi-scale features from RGB-D data. The component association enhancement module enhances the relationship features between components. The bidirectional information flow network integrates RGB and point cloud features. The component adaptive regression module predicts the object pose. The trajectory correction module further refines the pose estimation results.

[0048] Step 3: Fuse the segmented component information with the RGB-D image, and obtain the multi-scale RGB-D image features of the object through a feature extraction network.

[0049] like Figure 3 As shown, for the feature encoding network, it uses view transformation matrix calculation to render the 3D model from different perspectives:

[0050] ;

[0051] in, , , . , , These are the Z-axis, X-axis, and Y-axis of the camera coordinate system represented in the world coordinate system, respectively. The coordinates of the camera center in the world coordinate system; This is the transpose symbol.

[0052] Transformation matrix from world coordinate system to camera coordinate system:

[0053] , ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, and These are the base coordinate system transformation rotation matrix and the scene rotation matrix, respectively. The scene translation vector; This is the final composite transformation matrix, used to transform points from the world coordinate system to the camera coordinate system; The symbol represents matrix multiplication.

[0059] In the keypoint extraction process, the distance calculation formula using the uniform sampling algorithm is as follows:

[0060] ;

[0061] Step 4: Use the point cloud feature enhancement module to enhance the relationship features between components and extract the global point cloud features of the objects.

[0062] The point cloud feature enhancement module adopts an improved structure based on the self-attention algorithm. The input point cloud features are first aggregated into local features and updated into geometric features through edge convolution operation. Then, the global point cloud features are obtained by interacting with the query weight matrix and the key value features containing normalization operation.

[0063] Figure 3 In this context, ORB-FPS refers to ORB-guided Farthest Point Sampling, where ORB stands for Oriented Fast and Rotated BRIEF, representing a combination of oriented Fast corner detection and rotation-invariant BRIEF descriptors; FLANN stands for Fast nearest neighbor search algorithm; MLP stands for Multilayer Perceptron; and Softmax is the activation function. Input the number of points in the point cloud; , , These are the channel dimensions for hidden features, value features, and key features, respectively. , , These are the learnable weight matrices corresponding to queries, keys, and values, respectively.

[0064] The mathematical expression for nearest neighbor aggregation of a point cloud is:

[0065] ;

[0066] Feature matrix splitting and concatenation:

[0067] ;

[0068] ;

[0069] The mathematical expression of the self-attention mechanism is:

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] in, , and These are the query matrix, key matrix, and value matrix, respectively. For activation functions; For edge convolution operations; For tensor flattening operations, high-dimensional feature maps are converted into two-dimensional matrices for matrix multiplication. The feature map is input to the self-attention module; and These are the learnable weight matrices used to generate keys and values, respectively. , , and These are the m-th output element, the m-th query vector, the m-th key vector, and the m-th value vector, respectively.

[0076] This design enables the module to better extract semantic features of components and integrate global information.

[0077] Step 5: Fuse multi-scale RGB-D image features and global point cloud features through a bidirectional information flow network to obtain a comprehensive feature representation.

[0078] As shown in Figures 4(a) and 4(b), the bidirectional information flow network realizes the mutual fusion of image features and point cloud features. In the direction of integrating image features into point cloud features, the module collects the neighborhood points of feature points in the coordinate map, extracts the corresponding image texture and color features, uses an improved attention mechanism to extract important aggregate features, and concatenates them with the original point cloud features before encoding them through a multilayer perceptron to obtain the fused point cloud feature representation.

[0079] In Figures 4(a) and 4(b), (FR) represents the multi-scale RGB-D image features of the input. (FP) represents the global point cloud features obtained in step 4. Wherein, Corresponding to the formula , Corresponding to the formula ;

[0080] In this embodiment, the nearest neighbor search operator is denoted as... In practice, fast approximate nearest neighbor search algorithms such as FLANN (FastLibrary for Approximate Nearest Neighbors) can be used for implementation. (Note: The last part, "notation," appears to be a separate, unrelated statement and is left untranslated.) Indicates the first point in the point cloud Location code of each point For querying, in the image feature set Find its nearest neighbor features .

[0081] This is an aggregation function used to perform aggregation operations such as weighted summation or concatenation of features within the neighborhood. In this embodiment, it is preferably a weighted summation of attention-weighted features.

[0082] This refers to the Fast Attention function based on the query-key-value mechanism. In the original text, LAttn is consistently denoted as... .

[0083] The point cloud features are obtained by fusing image information.

[0084] The final image features are obtained by fusing point cloud information.

[0085] Indicates the first in the image Position encoding of pixels (or pixel regions) For querying, in the point cloud feature set Finding nearest neighbor point cloud features .

[0086] The mathematical expression for integrating image features into point cloud features is:

[0087] ;

[0088] ;

[0089] In the direction of integrating point cloud features into image features, the process is roughly the opposite, and the final mathematical expression of the fused image features is as follows:

[0090] ;

[0091] ;

[0092] in, A multilayer perceptron representing shared weights, This represents the attention mechanism. This refers to the information flow that maps from the image feature space to the point cloud feature space. Features of multi-scale RGB-D images; Let i be the i-th point in the point cloud; For point Location encoding; It is a set of feature points in a point cloud; The final point cloud features after fusing image information; The global point cloud features output from step 4; This refers to the information flow that maps from the point cloud feature space to the image feature space. The features of the i-th pixel region in the image; Let i be the i-th pixel in the image; It is the set of all pixels in the image; The final image features are obtained by fusing point cloud information.

[0093] The i-th point in the point cloud feature point set corresponds to the i-th pixel region of the feature in the image and the i-th pixel in that pixel region. The i in the above 'i-th point', 'i-th pixel region' and 'i-th pixel' are in one-to-one correspondence.

[0094] Step 6: Use the component adaptive regression module to predict the pose information of the object and its key components.

[0095] The component adaptive regression module projects each feature point onto a 2D RGB image and a depth image using a projection matrix. It then pairs and stitches appearance and geometric features through mapping relationships to obtain the final features. These features are then input into the semantic classification unit, the center inference unit, and the feature point localization unit, respectively.

[0096] The semantic classification loss function is defined as:

[0097] ;

[0098] in, This represents the number of semantic tags for all components of each instance after excluding the scene background. This indicates that a pixel given by the instance segmentation module belongs to the label. The probability, Indicates label Weighting factors It is the modulation factor; This is a focusing parameter used to adjust the rate at which the weights of easily distinguishable samples decrease.

[0099] The loss function for center inference and feature point localization is defined as follows:

[0100] ;

[0101] in, Indicates the sample excluding the background. The estimated unit vector value of each pixel at a point in the model's 3D space pointing to the instance's center point or keypoint. express The true value of the unit vector at the corresponding point. express and The angle between them.

[0102] The total loss function is:

[0103] ;

[0104] Unlike directly regressing the entire target object, this invention focuses more on specific components when processing pose estimation results. Pose estimation is performed using component features, and the pose estimation results of different components within the same instance are weighted, with the weight value proportional to the feature significance of the component in the pose estimation. For the case where there are w components in the same instance I, the mathematical expression of the final pose estimation result is as follows:

[0105] ;

[0106] in, For a given object instance I, the final estimated 6D pose; and Let be the rotation matrix and translation vector predicted for the t-th component in Instance I, respectively. This is an aggregation function used to fuse the pose predictions of all w parts of an instance into the final pose of that instance.

[0107] Step 7: Obtain the final 6D pose of the object through precise point-to-point optimization algorithm and trajectory correction strategy.

[0108] First, by using control point representation, the 3D reference points of the object are parameterized as a weighted sum of four virtual control points, transforming the complex pose determination problem into solving the coordinates of these four control points in the camera coordinate system. Then, a projection equation is established using the camera projection model, which correlates the projection of each 3D reference point with its spatial position after the object's pose transformation. Finally, the projection equations of all reference points are combined in matrix form to form an overdetermined homogeneous linear equation system. By solving this homogeneous linear equation system, the coordinates of the control points are obtained, thereby recovering the object's accurate 6D pose.

[0109] like Figure 5 As shown, the pose optimization strategy for conveyor belt scenarios mainly consists of two parts: translation matrix optimization and rotation matrix optimization. In the spatial linear motion trajectory iterative algorithm, the adjustment threshold and iteration count variable are first initialized, the parameter matrix is ​​iteratively updated based on the observation information, the error is calculated, and the iteration is completed when the error is less than the threshold. In the precise point pair optimization algorithm, the control points are represented as:

[0110] ;

[0111] in, As the reference point coordinates, For the coordinates of the control points, These are the weighting coefficients;

[0112] The projection equation is:

[0113] ;

[0114] Matrix form:

[0115] ;

[0116] in, It is a 2n×12 matrix. This is the coordinate vector of the control point in the camera coordinate system.

[0117] For optimization of the rotation matrix, the rotation matrix parameters for the final position are obtained by calculating the average of the three single-axis rotation angles given by all sampling points:

[0118] ;

[0119] ;

[0120] ;

[0121] in, This represents the number of matching pairs between reference points and projected points used for optimization. , , These are the average values ​​of the three single-axis rotation angles calculated for all matching pairs; , These are the single-axis rotation angles obtained by inverse solving the pose estimation of the k-th matching pair.

[0122] This step is a pose optimization process based on the PnP (Perspective-n-Point) concept. "Control point representation" is an efficient parameterization method that represents countless points on an object as a linear combination of a few virtual "control points," thus reducing the number of optimization variables. "Projection equations" establish the geometric constraints between these 3D reference points (represented as control points) and their projected pixels on the 2D image. "Matrix form" organizes the projection equations of all points into a large homogeneous linear system Ax=0. By solving this system, the coordinates of the control points in the camera coordinate system can be analytically or iteratively determined, thereby recovering the object's precise rotation matrix R_rot and translation vector T_trans.

[0123] Through this design, the method of this invention fully utilizes precise segmentation capabilities and high-precision pose estimation capabilities, achieving more accurate and robust pose estimation for objects, especially demonstrating excellent performance in complex scenes and under partial occlusion conditions. Experimental results show that, compared with existing technologies, the method of this invention significantly improves pose estimation accuracy.

[0124] In practical applications, the method of this invention can be flexibly applied to various scenarios such as industrial robot grasping, autonomous driving object recognition, and augmented reality scene understanding, and has broad application prospects.

Claims

1. A method for pose estimation of a three-dimensional object based on component segmentation, the method comprising: The method comprises the following steps: Step 1: pre-processing the input RGB-D image to obtain the color image and depth information of the object; Step 2: multi-level component segmentation of the three-dimensional object based on a multi-granularity segmentation network, and obtaining the component-level representation of the RGB and depth images of the segmented object through the pre-processed color image and depth information; Step 3: fusing the segmented component-level representation with the RGB-D image to obtain the multi-scale RGB-D image features of the object through a feature extraction network; Step 4: enhancing the relationship features between components by using a point cloud feature enhancement module, and extracting the global point cloud features of the object; the point cloud feature enhancement module adopts an improved structure based on a self-attention algorithm, aggregates local features and updates geometric features through edge convolution operation on the input point cloud features, and then obtains global point cloud features by using a query weight matrix to interact with key-value features containing normalization operation; Step 5: fusing the multi-scale RGB-D image features and the global point cloud features through a bidirectional information flow network to obtain comprehensive feature representation; Step 6: predicting the pose information of the object and its key components by using a component adaptive regression module; Step 7: obtaining the final 6D pose of the object through an accurate point pair optimization algorithm and a trajectory correction strategy.

2. The component-segmentation-based three-dimensional object pose estimation method of claim 1, wherein, The step 1 specifically comprises: Step 1.1: normalizing the RGB-D image; ； wherein, is the normalized RGB-D image, is the original RGB-D image, and are the mean and standard deviation of the image, respectively. Step 1.2: performing bilateral filtering on the normalized RGB-D image; ； wherein, is the filtered depth map, is a normalization factor, is a spatial domain Gaussian function, is a value domain Gaussian function, is a window region; , are the center pixel coordinate and the neighborhood pixel coordinate in the image, respectively; and are the depth values corresponding to the pixel x and the pixel y in the original depth map, respectively.

3. The component-segmentation-based three-dimensional object pose estimation method of claim 1, wherein, The step 2 specifically comprises: Step 2.1: a large-scale pre-training stage, learning 3D feature coding 3D backbone network from unlabeled 3D data; Using the disclosed three-dimensional model dataset as training data, learning 3D prior knowledge by distilling visual features from a visual feature network to a 3D backbone network; during the training process, randomly sampling point clouds from the mesh surface of the 3D object as the input of the 3D backbone network, enhancing the visual features through feature enhancement technology, and performing 3D feature extraction as point-level supervision; the loss function of the pre-training stage is is: ； wherein, is the average of the two-dimensional visual features, as follows: is the average of the two-dimensional visual features, as follows: ； wherein, is the total number of virtual camera perspectives employed in rendering the 3D object, is the two-dimensional visual feature corresponding to the projected position of the three-dimensional point under the lth perspective. Step 2.2: a target-specific optimization stage, training a lightweight neural network to realize scale condition grouping; A base mechanism is introduced to control the granularity of segmentation by calculating 3D scales from the masks generated by 2D segmentation ; the 3D scales are fed into a lightweight neural network combined with a long-range connection module in the network to adjust and capture the low-level features of the point cloud under different segmentation granularities, and the scale is calculated by the formula: ； wherein , , is the standard deviation of the X, Y, Z direction coordinates, is a scaling factor; 3D scales computed from 2D masks are trained with a light-weight neural network by a contrastive learning method As a conditional input, the point cloud features are fed into a light-weight neural network; the light-weight neural network, driven by contrastive learning, learns to pull point features with similar scale context closer and push point features with different scale context further, guiding the network to output different granularities, i.e., different groupings or segmentation levels of components; Step 2.3: a semantic understanding stage, using a multi-modal language model to assign semantic labels to each component; Through 3D to 2D mapping, the corresponding 2D area of each 3D component in multi-view rendering is identified, and a multi-modal language model is used for semantic query of the highlighted component to obtain the semantic label of the component.

4. The part-segment-based three-dimensional object pose estimation method of claim 1, wherein, The step 3 specifically comprises: Applying an efficient feature point extraction algorithm to calculate the view transformation matrix: ； wherein is a view transformation matrix; , , , , , are the representation vectors of the Z-axis, X-axis and Y-axis of the camera coordinate system in the world coordinate system, respectively; is the position coordinate of the camera center in the world coordinate system; is the transpose symbol; The transformation matrix from the world coordinate system to the camera coordinate system: ， ； ； ； ； ； wherein, and are a base coordinate system transformation rotation matrix and a scene rotation matrix, respectively; is a scene translation vector; is a final composite transformation matrix for transforming a point from a world coordinate system to a camera coordinate system; the symbol denotes matrix multiplication; The distance calculation formula of the uniform sampling algorithm: ； wherein, denotes the square of the shortest Euclidean distance from point p to the set S, , , are, respectively, a set of sampled points, a candidate point, a point in the set S of sampled points.

5. The part-segment-based three-dimensional object pose estimation method of claim 1, wherein, In the step 4, the point cloud feature enhancement module enhances the local geometric features through the following process: First, by the point cloud of the neighborhood aggregation, for each center point in the space and feature dimension to find a neighborhood point to capture its local neighborhood information; second, by the splitting and splicing of the feature matrix, the center point feature, the neighboring point feature and the relative position and feature difference between them are encoded and combined to form a local feature descriptor, which is encoded by a multilayer perceptron to update the geometric feature; Specifically, the neighbor aggregation representation of the point cloud is: ； wherein, is a matrix of features of the point cloud after the aggregation, is a nearest neighbor search algorithm, is a matrix of features of the input point cloud, is the number of points in the point cloud processed by the previous layer, is the dimension of the input features, is the number of selected nearest neighbors for each point; The feature matrix splitting and splicing is represented as: ； ； wherein, is the concatenated feature tensor, is the original feature of each point copied is the tensor formed after the concatenation with the neighborhood features; is the feature tensor output by the multi-layer perceptron, and are respectively are two parts of the tensor in the feature dimension, is a multi-layer perceptron network module; is the hidden layer feature dimension output by the multi-layer perceptron network module; The point cloud feature enhancement module adopts an improved structure based on a self-attention algorithm, and the self-attention mechanism calculation formula is: ； ； ； ； ； wherein, , and are a query matrix, a key matrix and a value matrix, respectively; is an activation function; is an edge convolution operation; is a tensor flattening operation to convert a high-dimensional feature map into a two-dimensional matrix for matrix multiplication; is a feature map input to the self-attention module; and are learnable weight matrices for generating keys and values, respectively; , , and are an m-th output element, an m-th query vector, an m-th key vector and an m-th value vector, respectively.

6. The part-segment-based three-dimensional object pose estimation method of claim 5, wherein, The mathematical expression of the bidirectional information flow network in the step 5 for fusing the multi-scale RGB-D image features into the global point cloud features is: ； ； The mathematical expression of the global point cloud features fusing into the multi-scale RGB-D image features is: ； ； wherein, represents a multi-layer perceptron for sharing weights, represents an attention mechanism; is an information flow from image feature space to point cloud feature space; is a multi-scale RGB-D image feature; is the i-th point in the point cloud; is the position encoding of the point is a set of point cloud feature points; is the final point cloud feature after fusing image information; is the global point cloud feature outputted in step 4; is an information flow from point cloud feature space to image feature space; is the i-th pixel region feature in the image; is the i-th pixel in the image; is a set of all pixels in the image; is the final image feature after fusing point cloud information.​ 7. The part-segment-based three-dimensional object pose estimation method of claim 6, wherein, In the step 6, the component adaptive regression module projects each feature point into a two-dimensional RGB image and a depth image through a projection matrix, pairs and splices the appearance features and geometric features through the mapping relationship, and obtains the final features; The final features are respectively input into a semantic classification unit, a center reasoning unit and a feature point positioning unit; the component self-adaptive regression module comprises a semantic classification loss : ； wherein, denotes the number of semantic labels of all components of each instance excluding the context background, denotes the probability that a certain pixel belongs to label given by the instance segmentation module, denotes the weight factor of label , is a modulation factor; is a focus parameter for adjusting the weight reduction rate of easy-to-segment samples; center reasoning loss and feature point positioning loss : ； wherein, represents the estimated value of the unit vector pointing from the model three-dimensional space point corresponding to the pixel in the sample other than the background to the instance center point or the instance key point, represents the true value of the unit vector at the corresponding point, represents the angle between and ;​​ Total loss function : ； wherein , and are the corresponding weight coefficients; For the case that there are w components in the same instance I, the mathematical representation of the final pose estimation result is: ； wherein, is the final estimated 6D pose for an instance of an object I; and are the predicted rotation matrix and translation vector for the t-th part of instance I, respectively; is an aggregation function that fuses the pose predictions of all w parts of an instance into a final pose for that instance.

8. The part-segment-based three-dimensional object pose estimation method of claim 7, wherein, The precise point pair optimization algorithm in the step 7 is specifically a pose solving method based on control points, and the process is as follows: Firstly, through control point representation, the three-dimensional reference point parameters of an object are parameterized as a weighted sum of four virtual control points, and a complex pose solving problem is converted into solving the coordinates of the four control points in a camera coordinate system; then, a projection equation is established by using a camera projection model, the projection equation relates the projection of each three-dimensional reference point to the spatial position after the object pose transformation; finally, the projection equations of all reference points are combined in a matrix form to form an overdetermined homogeneous linear equation group, and the control point coordinates are obtained by solving the homogeneous linear equation group, and then the accurate 6D pose of the object is restored; The control point representation is as follows: ； wherein, is a reference point coordinate, is a control point coordinate, is a weight coefficient; The projection equation is as follows: ； wherein, is the depth value of the kth reference point in the camera coordinate system; and are the x and y coordinates of the kth reference point observed on the image; is the intrinsic matrix of the camera; and are the rotation matrix and the translation vector of the object to be optimized, respectively; is a three-dimensional reference point in the object model coordinate system; The matrix form is as follows: ； wherein, is a 2n x 12 matrix, is a coordinate vector of the control points in the camera coordinate system; For the optimization of the rotation matrix, the average value of the three single-axis rotation angles given by all sampling points is calculated to obtain the rotation matrix parameters of the final position: ； ； ； wherein, is the number of matching pairs of reference and projection points for optimization; , , are the average values of the three single-axis rotation angles calculated for all matching pairs, respectively; , are the single-axis rotation angles back-solved from the pose estimation by the k-th matching pair.

Images