Target 6D attitude estimation method guided by neighborhood perception information

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a target 6D attitude estimation method guided by neighborhood perception information. In order to solve the problem that a traditional 6D pose estimation method based on sparse key points is poor in performance in a scene with large occlusion and visual angle change, neighborhood geometric information is introduced from a depth map, neighborhood aggregation and a position coding attention mechanism are combined, local geometric features of an object are fully excavated, and the position of the object is estimated. And the robustness and accuracy of pose estimation are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and specifically relates to a method for 6D pose estimation of a target guided by neighborhood perception information, and a method for improving the accuracy of 6D pose estimation by aggregating neighborhood geometric information using graph convolution. Background Art

[0002] With the rapid development of intelligent technology, 6D pose estimation has gradually become a core technology in many fields such as intelligent driving, industrial robots, virtual reality and automated services. In robot grasping, 6D pose estimation can accurately perceive the position and direction of objects in three-dimensional space, thereby enabling the efficient grasping of objects by the robotic arm. In AR / VR, 6D pose estimation can track the position and direction of users or objects in virtual space in real time, ensure the accurate superposition of virtual content and real scenes, and provide a more natural interactive experience; in autonomous driving, 6D pose estimation helps vehicles perceive the spatial position and direction of traffic signals, pedestrians and dynamic obstacles, and provides technical support for path planning and safe driving. These wide range of application scenarios fully reflect the core value and development potential of 6D pose estimation.

[0003] Traditional pose estimation methods based on sparse key points usually rely on a small number of feature points of an object. Although this method has a small amount of computation, it faces many challenges in complex scenes. For example, occlusion and perspective changes will cause feature point extraction failure, thereby reducing the accuracy of pose estimation. In addition, the key point regression method based on deep learning learns the spatial distribution of feature points through neural networks to predict the 6D pose of an object. Although it alleviates the problem of insufficient feature points to a certain extent, its global semantic understanding of feature points is weak and it is difficult to fully capture the local geometric information of objects. This still has performance bottlenecks in complex environments such as occlusion, lighting changes, and background interference.

[0004] Therefore, introducing neighborhood geometry information becomes an important research direction to improve the robustness of 6D pose estimation. By capturing the local geometric features of the object surface and guiding the model to learn richer spatial information, the performance of the model in complex scenes such as occlusion and perspective changes can be effectively enhanced. On this basis, a 6D pose estimation method based on neighborhood perception information guidance is proposed, which not only has important scientific significance, but also has wide application value in robot operation, autonomous driving and other practices. Summary of the invention

[0005] In view of the problem that the traditional 6D pose estimation method based on sparse key points performs poorly in scenes with large occlusion and perspective changes, this paper proposes a target 6D pose estimation method guided by neighborhood perception information. Traditional methods rely on a small number of feature points, and it is difficult to obtain sufficient feature information for accurate recognition in complex environments. This paper introduces neighborhood geometry information from the depth map, combines neighborhood aggregation and position encoding attention mechanism, fully exploits the local geometric features of the object, improves the robustness and accuracy of pose estimation, and provides an efficient and reliable solution for target recognition and pose estimation in complex scenes.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a method for estimating a target 6D posture guided by neighborhood perception information, comprising the following steps:

[0008] Step 1, feature extraction: It is divided into three branches: RGB image branch, depth image branch and grid model branch, which are used to extract RGB features F rgb , point cloud local features F cloud and Mesh global feature F global ;

[0009] Step 2: Acquisition of dense correspondence: local features F of point cloud cloud and Mesh global feature F global Perform position encoding, apply the self-attention mechanism to the position-encoded features, and then perform cross-modal cross-attention fusion to obtain the fused feature F fused ;

[0010] Step 3, pose solution: Fusion feature F fused With RGB feature F rgb Feature similarity calculation is performed to establish the 2D-3D correspondence, and then the PnP+RANSAC algorithm is used to solve the 6D pose of the object.

[0011] Furthermore, the RGB image branch in step 1 uses a multi-layer fully convolutional network to extract the RGB features F in the RGB image layer by layer. rgb , specifically:

[0012] The ResNet network is used as the backbone network for feature extraction. The input RGB image is downsampled using the maximum pooling and cross-row convolution layers to obtain the feature map. The bilinear upsampling layer is used to enlarge the feature map to the same size as the input RGB image to obtain the semantic information M of the target object. seg , jump-connect the residual module feature maps before downsampling and after the bilinear upsampling layer to obtain the RGB feature F rgb .

[0013] Furthermore, the depth image branch in step 1 is specifically:

[0014] The depth image is input into the Pointnet++ network and converted into a point cloud P = {P1,P2,...,P N}∈R N×(3+c) , indicating that the point cloud P is composed of N points, and the point cloud P includes spatial coordinates (x, y, z) and point cloud features;

[0015] The relationship between the center point in the point cloud P and its adjacent points is used to establish a graph structure between points. For the i-th point p i , its neighboring points use the nearest neighbor algorithm based on the combination of Euclidean distance and normal vector similarity to construct the graph structure, the adjacency relationship is: G = (V, E), vertex V = {v1,v2,...,v N}, represents the vertex in the point cloud, and the edge E represents the relationship between point pairs, which is constructed by the nearest neighbor algorithm as follows:

[0016]

[0017] Among them, K-NN is a nearest neighbor algorithm;

[0018] For each edge Eij, calculate the weight:

[0019]

[0020] Among them, w ij represents the weight of the edge between point i and point j, x i ,x j Represent the spatial coordinates of i and j respectively, is a distance-based weight, is the weight based on the normal vector similarity, α and β represent the proportional factors of the distance weight and the normal vector similarity weight respectively, and n i ,n j Respectively represent the normal vectors of points i and j, |||| represents the modulus of the normal vector;

[0021] For point p i The selected neighboring points are expressed as

[0022] The point cloud features of the constructed graph structure are passed into SplineCNN, and the geometric information on the edge is modeled through the parameterized spline function. d The feature update formula is:

[0023]

[0024] Among them, e ijrepresents the geometric properties of the edge, B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, and f j Indicates i d The characteristics of the neighboring point j;

[0025] When constructing the local features of the point cloud, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, and gradually extracts and aggregates local geometric information through layer-by-layer convolution operations, thereby capturing richer local geometric feature representations. The feature stacking formula of the multi-layer SplineCNN is:

[0026] F cloud =F d (l+1) =SplineConv(F d (l) , E, B)(3)

[0027] Among them, F d (l) Represents the input features of the lth layer, and the output point cloud local features are N c is the number of point cloud points, and C is the feature dimension.

[0028] Furthermore, the grid model branch in step 1 is specifically:

[0029] The Mesh model is sent as input to SplineCNN. The vertices of the Mesh model V = {v1,v2,...,v N}, each vertex contains its coordinates and initial features; the edge E = {(i, j)} of the Mesh model represents the connection relationship between vertices, which is defined by the topology of the triangular mesh; for point i m The feature update formula is:

[0030]

[0031] Where B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, e ij Represents the geometric information of the edge, w ij represents the weight between vertex i and vertex j, f j Indicates point i m Input features of neighboring point j;

[0032] When constructing the Mesh global features, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, gradually extracting and aggregating higher-level global geometric information. The feature stacking formula of the multi-layer SplineCNN is:

[0033] F global =F m(l+1) =SplineConv(F m (l) , E, B)(5)

[0034] Among them, F m (l) The input features of the lth layer and the local features of the final output are N m is the number of points in the Mesh model, C is the feature dimension, and B represents the parameterized spline basis function.

[0035] Furthermore, the acquisition of the dense correspondence in step 2 is specifically as follows:

[0036] For the local feature F of the point cloud cloud Apply the embedding operation to generate the positional encoding, that is, use an embedding matrix E cloud ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the coordinate value of the point to the index through a mapping function, which is defined as:

[0037] I i cloud =IndexMap(F i cloud )(6)

[0038] Among them, I i cloud is the embedding index of point cloud point i, F i cloud It is the local feature of the point cloud obtained by multi-layer SplineCNN of the i-th point;

[0039] The positional encoding is defined as:

[0040]

[0041] Generate the position encoding after embedding matrix for all point cloud points i:

[0042] P embed,cloud ∈R Nc×D (8);

[0043] For the Mesh global feature F global Apply the embedding operation to generate the positional encoding using an embedding matrix E global ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the i-th feature point in the Mesh feature point to the index through a mapping function:

[0044] I i global=IndexMap(F i global )(9)

[0045] in, is the embedding index of Mesh point i, F i global It is the Mesh global feature obtained by multi-layer SplineCNN at the i-th point;

[0046] The positional encoding is defined as:

[0047]

[0048] Generate the position encoding after embedding matrix for all points i in the Mesh model:

[0049] P embed,global ∈R Nm×D (11);

[0050] Position encoding of point cloud local features and Mesh global features:

[0051]

[0052] Features and Features The self-attention mechanism is applied respectively to obtain the enhanced point cloud local features and Mesh global features:

[0053]

[0054]

[0055] The enhanced Mesh global features are used as Query, and the enhanced point cloud local features are used as Key and Value to perform a cross-modal cross-attention mechanism:

[0056]

[0057] in, W q ,W k ,W v represents the learnable linear projection matrix, d k represents the dimension of the key vector, and T represents the transpose of the matrix;

[0058] The results of the fusion features are as follows:

[0059]

[0060] Furthermore, the total loss of the method is shown as follows:

[0061] Ltotal =ω1L seg +ω2L match +ω3L select (17)

[0062] Among them, ω1, ω2, ω3 are the weight factors of each loss, L seg is the segmentation loss for segmenting the object from the image background, L match is the estimated loss of the 2D-3D point pair matching relationship, L select is the matching loss of the point-to-point matching relationship;

[0063] The segmentation loss is:

[0064] L seg =CEloss(SoftMax(M seg ), M gt ) (18)

[0065] Among them, M gt It means projecting the 3D model of an object onto a 2D image through its true pose;

[0066] The estimated losses are:

[0067] L match =CEloss(SoftMax(F rgb (F fused ) T ), S gt ) (19)

[0068] Among them, S gt is the obtained true 2-D-3-D correspondence;

[0069] The matching loss includes weighted optimization of positive and negative samples:

[0070]

[0071] Among them, Y ij represents the true matching label, Y ij =1 means that Mesh point i and point cloud point j are truly matched, N i ={j|Y ij =0, S ij >δ} means that for each Mesh point i, a set of negative samples is selected; δ represents the similarity threshold, indicating potential false positive pairs; λ represents the weight ratio controlling the loss of positive and negative samples, P ij Represents the matching probability of Mesh point i and point cloud point j. The calculation formula is as follows:

[0072]

[0073]

[0074]

[0075] Among them, S ik represents the similarity score between point cloud point k and Mesh point i, exp(S ik ) is used to convert the similarity score into a non-negative value, indicating the weight of the point pair in the total match, S ij represents the similarity score between point cloud point j and Mesh point i, F i global ∈R C Represents the characteristics of Mesh point i, represents the feature of point j in the point cloud, and ||·|| represents the L2-norm of the feature vector, which is used to normalize the similarity.

[0076] Compared with the prior art, the present invention has the following advantages:

[0077] 1. It performs more stably in scenes with occlusion and perspective changes, has stronger anti-interference ability than the traditional sparse key point method, establishes dense feature correspondence between 2D-3D and 3D-3D spaces, improves the richness and accuracy of feature points, and thus improves the accuracy of 6D pose estimation.

[0078] 2. Through neighborhood aggregation and attention mechanism, multimodal information can be efficiently integrated, which can not only capture the surface details of the object, but also represent its overall topological structure, further improving the robustness and accuracy of pose estimation.

[0079] 3. The method of the present invention is easy to implement, and its application value is mainly reflected in: 1) In the industrial field, by accurately identifying the position and direction of objects in space, industrial robots can grasp and manipulate objects more accurately, effectively reduce the failure rate of grasping, and improve the efficiency of automated operations. 2) In the medical field, the complexity of the environment (such as lighting changes, dynamic object interference, etc.) often affects the robot's visual perception ability. Through the adaptive fusion of multimodal features, this technology enables the robot to maintain stable recognition and grasping capabilities of objects under different environmental conditions, enabling it to provide continuous and efficient services in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is the overall flow chart of the present invention;

[0081] Figure 2 It is a network framework diagram of the method proposed in the present invention;

[0082] Figure 3 This is the network framework diagram for the RGB image branch. DETAILED DESCRIPTION

[0083] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0084] like Figure 1 and Figure 2 As shown, a method for estimating a target 6D posture guided by neighborhood perception information in this embodiment includes the following steps:

[0085] Step 1, feature extraction: It is divided into three branches: RGB image branch, depth image branch and grid model branch, which are used to extract RGB features F rgb , point cloud local features F cloud and Mesh global feature F global ;

[0086] Among them, the RGB image branch (such as Figure 3 As shown in Figure 2, the RGB features F in the RGB image are extracted layer by layer using a multi-layer fully convolutional network. rgb Specifically, the ResNet network is used as the backbone network for feature extraction, and the input RGB image is downsampled using the maximum pooling and cross-row convolution layers to obtain the feature map; the feature map is enlarged to the same size as the input RGB image using the bilinear upsampling layer, and the residual jump connection is used before downsampling and after the bilinear upsampling layer to retain the original information of the image and obtain the semantic information M of the target object. seg And RGB feature F rgb .

[0087] Depth image branch: Input the depth image into the Pointnet++ network and convert it into a point cloud P = {P1,P2,...,P N}∈R N×(3+c) , indicating that the point cloud P is composed of N points, and the point cloud P includes spatial coordinates (x, y, z) and point cloud features;

[0088] The relationship between the center point in the point cloud P and its adjacent points is used to establish a graph structure between points. For the i-th point p i , its neighboring points use the nearest neighbor algorithm based on the combination of Euclidean distance and normal vector similarity to construct the graph structure, the adjacency relationship is: G = (V, E), vertex V = {v1, v2, ..., v N}, represents the vertex in the point cloud, and the edge E represents the relationship between point pairs, which is constructed by the nearest neighbor algorithm as follows:

[0089]

[0090] Among them, K-NN is a nearest neighbor algorithm;

[0091] For each edge Eij, calculate the weight:

[0092]

[0093] Among them, w ij represents the weight of the edge between point i and point j, x i ,x j Represent the spatial coordinates of i and j respectively, is a distance-based weight, is the weight based on the normal vector similarity, α and β represent the proportional factors of the distance weight and the normal vector similarity weight respectively, and n i ,n j Represent the normal vectors of point i and j respectively, and || || represents the modulus of the normal vector. α and β are dynamically learned during training to allow the network to adaptively select more appropriate weights.

[0094] For point p i The selected neighboring points are expressed as Comprehensively considering the spatial distribution and local geometric characteristics of points helps to build a more robust graph structure;

[0095] The point cloud features of the constructed graph structure are passed into SplineCNN, and the geometric information on the edge is modeled through the parameterized spline function. d The feature update formula is:

[0096]

[0097] Among them, e ij represents the geometric properties of the edge, B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, and f j Indicates i d The characteristics of the neighboring point j;

[0098] When constructing the local features of the point cloud, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, and gradually extracts and aggregates local geometric information through layer-by-layer convolution operations, thereby capturing richer local geometric feature representations. The feature stacking formula of the multi-layer SplineCNN is:

[0099] F cloud =F d (l+1) =SplineConv(F d (l) , E, B)(3)

[0100] Among them, F d (l) Represents the input features of the lth layer, and the output point cloud local features are N cis the number of point cloud points, C is the feature dimension; the local feature of each point is updated through graph convolution to contain more geometric context information.

[0101] Mesh model branch: The Mesh model is sent as input to SplineCNN. The vertices of the Mesh model V = {v1,v2,...,v N}, each vertex contains its coordinates and initial features; the edge E = {(i, j)} of the Mesh model represents the connection relationship between vertices, which is defined by the topology of the triangular mesh; for point i m The feature update formula is:

[0102]

[0103] Where B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, e ij Represents the geometric information of the edge, w ij represents the weight between vertex i and vertex j, f j Indicates point i m Input features of neighboring point j;

[0104] When constructing the Mesh global features, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, gradually extracting and aggregating higher-level global geometric information. This layer-by-layer feature aggregation method can effectively capture complex spatial structure information and provide richer global geometric feature representation for posture estimation. The feature stacking formula of the multi-layer SplineCNN is:

[0105] F global =F m (l+1) =SplineConv(F m (l) , E, B)(5)

[0106] Among them, F m (l) The input features of the lth layer and the local features of the final output are N m is the number of points in the Mesh model, C is the feature dimension, and B represents the parameterized spline basis function. The global features of the Mesh and the local features of the point cloud are enhanced through the self-attention mechanism to extract their internal dependencies.

[0107] Step 2: Acquisition of dense correspondence: local features F of point cloud cloud and Mesh global feature F global Perform position encoding, apply the self-attention mechanism to the position-encoded features, and then perform cross-modal cross-attention fusion to obtain the fused feature F fused .

[0108] In the acquisition of dense correspondence, the coordinates of the point cloud need to be processed first to lay the foundation for subsequent feature fusion. By generating a position code for each point in the point cloud, the spatial geometric information of the point cloud data can be captured to ensure that the local features of the point cloud can maintain consistency with the global features. This operation maps the point cloud coordinates to a high-dimensional feature space through an embedding matrix, providing key support for feature alignment between the point cloud and the Mesh model.

[0109] For the local feature F of the point cloud cloud Apply the embedding operation to generate the positional encoding, that is, use an embedding matrix E cloud ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the coordinate value of the point to the index through a mapping function, which is defined as:

[0110]

[0111] in, is the embedding index of point cloud point i, F i cloud It is the local feature of the point cloud obtained by multi-layer SplineCNN of the i-th point;

[0112] The positional encoding is defined as:

[0113]

[0114] Generate the position encoding after embedding matrix for all point cloud points i:

[0115] P embed,cloud ∈R Nc×D (8);

[0116] After the embedding matrix of the point cloud is generated, each point cloud point is assigned a unique feature representation through position encoding to characterize its position relationship in three-dimensional space. These position encodings serve as important inputs for subsequent feature fusion and provide geometric information support for the enhancement of local features of the point cloud.

[0117] Next, a similar embedding operation is applied to the indices of the Mesh points to build a spatial representation of the Mesh model points by generating position encodings.

[0118] For the Mesh global feature F global Apply the embedding operation to generate the positional encoding using an embedding matrix E global ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the i-th feature point in the Mesh feature point to the index through a mapping function:

[0119]

[0120] in, is the embedding index of Mesh point i, F i global It is the Mesh global feature obtained by multi-layer SplineCNN at the i-th point;

[0121] The positional encoding is defined as:

[0122]

[0123] Generate the position encoding after embedding matrix for all points i in the Mesh model:

[0124] P embed,global ∈R Nm×D (11);

[0125] Position encoding of point cloud local features and Mesh global features:

[0126]

[0127] Features and Features The self-attention mechanism is applied respectively to obtain the enhanced point cloud local features and Mesh global features:

[0128]

[0129]

[0130] The enhanced Mesh global features are used as Query, and the enhanced point cloud local features are used as Key and Value to perform a cross-modal cross-attention mechanism:

[0131]

[0132] in, W q ,W k ,W v represents the learnable linear projection matrix, d k represents the dimension of the key vector, and T represents the transpose of the matrix;

[0133] The results of the fusion features are as follows:

[0134]

[0135] The calculation formula of the self-attention mechanism in this embodiment is as follows:

[0136]

[0137] Q=W q F pos ,K=W k F pos ,V=W v F pos .

[0138] Step 3, pose solution: Fusion feature F fused With RGB feature F rgb Feature similarity calculation is performed to establish the 2D-3D correspondence, and then the PnP+RANSAC algorithm is used to solve the 6D pose of the object.

[0139] The feature similarity is calculated by matrix multiplication.

[0140] The calculation formula is as follows:

[0141] S=F fused ·(F rgb ) T

[0142] Among them, S represents the similarity between features, · represents the dot product, and T represents the transpose of the matrix.

[0143] Furthermore, the total loss of the method of this embodiment is shown as follows:

[0144] L total =ω1L seg +ω2L match +ω3L select (17)

[0145] Among them, ω1, ω2, ω3 are the weight factors of each loss, L seg is the segmentation loss for segmenting the object from the image background, L match is the estimated loss of the 2D-3D point pair matching relationship, L select is the matching loss of the point-to-point matching relationship;

[0146] The segmentation loss is:

[0147] L seg =CEloss(SoftMax(M seg ), M gt )(18)

[0148] Among them, M gt It means projecting the 3D model of an object onto a 2D image through its true pose;

[0149] The estimated losses are:

[0150] L match=CEloss(SoftMax(F rgb (F fused ) T ), S gt ) (19)

[0151] Among them, S gt is the obtained true 2-D-3-D correspondence;

[0152] The matching loss optimizes the point pair matching relationship through cross entropy loss, so that the similarity of the true matching pairs is maximized and the similarity of the non-matching pairs is minimized. The complete matching loss includes weighted optimization of positive samples and negative samples:

[0153]

[0154] Among them, Y ij represents the true matching label, Y ij =1 means that Mesh point i and point cloud point j are truly matched, N i ={j|Y ij =0, S ij >δ} means that for each Mesh point i, a set of negative samples is selected; δ represents the similarity threshold, indicating potential false positive pairs; λ represents the weight ratio controlling the loss of positive and negative samples, P ij Represents the matching probability of Mesh point i and point cloud point j. The calculation formula is as follows:

[0155]

[0156]

[0157]

[0158] Among them, S ik represents the similarity score between point cloud point k and Mesh point i, exp(S ik ) is used to convert the similarity score into a non-negative value, indicating the weight of the point pair in the total match, S ij represents the similarity score between point cloud point j and Mesh point i, F i global ∈R C Represents the characteristics of Mesh point i, represents the feature of point j in the point cloud, and ||·|| represents the L2-norm of the feature vector, which is used to normalize the similarity.

[0159] The above shows and describes the main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0160] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for estimating a target 6D posture guided by neighborhood perception information, characterized in that: The following steps are involved: Step 1, feature extraction: It is divided into three branches: RGB image branch, depth image branch and grid model branch, which are used to extract RGB features F rgb , point cloud local features F cloud and Mesh global feature F global ; Step 2: Acquisition of dense correspondence: local features F of point cloud cloud and Mesh global feature F global Perform position encoding, apply the self-attention mechanism to the position-encoded features, and then perform cross-modal cross-attention fusion to obtain the fused feature F fused ; Step 3, pose solution: Fusion feature F fused With RGB feature F rgb Feature similarity calculation is performed to establish the 2D-3D correspondence, and then the PnP+RANSAC algorithm is used to solve the 6D pose of the object.

2. The method for estimating a target 6D posture guided by neighborhood perception information according to claim 1, characterized in that: In step 1, the RGB image branch uses a multi-layer fully convolutional network to extract the RGB features F in the RGB image layer by layer. rgb , specifically: The ResNet network is used as the backbone network for feature extraction. The input RGB image is downsampled using the maximum pooling and cross-row convolution layers to obtain the feature map. The feature map is enlarged to the same size as the input RGB image using the bilinear upsampling layer. Residual skip connections are used before downsampling and after the bilinear upsampling layer to obtain the semantic information M of the target object. seg And RGB feature F rgb .

3. The method for estimating a target 6D posture guided by neighborhood perception information according to claim 1, characterized in that: The depth image branch in step 1 is specifically: The depth image is input into the Pointnet++ network and converted into a point cloud P = {P1,P2,...,P N }∈R N×(3+c) , indicating that the point cloud P is composed of N points, and the point cloud P includes spatial coordinates (x, y, z) and point cloud features; The relationship between the center point in the point cloud P and its adjacent points is used to establish a graph structure between points. For the i-th point p i , its neighboring points use the nearest neighbor algorithm based on the combination of Euclidean distance and normal vector similarity to construct the graph structure, the adjacency relationship is: G = (V, E), vertex V = {v1,v2,...,v N }, represents the vertex in the point cloud, and the edge E represents the relationship between point pairs, which is constructed by the nearest neighbor algorithm as follows: Among them, K-NN is a nearest neighbor algorithm; For each edge Eij, calculate the weight: Among them, w ij represents the weight of the edge between point i and point j, x i ,x j Represent the spatial coordinates of i and j respectively, is a distance-based weight, is the weight based on the normal vector similarity, α and β represent the proportional factors of the distance weight and the normal vector similarity weight respectively, and n i ,n j Respectively represent the normal vectors of points i and j, || || represents the modulus of the normal vector; For point p i The selected neighboring points are expressed as The point cloud features of the constructed graph structure are passed into SplineCNN, and the geometric information on the edge is modeled through the parameterized spline function. d The feature update formula is: Among them, e ij represents the geometric properties of the edge, B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, and f j Indicates i d The characteristics of the neighboring point j; When constructing the local features of the point cloud, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, and gradually extracts and aggregates local geometric information through layer-by-layer convolution operations, thereby capturing richer local geometric feature representations. The feature stacking formula of the multi-layer SplineCNN is: F cloud =F d (l+1) =SplineConv(F d (l) ,E,B)(3) Among them, F d (l) Represents the input features of the lth layer, and the output point cloud local features are N c is the number of point cloud points, and C is the feature dimension.

4. The method for estimating a target 6D posture guided by neighborhood perception information according to claim 1, characterized in that: The grid model branch in step 1 is specifically: The Mesh model is sent as input to SplineCNN. The vertices of the Mesh model V = {v1,v2,...,v N }, each vertex contains its coordinates and initial features; the edge E = {(i, j)} of the Mesh model represents the connection relationship between vertices, which is defined by the topology of the triangular mesh; for point i m The feature update formula is: Where B(·) represents the parameterized spline basis function, which is used to model the geometric information of the edge, e ij Represents the geometric information of the edge, w ij represents the weight between vertex i and vertex j, f j Indicates point i m Input features of neighboring point j; When constructing the Mesh global features, the multi-layer SplineCNN uses the output of the previous layer as the input of the next layer, gradually extracting and aggregating higher-level global geometric information. The feature stacking formula of the multi-layer SplineCNN is: F global =F m (l+1) =SplineConv(F m (l) ,E,B)(5) Among them, F m (l) The input features of the lth layer and the local features of the final output are N m is the number of points in the Mesh model, C is the feature dimension, and B represents the parameterized spline basis function.

5. The method for estimating a target 6D posture guided by neighborhood perception information according to claim 1, characterized in that: The acquisition of the dense correspondence in step 2 is specifically as follows: For the local feature F of the point cloud cloud Apply the embedding operation to generate the positional encoding, that is, use an embedding matrix E cloud ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the coordinate value of the point to the index through a mapping function, which is defined as: in, is the embedding index of point cloud point i, F i cloud It is the local feature of the point cloud obtained by multi-layer SplineCNN of the i-th point; The positional encoding is defined as: Generate the position encoding after embedding matrix for all point cloud points i: P embed,cloud ∈R Nc×D (8); For the Mesh global feature F global Apply the embedding operation to generate the positional encoding using an embedding matrix E global ∈R K×D , K represents the size of the embedding dictionary, D represents the dimension of the position encoding, and the position mapping maps the i-th feature point in the Mesh feature point to the index through a mapping function: in, is the embedding index of Mesh point i, F i global It is the Mesh global feature obtained by multi-layer SplineCNN at the i-th point; The positional encoding is defined as: Generate the position encoding after embedding matrix for all points i in the Mesh model: P embed,global ∈R Nm×D (11); Position encoding of point cloud local features and Mesh global features: Features and Features The self-attention mechanism is applied respectively to obtain the enhanced point cloud local features and Mesh global features: The enhanced Mesh global features are used as Query, and the enhanced point cloud local features are used as Key and Value to perform a cross-modal cross-attention mechanism: in, W q ,W k ,W v represents the learnable linear projection matrix, d k represents the dimension of the key vector, and T represents the transpose of the matrix; The results of the fusion features are as follows:

6. The method for estimating a target 6D posture guided by neighborhood perception information according to claim 1, characterized in that: The total loss of the method is as follows: L total =ω1L seg +ω2L match +ω3L select (17) Among them, ω1, ω2, ω3 are the weight factors of each loss, L seg is the segmentation loss for segmenting the object from the image background, L match is the estimated loss of the 2D-3D point pair matching relationship, L select is the matching loss of the point-to-point matching relationship; The segmentation loss is: L seg =CEloss(SoftMax(M seg ),M gt )(18) Among them, M gt It means projecting the 3D model of an object onto a 2D image through its true pose; The estimated losses are: L match =CEloss(SoftMax(F rgb (F fused ) T ),S gt )(19) Among them, S gt is the obtained true 2-D-3-D correspondence; The matching loss includes weighted optimization of positive and negative samples: Among them, Y ij represents the true matching label, Y ij =1 means that Mesh point i and point cloud point j are truly matched, N i ={jY ij =0, S ij >δ} means that for each Mesh point i, a set of negative samples is selected; δ represents the similarity threshold, indicating potential false positive pairs; λ represents the weight ratio controlling the loss of positive and negative samples, P ij Represents the matching probability of Mesh point i and point cloud point j. The calculation formula is as follows: Among them, S ik represents the similarity score between point cloud point k and Mesh point i, exp(S ik ) is used to convert the similarity score into a non-negative value, indicating the weight of the point pair in the total match, S ij represents the similarity score between point cloud point j and Mesh point i, F i global ∈R C Represents the characteristics of Mesh point i, represents the feature of point j in the point cloud, and ||·|| represents the L2-norm of the feature vector, which is used to normalize the similarity.

Images