A semantic component pose estimation method based on deep learning

Abstract

The present invention discloses a semantic component pose estimation method based on deep learning, and proposes a SURF-FPS algorithm and a CPFNet model. The SURF-FPS algorithm is a three-dimensional key point sampling method for object models combined with the SURF algorithm, which is used in the data preprocessing stage. It can comprehensively consider the geometric features and potential surface texture features of the object, so that the selected object key points have more stable feature representativeness; the CPFNet model can make full use of the RGB image and depth information of the sample, and perform high-dimensional feature fusion between image information and point cloud information between each encoding layer and decoding layer; the point cloud feature encoding module merges the spatial information of the point cloud with the corresponding high-dimensional feature information, and the module can more completely extract the contextual component features of the target object, and perform global information fusion in the subsequent attention module, so as to comprehensively improve the pose estimation effect by utilizing key components with more obvious features.

Description

Technical Field

[0001] The present invention relates to the technical field of 6D pose estimation, and in particular to a semantic component pose estimation method based on deep learning. Background Art

[0002] As the trend toward intelligent urban transportation continues to gain momentum, enthusiasm for autonomous driving is growing. While there's still a gap to fully autonomous driving, and the autonomous technology currently used in cars can only be considered an assisted driving system, unable to completely replace human drivers, technological advancements are already far greater than ever before. Semi-autonomous vehicles equipped with lidar and multiple cameras are now undergoing human-supervised driving tests on real roads both domestically and internationally. This excitement is driven by the power of 6D pose estimation technology. Automatic obstacle avoidance and lane changes, actions that are easily accomplished by human drivers, require the detection and estimation of all surrounding objects. Only by accurately detecting and understanding the surroundings can the car maintain safe and continuous driving.

[0003] In recent years, the computing power of various electronic devices, large and small, has been growing exponentially, roughly in line with Moore's Law in the chip industry. This has led to a surge in the popularity of deep learning and neural networks, which rely on powerful computing power. This powerful computing power allows researchers to prioritize optimization when designing neural network models, rather than relying on computing power as a primary constraint. This allows users to achieve greater ease in practical applications. Currently, deep learning-based 6D pose estimation methods can be broadly categorized into three types: overall image estimation, image keypoint matching, and image-based depth information integration.

[0004] Holistic image estimation involves a neural network directly estimating the 6D pose of an object in view based on an input RGB image. This is an end-to-end estimation process, where a single network model completes the entire pose estimation in a single step. Do et al. proposed the Deep-6DPose network, which is divided into two main parts: object detection and segmentation, and estimation. The former uses region proposal networks (RPNs) to detect and segment various objects, while the latter performs 6D pose estimation on the segmented objects. During estimation, the parameters to be estimated within the transformation matrix are decoupled into translation and rotation parameters. This allows the rotation parameters to be regressed using Lie algebraic representations. The end-to-end network achieves inference speeds of up to 10 FPS, meeting the requirements for practical robotics. However, because the Deep-6DPose network actually predicts the pose of objects processed by the RPN, rather than directly predicting the pose of the target object in the input image, it is not very effective for estimating the 6D pose of small or symmetrical objects.

[0005] Image keypoint matching methods establish a mapping relationship between 2D points in an image and 3D points on an object model, and then derive the pose information of the target object through a series of mapped keypoints. Specifically, this method consists of two stages: first, detecting 2D keypoints in the input image; second, using a PnP algorithm to calculate the 6D pose of the target object. Because the step of extracting image keypoints already relies on neural network processing, and the detected keypoints must be calculated through a fixed algorithm to estimate the pose of the target object, this type of image keypoint matching method is generally slower than the whole image estimation method.

[0006] The above two methods mainly use RGB images as input information of the 6D pose estimation network. However, for the complex pose estimation challenges that are increasingly close to real scenes, only using one-sided images cannot achieve a relatively ideal and usable pose estimation accuracy. In some relatively favorable scenarios, when acquiring the image of the target object, its depth information can be obtained at the same time through the depth detection device. Moreover, even if it is impossible to obtain its depth information in real time while taking the image of the target object in actual application, RGB-D data can be used as input when training the network to provide the pose estimation network with richer color and depth information for learning the analysis pattern, so that it can have better performance in actual reasoning. Summary of the Invention

[0007] To address the above issues, the present invention aims to provide a semantic component pose estimation method based on deep learning. By utilizing a high-quality keypoint selection method, a point cloud encoding module that significantly enhances local contextual information, an attention mechanism module that aggregates high-quality global features, and an attention fusion mechanism for component poses, this method can more preferentially utilize key components with more distinct features to comprehensively improve pose estimation. The technical solution is as follows:

[0008] A method for estimating the pose of a semantic component based on deep learning, comprising the following steps:

[0009] Step 1: Build a component-level object pose estimation dataset that can be used for pose estimation based on the existing dataset of the original object model, and select a model with a separable structure as the target model;

[0010] Step 2: Considering the geometric feature information and potential surface texture information of the target model, the SURF-FPS algorithm is used to select the 3D key points of the target model;

[0011] Step 3: Construct the object's 6D pose estimation network model CPFNet based on the 3D key points of the target model. The Transformer structure with an attention mechanism is used as the architecture of the network model. In the Transformer's point cloud feature encoding module, the input point cloud spatial data and the corresponding multi-dimensional feature data are first enhanced to represent the local contextual features of the object parts. The enhanced features of each component are then sent to the attention mechanism module for overall feature integration and extraction.

[0012] In addition, a bidirectional fusion module of image and point cloud is added between the encoding module and decoding module of image features and point cloud features. This allows the fusion of RGB image information and depth point cloud information to not only consider the spatial position of the point cloud, but also make full use of the comprehensive features of the point cloud given by the aforementioned point cloud feature encoding module.

[0013] After obtaining the feature vector containing all the comprehensive features of the sample, the instance semantic segmentation module, instance center point voting module and key point detection module are used in the component-level adaptive posture regression module for processing, and the least squares method is used to fit the posture of the target object.

[0014] Furthermore, the step 2 specifically includes:

[0015] Step 2.1: Render the target model from multiple angles on the viewpoint sampling sphere to generate images covering all orientations of the target model and generate pseudocode for the pose of the target object in the camera coordinate system;

[0016] Step 2.2: Use the SURF algorithm to find 2D key points with distinct features in each image. Then, use the mapping relationship generated during rendering to find the original 3D points of the model corresponding to the 2D key points in all viewpoint images.

[0017] Step 2.3: Obtain the final target model key points from the filtered point set using the FPS algorithm.

[0018] Furthermore, the SURF algorithm specifically includes:

[0019] Step 2.2.1: Search the images of the target model in all scale spaces and construct the Hessian matrix to identify potential extreme interest points that are invariant to scale and selection;

[0020] First, perform Gaussian filtering on the image. The Hessian matrix after filtering is shown as follows:

[0021]

[0022] Where (x, y) represents the pixel position in image I, L xy (x,y,σ)=G(σ)*I(x,y) represents the Gaussian scale space obtained by convolving the image I with the Gaussian second-order mixture partial derivative, σ represents the scale of the Gaussian template used for convolution, G(σ) represents the Gaussian second-order partial derivative operation of the corresponding parameter scale, I(x,y) represents the value of the image I at (x,y), and g(δ) represents the Gaussian template function; L xx (x,y,σ) and L yy (x, y, σ) corresponds to the Gaussian scale space obtained by convolving the image I with the Gaussian second-order partial derivative in the x and y directions respectively;

[0023] Set the Hessian matrix discriminant to the following:

[0024] det(H)=L xx *L yy -(0.9L xy ) 2

[0025] Among them, L xx , L yy and L xy Respectively with L xx (x,y,σ),L yy (x,y,σ) and L xy (x, y, σ) have the same meaning;

[0026] Step 2.2.2: Construct the scale space, filter the feature points and perform precise positioning;

[0027] By comparing the det(H) of each point with its surrounding pixels in the image domain and the pixels in the adjacent scale domain, the extreme points are found as the initial filtered feature points, and then the stable feature points are obtained by eliminating the wrong positioning points and the points with weak energy;

[0028] Step 2.2.3: Assign feature direction;

[0029] With the stable feature point as the center, the Harr wavelet features in the 60° sector neighborhood are counted with a rotation step of 15°, and the sector direction with the largest eigenvalue is taken as the feature direction;

[0030] Step 2.2.4: Generate feature description

[0031] Several rectangular sub-regions are taken along the main direction of each feature point, and the Harr wavelet features of several pixel points in the horizontal and vertical main directions and their absolute values ​​are added inside each sub-region, and the sum is weighted according to the second-order Gaussian function. The Gaussian template and features in each region are convolved and normalized.

[0032] Furthermore, the specific processing steps in the object 6D pose estimation network model CPFNet are as follows:

[0033] Step 3.1: The point cloud feature encoding module first enhances the input point cloud spatial data and the corresponding multi-dimensional feature data through the context feature enhancement submodule to enable it to represent the local context features of the object parts. The enhanced component features are then sent to the attention mechanism module for overall feature integration and extraction.

[0034] Step 3.2: The image and point cloud bidirectional fusion module fuses the appearance features of the object model contained in the image with the geometric features of the object model contained in the point cloud. After each layer of encoding or decoding module extracts and aggregates the sample information, the further processed image features are integrated into the point cloud features, and the further processed point cloud features are integrated into the image features. Finally, the entire Transformer structure obtains the final features of all the information of the integrated object model.

[0035] Step 3.3: The final features are then passed into the relatively independent instance semantic segmentation module, center point voting module and key point detection module respectively. The instance semantic segmentation module gives semantic labels to the points in the scene belonging to different target object instances respectively. The instance center point voting module gives the offset direction of the center point of the target object instance to which it belongs point by point. The key point detection module detects the pre-selected key points in each target object module. The results output by the three modules work together and are supplemented by MeanSift's center clustering filter and least squares fitting to generate the final object pose estimation result.

[0036] Furthermore, the step 3.1 specifically includes:

[0037] Step 3.1.1: The input of each layer’s point cloud feature encoding module is an N×(3+C in ), where N represents the total number of points in the point cloud that need to be processed in the current layer, (3+C in ) represents the dimension of the spatial coordinates of the point and the dimension of the point cloud features fused with RGB features in the previous module;

[0038] Step 3.1.2: Perform ORB-FPS operation on the point cloud to obtain a sub-point set with geometric features and potential texture features, that is, S×(3+C in ) dimension matrix D0, which is then fed into the context feature enhancement submodule to extract the component context features of the object model; where S represents the number of new samples after the ORB-FPS operation;

[0039] Step 3.1.3: After obtaining the features of the integrated component semantic information, it is sent to the global feature extraction module. The attention module used here is based on the Lambda attention algorithm using linear attention and kernel factorization.

[0040] Atnn(Q,K,V)=Q(softmax(K) T V)

[0041] Among them, Q, K, V represent the weight matrices W of the input feature matrix and the Query, Key and Value defined in the attention mechanism respectively. q 、W k and W v Feature result after dot product;

[0042] Step 3.1.4: Use the improved Lambda attention mechanism LAttn(·) to combine the input features, that is, the output features of the context feature enhancement module, with the query weight matrix W q Before performing linear mapping, the convolution operation is used to aggregate local features and update geometric features, and the global features are obtained by interacting with the key-value features including the softmax operation using Q. The remaining Key weight matrix W k and Value weight matrix W v The corresponding calculation dimension transformation is also made.

[0043] Furthermore, the step 3.1.2 specifically includes:

[0044] Step a): Use the FLANN algorithm to aggregate the features of the context area of ​​each point to form a matrix D1, so that the data dimension is expanded to S×(3+C in)×K; where K represents the neighbor parameter set in the FLANN algorithm;

[0045] Step b): D0 before amplification is directly copied K times to form matrix D2 to match the dimension of the matrix after amplification. The result of subtraction between the two is then concatenated with the copied matrix D2 to obtain the characteristic matrix D3 of S×(6+2Cin)×K dimensions. The characteristic matrix D3 is essentially divided into submatrices D of S×6×K dimensions. 31 and S×2C in ×K-dimensional submatrix D 32 ;

[0046] Step c): Let the submatrix D 31 Encode through the MLP layer and combine the encoded spatial features with the sub-matrix D 32 The concatenation is performed to form the final output of the context feature enhancement module S×(C h +2C in )×K dimensional feature matrix D4; where C h Represents the output dimension set by the MLP layer.

[0047] Furthermore, the step 3.2 specifically includes:

[0048] Step 3.2.1: When the image and point cloud bidirectional fusion module first receives input, it uses the renderer's intrinsic parameter matrix to upscale the sample's depth map into a point cloud in the camera coordinate system. The 3D point cloud is used as a bridge between pixels and depth points. This results in a coordinate map that is fully pixel-aligned with the image feature map and a point cloud that fully corresponds to the point cloud features.

[0049] Step 3.2.2: In the direction of fusing image features into point cloud features, the FLANN algorithm is used to collect feature points s in the point cloud in the coordinate map. i Several neighboring points of the point cloud, the feature vector used for approximate nearest neighbor matching is the spatial coordinate s of the feature point in the point cloud i (x i ,y i ,z i ) and its corresponding point feature F Pi The texture and color features F are collected from the image feature maps corresponding to the neighboring points Ri To avoid the feature destruction caused by the hard integration of the neighboring features by the maximum pooling; the improved Lambda attention mechanism LAttn(·) is used to collect the aggregated features, and then the multi-layer perceptron σ(·) with shared weights is used to encode the integrated features to the same dimension as the point cloud features, and finally the multi-layer perceptron is used to combine it with the original point cloud features F P The spliced ​​features are encoded to obtain the point cloud features F after fusion of image featuresPfused ;

[0050] Step 3.2.3: In the direction of fusing point cloud features to image features, the FLANN algorithm is used to find each pixel r in the image features that has the corresponding coordinate map features. i Several neighboring points in the corresponding point cloud, collect the point features F corresponding to the neighboring points Pi , the feature vector used for approximate nearest neighbor matching is the spatial coordinate s in the pixel point corresponding coordinate map i (x i ,y i ,z i ) and its corresponding pixel feature F in the image feature Ri The splicing is then performed using a multi-layer perceptron with shared weights to encode it into the same dimension as the image feature, and then the attention mechanism is used to extract the aggregated features and combine them with the original image feature F R The splicing is then performed, and the splicing features are encoded through a multi-layer perceptron with shared weights to obtain the image features F after the fusion of point cloud features. Rfused .

[0051] Furthermore, the object 6D pose estimation network model CPFNet also includes adding skip connections between the encoding layer and the corresponding decoding layer of the image features and point cloud features, and embedding appropriate original sample information when decoding highly concentrated features in the decoding layer, so that the generated features can collect real original neighborhood information while aggregating comprehensive features; in the skip connection of the point cloud features, before using the multi-layer perceptron for encoding, the original point cloud features of this layer before processing are first extracted through the extended point convolutional network.

[0052] Furthermore, in step 3.3, the loss function of the semantic instance segmentation module is:

[0053]

[0054] Where N represents the number of semantic labels of all parts of each instance after compact sorting except the scene background, and p i represents the probability that a pixel given by the instance segmentation module belongs to label i, α i Represents the weight factor of label i. This hyperparameter is used to adjust the ratio between the loss of pixels belonging to this label and those not belonging to this label, (1-p i ) γ is a modulation factor used to reduce the loss contribution of easily classified pixels;

[0055] The loss function of the center point voting module and the loss function of the key point detection module are:

[0056]

[0057] Where V represents the estimated value of the unit vector pointing to the instance center point or instance key point at the three-dimensional space point of the target model corresponding to the m pixels in the sample excluding the background, express The true value of the unit vector at the corresponding point, express and The angle between

[0058] After integrating the loss functions of each module, the total loss function is shown as follows:

[0059] L=λ1L InsSeg +λ2L CtrOfst +λ3L KpOfst

[0060] Among them, λ1, λ2 and λ3 are the loss functions L of the instance semantic segmentation module respectively. InsSeg , the center point voting module loss function L CtrOfst And the loss function L of the key point detection module KpOfst The weight parameter of .

[0061] The beneficial effects of the present invention are

[0062] 1) The method of the present invention includes a more effective method for selecting three-dimensional key points of object models, SURF-FPS, which can take into account the potential weak texture information on the surface of the object model and the geometric constraint information of the model vertices.

[0063] 2) The CPFNet network model proposed in this paper takes RGB images and their corresponding depth images as input, adopts the Transformer architecture in the stage of extracting high-polymerization features, designs a point cloud feature encoding module that can more efficiently enhance the local context features of the target object, and adds a key approximate nearest neighbor selection algorithm to the image and point cloud bidirectional fusion module used to fuse object appearance information and geometric information between each encoding and decoding layer, so that this module can take into account higher-level feature information when fusing features.

[0064] 3) For the module that finally uses comprehensive features to derive the instance posture part, the present invention proposes a mechanism to obtain the overall posture of the instance by fusing the attention of each semantic component, because the components have different effects on the overall posture estimation of the instance.

[0065] 4) The present invention also proposes a loss function that matches the network model. This loss function can fully utilize the data characteristics of the feature streams of the image feature encoding branch and the point cloud feature encoding branch themselves, and effectively constrain each branch in the network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Schematic diagram of the overall architecture of the CPFNet model for 6D pose estimation of objects in the present invention.

[0067] Figure 2 Schematic diagram of the point cloud encoding module of the present invention.

[0068] Figure 3(a) is a schematic diagram of the fusion of image features into point cloud features in the image and point cloud bidirectional fusion module.

[0069] Figure 3(b) is a schematic diagram of the fusion of point cloud features into image features in the image and point cloud bidirectional fusion module.

[0070] Figure 4(a) shows the loss value curve of the CPFNet training curve during the model training phase.

[0071] Figure 4(b) shows the loss curve of the model verification stage of the CPFNet training curve. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The present invention proposes the SURF-FPS method and the CPFNet (Context Part Fusion Net) model. SURF-FPS is a three-dimensional key point sampling method for object models combined with the SURF algorithm. It is mainly used in the data preprocessing stage and can comprehensively consider the geometric features and potential surface texture features of the object, making the selected object key points more stable and representative. The CPFNet model uses the Transformer structure as the main architecture of the network model, which can make full use of the RGB image and depth information of the sample, and perform high-dimensional feature fusion between image information and point cloud information between each encoding layer and decoding layer.

[0074] Specifically, the present invention designs a point cloud feature encoding module. By merging the spatial information of the point cloud with the corresponding high-dimensional feature information, the module can more completely extract the contextual component features of the target object, and fuse the global information in the subsequent attention module, thereby utilizing key components with more obvious features to comprehensively improve the effect of posture estimation.

[0075] 1. Part-level object pose estimation dataset

[0076] The dataset of the present invention selects a common 6 types of object models in the ShapeNet dataset and the PartNet dataset. Since the main goal is to analyze the impact of object component segmentation on the accuracy of pose estimation, the selected object types can be obviously divided into different components based on common sense. A model with a separable structure is selected from each category. The number of each category is the number within the self-built dataset. The number of vertices and the number of meshes can reflect the fineness of the original model from the side. The number of sampling points is the number of sampling points provided by the model in the PartNet dataset. The total number of sampling points for each model is 10,000. The following table respectively counts the number of sampling points and meshes contained in each component. The specific information is shown in Table 1.

[0077] Table 1 Object original model information

[0078]

[0079] Considering that the coordinate representation of the models in ShapeNet and PartNet is offset, it is also necessary to calculate the transformation matrix between the models of the two data sets. The present invention adopts key point transformation calculation and verifies it with point set distance.

[0080] The total size of this dataset is 100,000, with a training sample to test sample ratio of 9:1. Each sample contains matching RGBD data and label data. The label data is divided into instance-level and component-level, which are used to evaluate the pose estimation effect at the instance level and component level respectively. The metadata of the dataset samples is shown in Table 2, where n represents the number of different instances.

[0081] Table 2 Dataset sample metadata information

[0082]

[0083] 2. Key point selection method

[0084] As the direct data for the final fitting of the object's posture, the object's key points play an extremely important role in the object's 6D posture estimation. In the preprocessing stage of the model data, if key points with high recognition can be selected, it will be very beneficial for the network model to detect and give key points.

[0085] Considering that the present invention does not process pure point cloud data, the selection of key points is combined with the texture or color information that may exist on the surface of the object model. Therefore, by rendering the corresponding RGB images of the target model from multiple angles during preprocessing, more unique key points can be assisted in the identification. The present invention adopts the SURF (Speeded Up Robust Features) algorithm. SURF is an efficient feature extraction and description algorithm with very stable features. It can remain unchanged not only for image brightness, rotation and scale transformation, but also has considerable stability against noise and perspective transformation. The main steps of the algorithm are as follows:

[0086] (1) We search images across all scale spaces and construct the Hessian matrix to identify potential extreme interest points that are invariant to scale and selection.

[0087] First, perform Gaussian filtering on the image. The Hessian matrix after filtering is shown as follows:

[0088]

[0089] Where (x, y) represents the pixel position in image I, L xy (x,y,σ)=G(σ)*I(x,y) represents the Gaussian scale space obtained by convolving the image I with the Gaussian second-order mixture partial derivative, σ represents the scale of the Gaussian template used for convolution, G(σ) represents the Gaussian second-order partial derivative operation of the corresponding parameter scale, I(x,y) represents the value of the image I at (x,y), and g(δ) represents the Gaussian template function; L xx (x,y,σ) and L yy (x, y, σ) corresponds to the Gaussian scale space obtained by convolving the image I with the Gaussian second-order partial derivative in the x and y directions respectively.

[0090] In order to speed up the calculation of the Hessian matrix determinant corresponding to each pixel in the image and use it to determine local feature points, SURF uses a box filter instead of a Gaussian filter. In order to balance the error caused by the box filter approximating the Gaussian kernel coefficient as it moves away from the center point, the Hessian matrix discriminant is set to:

[0091] det(H)=L xx *L yy -(0.9L xy ) 2

[0092] Among them, L xx , L yy and L xy Respectively with L xx (x,y,σ),L yy(x,y,σ) and L xy (x,y,σ) have the same meaning.

[0093] (2) Construct a scale space, filter feature points and perform precise positioning.

[0094] SURF's scale space consists of several groups and layers. The image size remains constant across groups, while the template size of the box filter used increases from group to group. The filter sizes seen in different layers within the same group remain the same, but the blur coefficients gradually increase, which also ensures that the computational cost of each layer is the same. This method compares the det(H) of each point with its surrounding pixels in the image domain and pixels in adjacent scale domains to identify extreme points as preliminary feature points. Stable feature points are then obtained by eliminating incorrectly located points and points with weaker energy.

[0095] (3) Feature direction assignment.

[0096] With the stable feature point as the center, the Harr wavelet features in the 60° sector neighborhood are counted with a rotation step of 15°, and the sector direction with the largest eigenvalue is taken as the feature direction;

[0097] (4) Generate feature description.

[0098] Several rectangular sub-regions are taken along the main direction of each feature point, and the Harr wavelet features of several pixel points in the horizontal and vertical main directions and their absolute values ​​are added within each sub-region, and the sum is weighted according to the second-order Gaussian function. In this process, the Gaussian template and features in each region are convolved and normalized to ensure illumination invariance.

[0099] Therefore, the present invention proposes the SURF-FPS algorithm. Specifically, the algorithm first renders the target model from multiple angles on the viewpoint sampling sphere to generate images covering all orientations of the object, and generates a pseudocode of the target object's posture in the camera coordinate system. Then, SURF is used in each image to find two-dimensional key points with obvious features, and then the original three-dimensional points of the model corresponding to the two-dimensional key points of all viewpoint images are found through the mapping relationship generated during rendering. Finally, the final target object key points are obtained from the screened point set through the FPS (Farthest point sampling) algorithm. In the pseudocode of the FPS algorithm, we select the center of gravity of the point cloud as the initial point s0 in the sampling point set S. The key points obtained by this method are more evenly distributed on the surface of the object and can point out more obvious features, which can reflect a better key point detection effect during the experiment.

[0100] 3. Network model design

[0101] The object 6D posture estimation model CPFNet proposed in the present invention is fitted based on the three-dimensional key points of the target model, so the main task of the network is to give the coordinates of each three-dimensional key point of the target object in the camera coordinate system. Considering that the samples provided by the posture estimation scene of the present invention are RGB images and their corresponding depth maps, the Transformer structure including the attention mechanism is adopted as the main architecture of the network model, and a module that can more efficiently fuse RGB image information and depth point cloud information is added between each encoding module and decoding module. After obtaining the feature vector containing all the comprehensive features of the sample, the instance semantic segmentation module, instance center point voting module and key point detection module are also used for processing, and finally the least squares method is used to fit the posture of the target object.

[0102] like Figure 1 The figure shows the overall architecture of the network model designed by the present invention. In view of the fact that the pose estimation target in the scene of this embodiment is a weakly textured object with relatively little RGB image information, and the target object has a strong relationship between components, the module for processing point clouds in the coding layer is targeted and redesigned, so that this module not only has a stronger ability to extract contextual local geometric features in processing point cloud feature information, but also adds an attention module that fuses and extracts global geometric features of point clouds, so that the network as a whole pays more attention to the features of object components. The module for fusing RGB image information and deep point cloud feature information has also been improved, so that when fusion is performed, not only the spatial position of the point cloud is considered, but also the comprehensive features of the point cloud given by the aforementioned point cloud feature encoding module are fully utilized, making the information fusion more characteristic. The present invention also improves the loss function of the network accordingly, so that the gradient descent direction of the entire network during training is more consistent with the semantic relationship of the object components.

[0103] (1) Point cloud feature encoding module

[0104] In the point cloud feature encoding module of the Transformer, the present invention first enhances the input point cloud spatial data and the corresponding multi-dimensional feature data through the context feature enhancement submodule to enable it to represent the local context features of the object parts, and then transmits the enhanced features of each component to the attention mechanism module for overall feature integration and extraction. The specific structure of this module is as follows: Figure 2 shown.

[0105] The input of each layer of the point cloud feature encoding module is an N×(3+C in ), where N represents the total number of points in the point cloud that need to be processed in this layer, (3+C in) represents the dimension of the spatial coordinates of the point and the dimension of the point cloud features that have been fused with RGB features in the previous module. First, the point cloud is subjected to an ORB-FPS operation to obtain a sub-point set with geometric features and potential texture features. ORB (Oriented Fast and Rotated Brief) is similar to the aforementioned SURF algorithm. It is an algorithm for feature point detection in two-dimensional images. It designs the Oriented FAST algorithm for feature extraction and the Rotated BRIEF algorithm for feature description. Its detection time is only 10% of that of the SURF algorithm, making it suitable for balancing feature detection speed and accuracy in network computing. The process of combining ORB and FPS is similar to the aforementioned SURF-FPS.

[0106] After ORB-FPS operation, we get S×(3+C in )-dimensional matrix D0, which is then sent to the context feature enhancement submodule to extract the component context features of the object model. Specifically, the FLANN (FastLibrary for Approximate Nearest Neighbors) algorithm is first used to aggregate the features of the context area of ​​each point to form a matrix D1, so that the data dimension is expanded to S×(3+Cin)×K, and then the matrix D0 before expansion is directly copied K layers to form a matrix D2 to match the matrix dimension after expansion. The result of the subtraction between the two is then concatenated with the copied matrix D2 to obtain a feature matrix D3 of S×(6+2Cin)×K dimensions. This matrix can be essentially divided into a submatrix D of S×6×K dimensions. 31 and a submatrix D of dimensions S×2Cin×K 32 , and then let the submatrix D 31 Encode through the MLP layer and combine the encoded spatial features with the sub-matrix D 32 The concatenation is performed to form the final output of the context feature enhancement module S×(C h +2C in )×K dimensional feature matrix D4.

[0107] In the feature flow process of the above-mentioned submodule, unlike the local spatial encoding submodule in RandLANet which only encodes the spatial coordinates of the points, the present invention also performs relative feature enhancement on the corresponding features of the points. In the entire process, the feature data matrix can essentially be divided into two parts: spatial position processing and feature processing of the point cloud. The subtraction and splicing operations on the feature part of the data matrix are to cooperate with the encoding operation of the spatial position, while the subtraction and splicing of the spatial position are to prepare for the encoding of the context neighborhood of the point through MLP. The subtraction of the spatial position of the point represents its relative position feature with similar spatial neighboring points. Then, splicing its original position can locate the spatial feature of the context area. The context feature encoding based on these two types of feature information can better enhance the topological relationship of the components.

[0108] The FLANN algorithm employed in this paper comprehensively considers all dimensional features of each point, including spatial coordinates, RGB color, corresponding normal vectors, and upper-layer fused point features, to select neighboring points that are most similar to the reference point in terms of high-dimensional, integrated features. FLANN, short for Fast Library for Approximate Nearest Neighbors, is an open-source library that primarily includes two algorithms—the Randomized KD Tree and the Hierarchical K-Means Tree—and their corresponding optimal algorithm selection mechanism. It enables nearest neighbor searches for large datasets containing high-dimensional features.

[0109] After obtaining the features of the integrated component semantic information, they are then sent to the global feature extraction module. The attention module used here is based on the Lambda attention algorithm using linear attention and kernel factor decomposition. Its simple expression is shown in the following formula:

[0110] Atnn(Q,K,V)=Q(softmax(K) T V)

[0111] Among them, Q, K, V represent the weight matrices W of the input feature matrix and the Query, Key and Value defined in the attention mechanism respectively. q 、W k and W v Feature result after dot product.

[0112] The attention module improves the Lambda attention algorithm to be more suitable for integrating features, that is, between the features and the query weight matrix W q Before performing linear mapping, the convolution operation is used to aggregate local features and update geometric features, and the global features are obtained by interacting with the key-value features including the softmax operation using Q. The weight matrix W of the remaining Key and Value is k Good Wv The corresponding calculation dimension transformation is also made. The specific calculation is shown in the following formula:

[0113] Q = EdgeConv(F)W q

[0114] K=Flatten(F)W k

[0115] V=Flatten(F)W v

[0116] y i =q i (softmax(k i ) T v i )

[0117] Among them, F represents the feature matrix after the semantic features of the components are enhanced, and EdgeConv is a convolution calculation method. is the output of the attention layer, q i 、k i and v i are the i-th subsequences of Q, K, and V respectively.

[0118] (2) Image and point cloud bidirectional fusion module

[0119] The image and point cloud bidirectional fusion module can fuse the appearance features of the object model contained in the image with the geometric features of the object model contained in the point cloud. After each layer of encoding or decoding module extracts and aggregates the sample information, the further processed image features are integrated into the point cloud features, and the further processed point cloud features are integrated into the image features. Both can provide additional features for the feature processing of the subsequent encoding or decoding layer. This can make the features finally obtained by the entire Transformer structure more comprehensive of all the information of the object model. The structure of this module is shown in Figure 3(a) and Figure 3(b).

[0120] Since the depth information of the sample flows between the point cloud feature encoding modules at different layers in the form of a depth map, when this module first accepts input, it will use the renderer's intrinsic parameter matrix to upgrade the sample's depth map into a point cloud in the camera coordinate system. These three-dimensional point clouds can serve as a bridge connecting pixel points and depth points, that is, obtaining a coordinate map that is completely pixel-aligned with the image feature map and a point cloud that completely corresponds to the point cloud features.

[0121] As shown in Figure 3(a), in the direction of fusing image features to point cloud features, feature points s in the point cloud are collected in the coordinate map. iSeveral neighboring points of the point cloud are collected by the FLANN algorithm mentioned above. The feature vector used for approximate nearest neighbor matching is the spatial coordinate s of the feature point in the point cloud. i (x i ,y i ,z i ) and its corresponding point feature F Pi The texture and color features F are collected from the image feature maps corresponding to these neighboring points Ri In order to avoid the feature destruction caused by the hard integration of neighboring features such as maximum pooling, the present invention adopts the aforementioned improved Lambda attention mechanism LAttn(·) to collect important aggregate features, and then uses a multi-layer perceptron σ(·) with shared weights to encode the integrated features to the same dimension as the point cloud features, and finally uses the multi-layer perceptron to combine them with the original point cloud features F P The spliced ​​features are encoded to obtain the point cloud features F after fusion of image features Pfused .

[0122] As shown in Figure 3(b), in the direction of fusing point cloud features to image features, the flow of features is approximately opposite to the above process. Specifically, the FLANN algorithm is used to find each pixel r in the image feature with the corresponding coordinate map feature R. i Several neighboring points in the corresponding point cloud, collect the point features F corresponding to the neighboring points Pi , the feature vector used for approximate nearest neighbor matching is the spatial coordinate s in the pixel point corresponding coordinate map i (x i ,y i ,z i ) and its corresponding pixel feature F in the image feature Ri Then, a multi-layer perceptron with shared weights is used to encode it into the same dimension as the image feature, and the same attention mechanism as above is used to extract important aggregate features and combine them with the original image feature F R The splicing is then performed, and the splicing features are encoded through a multi-layer perceptron with shared weights to obtain the image features F after the fusion of point cloud features. Rfused .

[0123] Because the dimensions of both point cloud features and image features decrease layer by layer during the encoding phase, it is necessary to maintain the correspondence between each point in the image features and the coordinate map to ensure that each pixel feature that survives compression encoding can be mapped to its original coordinate map. Considering that the convolution kernel mapping the original features is the cause of the reduction in image feature dimensionality, the center of the convolution kernel is directly used to approximate the new mapped coordinates. In the case where the image features do not match the feature dimensions of the coordinate map, the nearest interpolation algorithm is used to map the coordinate map to the same dimensions as the image features.

[0124] (3) Component-level adaptive posture regression module

[0125] With the help of the image and point cloud bidirectional fusion module, after the encoding and decoding process of the Transformer, high-density aggregated features that integrate the image appearance features and the point cloud geometric features can be obtained. The feature points are then projected into two-dimensional RGB images and depth images through the renderer's intrinsic parameter matrix. At the same time, the appearance features and geometric features are paired and spliced ​​through their mapping relationship to obtain the final features, and then the features are passed to the relatively independent instance semantic segmentation module, center point voting module and key point detection module respectively. These modules are composed of basic multi-layer perceptrons and convolutional layers. The instance semantic segmentation module is used to give semantic labels to the points in the scene belonging to different target object instances, the instance center point voting module is used to give the offset direction of the center point of the target object instance to which it belongs point by point, and the key point detection module is used to detect the pre-selected key points in each target object module. The results output by the three modules work together and are supplemented by MeanSift's center clustering filter and least squares fitting to generate the final object pose estimation result.

[0126] Different from other models that directly regress the entire target object, thanks to the component-level dataset dedicated to the experiment with precise component-level annotations, the three modules of the present invention that process the final features are targeted at specific components, that is, all components of each instance are segmented from the image, the center point of the voting direction is the center point of each component, and the key points detected are also based on component features and belong to each component. Since the pose estimation of the relevant component relative to the instance has both positive and negative effects, the pose estimation results K obtained by the present invention for each component of the same instance I are i =[R i ;T i ] is weighted, and the influence factor β of the posture estimation results of each component on the synthetic overall posture estimation result i It is variable and also changes with training as a set of special weight vectors. The weight values ​​after iteration are proportional to the significance of the features of the corresponding component in the pose estimation of the instance to which it belongs. Specifically, the translation matrix T in the pose transformation matrix of each component is i The synthesis is to directly take the average of the element values. i The synthesis requires solving the rotation angles in three directions, taking the average of each angle and then calculating the total rotation matrix. This process is recorded as σR i For instance I with n components, the mathematical representation of the final pose estimation result is shown below:

[0127]

[0128] (4) Skip Connection Module

[0129] Although the point cloud feature encoding module of the present invention can enhance the structural information of components by explicitly considering contextual features, due to the convolution of image features and the downsampling of point cloud features between the modules of each encoding layer, a large amount of original sample information is lost while aggregating and extracting high-concentration comprehensive features. Therefore, the present invention adds jump connections between the encoding layers and corresponding decoding layers of image features and point cloud features respectively. The jump connection can embed appropriate original sample information when the decoding layer decodes highly concentrated features, so that the generated features collect real original neighborhood information while aggregating comprehensive features.

[0130] Unlike multi-layer perceptrons (MLPs) that only use shared weights in image features, in the skip connection of point cloud features, features are first extracted from the original point cloud features of the current layer before encoding using the MLP. This network component can significantly increase the effective receptive neighborhood when extracting features of each point neighborhood by expanding the convolution of the point cloud, so that the contextual relationships of each point can be more preserved. Although increasing the number of corresponding residual structures can gradually increase the effective reception range of high-level receptive neighborhoods, making the information inserted by the skip connection richer, after comprehensively considering computational efficiency, the present invention ultimately limits the number of encoder and decoder layers to two.

[0131] Specifically, the ordinary point convolution algorithm directly calculates the k nearest neighbor points in the neighborhood of the reference point and performs convolution operations on them, as shown in the following formula:

[0132]

[0133] Among them, N j Indicates the reference point p j neighborhood points, ⊙ represents the Hadamard product, f(·) represents the function that maps the feature vector to the point coordinates, g(·) represents the kernel function that maps the relative position to the kernel weight, and the multilayer perceptron is used here to implement g(·).

[0134] The extended point convolution first calculates the k*d nearest neighbor points, then sorts them according to the distance to the reference point, takes a point every d nearest neighbor points, and finally obtains k points for convolution operation, using sparsity to expand the coverage of the convolution kernel in the neighborhood of the reference point.

[0135] (5) Loss function

[0136] When considering the loss function for the semantic instance segmentation module, given that the target object as the foreground only occupies a small part of the entire scene, segmenting the pixels contained in the target instance belongs to unbalanced data classification, and more attention needs to be paid to samples that are difficult to classify, such as instance edge discoloration. Since the samples of the present invention are multiple instances of a single scene, the semantic labels of the pixels in a single sample not only include several different instances and a background, but also include the component labels of each instance. Therefore, the instance semantic segmentation module actually completes a multi-classification task. The present invention uses the multi-classification version of FocalLoss as the loss function of the instance semantic segmentation module, as shown in the following formula:

[0137]

[0138] Where N represents the number of semantic labels of all parts of each instance after compact sorting except the scene background, and p i represents the probability that a pixel given by the instance segmentation module belongs to label i, α i Represents the weight factor of label i. This hyperparameter is used to adjust the ratio between the loss of pixels belonging to this label and those not belonging to this label, (1-p i ) γ is a modulation factor used to reduce the loss contribution of easily classified pixels.

[0139] For the center point voting module and the key point detection module, since the outputs of these two modules are vectors pointing to a certain point, their loss functions are combined and designed. Taking into account the particularity of vectors representing three-dimensional directions, if only the error vector obtained by subtracting the predicted vector from the true value vector is modulo, the directionality of the vector cannot be fully utilized. Therefore, the inner product form is adopted to make the loss function value of the angular error of the predicted vector exceed the value of the first power function, as shown in the following formula:

[0140]

[0141] Where V represents the estimated value of the unit vector pointing to the instance center point or instance key point at the three-dimensional space point of the target model corresponding to the m pixels in the sample excluding the background, express The true value of the unit vector at the corresponding point, express and Angle.

[0142] After integrating the loss functions of each module, the total loss function is shown as follows:

[0143] L=λ1L InsSeg +λ2L CtrOfst +λ3L KpOfst

[0144] Among them, λ1, λ2 and λ3 are the loss functions L of the instance semantic segmentation module respectively. InsSeg , the center point voting module loss function L CtrOfst And the loss function L of the key point detection module KpOfst The weight parameter of .

[0145] 4. Comparative Experiment on Multi-model Pose Estimation

[0146] The comparative experiment of this embodiment uses a total of five network models, and the comparative experiment is conducted under the same experimental conditions. Among them, CPFNet is an object posture estimation network model, and the FFB6D network is a benchmark model for precision analysis of object component relationships.

[0147] The PVN3D network is also a model dedicated to the task of estimating the 6D pose of an object from a single RGB-D image. This model does not directly regress the parameters of the pose matrix. Instead, like the FFB6D network, it uses the 3D key points of the target object as a bridge to estimate its pose in two stages. The first stage uses a deep Hough voting network to detect the 3D key points of the object, and the second stage estimates the 6D pose parameters by least squares fitting. This model innovatively extends the 2D key points used in traditional object 6D pose estimation to 3D, and can fully utilize the additional geometric constraints of rigid objects contained in the depth information.

[0148] The MaskedFusion network is a framework for estimating the 6D pose of objects using RGB-D data. Its architecture is a modular pipeline with multiple built-in subtasks that use different methods to achieve accurate 6D pose. It mainly expresses the characteristics of the object shape by including masks in the network and uses object masks to eliminate irrelevant data.

[0149] The main contribution of the PR-GCN network is the proposed point refinement graph convolutional network (PR-GCN), which uniformly addresses the two major limitations of RGB-D-based 6D pose estimation: poor representation of depth data and insufficient integration of different modalities. The network first introduces a point refinement network (PRN) to polish 3D point clouds, recovering missing parts while removing noise. It then proposes a multimodal fusion graph convolutional network (MMF-GCN) to enhance the combination of image and depth information. It captures multimodal correlations in geometric perception through local information propagation in the graph convolutional network. Experiments on several benchmark datasets demonstrate that the proposed PRN and MMF-GCN modules can be well generalized to other frameworks.

[0150] Table 3 Comparative experimental results of each network

[0151]

[0152] The training curves of the network models are shown in Figures 4(a) and 4(b). The pose estimation results of each network model on the component-level object pose estimation dataset of the present invention are shown in Table 3, with the best pose estimation result highlighted in bold. The experimental results show that for the component-level object pose estimation dataset of the present invention, since many objects contained in the sample scenes are distant and weakly textured, the image features that each network model can rely on are relatively insufficient. Therefore, in general, the benchmark models that have performed well in recent years used as comparisons have failed to match their performance in the widely used benchmark datasets.

[0153] The CPFNet network model proposed in this paper achieved the best results in the pose estimation task of each target object, and compared with the benchmark network model FFB6D used for pose estimation accuracy analysis, the average pose estimation accuracy of all instances was improved by 4.7% to 81.0%.

[0154] In summary, the method of the present invention includes a more effective method for selecting three-dimensional key points of object models, SURF-FPS, which can take into account the potential weak texture information on the object model surface and the geometric constraints of the model vertices. The CPFNet network model proposed in this invention takes an RGB image and its corresponding depth image as input. It adopts a Transformer architecture in the stage of extracting high-density features, designs a point cloud feature encoding module that can more efficiently enhance the local contextual features of the target object, and adds a key approximate nearest neighbor selection algorithm to the image and point cloud bidirectional fusion module used to fuse object appearance information and geometric information between the encoding and decoding layers, allowing this module to consider higher-level feature information when fusing features. For the module that finally uses comprehensive features to derive the instance pose, since the components have different effects on the overall instance pose estimation, a mechanism is proposed to obtain the overall instance pose through the attention fusion of each semantic component. The present invention also proposes a loss function that matches the network model. This loss function can fully utilize the data characteristics of the feature streams of the image feature encoding branch and the point cloud feature encoding branch to effectively constrain each branch in the network model.

Claims

1. A semantic component pose estimation method based on deep learning, characterized in that: The following steps are involved: Step 1: Build a component-level object pose estimation dataset that can be used for pose estimation based on the existing dataset of the original object model, and select a model with a separable structure as the target model; Step 2: Considering the geometric feature information and potential surface texture information of the target model, the SURF-FPS algorithm is used to select the 3D key points of the target model; Step 3: Construct the object's 6D pose estimation network model CPFNet based on the 3D key points of the target model. The Transformer structure with an attention mechanism is used as the architecture of the network model. In the Transformer's point cloud feature encoding module, the input point cloud spatial data and the corresponding multi-dimensional feature data are first enhanced to represent the local contextual features of the object parts. The enhanced features of each component are then sent to the attention mechanism module for overall feature integration and extraction. In addition, a bidirectional fusion module of image and point cloud is added between the encoding module and decoding module of image features and point cloud features. This allows the fusion of RGB image information and depth point cloud information to not only consider the spatial position of the point cloud, but also make full use of the comprehensive features of the point cloud given by the aforementioned point cloud feature encoding module. After obtaining the feature vector containing all the comprehensive features of the sample, the instance semantic segmentation module, instance center point voting module and key point detection module are used in the component-level adaptive posture regression module for processing, and the least squares method is used to fit the posture of the target object.

2. The semantic component pose estimation method based on deep learning according to claim 1, characterized in that The step 2 specifically includes: Step 2.1: Render the target model from multiple angles on the viewpoint sampling sphere to generate images covering all orientations of the target model and generate pseudocode for the pose of the target object in the camera coordinate system; Step 2.2: Use the SURF algorithm to find 2D key points with distinct features in each image. Then, use the mapping relationship generated during rendering to find the original 3D points of the model corresponding to the 2D key points in all viewpoint images. Step 2.3: Obtain the final target model key points from the filtered point set using the FPS algorithm.

3. The semantic component pose estimation method based on deep learning according to claim 2, characterized in that The SURF algorithm specifically includes: Step 2.2.1: Search the images of the target model in all scale spaces and construct the Hessian matrix to identify potential extreme interest points that are invariant to scale and selection; First, perform Gaussian filtering on the image. The Hessian matrix after filtering is shown as follows: Where (x, y) represents the pixel position in image I, L xy (x,y,σ)=G(σ)*I(x,y) represents the Gaussian scale space obtained by convolving the image I with the Gaussian second-order mixture partial derivatives, σ represents the scale of the Gaussian template used for convolution, G(σ) represents the Gaussian second-order partial derivative operation of the corresponding parameter scale, I(x,y) represents the value of the image I at (x,y), and g(δ) represents the Gaussian template function; L xx (x,y,σ) and L yy (x, y, σ) corresponds to the Gaussian scale space obtained by convolving the image I with the Gaussian second-order partial derivative in the x and y directions respectively; Set the Hessian matrix discriminant to the following: det(H)=L xx *L yy -(0.9L xy ) 2 Among them, L xx , L yy and L xy Respectively with L xx (x,y,σ),L yy (x,y,σ) and L xy (x, y, σ) have the same meaning; Step 2.2.2: Construct the scale space, filter the feature points and perform precise positioning; By comparing the det(H) of each point with its surrounding pixels in the image domain and the pixels in the adjacent scale domain, the extreme points are found as the initial filtered feature points, and then the stable feature points are obtained by eliminating the wrong positioning points and the points with weak energy; Step 2.2.3: Assign feature direction; With the stable feature point as the center, the Harr wavelet features in the 60° sector neighborhood are counted with a rotation step of 15°, and the sector direction with the largest eigenvalue is taken as the feature direction; Step 2.2.4: Generate feature description Several rectangular sub-regions are taken along the main direction of each feature point, and the Harr wavelet features of several pixel points in the horizontal and vertical main directions and their absolute values ​​are added inside each sub-region, and the sum is weighted according to the second-order Gaussian function. The Gaussian template and features in each region are convolved and normalized.

4. The semantic component pose estimation method based on deep learning according to claim 1, characterized in that The specific processing steps in the object 6D pose estimation network model CPFNet are as follows: Step 3.1: The point cloud feature encoding module first enhances the input point cloud spatial data and the corresponding multi-dimensional feature data through the context feature enhancement submodule to enable it to represent the local context features of the object parts. The enhanced component features are then sent to the attention mechanism module for overall feature integration and extraction. Step 3.2: The image and point cloud bidirectional fusion module fuses the appearance features of the object model contained in the image with the geometric features of the object model contained in the point cloud. After each layer of encoding or decoding module extracts and aggregates the sample information, the further processed image features are integrated into the point cloud features, and the further processed point cloud features are integrated into the image features, so that the entire Transformer structure finally obtains the final features of all the information of the comprehensive object model; Step 3.3: The final features are then passed into the relatively independent instance semantic segmentation module, center point voting module and key point detection module respectively. The instance semantic segmentation module gives semantic labels to the points in the scene belonging to different target object instances respectively. The instance center point voting module gives the offset direction of the center point of the target object instance to which it belongs point by point. The key point detection module detects the pre-selected key points in each target object module. The results output by the three modules work together and are supplemented by MeanSift's center clustering filter and least squares fitting to generate the final object pose estimation result.

5. The semantic component pose estimation method based on deep learning according to claim 4, characterized in that The step 3.1 specifically includes: Step 3.1.1: The input of each layer’s point cloud feature encoding module is an N×(3+C in ), where N represents the total number of points in the point cloud that need to be processed in the current layer, (3+C in ) represents the dimension of the spatial coordinates of the point and the dimension of the point cloud features fused with RGB features in the previous module; Step 3.1.2: Perform ORB-FPS operation on the point cloud to obtain a sub-point set with geometric features and potential texture features, that is, S×(3+C in ) dimension matrix D0, which is then fed into the context feature enhancement submodule to extract the component context features of the object model; where S represents the number of new samples after the ORB-FPS operation; Step 3.1.3: After obtaining the features of the integrated component semantic information, it is sent to the global feature extraction module. The attention module used here is based on the Lambda attention algorithm using linear attention and kernel factorization. Atnn(Q,K,V)=Q(softmax(K) T V) Among them, Q, K, V represent the weight matrices W of the input feature matrix and the Query, Key and Value defined in the attention mechanism respectively. q 、W k and W v Feature result after dot product; Step 3.1.4: Use the improved Lambda attention mechanism LAttn(·) to combine the input features, that is, the output features of the context feature enhancement module, with the query weight matrix W q Before performing linear mapping, the convolution operation is used to aggregate local features and update geometric features, and the global features are obtained by interacting with the key-value features including the softmax operation using Q. The remaining Key weight matrix W k and Value weight matrix W v The corresponding calculation dimension transformation is also made.

6. The method for semantic component pose estimation based on deep learning according to claim 5, characterized in that The step 3.1.2 specifically includes: Step a): Use the FLANN algorithm to aggregate the features of the context area of ​​each point to form a matrix D1, so that the data dimension is expanded to S×(3+C in )×K; where K represents the neighbor parameter set in the FLANN algorithm; Step b): D0 before amplification is directly copied K times to form matrix D2 to match the dimension of the matrix after amplification. The result of subtraction between the two is then concatenated with the copied matrix D2 to obtain the characteristic matrix D3 of S×(6+2Cin)×K dimensions. The characteristic matrix D3 is essentially divided into submatrices D of S×6×K dimensions. 31 and S×2C in ×K-dimensional submatrix D 32 ; Step c): Let the submatrix D 31 Encode through the MLP layer and combine the encoded spatial features with the sub-matrix D 32 The concatenation is performed to form the final output of the context feature enhancement module S×(C h +2C in )×K dimensional feature matrix D4; C h Represents the output dimension set by the MLP layer.

7. The method for semantic component pose estimation based on deep learning according to claim 5, characterized in that: The step 3.2 specifically includes: Step 3.2.1: When the image and point cloud bidirectional fusion module first receives input, it uses the renderer's intrinsic parameter matrix to upgrade the sample's depth map into a point cloud in the camera coordinate system, and uses the 3D point cloud as a bridge connecting pixel points and depth points. That is, a coordinate map that is completely pixel-aligned with the image feature map and a point cloud that completely corresponds to the point cloud features are obtained; Step 3.2.2: In the direction of fusing image features into point cloud features, the FLANN algorithm is used to collect feature points s in the point cloud in the coordinate map. i Several neighboring points of the point cloud, the feature vector used for approximate nearest neighbor matching is the spatial coordinate s of the feature point in the point cloud i (x i ,y i ,z i ) and its corresponding point feature F Pi The texture and color features F are collected from the image feature maps corresponding to the neighboring points Ri To avoid the feature destruction caused by the hard integration of the neighboring features by the maximum pooling; the improved Lambda attention mechanism LAttn(·) is used to collect the aggregated features, and then the multi-layer perceptron σ(·) with shared weights is used to encode the integrated features to the same dimension as the point cloud features, and finally the multi-layer perceptron is used to combine it with the original point cloud features F P The spliced ​​features are encoded to obtain the point cloud features F after fusion of image features Pfused ; Step 3.2.3: In the direction of fusing point cloud features to image features, the FLANN algorithm is used to find each pixel r in the image features that has the corresponding coordinate map features. i Several neighboring points in the corresponding point cloud, collect the point features F corresponding to the neighboring points Pi , the feature vector used for approximate nearest neighbor matching is the spatial coordinate s in the pixel point corresponding coordinate map i (x i ,y i ,z i ) and its corresponding pixel feature F in the image feature Ri The splicing is then performed using a multi-layer perceptron with shared weights to encode it into the same dimension as the image feature, and then the attention mechanism is used to extract the aggregated features and combine them with the original image feature F R The splicing is then performed, and the splicing features are encoded through a multi-layer perceptron with shared weights to obtain the image features F after the fusion of point cloud features. Rfused .

8. The method for semantic component pose estimation based on deep learning according to claim 4, characterized in that: The object 6D pose estimation network model CPFNet also includes adding skip connections between the encoding layers and corresponding decoding layers of the image features and point cloud features, embedding appropriate original sample information when decoding highly concentrated features in the decoding layer, so that the generated features collect real original neighborhood information while aggregating comprehensive features; in the skip connection of the point cloud features, before using the multi-layer perceptron for encoding, the original point cloud features of the current layer before processing are first extracted through the extended point convolutional network.

9. The semantic component pose estimation method based on deep learning according to claim 4, characterized in that In step 3.3, the loss function of the semantic instance segmentation module is: Where N represents the number of semantic labels of all parts of each instance after compact sorting except the scene background, and p i represents the probability that a pixel given by the instance segmentation module belongs to label i, α i Represents the weight factor of label i. This hyperparameter is used to adjust the ratio between the loss of pixels belonging to this label and those not belonging to this label, (1-p i ) γ is a modulation factor used to reduce the loss contribution of easily classified pixels; The loss function of the center point voting module and the loss function of the key point detection module are: Where V represents the estimated value of the unit vector pointing to the instance center point or instance key point at the three-dimensional space point of the target model corresponding to the m pixels in the sample excluding the background, express The true value of the unit vector at the corresponding point, express and The angle between After integrating the loss functions of each module, the total loss function is shown as follows: L=λ1L InsSeg +λ2L CtrOfst +λ3L KpOfst Among them, λ1, λ2 and λ3 are the loss functions L of the instance semantic segmentation module respectively. InsSeg , the center point voting module loss function L CtrOfst And the loss function L of the key point detection module KpOfst The weight parameter of .

Images