A six-DOF pose estimation method for weakly textured objects

Abstract

This invention discloses a six-DOF pose estimation method for weakly textured objects. The method generates an object dataset using physically based rendering based on the object's CAD model. A convolutional neural network model with a single encoder and multiple decoders is constructed to obtain the object's prediction mask and high-dimensional feature map. A fully connected network is established to map the input object point cloud into a high-dimensional feature representation. The correlation matrix of the network's output feature map is calculated, and candidate poses are obtained through sampling, calculation, and scoring. The final pose is then refined. This invention completes model training using synthetic data, achieving excellent generalization ability without real-world scene annotation. By modeling the surface of weakly textured objects, it significantly improves the detection accuracy for a large number of industrial parts, demonstrating significant application value.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a method for six-degree-of-freedom pose estimation of objects with weak texture using machine vision technology. Background Technology

[0002] Determining the 3D translation and rotation transformation parameters (six degrees of freedom) of objects in a scene relative to a camera is a classic problem in computer vision. In recent years, with the development of emerging applications such as augmented reality, autonomous driving, and visual robots, there has been a greater demand for higher accuracy and speed in six-DOF pose estimation.

[0003] While the development of deep learning algorithms has led to an increasing number of pose estimation methods leveraging the powerful learning capabilities of neural networks to improve accuracy, traditional six-DOF pose estimation algorithms based on RGB images primarily rely on keypoint and line matching to establish a 2D-3D feature correspondence between the image and the 3D model. Then, based on this 2D-3D correspondence, the projection error function is optimized to estimate the target pose parameters. These methods are suitable for objects with rich texture information and high-quality images, achieving high pose estimation accuracy. However, their performance is less than ideal for objects with weak texture information and for scenes with occlusion or varying lighting conditions, resulting in significant estimation errors. Currently, many industrial workpieces have low texture and reflective surfaces, making this traditional method unsuitable for practical applications. Summary of the Invention

[0004] To overcome the shortcomings of the existing technology, this invention provides a six-degree-of-freedom pose estimation method for weakly textured objects based on deep learning. This method improves the problem of inaccurate pose estimation caused by the lack of surface features of objects, enhances the recognition effect of weak textures, and enables it to play a role in grasping scattered industrial parts.

[0005] To achieve its objectives, the present invention employs the following technical solution:

[0006] The six-DOF pose estimation method for weakly textured objects in this invention is characterized by the following steps:

[0007] Step 1: Perform offline training as follows, including training the object detection network YOLOX, the convolutional neural network, and the fully connected network:

[0008] Step 11: Construct a CAD model of the object to be identified, and use the BlenderProc program to generate a dataset of objects in different scenes through physical rendering, which will serve as the training data set;

[0009] Step 12: Use the images in the training dataset and the category labels in the images to complete the training of the object detection network YOLOX;

[0010] Step 13: Build a convolutional neural network model with a single encoder and multiple decoders to perform surface encoding based on the input image. Load the object categories and two-dimensional detection boxes in the training dataset as input. After passing through the encoder and the corresponding decoder, a thirteen-dimensional feature map is obtained. The first dimension is the prediction mask of the object in the image, and the other twelve dimensions are the predicted high-dimensional feature maps, denoted as high-dimensional feature map A1.

[0011] The encoder is used to extract high-dimensional features from the input and reduce the feature size. The multi-decoder provides a separate decoder for each category of object, with N categories of objects corresponding to N decoders.

[0012] Step 14: Build a fully connected network model, convert the points sampled from the CAD model of the object to be identified into a high-dimensional feature representation, and obtain the feature map of the rendered object in real pose, denoted as feature map B1;

[0013] Step 15: The training loss function consists of two parts. The first part is the cross-entropy loss between the predicted mask and the real mask. The second part is the comparison loss between the feature points in the high-dimensional feature map A1 sampled from the real mask and the feature points in the feature map B1. The sum of the first and second parts is used as the final loss. The neural network parameters are optimized by the ADAM optimizer. The training of the convolutional neural network and the fully connected network is completed.

[0014] Step 2: Perform six-degree-of-freedom pose estimation using linear inference as follows:

[0015] Step 21: For the RGB image of the scene captured by the camera, use the trained object detection network YOLOX to obtain the two-dimensional detection box and category information of the target object on the RGB image;

[0016] Step 22: Select an image region contained in a two-dimensional detection box and feed it into a convolutional neural network to obtain thirteen different feature maps. The first dimension of the feature map is the prediction mask of the object in the region, and the other twelve dimensions are the high-dimensional feature maps of the object in the region, thus obtaining the feature map matrix.

[0017] Step 23: Using the target object category information obtained in Step 21, load the 3D model points of the target object and input them into the fully connected neural network to obtain the high-dimensional information of the object point cloud, that is, obtain the high-dimensional information matrix of each object point, which has twelve dimensions.

[0018] Step 24: Multiply the feature map matrix with the high-dimensional information matrix of each object point to obtain the correlation matrix between the 2D image feature map points and the 3D object points. The correlation matrix is ​​then weighted with the probability confidence of the prediction mask to obtain the final correlation matrix. The final correlation matrix is ​​sampled multiple times, with 4 pairs of 2D-3D point pairs sampled each time.

[0019] Step 25: Calculate the candidate poses for the 2D-3D point pairs obtained from multiple samplings using the PnP algorithm;

[0020] Step 26: Score the candidate poses based on the confidence level of the predicted mask and the value of the correlation matrix, and select the candidate pose with the highest score; refine the candidate pose using an optimization algorithm to obtain the final pose, thus completing the pose estimation.

[0021] The six-DOF pose estimation method for weakly textured objects in this invention is also characterized by the following: In the single encoder and multiple decoder structure, the encoder is an encoder with an attention mechanism module, and the number of decoders corresponds to the object category and is one-to-one. The encoding and decoding process is as follows:

[0022] Step A: Adjust the size of the input RGB image to 3×224×224, and use it as the image to be encoded;

[0023] Step B: Input the image to be encoded into the encoder to obtain five feature maps of different sizes;

[0024] Step C: Select the corresponding decoder according to the object category, input the five feature maps into the decoder for feature extraction and fusion, and complete the decoding.

[0025] The six-DOF pose estimation method for weakly textured objects in this invention is also characterized by the following process: Step B obtains five feature maps of different sizes as follows:

[0026] Step B1: Input the image to be encoded into the Conv-BN-RELU convolutional layer to obtain the first feature map with a size of 64×112×112;

[0027] The Conv-BN-RELU convolutional layer has a kernel size of 7*7, a stride of 2, padding of 3, and 64 output channels.

[0028] Step B2: Input the first feature map into the max pooling layer, pass through the first-level residual block, and then weight the 64 channels of the first-level residual block through the attention mechanism module to obtain a second feature map with a size of 64×56×56.

[0029] The kernel size of the max pooling layer is 3*3, the stride is 2, and the padding is 1.

[0030] The first-level residual block consists of three residual blocks, each of which is composed of two convolutions. The number of convolution kernels is 64, and the size is 3*3.

[0031] Step B3: The second feature map is processed through a two-level residual block, and the 128 channels of the residual block are weighted by the attention mechanism module to obtain a third feature map with a size of 128×28×28.

[0032] The second-level residual block consists of four residual blocks, each of which is composed of two convolutions with 128 convolution kernels and a size of 3*3.

[0033] Step B4: The third feature map is processed through a three-level residual block, and the 256 channels of the residual block are weighted by the attention mechanism module to obtain a fourth feature map with a size of 256×14×14.

[0034] The third-level residual block consists of six residual blocks, each of which is composed of two convolutions with 256 convolution kernels of size 3*3.

[0035] Step B5: The fourth feature map is processed through a four-level residual block, and the 512 channels of the residual block are weighted by the attention mechanism module to obtain a fifth feature map with a size of 512×7×7.

[0036] The fourth-level residual block consists of three residual blocks, each of which is composed of two convolutions. The number of convolution kernels is 512, and the size is 3*3.

[0037] The characteristic of the six-DOF pose estimation method for weakly textured objects in this invention is that step C involves feature extraction and fusion according to the following process:

[0038] Step C1: Select the appropriate decoder based on the object category information output by the YOLOX object detection network;

[0039] Step C2: The fifth feature map is used as the input to the decoder. After passing through the convolutional layer and the upsampling layer, the first feature map of the decoder with a size of 512×14×14 is obtained, which is denoted as feature map T1.

[0040] Step C3: Pass the fourth feature map through a convolutional layer, then concatenate it with feature map T1, and then pass it through a convolutional layer and an upsampling layer to obtain the second feature map of the decoder with a size of 512×28×28, denoted as feature map T2;

[0041] Step C4: Pass the third feature map through a convolutional layer, then concatenate it with feature map T2, and then pass it through a convolutional layer and an upsampling layer to obtain the decoder's third feature map with a size of 256×56×56, denoted as feature map T3;

[0042] Step C5: Pass the second feature map through a convolutional layer, then concatenate it with feature map T3, and then pass it through a convolutional layer and an upsampling layer to obtain the fourth feature map of the decoder with a size of 256×112×112, denoted as feature map T4;

[0043] Step C6: Pass the first feature map through a convolutional layer, then concatenate it with feature map T4, and then pass it through a convolutional layer and an upsampling layer to obtain the fifth feature map of the decoder with a size of 128×224×224, denoted as feature map T5;

[0044] Step C7: Pass feature map T5 through a convolutional layer to obtain the sixth feature map of the decoder with a size of 64×224×224, denoted as feature map T6;

[0045] Step C8: Pass feature map T6 through a convolutional layer to obtain the seventh feature map of the decoder with a size of 13×224×224, denoted as feature map T7, thus completing feature extraction and fusion.

[0046] The six-DOF pose estimation method for weakly textured objects in this invention is also characterized by:

[0047] High-dimensional information of object point clouds can be obtained using a fully connected neural network as follows:

[0048] Step 51: Load the object point cloud according to the object category. If the number of point clouds is M, then the network input size is M×3.

[0049] Step 52: Build a four-layer fully connected network for each type of object;

[0050] Step 53: Load the corresponding fully connected network according to the object category and take the object point cloud as input. After passing through the fully connected network, the M×12 dimension feature map is obtained, which is the high-dimensional information of the object point cloud.

[0051] The six-DOF pose estimation method for weakly textured objects in this invention is also characterized by:

[0052] The four-layer fully connected network mentioned in step 52 is:

[0053] The first layer of the network is a fully connected layer with an input dimension of 3 and an output feature size of M×256.

[0054] The second layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of 256.

[0055] The third layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of 256.

[0056] The fourth layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of M×12.

[0057] The six-DOF pose estimation method for weakly textured objects in this invention is also characterized by:

[0058] The training loss function of convolutional neural networks and fully connected networks consists of the sum of two functions. The Adam optimizer is used to optimize the network parameters through gradient descent to minimize the overall L value.

[0059]

[0060] in:

[0061] L is the loss of the convolutional neural network and the fully connected network.

[0062] It is the set of coordinates uniformly sampled from the object mask in the current scene;

[0063] u is any coordinate uniformly sampled from the object mask in the current scene;

[0064] It is the transpose of the data values ​​of the feature map output by the input image through the convolutional neural network in coordinate u;

[0065] p u It is the coordinates of the original object surface points in coordinate u after the object point cloud is rendered into an image based on the actual pose.

[0066] k u It is p u Data values ​​output through a fully connected network;

[0067] It is a set of points uniformly sampled from the surface S of the CAD model of the object;

[0068] p i It is a set of points For any surface point in the middle, k i It is p i The value output after a fully connected network;

[0069] y i The label indicating whether an object exists in the current pixel is 0 or 1;

[0070] It is the probability that an object exists in the current pixel of the mask channel of the convolutional neural network;

[0071] I is the set of image pixel coordinates.

[0072] Compared with existing technologies, the beneficial effects of this invention are reflected in:

[0073] 1. The method of the present invention effectively improves the problem of inaccurate pose estimation caused by the lack of surface features of objects, improves the recognition effect of weak textures, and enables it to play a role in grasping scattered industrial parts.

[0074] 2. In this invention, the decoder part and the fully connected network part of the convolutional neural network are independent for a certain category of objects. This helps to improve the accuracy of object pose estimation, especially to reduce the problem of low 2D-3D point matching accuracy in weakly textured objects due to fewer features.

[0075] 3. In this invention, the training dataset is synthesized by rendering the object's CAD model, which eliminates the need for manual annotation of real scene data and simplifies the tedious steps of data annotation.

[0076] 4. This invention introduces YOLOX as the target detection network, which has a fast detection speed and high accuracy, and has a significant advantage over the commonly used two-stage networks such as Mask R-CNN.

[0077] 5. This invention uses convolutional neural networks and fully connected networks to generate dense 2D-3D point correspondences, which has better accuracy and robustness compared to using voting methods or regressing specific points of objects. Attached Figure Description

[0078] Figure 1 A flowchart for generating the training dataset in the method of this invention;

[0079] Figure 2 This is a flowchart of the network model training phase in the method of the present invention;

[0080] Figure 3 This is a flowchart of the network model inference stage in the method of the present invention;

[0081] Figure 4 This is a flowchart illustrating the implementation of the attention mechanism in the convolutional neural network in the method of this invention.

[0082] Figure 5 This is a diagram of the fully connected network structure in the method of the present invention; Specific implementation methods

[0083] In this embodiment, the six-DOF pose estimation method for weakly textured objects is performed according to the following steps:

[0084] Firstly according to Figure 2 The process shown involves offline training as follows: training the object detection network YOLOX, the convolutional neural network, and the fully connected network:

[0085] Step 11: Construct a CAD model of the object to be identified, build a physical simulation environment, and use the BlenderProc program to generate a dataset of objects in different scenes through physical rendering of the CAD model, which will serve as the training data set.

[0086] In this embodiment, the physics simulation engine is Blender, such as Figure 1 As shown, the rendering parameters are configured through the BlenderProc program: object CAD model, random object pose, random camera angle and lighting, and random scene. BlenderProc internally calls the Blender engine, which generates information including the object's RGB image and depth map, object category label, object bounding box and mask, and object pose. In this embodiment, the training set generates 50 scenes, each with 1000 images. For network training, the input data needs to be preprocessed. First, the RGB images in the training set are read, then the object category, bounding box, and mask information in the image are loaded. Each bounding box is enumerated, and the corresponding portion of the RGB image and mask image is cropped based on that box. After uniformly scaling the image to 224×224 size, it is used as the network input.

[0087] Step 12: Use the images in the training dataset and the category labels in the images to train the object detection network YOLOX.

[0088] Step 13: Construct a convolutional neural network model with a single encoder and multiple decoders to perform surface encoding based on the input image portion; load the object categories and two-dimensional detection boxes from the training dataset as input, and obtain a thirteen-dimensional feature map after passing through the encoder and the corresponding decoder. The first dimension is the prediction mask of the object in the predicted image, and the other twelve dimensions are the predicted high-dimensional feature maps, denoted as high-dimensional feature map A1; the encoder is used to extract high-dimensional features from the input and reduce the feature size, and the multiple decoders provide a separate decoder for each category of object to decode, with N categories of objects corresponding to N decoders.

[0089] The input to the convolutional neural network is a preprocessed RGB image of size 3×224×224. The image first passes through a Conv-BN-RELU layer with a 7*7 kernel, a stride of 2, padding of 3, and 64 output channels, producing a first feature map of size 64×112×112. This first feature map is then input to a max pooling layer, followed by a first-level residual block. An attention mechanism then weights the 64 channels of the first-level residual block to obtain a second feature map of size 64×56×56. The first-level residual block contains three residual blocks, each consisting of two convolutional layers with 64 kernels of size 3*3. The attention mechanism weights the 64 channels of the first-level residual block's output, enhancing the algorithm's robustness. Specifically, this layer... Figure 4 As shown, this layer first performs global average pooling on the feature map with input channel C, reducing the dimensions of H and W to 1 and retaining only the channel dimension. Then, it performs one-dimensional convolution, allowing each channel to interact with the channels of adjacent layers and share weights. After processing by the Sigmoid function, the feature map is finally multiplied by the weights of the corresponding channels.

[0090] Then, the data passes through a second-level residual block, and the attention mechanism module weights the 128 channels of the second-level residual block to obtain a third feature map with a size of 128×28×28. The second-level residual block contains four residual blocks, each consisting of two convolutions with 128 kernels of size 3*3. The attention mechanism module weights the 128-dimensional channels of the output of the second-level residual block. Next, the data passes through a third-level residual block, and the attention mechanism module weights the 256 channels of the third-level residual block to obtain a fourth feature map with a size of 256×14×14. The third-level residual block contains six residual blocks, each... Each residual block consists of two convolutions with 256 kernels and a size of 3*3. The attention mechanism module is used to weight the 256-dimensional channels of the output of the third-level residual block. Then, it passes through a fourth-level residual block, and the attention mechanism module weights the 512 channels of the fourth-level residual block to obtain a fifth feature map with a size of 512×7×7. The second-level residual block contains four residual blocks, each consisting of two convolutions with 128 kernels and a size of 3*3. The attention mechanism module is used to weight the 512-dimensional channels of the output of the fourth-level residual block. The encoder stage is now complete.

[0091] For feature map decoding, this embodiment generates a corresponding decoder for each type of object that needs pose estimation. During decoding, one decoder is selected based on the category of the object in the input image. The final output of the decoder is 13×224×224. The first channel is the mask in the RGB image predicted by the convolutional neural network, and the other twelve channels are the high-dimensional feature representation of the original RGB image at this pixel.

[0092] Step 14: Construct a fully connected network model to convert the points sampled from the CAD model of the object to be identified into a high-dimensional feature representation, obtaining the feature map of the rendered object in its true pose, denoted as feature map B1; specifically as follows... Figure 5 As shown, the CAD model of the object, after discrete sampling, has a size of M×3, where 3 represents the object's X, Y, and Z dimensions. It is then input into a fully connected network with an input feature dimension of 3 and an output feature dimension of 256. This is followed by two more fully connected layers with an input dimension of 256 and an output dimension of 256. Finally, it passes through a fully connected layer with an input dimension of 256 and an output dimension of 12, resulting in a final data size of M×12.

[0093] Step 15: The training loss function consists of two parts. The first part is the cross-entropy loss between the predicted mask and the real mask. The second part is the comparison loss between the feature points in the high-dimensional feature map A1 with the coordinates of the feature points in the feature map B1 sampled from the real mask. The sum of the first and second parts is used as the final loss. The neural network parameters are optimized by the ADAM optimizer. The training of the convolutional neural network and the fully connected network is completed.

[0094] In practice, the training loss function of convolutional neural networks and fully connected networks consists of the sum of two functions. The Adam optimizer is used to optimize the network parameters through gradient descent to minimize the overall L value.

[0095]

[0096] in:

[0097] L is the loss of the convolutional neural network and the fully connected network.

[0098] It is the set of coordinates uniformly sampled from the object mask in the current scene;

[0099] u is any coordinate uniformly sampled from the object mask in the current scene;

[0100] It is the transpose of the data values ​​of the feature map output by the input image through the convolutional neural network in coordinate u;

[0101] p u It is the coordinates of the original object surface points in coordinate u after the object point cloud is rendered into an image based on the actual pose.

[0102] k u It is p u Data values ​​output through a fully connected network;

[0103] It is a set of points uniformly sampled from the surface S of the CAD model of the object;

[0104] p i It is a set of points For any surface point in the middle, k i It is p i The value output after a fully connected network;

[0105] y i The label indicating whether an object exists in the current pixel is 0 or 1;

[0106] It is the probability that an object exists in the current pixel of the mask channel of the convolutional neural network;

[0107] I is the set of image pixel coordinates.

[0108] As the loss propagates back, the feature map generated by the convolutional neural network will approximate the fully connected model map; the fully connected network represents the surface feature information of the object, realizing surface encoding of objects with weak texture; the convolutional neural network generates surface information based on the input image, establishes a correlation matrix through the generated feature map in the online inference stage, obtains 2D-3D point pair information, and then obtains the pose through the PnP algorithm.

[0109] Then according to Figure 3 The following steps demonstrate how to achieve six-degree-of-freedom pose estimation using linear inference:

[0110] Step 21: For the RGB image of the scene captured by the camera, use the trained object detection network YOLOX to obtain the two-dimensional detection box, category information and confidence score of the target object on the RGB image.

[0111] Step 22: Select an image region contained in a two-dimensional detection box and feed it into a convolutional neural network to obtain thirteen different feature maps. The first dimension of the feature map is the prediction mask of the object in the region, and the other twelve dimensions are the high-dimensional feature maps of the object in the region, thus obtaining the feature map matrix.

[0112] Step 23: Using the target object category information obtained in Step 21, load the 3D model points of the target object and input them into a fully connected neural network to obtain the high-dimensional information of the object point cloud, that is, obtain the high-dimensional information matrix of each object point, which has twelve dimensions.

[0113] Step 24: Multiply the feature map matrix with the high-dimensional information matrix of each object point to obtain the correlation matrix between the 2D image feature map points and the 3D points of the object. The correlation matrix is ​​then weighted with the probability confidence of the prediction mask to obtain the final correlation matrix. The final correlation matrix is ​​sampled multiple times, with 4 pairs of 2D-3D point pairs sampled each time.

[0114] Step 25: Calculate the candidate poses for the 2D-3D point pairs obtained from multiple samplings using the PnP algorithm.

[0115] Step 26: Score the candidate poses based on the confidence level of the predicted mask and the value of the correlation matrix, and select the candidate pose with the highest score; refine the candidate pose using an optimization algorithm, such as the quasi-Newton method, to obtain the final pose and complete the pose estimation.

[0116] In the single encoder and multiple decoder architecture, the encoder is an encoder with an attention mechanism module, and the number of decoders corresponds to the object category in a one-to-one manner. The encoding and decoding process is as follows:

[0117] Step A: Adjust the size of the input RGB image to 3×224×224 as the image to be encoded.

[0118] Step B: Input the image to be encoded into the encoder to obtain five feature maps of different sizes.

[0119] Step C: Select the corresponding decoder according to the object category, input the five feature maps into the decoder for feature extraction and fusion, and complete the decoding.

[0120] In this embodiment, step B involves obtaining five feature maps of different sizes as follows:

[0121] Step B1: Input the image to be encoded into the Conv-BN-RELU convolutional layer to obtain the first feature map with a size of 64×112×112; the Conv-BN-RELU convolutional layer has a kernel size of 7*7, a stride of 2, padding of 3, and 64 output channels.

[0122] Step B2: Input the first feature map into the max pooling layer, pass it through the first-level residual block, and then pass it through the attention mechanism module to weight the 64 channels of the first-level residual block to obtain a second feature map with a size of 64×56×56; the convolution kernel size of the max pooling layer is 3*3, the stride is 2, and the padding is 1; the first-level residual block consists of three residual blocks, each residual block is composed of 2 convolutions, the number of convolution kernels is 64, and the size is 3*3.

[0123] Step B3: The second feature map is processed through a secondary residual block, and the 128 channels of the residual block are weighted by the attention mechanism module to obtain a third feature map with a size of 128×28×28. The secondary residual block consists of four residual blocks, each of which is composed of two convolutions with 128 convolution kernels and a size of 3*3.

[0124] Step B4: The third feature map is processed through a three-level residual block, and the 256 channels of the residual block are weighted by the attention mechanism module to obtain a fourth feature map with a size of 256×14×14; the three-level residual block consists of six residual blocks, each of which is composed of two convolutions with 256 convolution kernels of size 3*3.

[0125] Step B5: The fourth feature map is processed through a four-level residual block, and the 512 channels of the residual block are weighted by the attention mechanism module to obtain a fifth feature map with a size of 512×7×7; the four-level residual block consists of three residual blocks, each of which is composed of two convolutions with 512 convolution kernels of size 3*3.

[0126] In this embodiment, step C involves feature extraction and fusion as follows:

[0127] Step C1: Select the appropriate decoder based on the object category information output by the YOLOX object detection network.

[0128] Step C2: The fifth feature map is used as the input to the decoder. After passing through the convolutional layer and the upsampling layer, the first feature map of the decoder with a size of 512×14×14 is obtained, which is denoted as feature map T1; the number of convolutional kernels in the convolutional layer is 512, and the kernel size is 1*1.

[0129] Step C3: The fourth feature map is first passed through a convolutional layer, then concatenated with feature map T1, and then passed through a convolutional layer and an upsampling layer to obtain the second feature map of the decoder with a size of 512×28×28, denoted as feature map T2; the first convolutional layer has 256 convolutional kernels with a kernel size of 1*1, and the second convolutional layer has 512 convolutional kernels with a kernel size of 3*3.

[0130] Step C4: The third feature map is first passed through a convolutional layer, then concatenated with feature map T2, and then passed through a convolutional layer and an upsampling layer to obtain the decoder's third feature map with a size of 256×56×56, denoted as feature map T3; the first convolutional layer has 128 convolutional kernels with a kernel size of 1*1, and the second convolutional layer has 256 convolutional kernels with a kernel size of 3*3.

[0131] Step C5: The second feature map is first passed through a convolutional layer, then concatenated with feature map T3, and then passed through a convolutional layer and an upsampling layer to obtain the fourth feature map of the decoder with a size of 256×112×112, denoted as feature map T4; the first convolutional layer has 64 convolutional kernels with a kernel size of 1*1, and the second convolutional layer has 128 convolutional kernels with a kernel size of 3*3.

[0132] Step C6: The first feature map is first passed through a convolutional layer, then concatenated with feature map T4, and then passed through a convolutional layer and an upsampling layer to obtain the fifth feature map of the decoder with a size of 128×224×224, denoted as feature map T5; the first convolutional layer has 64 convolutional kernels with a kernel size of 1*1, and the second convolutional layer has 128 convolutional kernels with a kernel size of 3*3.

[0133] Step C7: Pass feature map T5 through a convolutional layer to obtain the sixth feature map of the decoder with a size of 64×224×224, denoted as feature map T6. The number of convolutional kernels in the convolutional layer is 64, and the kernel size is 3*3.

[0134] Step C8: Pass feature map T6 through a convolutional layer to obtain the seventh feature map of the decoder with a size of 13×224×224, denoted as feature map T7. The number of convolutional kernels in the convolutional layer is 13, and the kernel size is 1*1, thus completing feature extraction and fusion.

[0135] In this embodiment, high-dimensional information of the object point cloud is obtained using a fully connected neural network in the following manner:

[0136] Step 51: Load the object point cloud according to the object category. If the number of point clouds is M, then the network input size is M×3.

[0137] Step 52: Build a four-layer fully connected network for each type of object.

[0138] Step 53: Load the corresponding fully connected network according to the object category and take the object point cloud as input. After passing through the fully connected network, the M×12 dimension feature map is obtained, which is the high-dimensional information of the object point cloud.

[0139] In step 52, the four-layer fully connected network is as follows:

[0140] The first layer of the network is a fully connected layer with an input dimension of 3 and an output feature size of M×256.

[0141] The second layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of 256.

[0142] The third layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of 256.

[0143] The fourth layer of the network is a fully connected layer with an input dimension of 256 and an output feature size of M×12.

[0144] This invention trains models using synthetic data, achieving excellent generalization capabilities without requiring real-world scene annotations. By modeling the surfaces of weakly textured objects, it significantly improves the detection accuracy for a large number of industrial parts, demonstrating great application value.

Claims

1. A method for six degrees of freedom pose estimation of weakly textured objects, characterized in that The following steps are performed: Step 1: offline training is performed according to the following steps, including: training a target detection network YOLOX, a convolutional neural network, and a fully connected network: Step 11: a CAD model of the object to be identified is constructed, and a BlenderProc program is used to generate a dataset of the object in different scenes through physical rendering, serving as a training data set; Step 12: the training of the target detection network YOLOX is completed using the image and class label in the image in the training data set; Step 13: a convolutional neural network model with a single encoder and multiple decoders is built, so that the surface is encoded according to the input image part, the object category and two-dimensional detection box in the training data set are loaded as input, and thirteen-dimensional feature maps are obtained through the encoder and the corresponding decoder, wherein the first dimension is the predicted mask of the object in the image, and the other twelve dimensions are high-dimensional feature maps, denoted as high-dimensional feature map A1; The encoder is used to extract high-dimensional features from the input and reduce the feature size, and the multiple decoders provide a separate decoder for each category of object for decoding, and N categories of objects correspond to N decoders; Step 14: a fully connected network model is built, the points sampled from the CAD model of the object to be identified are converted into high-dimensional feature representations, and the feature map obtained by rendering the object with the real pose is obtained, denoted as feature map B1; Step 15: the loss function of the training is composed of two parts, the first part is the cross-entropy loss of the predicted mask and the real mask, and the second part is the comparison loss of the feature points corresponding to the coordinates in the high-dimensional feature map A1 and the feature points corresponding to the coordinates in the feature map B1; the sum of the losses of the first part and the second part is taken as the final loss, and the neural network parameters are optimized by the ADAM optimizer; the training of the convolutional neural network and the fully connected network is completed; Step 2: six-degree-of-freedom pose estimation is achieved through online reasoning according to the following steps: Step 21: for the RGB image in the scene photographed by the camera, the target detection network YOLOX obtained by training is used to obtain the two-dimensional detection box and category information of the target object on the RGB image; Step 22: select a picture area contained in a two-dimensional detection box and input it into the convolutional neural network to obtain thirteen different feature maps, wherein the first dimension of the feature is the predicted mask of the object in the region, and the other twelve dimensions are high-dimensional feature maps of the object in the region, thereby obtaining a feature map matrix; Step 23: using the target object category information obtained in step 21, the three-dimensional model points of the target object are loaded and input into the fully connected neural network to obtain high-dimensional information of the object point cloud, i.e., to obtain a high-dimensional information matrix of each object point, with a dimension of twelve dimensions; Step 24: multiply the feature map matrix and the high-dimensional information matrix of each object point to obtain a correlation matrix of 2D image feature points and object 3D points, and the correlation matrix is weighted with the probability confidence of the predicted mask to obtain a final correlation matrix, and the final correlation matrix is sampled multiple times, with 4 pairs of 2D-3D point pairs sampled each time; Step 25: Calculate the candidate poses according to the PnP algorithm for the 2D-3D point pairs obtained by multiple samplings; Step 26: Score the candidate poses according to the confidence of the prediction mask and the value of the final correlation matrix, and select the candidate pose with the highest score according to the score; refine the candidate pose through the optimization algorithm to obtain the final pose, and complete the pose estimation.

2. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 1, characterized in that In the single-encoder and multi-decoder structure, the encoder is an encoder with an attention mechanism module, the number of decoders corresponds to the object categories, and each decoder corresponds to one object category, and the encoding and decoding process is as follows: Step A, adjust the size of the input RGB image to 3x224x224 as the image to be encoded; Step B, input the image to be encoded into the encoder to obtain five feature maps of different sizes; Step C, select the corresponding decoder according to the category of the object, input the five feature maps into the decoder for feature extraction and fusion, and complete decoding.

3. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 2, characterized in that: Step B is to obtain five feature maps of different sizes according to the following process: Step B1, input the image to be encoded into the Conv-BN-RELU convolution layer to obtain a first feature map with a size of 64x112x112; The convolution kernel size of the Conv-BN-RELU convolution layer is 7*7, the step is 2, the padding is 3, and the output channel is 64; Step B2, input the first feature map into the max pooling layer, pass through a one-level residual block, and pass through an attention mechanism module to weight the 64-layer channel of the one-level residual block to obtain a second feature map with a size of 64x56x56; The convolution kernel size of the max pooling layer is 3*3, the step is 2, and the padding is 1; The one-level residual block is composed of three residual blocks, each residual block is composed of two convolutions, the number of convolution kernels is 64, and the size is 3*3; Step B3, pass the second feature map through a two-level residual block, and pass through an attention mechanism module to weight the 128-layer channel of the residual block to obtain a third feature map with a size of 128x28x28; The two-level residual block is composed of four residual blocks, each residual block is composed of two convolutions, the number of convolution kernels is 128, and the size is 3*3; Step B4: pass the third feature map through a three-level residual block, and pass through an attention mechanism module to weight the 256-layer channel of the residual block to obtain a fourth feature map with a size of 256x14x14; The three-level residual block is composed of six residual blocks, each residual block is composed of two convolutions, the number of convolution kernels is 256, and the size is 3*3; Step B5: pass the fourth feature map through a four-level residual block, and pass through an attention mechanism module to weight the 512-layer channel of the residual block to obtain a fifth feature map with a size of 512x7x7; The four-level residual block is composed of three residual blocks, each residual block is composed of two convolutions, the number of convolution kernels is 512, and the size is 3*3.

4. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 3, characterized in that: The step C is characterized by feature extraction and fusion in the following process: Step C1: selecting a corresponding decoder according to the object category information output by the target detection network YOLOX; Step C2: taking the fifth feature map as the input of the decoder, and obtaining a decoder first feature map with a size of 512x14x14 through a convolution layer and an upsampling layer, denoted as feature map T1; Step C3: taking the fourth feature map as the input of the decoder, and obtaining a decoder second feature map with a size of 512x28x28 through a convolution layer, splicing with the feature map T1, and then a convolution layer and an upsampling layer, denoted as feature map T2; Step C4: taking the third feature map as the input of the decoder, and obtaining a decoder third feature map with a size of 256x56x56 through a convolution layer, splicing with the feature map T2, and then a convolution layer and an upsampling layer, denoted as feature map T3; Step C5: taking the second feature map as the input of the decoder, and obtaining a decoder fourth feature map with a size of 256x112x112 through a convolution layer, splicing with the feature map T3, and then a convolution layer and an upsampling layer, denoted as feature map T4; Step C6: taking the first feature map as the input of the decoder, and obtaining a decoder fifth feature map with a size of 128x224x224 through a convolution layer, splicing with the feature map T4, and then a convolution layer and an upsampling layer, denoted as feature map T5; Step C7: taking the feature map T5 as the input of the decoder, and obtaining a decoder sixth feature map with a size of 64x224x224 through a convolution layer, denoted as feature map T6; Step C8: taking the feature map T6 as the input of the decoder, and obtaining a decoder seventh feature map with a size of 13x224x224 through a convolution layer, denoted as feature map T7, to complete the feature extraction and fusion.

5. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 1, characterized in that: The object point cloud high-dimensional information is obtained by using a fully connected neural network in the following manner: Step 51: loading the object point cloud according to the object category, and the number of point clouds is M, so the network input size is Mx3; Step 52: building a four-layer fully connected network for each object category; Step 53: loading the corresponding fully connected network according to the object category and taking the object point cloud as the input, and obtaining a feature map with a dimension of Mx12 through the fully connected network, which is the object point cloud high-dimensional information.

6. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 5, characterized in that: The four-layer fully connected network in step 52 is as follows: The first layer of the network is a fully connected layer with an input dimension of 3 and an output feature of 256, and the output feature size is Mx256; The second layer of the network is a fully connected layer with an input dimension of 256 and an output feature of 256, and the output feature size is Mx256; The third layer of the network is a fully connected layer with an input dimension of 256 and an output feature of 256, and the output feature size is Mx256; The fourth layer of the network is a fully connected layer with an input dimension of 256 and an output feature of 12, and the output feature size is Mx12.

7. The six-degree-of-freedom pose estimation method of weakly textured objects according to claim 1, characterized in that: The training loss function of the convolutional neural network and the fully connected network is composed of two functions, an Adam optimizer is adopted, and the network parameters are optimized by gradient descent to minimize the overall L value. Wherein: L is the loss of the convolutional neural network and the fully connected network; is a set of coordinates uniformly sampled from the object mask in the current scene; U is an arbitrary coordinate uniformly sampled from the object mask in the current scene; is the transpose of the data value of the feature map output by the convolutional neural network at coordinate u of the input image; p u It refers to the coordinates of the original object surface points in coordinate u after the object's point cloud is rendered into an image based on its actual pose; k u It is p u Data values ​​output through a fully connected network; is a set of points sampled uniformly from the surface S of the object CAD model; p i is a set of points any surface point in k i is p i value output by the fully connected network; y i a label indicating whether an object is present at the current pixel, 0 or 1; is the probability that the current pixel in the mask channel of the convolutional neural network exists an object; I is a set of image pixel coordinates.

Images