Weak texture object pose estimation method based on deep learning and synthetic data

Abstract

The application discloses a weak texture object pose estimation method based on deep learning and synthetic data, renders a non-texture CAD model to obtain synthetic data, obtains a pose potential feature extraction network through a self-encoder network composed of five convolution layers of a CBAM module, makes a codebook, inputs a real test image into an improved Mask R-CNN instance segmentation network to obtain bounding box information, performs cutting on the bounding box information, inputs the bounding box information into the pose potential feature extraction network to obtain a potential feature vector, calculates a cosine similarity according to the potential feature vector and a template potential vector, adopts a k nearest neighbor algorithm to obtain a 3D rotation pose, obtains a z-axis translation estimation value according to boundary box information of CAD model data in the real test image, adopts a conversion relationship between an image coordinate system and a camera coordinate system to obtain x-axis and y-axis translation amounts, improves the network expression feature capability, reduces the calculation complexity, and improves the 6D pose estimation recognition precision while reducing the model parameter amount.

Description

Technical Field

[0001] This invention relates to 6D pose estimation technology for objects in the fields of computer vision and robotic arm grasping, and particularly to a method for 6D pose estimation of objects with weak texture. Background Technology

[0002] 6D object pose estimation helps operators obtain 3D translation and rotation information of rigid bodies in the camera coordinate system, thereby improving operator accuracy and efficiency. Traditional object pose estimation methods are generally based on feature point correspondence and template matching. These methods perform poorly in occlusion situations and with weakly textured objects, making them unsuitable for today's complex industrial environments. With the development of deep learning technology, more and more deep learning-based 6D pose estimation methods have been proposed.

[0003] Chinese patent application number 202010619800.7 proposes a single-image robot unordered target grasping method based on pose estimation and correction. However, this method requires a large amount of real-world datasets, resulting in high cost, heavy workload, and difficult production. Chinese patent application number CN202110921177.5 proposes a pose estimation method and system for weakly textured objects, utilizing a point rendering segmentation network with an embedded instance segmentation method. However, due to network limitations, this pose estimation method relies on depth information and real-world images in practical tasks, lacking flexibility. Currently, although deep learning-based methods outperform traditional methods, they typically require labeled real-world datasets, resulting in significant dataset production costs. 6D pose estimation methods for weakly textured and synthetic data struggle to achieve both high accuracy and speed. Summary of the Invention

[0004] The purpose of this invention is to address the problems of low accuracy, slow speed, and difficulty in dataset creation for 6D pose estimation of weakly textured objects in current industrial scenarios. A method for weakly textured object pose estimation based on deep learning and synthetic data is proposed. Synthetic data is used for training to overcome the data acquisition difficulty, and algorithmic fusion improves detection accuracy while maintaining detection speed.

[0005] To achieve the above objectives, the technical solution of the present invention, a weakly textured object pose estimation method based on deep learning and synthetic data, is as follows: obtaining a T-LESS dataset of several textureless objects to obtain a textureless CAD model for each object, and further including the following steps:

[0006] Step 1): Render the textureless CAD model to obtain the composite data and the bounding box information corresponding to each data point;

[0007] Step 2): Decouple the 6D pose of the object into a combination of a rotation pose and a translation amount. Each data in the synthesized data corresponds to a rotation pose and a translation amount.

[0008] Step 3): The synthesized data is passed through an autoencoder network consisting of 5 convolutional layers fused with the CBAM module to obtain the template latent vector z. i After passing through 5 more deconvolutional layers to complete one full iteration, a pose latent feature extraction network is trained. The synthetic data is then input into this network to create a codebook, where the template latent vector z is located. i Each data point in the synthetic data corresponds one-to-one with the rotational pose of each data point.

[0009] Step 4): Input the real test image into the improved Mask R-CNN instance segmentation network that integrates Swin Transformer to obtain feature maps. Set a predetermined ROI for each point in the feature map. ROIs are filtered by RPN to obtain candidate ROIs. Perform ROIAlign on the candidate ROIs to obtain bounding box information.

[0010] Step 5) Crop the bounding box information described in Step 4) and input it into the pose latent feature extraction network described in Step 3) to obtain the corresponding latent feature vector z. test ;

[0011] Step 6): Based on the latent feature vector z test With the template latent vector z i The cosine similarity is calculated, and then the k-nearest neighbor algorithm is used to find the k templates that are closest to the pose of the object image. The poses corresponding to these k templates are the 3D rotation poses of the object image.

[0012] Step 7): First, obtain the estimated value of the z-axis translation based on the bounding box information of the CAD model data in the synthetic data corresponding to the object in the real test image and the bounding box information mentioned in Step 4). Then, use the transformation relationship between the image coordinate system and the camera coordinate system to obtain the translation of the object in the real test image on the x-axis and y-axis.

[0013] The advantages of this invention using the above technical solution are as follows:

[0014] (1) This invention designs a denoising autoencoder network that integrates CBAM (Convolutional Attention Module) for 6D pose estimation tasks. It is obtained by incorporating CBAM into existing denoising autoencoder algorithms. This module suppresses irrelevant features from both the channel and spatial dimensions, highlights object features, suppresses irrelevant features, and improves the network's ability to express features. Therefore, the denoising autoencoder can better extract latent features.

[0015] (2) This invention designs an instance segmentation network for 6D pose estimation, integrating the Swin Transformer into the improved instance segmentation model Mask R-CNN. Addressing the low accuracy and slow speed of the traditional Mask R-CNN instance segmentation network, the backbone network is replaced with a Swin Transformer network structure. This reduces the number of network parameters while improving detection accuracy. Due to the sliding window operation of the Swin Transformer, the computational complexity is greatly reduced, enabling the network to handle large-size image inputs while maintaining low memory usage and significantly reducing the number of model parameters. Furthermore, the improved Mask R-CNN instance segmentation network addresses the lack of low-level information in the feature map information passed from the FPN to the RPN by adding bottom-up backward connection channels to the traditional FPN structure, thus improving the accuracy of the instance segmentation network in this invention.

[0016] (3) This invention is a 6D pose estimation of weakly textured objects based on synthetic data. It uses virtual rendering, spatial sampling and domain randomization techniques to train the improved denoising autoencoder network of this invention, thereby reducing the domain gap between the virtual model and the real model and greatly reducing the implementation cost of the 6D pose estimation method. This invention integrates the improved algorithm into the whole model, so that this invention can improve the 6D pose estimation recognition accuracy while reducing the number of model parameters. Attached Figure Description

[0017] Figure 1 This is a flowchart of a weakly textured object pose estimation method based on deep learning and synthetic data proposed in this invention;

[0018] Figure 2 This is a structural diagram of the convolutional noise reduction autoencoder that incorporates CBAM in this invention;

[0019] Figure 3 This is a diagram of the instance segmentation network structure that incorporates the improved Swing Transformer backbone network in this invention. Detailed Implementation

[0020] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0021] First, a T-LESS dataset of at least 30 textureless industrial objects was acquired. These objects exhibit symmetry, with some sharing similar shapes. Data acquisition scenarios included non-clustered, single-object instances, as well as complex cases involving clustering, occlusion, and multiple object instances. The T-LESS dataset provides textureless CAD models for each object, with images acquired using various RGB-D sensors such as Microsoft v2. When evaluating 6D pose estimation algorithms, images of industrial objects acquired using a Primesense sensor are typically chosen as the test set, providing a mask image and pose for each object.

[0022] like Figure 1 As shown, for a textureless CAD model, OpenGL rendering tools are used to render it, obtaining a large amount of composite data A, and recording the pose information in virtual space. Specifically, the textureless CAD model is first opened in Meshlab, and the center of the CAD model is translated to the center of the world coordinate system origin, thus ensuring that the rendered viewpoint is close to the test data. Then, OpenGL is used to load the .obj format CAD model point cloud data, and after determining the camera focal length parameters, principal point coordinates, and camera extrinsic parameters, composite data B against a pure background can be rendered. At this time, except for the CAD model, the background is displayed as pure black. After binarization, the mask information can be obtained. Then, the white area is superimposed on the pure black background using the mask information, and after Fibonacci mesh sampling, the required large amount of composite data A is finally obtained. Composite data A records the bounding box information (u) corresponding to each data point. j ,v j ,w j ,h j ), which is the pose information in virtual space, 1≤j≤n, where n is the number of sampled viewpoints, u: represents the horizontal coordinate of the top left corner of the bounding box; v: represents the vertical coordinate of the top left corner of the bounding box; w: represents the width of the bounding box; h: represents the height of the bounding box.

[0023] Fibonacci grid sampling specifically involves sampling n viewpoints, each with coordinates (x, y, y). n y n , z n ), obtained from the following formula:

[0024]

[0025] in, It is the golden ratio.

[0026] Then, a decoupling analysis is performed on the 6D pose estimation problem of an object: 6D pose estimation of an object refers to determining the rotation and translation of the object in three-dimensional space, which is usually represented as a combination of rotation pose and translation. Therefore, each data point in the synthetic data A corresponds to a rotation pose R. i It involves a translation. However, rotation domain translation consists of two relatively independent transformations, decoupling the 6D pose estimation of an object into 3D rotation and 3D translation, thus simplifying and making the 6D pose estimation problem more controllable.

[0027] Then, a denoising autoencoder incorporating CBAM is used as the feature extraction network. The latent feature extraction network is trained using synthetic data A with domain randomization, and a codebook is constructed by combining the corresponding vectors obtained from training on synthetic data A. Specifically: Figure 2 As shown, 20,000 synthetic data A images with a resolution of 128×128 are generated using a textureless CAD model of the object and trained using domain randomization, with a batch size of 64. The synthetic data A processed by the domain randomization method is input into a denoising autoencoder network fused with CBAM. After passing through an encoder structure consisting of 5 convolutional layers of the CBAM module, latent vectors are obtained, followed by 5 deconvolutional layers to complete one complete iteration. After 40,000 iterations, the model learns important features, and finally a pose latent feature extraction network is trained. Then, the synthetic data A is input into this pose latent feature extraction network to create a codebook.

[0028] The denoising autoencoder network incorporating CBAM integrates the first three layers of the encoder in a convolutional denoising autoencoder with CBAM. The purpose of integrating CBAM is to improve the perceptual and representational capabilities of the CNN, enabling it to better adapt to complex and varied visual tasks. By introducing channel and spatial attention mechanisms, CBAM can automatically select and adjust the features extracted by the network, making them more accurate and targeted.

[0029] The domain randomization method is as follows: During the training process, 20,000 synthetic data A images with a resolution of 128×128 are generated from the textureless CAD model of the object. Random scaling and translation transformations are added to the template images as noise. In order to obtain more effective data, the domain randomization method is used to perform data augmentation as follows: (1) Randomly replace the background with images from the MS COCO and PASCAL VOC datasets; (2) Randomly add a certain degree of black square occlusion; (3) Randomly change the image contrast, brightness and color distortion; (4) Randomly adjust the Gaussian blur; (5) Randomly apply additional enhancements to the input training images: render using random light positions and random diffuse and specular reflections. Among them, the first four data augmentation methods are applied to the synthetic data A with a 50% probability, and all are processed by the fifth enhancement method.

[0030] The codebook is constructed as follows: In the final synthetic data A obtained through Fibonacci grid sampling, the pose feature represented by each data point is matched one-to-one with its corresponding 128-dimensional template latent vector to create a codebook. The codebook is the template latent vector z obtained from data point i in synthetic data A through a latent feature extraction network. i The rotation matrix R corresponding to each data point in the synthetic data A i A one-to-one correspondence is established and recorded in the codebook, where 1 ≤ i ≤ n.

[0031] like Figure 3 As shown, this invention inputs real test images into a modified Mask R-CNN instance segmentation network that integrates Swin Transformer for object detection and segmentation. Specifically, composite structures 1-4 are the Swin-T structure in the Swin Transformer network. First, the images pass through a backbone network Mask R-CNN composed of four layers of composite structures 1-4 and a modified FPN structure to obtain a feature map. Then, a predetermined ROI (Region of Interest) is assigned to each point in this feature map, resulting in multiple ROIs. These ROIs are then fed into the RPN (Region Proposal Network) to filter out some ROIs, obtaining candidate ROIs. Next, ROIAlign (feature alignment) is performed on the candidate ROIs. The aligned candidate ROIs are then classified and bounding box regressed to generate a mask, completing object detection and segmentation. At this point, the bounding box information of the bounding box corresponding to the real test image is represented as (u w ,v w ,w w ,h w Let u represent the horizontal coordinate of the top-left corner of the bounding box, v represent the vertical coordinate of the top-left corner of the bounding box, w represent the width of the bounding box, and h represent the height of the bounding box. Perform an FCN operation to generate a mask within each ROI.

[0032] The conventional FPN structure only contains feature information from the current and upper layers in the feature map input to the RPN, lacking feature information from lower layers, while the lower-level feature maps contain more detailed information. Therefore, this invention adopts an improved FPN structure, adding channels that connect from bottom to top and then backward. Wherein, P i (i = 2, 3, 4, 5, 6) are the features of the feature pyramid of the FPN structure, and the added bottom-up path merges the low-level feature maps M. i and high-level feature map P i+1 Generate a new feature map M i+1 Specifically, as shown in Equation 2:

[0033]

[0034] in, This represents a convolution with a stride of 2 and a kernel size of 3×3. This represents a convolution with a stride of 1 and a kernel size of 3×3; P i+1 Represents a high-level feature map; M i M represents a low-level feature map; i+1 This represents the newly obtained feature map.

[0035] During model training, in order to simulate the scene of objects gathering and scattering in reality, this invention performs additional data augmentation on the synthetic data A. The specific four data augmentation methods are as follows: (1) Randomly replace the background images of the MS COCO dataset and PASCALVOC dataset, and paste the objects from the training set with random translation, scaling and in-plane rotation; (2) Randomly adjust the contrast, color saturation and brightness; (3) Randomly add Gaussian blur and Gaussian noise; (4) Randomly apply additional augmentation to the input training attempt: render using random light positions and random diffuse and specular reflection. Among them, the first two data augmentation methods are applied to the training data with a 50% probability, the third data augmentation method is applied to the training data with a 20% probability, and all data are processed by the fourth augmentation method. At the same time, this invention sets Adam as the gradient descent optimization algorithm for setting model parameters.

[0036] The bounding box information obtained by combining the instance segmentation network (u w ,v w ,w w ,h w ), for object bounding box information (u w ,v w ,w w ,h w The cropped image is then input into the trained preprocessor. Figure 1 The latent feature extraction network shown yields the corresponding latent feature vector z. test .

[0037] The latent feature vector z of the object obtained from the real test image test The initial 3D rotation pose is obtained by calculating cosine similarity with the codebook data and using the K-nearest neighbor algorithm to achieve template matching. Specifically, this involves combining all the template latent vectors z recorded in the codebook. i The latent feature vector z of the pose of the object image test Cosine similarity is used to measure the feature similarity between an object image and a template image, calculated as follows:

[0038]

[0039] After calculating the similarity between the object image and all templates, the k-nearest neighbor algorithm is used to find the k templates whose poses are closest to the object image. The poses R corresponding to these k templates are... k (1≤k≤n) represents the 3D rotation pose of the object image, and k = 1 is often taken in practical applications.

[0040] Finally, based on the synthetic data A corresponding to the object in the real test image, and its corresponding CAD model data's bounding box information (u j ,v j ,w j ,h j The bounding box information of objects in real test images obtained by the improved Mask R-CNN instance segmentation network, which incorporates the Swin Transformer, is used to obtain (u w ,v w ,w w ,h w Based on the obtained template information, the initial 3D translation pose is calculated. Specifically:

[0041] First, based on the camera's imaging model, we obtain:

[0042]

[0043] Among them, t w，z t represents the z-axis translation of an object in a real test image. j，z f represents the z-axis translation of the object in the composite data A. w and f j Let represent the focal lengths of the real camera and the rendered camera, respectively. Then the z-axis translation t w，z The estimated value is:

[0044]

[0045] w w h w These are the bounding box information of the actual test image (u w ,v w ,w w ,h w The width and height of the bounding box in () j h j These are the bounding box information (u) corresponding to each data point recorded in synthetic data A. j ,v j ,w j ,h j The width and height of the bounding box in ().

[0046] To calculate the translation t of the object on the x and y axes in a real test image. w，x and t w，y First, estimate the position x of the center of the object in the real test image on the plane. w y w for:

[0047]

[0048] Among them, c x，w With c y，w These represent the offsets in the image plane coordinates, respectively. Similarly, the position (x, y) of the object center in the synthetic data A on the image plane can be obtained. j y j The transformation relationship between the image coordinate system and the camera coordinate system is as follows:

[0049]

[0050] Based on position x w y w The transformation relationship between the image coordinate system and the camera coordinate system is used to obtain the translation of the object on the x-axis and y-axis in the real test image:

[0051]

[0052] When depth information is available, the pose is optimized using the object's point cloud. The point cloud of the object in the real test image is denoted as P. w Its center of mass is The corresponding point cloud in the synthetic data A is denoted as P. j Its centroid is memorized as The z-axis translation of the object in the actual test image is:

[0053]

[0054] Finally, based on the transformation relationship between the image coordinate system and the camera coordinate system... To recalculate the X-axis and y-axis translations of the object in the real test image, the 3D translation pose of the object in the real test image is: [t w，x , t w，y , t w，z The final result is a 3D rotation pose and a 3D translation pose, where the 3D rotation pose is R. k The 3D translation pose is [t] w，x , t w，y , t w，z Together, these two elements constitute the initial 6D pose of the object in the real test image.

[0055] The preferred embodiment of this invention is to combine depth information and a mask with an iterative nearest-point algorithm to refine the pose. Specifically, in the traditional approach, the ICP algorithm directly performs fine registration between the object point cloud and the source point cloud. However, since only a portion of the scene is visible, the registration accuracy is low. The iterative nearest-point optimization first involves a series of preprocessing steps on the initial point cloud information, such as hole filling and downsampling. Then, the ICP algorithm is used for iterative optimization, repeatedly executing the following steps until convergence: based on the initial pose information, the source point cloud is transformed into the object point cloud; then, the point correspondence is recalculated using nearest-point matching; finally, the rigid body transformation is recalculated based on the new point correspondence. Through multiple iterations, the ICP algorithm gradually optimizes the initial pose estimation, resulting in more accurate registration. During the pose refinement process, parameters can be flexibly adjusted to balance the number of iterations and registration accuracy to meet specific application requirements.

[0056] After fine registration using the ICP algorithm, a more accurate 3D rotation pose of the object in the real test image can be obtained as R. k ′ and 3D translation pose are [t w，x ′, t w，y ′, t w，z ′).

[0057] The following is a verification example of the present invention:

[0058] The experiments evaluated the mAP of IoU for thresholds of 0.5 and 0.75. Furthermore, the number of parameters was used to measure the model complexity. The results of the instance segmentation network experiments are shown in Table 1.

[0059] Table 1

[0060]

[0061] Table 1 lists the comparisons between the backbone networks of ResNet50 and ResNet101 and the backbone network of the present invention, which uses Swin-T and the improved FPN structure model. The results demonstrate the effectiveness and advancement of the improved instance segmentation algorithm of the present invention in target detection and segmentation tasks.

[0062] During training, this invention uses 3D model views without pose annotations as training data. This method belongs to self-supervised learning, which saves a significant amount of cost compared to supervised learning. For objects with no or weak texture, this invention applies the Visible Surface Difference (VSD) metric to quantify the prediction results (measure recall):

[0063]

[0064] in, This represents the distance from the camera center to the 3D projection point obtained after estimating the object model; err vsd The distance between the estimated depth of the object and the actual visible surface depth on the ground was determined. In the test experiment, a threshold τ = 20 mm and θ = 0.3 were selected. As shown in Table 2, the error rate of object recall... vsd <0.3:

[0065] Table 2

[0066]

[0067] The base model (noise reduction autoencoder and Mask R-CNN) without the improvements of this invention, with the addition of CBAM, achieved an average accuracy improvement of 3.29%; improving only the backbone network of Mask R-CNN resulted in an average accuracy improvement of 9.16%; the complete model achieved an average accuracy improvement of 12.33%; and the average recognition time was 0.6s.

Claims

1. A weak texture object pose estimation method based on deep learning and synthetic data, a T-LESS dataset of several textureless objects is obtained, and a textureless CAD model of each object is obtained, characterized in that Comprising the following steps: Step 1): rendering the non-textured CAD model to obtain synthetic data and the bounding box information corresponding to each data; Step 2): decoupling the 6D pose of the object into a combination of a rotation pose and a translation amount, wherein each data in the synthetic data corresponds to a rotation pose and a translation amount; Step 3): the synthetic data is input into the auto-encoder network composed of the 5-layer convolutional layer of the CBAM module to obtain a template latent vector z i After being input into the 5-layer deconvolutional layer, one complete iteration is completed, and a pose latent feature extraction network is trained, and the synthetic data is input into the pose latent feature extraction network to make a codebook, wherein the template latent vector z in the codebook i The rotation pose corresponding to each data in the synthetic data is one-to-one correspondence. Step 4): inputting the real test image into the improved Mask R-CNN instance segmentation network based on the fusion Swin Transformer to obtain a feature map, setting a predetermined ROI for each point in the feature map, filtering the ROI through the RPN to obtain a candidate ROI, and performing ROIAlign on the candidate ROI to obtain the bounding box information; Step 5) cropping the bounding box information in step 4), inputting the pose latent feature extraction network in step 3) to obtain the corresponding latent feature vector z test ; Step 6): According to the potential feature vector z test Cosine similarity is calculated with the template potential vector z, and the k nearest neighbors algorithm is used to find the k templates closest to the object image pose. The poses corresponding to the k templates are the 3D rotation poses of the object image. Step 7): first obtaining the estimated value of the z-axis translation amount according to the bounding box information of the CAD model data in the synthetic data corresponding to the object in the real test image and the bounding box information in step 4), and then using the conversion relationship between the image coordinate system and the camera coordinate system to obtain the translation amount of the object in the real test image in the x-axis and y-axis.

2. The weakly textured object pose estimation method of claim 1, wherein: The estimated value of the z-axis translation amount in step 7) is t j，z f w f j f w w w w j h j h are the width and height of the bounding box in the bounding box information of each data pair recorded in the synthetic data, respectively.

3. The weakly textured object pose estimation method of claim 2, wherein: When depth information is available, the z-axis translation of the object in the real test image is P w Let P be the point cloud of the object in the real test image w , Let C be the centroid of the point cloud of the object in the real test image, Let C be the centroid of the corresponding data point cloud in the synthetic data, t j，z Let T be the z-axis translation of the object in the synthetic data.

4. The weakly textured object pose estimation method of claim 3, wherein: According to the conversion relationship between the image coordinate system and the camera coordinate system, the x-axis and y-axis translation amounts of the object in the real test image are recalculated to obtain the 3D translation pose of the object in the real test image.

5. The weakly textured object pose estimation method of claim 4, wherein: The initial point cloud information is preprocessed, and the ICP algorithm is used for iterative optimization of the initial pose estimation to obtain the 3D rotation pose and translation amount.

6. The weakly textured object pose estimation method of any one of claims 1-5, wherein: In step 1), the.obj format CAD model point cloud data is loaded using OpenGL, and the data under pure background is rendered, the white area is superimposed with the pure black background, and then the Fibonacci grid sampling is performed to obtain the synthetic data.

7. The weakly textured object pose estimation method of any one of claims 1-5, wherein: In step 3), the synthetic data is trained by domain randomization, and then input into the autoencoder network, wherein the first three layers of the encoder in the convolutional denoising autoencoder are respectively integrated into CBAM.

8. The weakly textured object pose estimation method of any one of claims 1-5, wherein: In step 4), the improved Mask R-CNN instance segmentation network is composed of a 4-layer composite structure and an improved FPN structure, the improved FPN includes the current layer, the upper layer, and the channels connected from bottom to top and backward, and the low-level feature map and the high-level feature map are combined to generate a new feature map.

9. The weakly textured object pose estimation method of claim 8, wherein: The new feature map P i (i = 2, 3, 4, 5, 6) are features of the feature pyramid of the FPN structure, the low-level feature map is M i , and the high-level feature map is P i+1 , represents a convolution with a stride of 2 and a convolution kernel size of 3x3; represents a convolution with a stride of 1 and a convolution kernel size of 3x3.

10. The weakly textured object pose estimation method of claim 1, wherein: In step 6), the cosine similarity

Images