Zero-sample unknown object 3D pose estimation method based on two-stage RGB-D fusion

Abstract

The invention discloses a zero-sample unknown object 3D pose estimation method based on two-stage RGB-D fusion. Comprising the following steps: giving a 3D model of an unknown object, and rendering the 3D model of the unknown object under different viewpoints to generate a series of RGB-D template images; based on a feature embedded network PoseFusion, texture features and geometric features are extracted from an RGB template image and a Depth template image through a modified ResNet50 network, and then a series of template scene features are generated through two-stage fusion of the texture features and the geometric features. Similarly, a real RGB-D image obtained by shooting an unknown object is also embedded into a real scene feature through PoseFusion. And finally, matching the real scene features with the series of template scene features by calculating occlusion local similarity, so that the 3D pose of the unknown object is determined as the matched template 3D pose.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a zero-sample unknown object 3D pose estimation method based on two-stage RGB-D fusion. Background Art

[0002] Object 3D pose estimation is based on computer vision technology to determine the three-dimensional rotation transformation parameters of the target object relative to the camera. It is a key issue in many fields such as robot environment perception, robot grasping, car starting and driving, augmented reality, etc.

[0003] Existing object pose estimation methods can be divided into known object pose estimation methods and unknown object pose estimation methods, depending on whether the target object appears during the model training phase. Thanks to the development of deep learning technology, existing known object pose estimation methods can achieve good pose prediction results. However, for real-world applications such as robotic grasping and autonomous driving, it is more meaningful and more urgent to be able to accurately estimate the pose of unknown objects without retraining the model.

[0004] Existing methods for estimating the pose of unknown objects mainly include methods based on point cloud registration, methods based on 3D reconstruction, and methods based on RGB image template matching. However, these methods all have some shortcomings. The method based on point cloud registration estimates the pose by detecting 3D key points. However, depth sensors often generate depth maps with noise and omissions, making it difficult to accurately detect 3D key points and then estimate the pose. The method based on 3D reconstruction reconstructs a 3D model and then solves the object pose, but the error of 3D reconstruction will directly affect the accuracy of pose estimation. The method based on RGB image template matching retrieves the most similar template RGB image by matching the target object RGB image and the template RGB image, but this method is not effective for objects with weak texture or no texture. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a zero-sample unknown object 3D pose estimation method based on two-stage RGB-D fusion, which can accurately estimate the pose parameters of unknown objects without retraining.

[0006] The technical solution adopted in the present invention is as follows:

[0007] (1) Using BlenderProc to generate a series of RGB-D template images (including RGB template images and Depth template images) from the CAD model of the unknown object at different viewpoints;

[0008] (2) The RGB-D image is extracted into high-dimensional scene features for matching through the scene feature extraction network PoseFusion. Specifically, for the RGB-D template image, the texture features and geometric features of the RGB template image and the Depth template image are extracted through a modified ResNet50 network, and then these texture features and geometric features are fused in two stages to generate a series of template scene features. Similarly, the real RGB-D image obtained by shooting the unknown object is also embedded into the real scene features through PoseFusion;

[0009] (3) matching the real scene features with a series of template scene features by calculating the occlusion local similarity to obtain the 3D pose of the unknown object;

[0010] In step (1), the specific steps of generating the RGB-D template image are:

[0011] (1.1) Template rendering viewpoint generation: The viewpoints generated by the template image are first initialized to the vertices of a regular icosahedron. Each triangle of the regular icosahedron is then divided into four smaller triangles. After two such divisions, the number of vertices of the regular icosahedron increases to 602. Finally, the viewpoints in the lower half of the plane are removed, resulting in 301 viewpoints.

[0012] (1.2) RGB-D template image rendering: Use BlenderProc to render the unknown object at different viewing angles to obtain a series of RGB-D template images (including RGB template images and Depth template images).

[0013] In step (2), the specific steps of scene feature extraction are:

[0014] (2.1) Template scene feature extraction: The RGB-D template image is extracted into a series of template scene features using the scene feature extraction network PoseFusion. The scene feature extraction network PoseFusion mainly consists of a feature extraction module and a feature fusion module.

[0015] The feature extraction module uses two modified ResNet50 networks that do not share weights to extract texture features and geometric features from the RGB template image and the Depth template image respectively. After feature extraction, the RGB template image is mapped from an image of size 3×224×224 to a high-dimensional texture feature of size 64×28×28, and the Depth template image is mapped from an image of size 1×224×224 to a high-dimensional geometric feature of size 64×28×28.

[0016] The feature fusion module uses the cross-modal alignment submodule and the geometric guidance fusion submodule to fuse the above-extracted texture features and geometric features to obtain scene features.

[0017] The cross-modal alignment submodule is used to eliminate noise in a single modality. In order to obtain aligned texture features, the texture features are input into the module as queries, and the geometric features are input into the cross-modal alignment submodule as keys and values.

[0018] This submodule first aligns the texture features with the geometric features in space. Specifically, this submodule first uses the texture features A∈R C×H×W Generate Q SA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G s ∈R C×H×W , and then further utilize G through multi-layer perceptron s Generate K SG , V SG ∈R C×H×W , and finally calculate the cross-modal attention weight W S Generate spatially aligned texture features A SA ∈R C×H×W :

[0019] W S =softMax(Q SA .T×K SG )

[0020] A SA =MLP(G S -(V SG W S )+G S

[0021] Where MLP stands for multi-layer perceptron. Similarly, the cross-modal alignment submodule also aligns the texture features and geometric features on the channel. First, the texture feature A∈R is used by the multi-layer perceptron to C×H×W Generate Q CA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G c ∈R C×H×W , and then further utilize G through multi-layer perceptron c Generate K CG , V CG ∈R C×H×W , and then calculate the cross-modal attention weight W C Generate channel-aligned texture features A CA ∈RC×H×W :

[0022] W G =softMax(Q CA .T×K CG )

[0023] A CA =MLP(G C -(V CG W C ))+G C

[0024] Where MLP stands for multi-layer perceptron. After spatial alignment and channel alignment, the cross-modal alignment submodule transforms the spatially aligned texture features A SA Texture feature A aligned with the channel CA Add each element one by one, then obtain a new feature through convolution, aggregate this feature with the initial texture feature A, and finally output the aligned texture feature through convolution.

[0025] Similarly, the geometric features are input into the module as queries, and the texture features are input into the cross-modal alignment sub-module as keys and values ​​to finally obtain the aligned geometric features.

[0026] After obtaining the aligned texture features and aligned geometric features, the feature fusion module uses the geometry-guided fusion submodule to fuse these two features to obtain scene features. This submodule uses the aligned geometric features as queries and the aligned texture features as keys and values, fuses these two features using a cross-attention mechanism, and finally transforms the feature dimensions using a multi-layer perceptron. Through the geometry-guided fusion submodule, the aligned texture features and aligned geometric features are ultimately fused into a template scene feature with a size of 16×28×28.

[0027] (2.2) Real Scene Feature Extraction: First, an RGB-D real image of the unknown object (including the real RGB image and the real depth image) is captured using an RGB-D camera. Then, after cropping and scaling, a real RGB image of 3×224×224 and a real depth image of 1×224×224 are obtained. Finally, the scene feature extraction network PoseFusion is used to abstract the real RGB-D image of the unknown object into real scene features of 16×28×28.

[0028] In step (3), the specific steps of pose estimation are: calculating the mask local similarity between the real scene feature q calculated above and each template scene feature t in the series of template scene features obtained above:

[0029]

[0030] Where M is the 2D mask of the RGB-D template image, which can reduce the influence of cluttered background on pose estimation. The serial number l represents the 2D grid position of the pixel, q (l) and t (l) Indicates the feature of the position. In addition, S represents the cosine similarity, which can be expressed as:

[0031]

[0032] By calculating the similarity between the real scene features and a series of template scene features, the RGB-D template image that best matches the RGB-D real image of the unknown object can be retrieved, so the pose parameters of the unknown object can be determined as the pose parameters corresponding to the template image.

[0033] Compared with the prior art, the advantages of the present invention are:

[0034] By calculating the similarity between the real image and the generated template image, the present invention can directly estimate the 3D pose of unknown objects that have not been seen during the model training phase. Compared with most existing pose estimation methods, there is no need to retrain unknown objects, saving time and avoiding unnecessary production operation pauses and data set preparation.

[0035] The present invention extends the traditional RGB image-based template matching method to an RGB-D image-based template matching method. By extracting features separately and then fusing features, the method proposed in the present invention can extract features of the target object in two different dimensions: texture and shape. Therefore, it can also achieve good pose estimation effects for weakly textured parts commonly found in industry.

[0036] This paper proposes a two-stage RGB-D feature fusion module. By aligning texture features and geometric features in space and channels, and then fusing the aligned features using a cross-attention mechanism, the proposed method can remove noise existing in a single modality and achieve better generalization performance for unknown objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flowchart of unknown object pose estimation in an embodiment;

[0038] Figure 2 is a schematic diagram of a rich texture LINEMOD dataset in an embodiment;

[0039] Figure 3 is a schematic diagram of a weak texture T-LESS data set in an embodiment;

[0040] Figure 4 is a diagram of a scene feature extraction network structure in an embodiment;

[0041] Figure 5 is a structural diagram of a cross-modal feature alignment module in an embodiment; DETAILED DESCRIPTION

[0042] In order to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific examples. Figure 1 This is the unknown object pose estimation flowchart for this example.

[0043] (1) In order to train the scene feature extraction network PoseFusion, this example classifies the objects in the dataset used in this experiment. Among them, the objects Cat, Driller, Duck, Eggbox, Glue, Holepuncher, Iron, Lamp, and Phone in the LINEMOD dataset are classified as known objects for training PoseFusion, and the objects Ape, Benchvise, Camera, and Can are used to test the pose estimation effect of the model for unknown objects. Figure 2 In the T-LESS dataset, objects 1-18 are considered known objects for training PoseFusion, while objects 19-30 are used to test the pose estimation effect of the model on unknown objects, as shown in the figure. Figure 3 shown.

[0044] (2) This example first initializes the template image to generate viewpoints as the vertices of the regular icosahedron, and then divides each triangular facet of the regular icosahedron into four smaller triangular faces. After two such divisions, the number of vertices of the regular icosahedron increases to 602. Because for the T-LESS dataset, this example uses 602 viewpoints to render and generate the RGB-D template image. For the LINEMOD dataset, this example removes the viewpoints of the lower half of the plane and obtains 301 viewpoints for rendering the RGB-D template image. Finally, this example uses BlenderProc to generate RGB-D template images of different perspectives for each object.

[0045] Figure 4 : This is a structural diagram of the scene feature extraction network PoseFusion in the embodiment. The feature embedding network PoseFusion is mainly composed of a feature extraction module and a feature fusion module.

[0046] The feature extraction module uses two modified ResNet50 networks that do not share weights to extract texture features and geometric features from the RGB template image and the Depth template image respectively. After feature extraction, the RGB template image is mapped from an image of size 3×224×224 to a high-dimensional texture feature of size 64×28×28, and the Depth template image is mapped from an image of size 1×224×224 to a high-dimensional geometric feature of size 64×28×28.

[0047] The feature fusion module uses the cross-modal alignment submodule and the geometric guidance fusion submodule to fuse the above-extracted texture features and geometric features to obtain scene features.

[0048] The cross-modal alignment submodule is used to eliminate the noise in a single modality. Figure 5 : This is a structural diagram of the cross-modal feature alignment submodule in the embodiment. In order to obtain aligned texture features, the texture features are input into the module as queries, and the geometric features are input into the cross-modal alignment submodule as keys and values. This submodule first aligns the texture features with the geometric features in space. Specifically, this submodule first uses the texture features A∈R through a multi-layer perceptron to align the texture features. C×H×W Generate Q SA ∈R C×H×W , and using the geometric features G∈R C×H×W Generate K SG , V SG ∈R C×H×W , and then generate the spatially aligned texture feature A by calculating the cross-modal attention weight SA ∈R C×H×W :

[0049] This submodule first aligns the texture features with the geometric features in space. Specifically, this submodule first uses the texture features A∈R C×H×W Generate Q SA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G s ∈R C×H×W , and then further utilize G through multi-layer perceptron s Generate K SG , V SG ∈R C×H×W , and finally calculate the cross-modal attention weight W S Generate spatially aligned texture features A SA ∈R C×H×W :

[0050] W S =sofMax(Q SA .T×KSG )

[0051] A SA =MLP(G S -(V SG ·WS))+G S

[0052] Where MLP stands for multi-layer perceptron. Similarly, the cross-modal alignment submodule also aligns texture features and geometric features on the channel.

[0053] This submodule first uses the texture feature A∈R through a multi-layer perceptron C×H×W Generate Q CA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G c ∈R C×H×W , and then further utilize G through multi-layer perceptron c Generate K CG , V CG ∈R C×H×W , and then calculate the cross-modal attention weight W C Generate channel-aligned texture features A CA ∈R C×H×W :

[0054] W C =softMax(Q CA .T×K CG )

[0055] A CA =MP(G C -(V CG W C ))+G C

[0056] Where MLP stands for multi-layer perceptron. After spatial alignment and channel alignment, the cross-modal alignment submodule transforms the spatially aligned texture features A SA Texture feature A aligned with the channel CA Add each element one by one, then obtain a new feature through convolution, aggregate this feature with the initial texture feature A, and finally output the aligned texture feature through convolution.

[0057] Similarly, the geometric features are input into the module as queries, and the texture features are input into the cross-modal alignment sub-module as keys and values ​​to finally obtain the aligned geometric features.

[0058] After obtaining the aligned texture features and aligned geometric features, the feature fusion module uses the geometry-guided fusion submodule to fuse these two features to obtain scene features. This submodule uses the aligned geometric features as queries and the aligned texture features as keys and values, fuses these two features using a cross-attention mechanism, and finally transforms the feature dimensions using a multi-layer perceptron. Through the geometry-guided fusion submodule, the aligned texture features and aligned geometric features are ultimately fused into a template scene feature with a size of 16×28×28.

[0059] (2.2) Real Scene Feature Extraction: First, an RGB-D real image of the unknown object (including the real RGB image and the real depth image) is captured using an RGB-D camera. Then, after cropping and scaling, a real RGB image of 3×224×224 and a real depth image of 1×224×224 are obtained. Finally, the scene feature extraction network PoseFusion is used to abstract the real RGB-D image of the unknown object into real scene features of 16×28×28.

[0060] In step (3), the specific steps of pose estimation are: calculating the mask local similarity between the real scene feature q calculated above and each template scene feature t in the series of template scene features obtained above:

[0061]

[0062] Where M is the 2D mask of the RGB-D template image, which can reduce the influence of cluttered background on pose estimation. The serial number l represents the 2D grid position of the pixel, q (l) and t (l) Indicates the feature of the position. In addition, S represents the cosine similarity, which can be expressed as:

[0063]

[0064] By calculating the similarity between the real scene features and a series of template scene features, the RGB-D template image that best matches the RGB-D real image of the unknown object can be retrieved, so the pose parameters of the unknown object can be determined as the pose parameters corresponding to the template image.

[0065] The method proposed in the present invention takes a total of 3 hours to train on the LINEMOD dataset and 8 hours to train on the T-LESS dataset. When testing the pose estimation effect of the model on the LINEMOD dataset, if the pose error is less than 15° and the predicted object type is correct, the pose estimation result is judged to be correct. After testing, the pose estimation of known objects achieved an accuracy rate of 98.1%, and the pose estimation of unknown objects achieved an accuracy rate of 96.0. When testing the pose estimation effect of the model on the T-LESS dataset, we use VSD to determine whether the pose estimation is accurate. After testing, the pose estimation of known objects achieved an accuracy rate of 65.73%, and the pose estimation of unknown objects achieved an accuracy rate of 63.44.

[0066] The above, combined with the accompanying drawings, details the basic principles and main features of the present invention. Using the above invention, the pose parameters of unknown objects can be estimated without retraining. The method proposed in the present invention has good generalization for unknown objects and can achieve prediction results similar to those of known objects without training. Furthermore, the method proposed in the present invention can also achieve good results for objects with weak textures. However, these descriptions should not be construed as limiting the scope of the present invention, which is defined by the appended claims. Any modifications based on the claims fall within the scope of the present invention.

Claims

1. A zero-shot unknown object 3D pose estimation method based on two-stage RGB-D fusion, characterized by: The following steps are involved: 1) Use BlenderProc to generate a series of RGB-D template images from different viewpoints of the CAD model of the unknown object; 2) The feature embedding network PoseFusion extracts RGB-D images into high-dimensional features for matching, specifically: The RGB-D template image in step 1) is extracted through the ResNet50 network to obtain texture features and geometric features, and then the texture features and geometric features are fused in two stages to generate a series of template scene features; Similarly, the real RGB-D images obtained by shooting unknown objects are also embedded into real scene features through PoseFusion; 3) The real scene features and a series of template scene features are matched by calculating the occlusion local similarity to obtain the 3D pose of the unknown object.

2. The method for zero-sample unknown object 3D pose estimation based on two-stage RGB-D fusion according to claim 1, characterized in that: In step 1), the specific steps of generating the RGB-D template image are: 1.1) Template rendering viewpoint generation: The template rendering viewpoint is first initialized to the vertices of a regular icosahedron. Each triangle of the regular icosahedron is then divided into four smaller triangles, and each divided triangle is further divided into four smaller triangles. After two divisions, the number of vertices of the regular icosahedron increases to 602. Finally, the viewpoints in the lower half of the plane are removed, resulting in 301 viewpoints. 1.2) RGB-D template image rendering: BlenderProc is used to render the CAD model of the unknown object at different viewpoints to obtain a series of RGB-D template images; The different viewpoints are obtained by step 1.1), and the RGB-D template image includes an RGB template image and a Depth template image.

3. The method for zero-shot unknown object 3D pose estimation based on two-stage RGB-D fusion according to claim 1, characterized in that: The step 2) is specifically as follows: 2.1) Template Scene Feature Extraction: The RGB-D template image is extracted into a series of template scene features using the feature embedding network PoseFusion. The feature embedding network PoseFusion mainly consists of a feature extraction module and a feature fusion module. 2.2) Real Scene Feature Extraction: First, an RGB-D camera is used to capture real RGB-D images of unknown objects, including real RGB images and real depth images. Then, after cropping and scaling, the real RGB images are obtained to a size of 3 × 224 × 224 and the real depth images to a size of 1 × 224 × 224. Finally, the feature embedding network PoseFusion is used to abstract the RGB-D real image of the unknown object into a real scene feature q of size 16×28×28.

4. The method for zero-shot unknown object 3D pose estimation based on two-stage RGB-D fusion according to claim 3, characterized in that: The feature extraction module in step 2.1) uses the ResNet50 network to extract texture features and geometric features from the RGB template image and the Depth template image, respectively. After feature extraction, the RGB template image is mapped from an image of size 3×224×224 to a high-dimensional texture feature of size 64×28×28, and the Depth template image is mapped from an image of size 1×224×224 to a high-dimensional geometric feature of size 64×28×28.

5. The method for zero-sample unknown object 3D pose estimation based on two-stage RGB-D fusion according to claim 4, characterized in that: The feature fusion module in step 2.1) uses the cross-modal alignment submodule and the geometric guidance fusion submodule to fuse the texture features and geometric features extracted by the feature extraction module to obtain scene features; 2.1.1) The cross-modal alignment submodule is used to eliminate noise in a single modality. To obtain aligned texture features, the texture features are input as queries to the module, and the geometric features are input as keys and values ​​to the cross-modal alignment submodule. The cross-modal alignment submodule spatially aligns texture features with geometric features, specifically: First, the texture feature A∈R is used by a multi-layer perceptron C×H×W Generate Q SA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G s ∈R C×H×W , and then further utilize G through multi-layer perceptron S Generate K SG , V SG ∈R C×H×W , and finally calculate the cross-modal attention weight W S Generate spatially aligned texture features A SA ∈R C×H×W : W S =softMax(Q SA .T×K SG ) A SA =MLP(G S -(V SG ·W S ))+G S Among them, MLP represents multi-layer perceptron; Similarly, the cross-modal alignment submodule also aligns texture features and geometric features on the channel, specifically: First, the texture feature A∈R is used by a multi-layer perceptron C×H×W Generate Q CA ∈R C×H×W , and then use the geometric features G∈R through the multi-layer perceptron C×H×W Generate G c ∈R C×H×W , and then further utilize G through multi-layer perceptron C Generate K CG , V CG ∈R C×H×W , and then calculate the cross-modal attention weight W C Generate channel-aligned texture features A CA ∈R C×H×W : W C =softMax(Q CA .T×K CG ) A CA =MLP(G C -(V CG ·W C ))+G C After spatial alignment and channel alignment, the cross-modal alignment submodule converts the spatially aligned texture features A SA Texture feature A aligned with the channel CA Add each element one by one, then obtain a new feature through convolution, aggregate this feature with the initial texture feature A, and finally output the aligned texture feature through convolution; 2.1.2) Similarly, the geometric features are input into the module as queries, and the texture features are input into the cross-modal alignment submodule as keys and values ​​to finally obtain the aligned geometric features. 2.1.3) After obtaining the aligned texture features and aligned geometric features, the feature fusion module uses the geometry-guided fusion submodule to fuse these two features to obtain scene features. The geometry-guided fusion submodule uses the aligned geometric features as queries and the aligned texture features as keys and values, fuses these two features using a cross-attention mechanism, and finally transforms the feature dimensions through a multi-layer perceptron. Through the geometry-guided fusion submodule, the aligned texture features and aligned geometric features are ultimately fused into a template scene feature of size 16×28×28.

6. The method for zero-shot unknown object 3D pose estimation based on two-stage RGB-D fusion according to claim 1, characterized in that: In step 3), the specific steps of posture estimation are: The mask local similarity is calculated between the real scene feature q and each template scene feature t in a series of template scene features: Where M is the two-dimensional mask of the RGB-D template image; the serial number l represents the two-dimensional grid position of the pixel, q (l) and t (l) Indicates the characteristics of the location, M l Indicates the value of the two-dimensional mask at that position; in addition, S represents the cosine similarity, which is specifically expressed as By calculating the similarity between the real scene features and a series of template scene features, the RGB-D template image that best matches the unknown object's RGB-D real image can be retrieved. Therefore, the pose parameters of the unknown object can be determined as the pose parameters corresponding to the best matching RGB-D template image.

Images