Object pose estimation method and device based on single-view-angle reference template

Abstract

The invention relates to an object pose estimation method and device based on a single-view reference template, and the method comprises the steps: extracting the feature information of a single reference RGB-D image, and generating a reference segmentation mask of the single reference RGB-D image through a preset SAM algorithm; obtaining a target segmentation mask corresponding to the target RGB-D image, and performing back projection on the target segmentation mask and the reference segmentation mask to obtain a reference point cloud and a target point cloud; performing point cloud coarse matching on the reference point cloud and the target point cloud, establishing a first corresponding relation matrix of the reference point cloud and the target point cloud, and calculating coarse pose estimation information; based on the coarse pose estimation information, point cloud fine matching is carried out on the reference point cloud and the target point cloud, a second corresponding relation matrix of the reference point cloud and the target point cloud is established, fine pose estimation information is obtained, and then a final pose estimation result of the to-be-measured target is obtained. Therefore, the problems of high acquisition cost, long time consumption, poor generalization ability and the like of the pose estimation reference image in the existing object pose estimation method are solved.

Description

Technical Field

[0001] The present application relates to the technical field of object pose estimation, and in particular to a method and device for object pose estimation based on a single-view reference template. Background Art

[0002] Currently, the vast majority of work focuses on instance-level object pose estimation, where the training and test datasets consist of the same set of known object instances and CAD models of the objects are usually required to generate training images and labels. Some existing techniques extend this paradigm to the category level, aiming to estimate the pose of new instances within predefined categories without the need for CAD models of the target objects.

[0003] However, both paradigms have limitations in open-world applications because they are time-consuming and difficult to implement for labeling and training new objects beyond known categories. To alleviate this problem, existing techniques can estimate the pose of any new object, that is, the target object is not seen during training. Therefore, existing techniques require multiple views of the object in different poses as reference images. If a CAD model exists during the test phase, reference views can be easily generated by image rendering; or in the absence of a CAD model, a reference view can be obtained by taking a series of images of the object; however, these two settings may be very time-consuming due to the creation of a CAD model of the object or the labeling of a large number of object poses. In summary, although existing methods show potential transfer capabilities for pose estimation of new objects, they are obviously limited by their reliance on CAD models or multiple reference views.

[0004] Specifically, some existing pose estimation methods are as follows:

[0005] 1. Pose estimation method for specific objects:

[0006] In recent years, pose estimation methods for specific objects have received widespread attention in the fields of robotics and computer vision. This method aims to predict the 6DoF pose of a known object with the help of a CAD model (computer-aided design model); specifically, this method can use an indirect method to predict the 6DoF pose, and then use the PnP algorithm based on RANSAC to solve the 6D pose; in addition, this method can use a deep neural network to directly regress the 6DoF pose; PoseCNN and DeepIM use point matching loss functions, while CosyPose extends DeepIM by introducing multi-view information and realizing global scene pose optimization, SingleStage and GDR-Net infer intermediate geometric features and directly regress the 6DoF pose through the network.

[0007] 2. General object pose estimation method:

[0008] Universal object pose estimation is a model that can be trained on partial objects and generalized to other arbitrary rigid objects. Specifically, given a target image, OVE6D and MegaPose retrieve a viewpoint that is most similar to the target image from a pre-rendered panoramic image database as a rough pose estimate; then, a customized neural network further optimizes this rough pose estimate, and FoundationPose further constructs a pose ranking network to score each optimized pose hypothesis; the feature matching method constructs pixel-level or point cloud-level correspondences by learning feature descriptors and solves the pose; in addition, FoundPose and FreeZe directly generate robust visual features from 2D images or 3D partial point clouds for feature matching.

[0009] However, general object pose estimation methods require dense viewpoint maps of the target object as a reference. These dense views need to be generated by taking a series of images of the target object, or scanning the CAD model of the target object and using image rendering, which is costly and time-consuming.

[0010] 3. Relative pose estimation method:

[0011] Relative object pose estimation estimates the relative pose transformation of a generic object using only one reference image. For example, RelPose and RelPose++ infer the distribution of relative rotations by leveraging an energy-based formula; in addition, iFusion optimizes the unknown relative pose by inverting the new view synthesis diffusion model. In the field of relative 3DoF object pose estimation, given two RGB images, 3DHAV proposed a hypothesis-verification framework to score each relative pose hypothesis. On this basis, DVMNet introduces an assumption-free process to calculate relative pose through deep voxel matching.

[0012] However, existing relative pose estimation methods only use the RGB modality to predict 3-DOF rotations and have limited ability to predict relative translation and generalize beyond their training datasets.

[0013] In summary, the acquisition cost of the pose estimation reference image in the existing object pose estimation methods is high, time-consuming, and has poor generalization ability, which needs to be solved urgently. Summary of the invention

[0014] The present application provides a method and device for object pose estimation based on a single-view reference template to solve the problems of high cost, long time consumption, and poor generalization ability of obtaining pose estimation reference images in existing object pose estimation methods.

[0015] The first aspect of the present application provides an object pose estimation method based on a single-view reference template, comprising the following steps: obtaining a single reference RGB-D image and a target RGB-D image of a target to be measured, extracting feature information of the single reference RGB-D image, and generating a reference segmentation mask of the single reference RGB-D image based on the feature information; obtaining a target segmentation mask corresponding to the target RGB-D image, and performing a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; Marking point cloud; based on the preset geometric encoder and color encoder, performing a point cloud coarse matching operation on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculating coarse pose estimation information through the first correspondence matrix; based on the coarse pose estimation information, performing a point cloud fine matching operation on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtaining fine pose estimation information through the second correspondence matrix, and obtaining the final pose estimation result of the target to be measured according to the fine pose estimation information.

[0016] Optionally, in one embodiment of the present application, the extracting feature information of the single reference RGB-D image and generating a reference segmentation mask of the single reference RGB-D image based on the feature information includes: extracting feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; generating multiple candidate mask proposals of the single reference RGB-D image, and calculating feature similarity corresponding to the feature information, so as to score each candidate mask proposal in the multiple candidate mask proposals according to the feature similarity to obtain a mask proposal score corresponding to each candidate mask proposal; and determining a reference segmentation mask of the single reference RGB-D image according to the mask proposal score.

[0017] Optionally, in one embodiment of the present application, based on the preset geometric encoder and color encoder, a point cloud coarse matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix, including: transforming the reference point cloud and the target point cloud into a preset global reference system, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain spliced ​​features; performing a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; performing inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and performing SVD decomposition processing on the first correspondence matrix to obtain the coarse pose estimation information.

[0018] Optionally, in one embodiment of the present application, the performing a point cloud fine matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud includes: performing point cloud downsampling on the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively;

[0019] Construct a local pose robust reference system, and transform the neighborhood point cloud into the local pose robust reference system; encode the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and perform a geometric attention decoding operation on the geometric position code, the color feature and the geometric feature to respectively generate a second eigenvector corresponding to the reference point cloud and the target point cloud; perform an inner product processing on the second eigenvector corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

[0020] The second aspect of the present application provides an object pose estimation method based on a single-view reference template, including: a segmentation mask module, used to obtain a single reference RGB-D image and a target RGB-D image of the target to be measured, and extract feature information of the single reference RGB-D image, and based on the feature information, generate a reference segmentation mask of the single reference RGB-D image; a back-projection module, used to obtain a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image. cloud; a coarse estimation module, used to perform a point cloud coarse matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and a color encoder, so as to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first correspondence matrix; a fine estimation module, used to perform a point cloud fine matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information, so as to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtain fine pose estimation information through the second correspondence matrix, and obtain a final pose estimation result of the target to be measured according to the fine pose estimation information.

[0021] Optionally, in one embodiment of the present application, the segmentation mask module includes: an extraction unit, used to extract feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; a scoring unit, used to generate multiple candidate mask proposals for the single reference RGB-D image, and calculate the feature similarity corresponding to the feature information, so as to score each candidate mask proposal in the multiple candidate mask proposals according to the feature similarity to obtain a mask proposal score corresponding to each candidate mask proposal; and a determination unit, used to determine the reference segmentation mask of the single reference RGB-D image according to the mask proposal score.

[0022] Optionally, in one embodiment of the present application, the coarse estimation module includes: a sampling unit, used to transform the reference point cloud and the target point cloud into a preset global reference system, and sample the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; a splicing unit, used to extract color features and geometric features of the point cloud sampling points, and splice the color features and the geometric features to obtain spliced ​​features; a decoding unit, used to perform a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; a first inner product unit, used to perform inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and perform SVD decomposition processing on the first correspondence matrix to obtain the coarse pose estimation information.

[0023] Optionally, in one embodiment of the present application, the fine estimation module includes: a downsampling unit, used to downsample the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively; a construction unit, used to construct a local pose robust reference system and transform the neighborhood point cloud into the local pose robust reference system; an encoding unit, used to encode the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and perform a geometric attention decoding operation on the geometric position code, the color feature and the geometric feature to generate a second feature vector corresponding to the reference point cloud and the target point cloud, respectively; a second inner product unit, used to perform inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

[0024] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object pose estimation method based on a single-view reference template as described in the above embodiment.

[0025] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the above-mentioned object pose estimation method based on a single-view reference template.

[0026] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned object pose estimation method based on a single-view reference template.

[0027] Therefore, the embodiments of the present application have the following beneficial effects:

[0028] The embodiments of the present application can obtain a single reference RGB-D image and a target RGB-D image of the target to be measured, extract feature information of the single reference RGB-D image, and generate a reference segmentation mask of the single reference RGB-D image based on the feature information; obtain a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; based on a preset geometric encoder and a color encoder, perform a point cloud coarse matching operation on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first correspondence matrix; based on the coarse pose estimation information, perform a point cloud fine matching operation on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtain fine pose estimation information through the second correspondence matrix, and obtain a final pose estimation result of the target to be measured according to the fine pose estimation information. This application makes full use of the RGB-D modality to predict the relative pose of 6 degrees of freedom, while enhancing the network's generalization ability for novel objects and environments; in addition, this application combines depth information to more accurately capture the three-dimensional shape and position information of the object, improving the accuracy and robustness of pose estimation. This solves the problems of high cost, long time consumption, and poor generalization ability of pose estimation reference images in existing object pose estimation methods.

[0029] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0031] Figure 1 A flowchart of a method for estimating an object pose based on a single-view reference template according to an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a comparison between the reference images of the present application and the traditional method provided for an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the logical architecture of a two-stage point cloud matching paradigm provided for one embodiment of the present application;

[0034] Figure 4 This is an example diagram of an object pose estimation device based on a single-view reference template according to an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0036] Among them, 10-object pose estimation device based on single-view reference template; 100-segmentation mask module, 200-back projection module, 300-coarse estimation module, 400-fine estimation module; 501-memory, 502-processor, 503-communication interface. DETAILED DESCRIPTION

[0037] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] The following describes the object pose estimation method and device based on a single-view reference template of an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides an object pose estimation method based on a single-view reference template. In this method, a single reference RGB-D image and a target RGB-D image of the target to be measured are obtained, and the feature information of the single reference RGB-D image is extracted, and based on the feature information, a reference segmentation mask of the single reference RGB-D image is generated; a target segmentation mask corresponding to the target RGB-D image is obtained, and a back-projection operation is performed on the reference segmentation mask and the target segmentation mask to obtain a single reference RGB-D image and a target RGB-D image. The reference point cloud and the target point cloud corresponding to each other are matched; based on the preset geometric encoder and color encoder, the reference point cloud and the target point cloud are roughly matched to establish the first correspondence matrix between the reference point cloud and the target point cloud, and the rough pose estimation information is calculated through the first correspondence matrix; based on the rough pose estimation information, the reference point cloud and the target point cloud are finely matched to establish the second correspondence matrix between the reference point cloud and the target point cloud, and the fine pose estimation information is obtained through the second correspondence matrix, and the final pose estimation result of the target to be measured is obtained according to the fine pose estimation information. The present application makes full use of the RGB-D modality to predict the relative pose of 6 degrees of freedom, and at the same time enhances the generalization ability of the network for novel objects and environments; in addition, the present application combines depth information to more accurately capture the three-dimensional shape and position information of the object, thereby improving the accuracy and robustness of pose estimation. As a result, the problems of high acquisition cost, long time consumption, and poor generalization ability of the pose estimation reference map in the existing object pose estimation method are solved.

[0039] Specifically, Figure 1 A flowchart of a method for estimating an object pose based on a single-view reference template provided in an embodiment of the present application.

[0040] like Figure 1 As shown, the object pose estimation method based on a single-view reference template includes the following steps:

[0041] In step S101, a single reference RGB-D image of the target to be measured and a target RGB-D image are obtained, and feature information of the single reference RGB-D image is extracted, and a reference segmentation mask of the single reference RGB-D image is generated based on the feature information.

[0042] The embodiment of the present application can first extract the feature information of a single reference RGB-D image of the target to be measured, and combine it with the preset SAM algorithm to generate a reference segmentation mask of the single reference RGB-D image. In this way, even when there is only a single reference image, the target object can be effectively segmented from the complex background, providing accurate input for the subsequent pose estimation step.

[0043] Optionally, in one embodiment of the present application, feature information of a single reference RGB-D image is extracted, and based on the feature information, a reference segmentation mask of the single reference RGB-D image is generated, including: extracting feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; generating multiple candidate mask proposals of the single reference RGB-D image, and calculating feature similarity corresponding to the feature information, so as to score each candidate mask proposal in the multiple candidate mask proposals by the feature similarity, and obtain a mask proposal score corresponding to each candidate mask proposal; and determining a reference segmentation mask of the single reference RGB-D image according to the mask proposal score.

[0044] Those skilled in the art should understand that segmenting the query object from the cluttered background benefits from the powerful generalization ability of the visual base model, so that the CAD model can be effectively used to segment novel objects.

[0045] Different from the traditional method of generating multiple descriptors from multiple rendering views, when only a single reference image is accessible, the embodiments of the present application make full use of the global and local descriptors in DINOv2 to score the mask proposals of SAM.

[0046] Specifically, the embodiments of the present application can use the DINOv2 model to extract rich feature information from the reference image, including global features (describing the overall properties of the entire object) and local features (focusing on the specific details of the object); thereafter, the embodiments of the present application can combine the feature information with the candidate mask proposals generated by the SAM algorithm, and score each mask proposal by calculating the feature similarity or matching degree. The higher the score, the greater the possibility of the mask proposal becoming the target object of the segmentation.

[0047] Therefore, the embodiments of the present application realize for the first time the segmentation of unseen objects and 6-degree-of-freedom (6DoF) pose estimation using a single RGB-D reference image, which not only expands the application scope of object recognition and pose estimation, but also significantly improves the ability to process unseen objects in complex scenes, and is highly technologically advanced and practical.

[0048] In step S102, a target segmentation mask corresponding to the target RGB-D image is obtained, and a back-projection operation is performed on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to a single reference RGB-D image and a target RGB-D image.

[0049] In step S103, based on the preset geometric encoder and color encoder, a point cloud coarse matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix.

[0050] After the reference and target RGB-D images and their segmentation masks are given, the embodiments of the present application can further back-project the depth into the point cloud space to obtain the target point cloud and the reference point cloud; then, the embodiments of the present application can use the point cloud registration method to establish the correspondence matrix between the target point cloud and the reference point cloud, such as Figure 2 As shown, the relative pose transformation between the target point cloud and the reference point cloud is estimated.

[0051] Optionally, in one embodiment of the present application, based on a preset geometric encoder and a color encoder, a point cloud coarse matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix, including: transforming the reference point cloud and the target point cloud into a preset global reference system, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain spliced ​​features; performing a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; performing an inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain a first correspondence matrix, and performing SVD decomposition processing on the first correspondence matrix to obtain coarse pose estimation information.

[0052] Understandably, given only a single reference image, the relative pose to be predicted is arbitrary in the entire SE(3) space (i.e., the group of three-dimensional Euclidean transformations including rotations and translations), which poses a significant challenge in predicting robust correspondences.

[0053] Therefore, the embodiments of the present application can, given an arbitrary point cloud, transform it into the reference frame through a pose-robust global reference frame transformation strategy.

[0054] Specifically, the embodiment of the present application can set the origin of the global reference system at the center of the point cloud, normalize the maximum radius of the point cloud to 1, and design a rotation matrix related to the point cloud distribution as the three-coordinate axis orientation of the reference system; after transforming the point cloud to the reference system, the embodiment of the present application can sample some points from the original point cloud to obtain point cloud sampling points; extract color features and geometric features from the point cloud sampling points, and input the spliced ​​features obtained after splicing them into a preset geometric attention decoding module, such as Figure 3 As shown, the feature vectors of the reference point cloud and the target point cloud are output respectively; the inner product of the feature vectors can form a correspondence matrix (i.e., the first correspondence matrix), and the matrix is ​​decomposed by SVD to obtain the initial pose estimation (i.e., coarse pose estimation information) of the coarse matching stage.

[0055] In step S104, based on the coarse pose estimation information, a point cloud fine matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and the fine pose estimation information is obtained through the second correspondence matrix, and the final pose estimation result of the target to be measured is obtained according to the fine pose estimation information.

[0056] It should be noted that, in point cloud registration, the embodiments of the present application can use color and geometric cues to construct correspondences, and follow the widely used two-stage matching paradigm of coarse matching stage and fine matching stage, thereby solving the problem of novel object pose estimation based on a single-frame RGB-D reference image, and has broad application prospects in factory object assembly and robotic arm operation.

[0057] Therefore, the embodiments of the present application can design a new and extensive benchmark testing system, which is specifically used to evaluate the performance of algorithms for unseen object segmentation and pose estimation using a single reference image. This benchmark test not only provides a unified evaluation standard for researchers in related fields, but also promotes fair comparison and continuous improvement of technologies by including evaluations of multiple traditional methods and learning-based methods.

[0058] Optionally, in one embodiment of the present application, based on the coarse pose estimation information, a point cloud fine matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, including: downsampling the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively; constructing a local pose robust reference system, and transforming the neighborhood point cloud to the local pose robust reference system; encoding the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and performing a geometric attention decoding operation on the geometric position code, color features, and geometric features to generate second feature vectors corresponding to the reference point cloud and the target point cloud, respectively; performing an inner product process on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain a second correspondence matrix.

[0059] In actual execution, the fine matching stage is different from the coarse matching stage in the following two aspects:

[0060] 1. In the fine matching stage, more points need to be sampled to establish the correspondence matrix;

[0061] 2. The embodiment of the present application establishes a local pose robust reference system based on the neighborhood of each point and performs a local reference system transformation on the point cloud. After the neighborhood point cloud is encoded by the encoder, a geometric position code is formed, which is sent to the subsequent decoding module together with the color features and geometric features mentioned in the coarse matching stage. The module outputs the feature vectors of the reference point cloud and the target point cloud respectively, and the inner product of the feature vectors can form a correspondence matrix (i.e., the second correspondence matrix). The matrix is ​​decomposed by SVD to obtain the final pose estimate of the fine matching stage.

[0062] Therefore, the embodiments of the present application make full use of the RGB-D modality to predict the relative pose of 6 degrees of freedom (6DoF), while enhancing the network's generalization ability for novel objects and environments; in addition, the embodiments of the present application combine depth information to more accurately capture the three-dimensional shape and position information of the object, thereby improving the accuracy and robustness of pose estimation.

[0063] In summary, most traditional methods select the most similar reference view as the pose anchor to significantly reduce the pose prediction space. However, the relative pose that needs to be predicted in the embodiments of the present application may be spread throughout the entire SE(3) space. In addition, in part-to-part object matching, factors such as occlusion, sensor noise, and extreme geometric shapes may seriously interfere with the matching process. Traditional technologies find it difficult to use a single RGB-D reference image to estimate the pose of a novel object.

[0064] Therefore, the embodiment of the present application uses a new method and benchmark to estimate the pose of an unseen object based on a single reference image. The embodiment of the present application first uses the powerful generalization ability of the visual basic model to effectively segment the unseen objects; secondly, a coarse-to-fine correspondence matching paradigm is used to estimate the relative pose between the reference object and the query object; in order to establish a reliable correspondence in part-to-part object matching, the embodiment of the present application uses an overlap predictor to identify the overlapping areas between objects.

[0065] It can be understood that the embodiments of the present application introduce a local and global pose-invariant reference framework, which can eliminate the influence of object pose and size changes on object representation, thereby achieving a consistent and standardized description of object features, and using a two-stage point cloud matching paradigm to learn a reliable correspondence between reference objects and target objects, thereby achieving efficient and accurate pose estimation; in addition, the embodiments of the present application also creatively incorporate an overlap predictor, so that the network can automatically adjust the confidence of each correspondence, further improving the robustness and accuracy of pose estimation, and showing great potential and application value in solving the problem of pose estimation of unseen objects.

[0066] According to the object pose estimation method based on a single-view reference template proposed in an embodiment of the present application, a single reference RGB-D image and a target RGB-D image of the target to be measured are obtained, and feature information of the single reference RGB-D image is extracted, and based on the feature information, a reference segmentation mask of the single reference RGB-D image is generated; a target segmentation mask corresponding to the target RGB-D image is obtained, and a back-projection operation is performed on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; based on a preset geometric encoder and a color encoder, a point cloud coarse matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix; based on the coarse pose estimation information, a point cloud fine matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and fine pose estimation information is obtained through the second correspondence matrix, and a final pose estimation result of the target to be measured is obtained according to the fine pose estimation information. This application makes full use of the RGB-D modality to predict the relative pose of 6 degrees of freedom, while enhancing the network's generalization ability for novel objects and environments; in addition, this application combines depth information to more accurately capture the three-dimensional shape and position information of the object, thereby improving the accuracy and robustness of pose estimation.

[0067] Secondly, the object pose estimation device based on a single-view reference template proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0068] Figure 4 It is a block diagram of an object pose estimation device based on a single-view reference template according to an embodiment of the present application.

[0069] like Figure 4 As shown, the object pose estimation device 10 based on a single-view reference template includes: a segmentation mask module 100, a back-projection module 200, a coarse estimation module 300 and a fine estimation module 400.

[0070] The segmentation mask module 100 is used to obtain a single reference RGB-D image and a target RGB-D image of the target to be measured, extract feature information of the single reference RGB-D image, and generate a reference segmentation mask of the single reference RGB-D image based on the feature information.

[0071] The back-projection module 200 is used to obtain a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to a single reference RGB-D image and a target RGB-D image.

[0072] The rough estimation module 300 is used to perform a point cloud rough matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and a color encoder to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate the rough pose estimation information through the first correspondence matrix.

[0073] The fine estimation module 400 is used to perform fine point cloud matching operations on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtain fine pose estimation information through the second correspondence matrix, and obtain the final pose estimation result of the target to be measured based on the fine pose estimation information.

[0074] Optionally, in one embodiment of the present application, the segmentation mask module 100 includes: an extraction unit, a scoring unit and a determination unit.

[0075] The extraction unit is used to extract feature information of a single reference RGB-D image, wherein the feature information includes global feature information and local feature information.

[0076] A scoring unit is used to generate multiple candidate mask proposals for a single reference RGB-D image, and calculate the feature similarity corresponding to the feature information, so as to score each of the multiple candidate mask proposals by the feature similarity to obtain a mask proposal score corresponding to each candidate mask proposal.

[0077] A determination unit is used to determine a reference segmentation mask for a single reference RGB-D image based on the mask proposal score.

[0078] Optionally, in one embodiment of the present application, the coarse estimation module 300 includes: a sampling unit, a splicing unit, a decoding unit and a first inner product unit.

[0079] Among them, the sampling unit is used to transform the reference point cloud and the target point cloud into a preset global reference system, and sample the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud.

[0080] The stitching unit is used to extract the color features and geometric features of the point cloud sampling points, and stitch the color features and the geometric features to obtain stitching features.

[0081] A decoding unit is used to perform a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively.

[0082] The first inner product unit is used to perform inner product processing on the first eigenvectors corresponding to the reference point cloud and the target point cloud to obtain a first correspondence matrix, and perform SVD decomposition processing on the first correspondence matrix to obtain coarse pose estimation information.

[0083] Optionally, in one embodiment of the present application, the fine estimation module 400 includes: a downsampling unit, a construction unit, a coding unit and a second inner product unit.

[0084] The downsampling unit is used to downsample the reference point cloud and the target point cloud to obtain the neighborhood point clouds corresponding to the reference point cloud and the target point cloud respectively.

[0085] The construction unit is used to construct a local pose robust reference system and transform the neighborhood point cloud into the local pose robust reference system.

[0086] The encoding unit is used to encode the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and perform a geometric attention decoding operation on the geometric position code, color features and geometric features to generate a second feature vector corresponding to the reference point cloud and the target point cloud, respectively.

[0087] The second inner product unit is used to perform inner product processing on the second eigenvectors corresponding to the reference point cloud and the target point cloud to obtain a second correspondence matrix.

[0088] It should be noted that the aforementioned explanation of the embodiment of the object pose estimation method based on a single-view reference template is also applicable to the object pose estimation device based on a single-view reference template of this embodiment, and will not be repeated here.

[0089] According to the object pose estimation device based on the single-view reference template proposed in the embodiment of the present application, the device includes a segmentation mask module, which is used to obtain a single reference RGB-D image and a target RGB-D image of the target to be measured, and extract feature information of the single reference RGB-D image, and generate a reference segmentation mask of the single reference RGB-D image based on the feature information; a back-projection module, which is used to obtain a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a reference point cloud corresponding to the single reference RGB-D image and the target RGB-D image. Target point cloud; a coarse estimation module, used to perform point cloud coarse matching operations on the reference point cloud and the target point cloud based on a preset geometric encoder and color encoder, so as to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate the coarse pose estimation information through the first correspondence matrix; a fine estimation module, used to perform point cloud fine matching operations on the reference point cloud and the target point cloud based on the coarse pose estimation information, so as to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtain the fine pose estimation information through the second correspondence matrix, and obtain the final pose estimation result of the target to be measured based on the fine pose estimation information. This application makes full use of the RGB-D modality to predict the relative pose of 6 degrees of freedom, while enhancing the network's generalization ability for novel objects and environments; in addition, this application combines depth information to more accurately capture the three-dimensional shape and position information of the object, thereby improving the accuracy and robustness of pose estimation.

[0090] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0091] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0092] When the processor 502 executes the program, the object pose estimation method based on the single-view reference template provided in the above embodiment is implemented.

[0093] Furthermore, the electronic device further comprises:

[0094] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0095] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0096] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0097] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0098] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0099] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0100] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for estimating an object pose based on a single-view reference template.

[0101] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned object pose estimation method based on a single-view reference template.

[0102] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0103] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0104] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0106] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0109] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for estimating object pose based on a single-view reference template, characterized in that: The following steps are involved: Acquire a single reference RGB-D image and a target RGB-D image of the target to be measured, extract feature information of the single reference RGB-D image, and generate a reference segmentation mask of the single reference RGB-D image based on the feature information; Acquire a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; Based on a preset geometric encoder and a color encoder, a point cloud coarse matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix; Based on the coarse pose estimation information, a point cloud fine matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and fine pose estimation information is obtained through the second correspondence matrix, and a final pose estimation result of the target to be measured is obtained based on the fine pose estimation information.

2. The method according to claim 1, characterized in that: The extracting feature information of the single reference RGB-D image and generating a reference segmentation mask of the single reference RGB-D image based on the feature information includes: Extracting feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; generating a plurality of candidate mask proposals for the single reference RGB-D image, and calculating a feature similarity corresponding to the feature information, so as to score each of the plurality of candidate mask proposals according to the feature similarity, and obtain a mask proposal score corresponding to each candidate mask proposal; A reference segmentation mask for the single reference RGB-D image is determined according to the mask proposal score.

3. The method according to claim 2, characterized in that The method of performing a point cloud coarse matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and a color encoder to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculating coarse pose estimation information through the first correspondence matrix includes: The reference point cloud and the target point cloud are transformed into a preset global reference system, and the reference point cloud and the target point cloud are sampled respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; Extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain splicing features; Performing a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; An inner product process is performed on the first eigenvectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and an SVD decomposition process is performed on the first correspondence matrix to obtain the coarse pose estimation information.

4. The method according to claim 3, characterized in that: The step of performing a point cloud fine matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud includes: Performing point cloud downsampling on the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively; Constructing a local pose robust reference system, and transforming the neighborhood point cloud into the local pose robust reference system; Encoding the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and performing a geometric attention decoding operation on the geometric position code, the color feature, and the geometric feature to generate second feature vectors corresponding to the reference point cloud and the target point cloud, respectively; An inner product process is performed on the second eigenvectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

5. An object pose estimation device based on a single-view reference template, characterized in that: include: A segmentation mask module, used to obtain a single reference RGB-D image and a target RGB-D image of the target to be measured, extract feature information of the single reference RGB-D image, and generate a reference segmentation mask of the single reference RGB-D image based on the feature information; A back-projection module, used to obtain a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; A coarse estimation module, configured to perform a point cloud coarse matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and a color encoder, so as to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first correspondence matrix; A fine estimation module is used to perform a point cloud fine matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud, and obtain fine pose estimation information through the second correspondence matrix, and obtain a final pose estimation result of the target to be measured based on the fine pose estimation information.

6. The device according to claim 5, characterized in that The segmentation mask module comprises: An extraction unit, configured to extract feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; a scoring unit, configured to generate a plurality of candidate mask proposals for the single reference RGB-D image, and calculate a feature similarity corresponding to the feature information, so as to score each of the plurality of candidate mask proposals according to the feature similarity, and obtain a mask proposal score corresponding to each candidate mask proposal; A determining unit is used to determine a reference segmentation mask of the single reference RGB-D image according to the mask proposal score.

7. The device according to claim 6, characterized in that The rough estimation module comprises: A sampling unit, used to transform the reference point cloud and the target point cloud into a preset global reference system, and to sample the reference point cloud and the target point cloud respectively, so as to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; A splicing unit, used for extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain splicing features; A decoding unit, configured to perform a geometric attention decoding operation on the spliced ​​features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; The first inner product unit is used to perform inner product processing on the first eigenvectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and perform SVD decomposition processing on the first correspondence matrix to obtain the coarse pose estimation information.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object pose estimation method based on a single-view reference template as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the object pose estimation method based on a single-view reference template as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the object pose estimation method based on a single-view reference template as described in any one of claims 1 to 4.

Images