A Method and Apparatus for Object Pose Estimation Based on a Single-View Reference Template

Abstract

The application relates to an object pose estimation method and device based on a single-view reference template, wherein the method comprises the following steps: extracting feature information of a single reference RGB-D image, and generating a reference segmentation mask of the single reference RGB-D image through a preset SAM algorithm; obtaining a target segmentation mask corresponding to a target RGB-D image, and performing back projection on the target segmentation mask and the reference segmentation mask to obtain a reference point cloud and a target point cloud; performing point cloud coarse matching on the reference point cloud and the target point cloud, establishing a first corresponding relationship matrix of the reference point cloud and the target point cloud, and calculating coarse pose estimation information; based on the coarse pose estimation information, performing point cloud fine matching on the reference point cloud and the target point cloud, establishing a second corresponding relationship matrix of the reference point cloud and the target point cloud, and obtaining fine pose estimation information, and then obtaining a final pose estimation result of a target to be measured. Therefore, the problems of high acquisition cost, long time consumption and poor generalization ability of a pose estimation reference image in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of object pose estimation technology, and in particular to an object pose estimation method and apparatus based on a single-view reference template. Background Technology

[0002] Currently, most work focuses on instance-level object pose estimation, where the training and testing datasets consist of a set of identical known object instances, and typically require CAD models of the objects to generate training images and labels. Some existing techniques extend this paradigm to the category level, aiming to estimate the pose of new instances within a predefined category without requiring CAD models of the target object.

[0003] However, both paradigms are time-consuming and potentially difficult to implement for labeling and training on new objects beyond known categories, thus limiting their application in open worlds. To mitigate this issue, existing techniques can estimate the pose of any new object, i.e., the target object is unseen during training. Therefore, these techniques require multiple views of the object in different poses as reference images. If a CAD model exists during the testing phase, reference views can be easily generated through image rendering; or, in the absence of a CAD model, a series of images of the object can be taken to obtain reference views. However, both of these setups can be very time-consuming due to the creation of the object's CAD model or the labeling of a large number of object poses. In summary, while existing methods demonstrate potential transferability for pose estimation of new objects, they are clearly limited by their dependence on CAD models or multiple reference views.

[0004] Specifically, existing partial pose estimation methods are described below:

[0005] 1. Pose estimation methods for specific objects:

[0006] In recent years, pose estimation methods for specific objects have received widespread attention in the fields of robotics and computer vision. These methods aim to predict the 6DoF pose of a known object using a CAD model (Computer-Aided Design Model). Specifically, one approach is to predict the 6DoF pose indirectly and then solve for the 6DoF pose using a RANSAC-based PnP algorithm. Another approach is to directly regress the 6DoF pose using a deep neural network. PoseCNN and DeepIM utilize point matching loss functions, while CosyPose extends DeepIM by introducing multi-view information and optimizing the global scene pose. SingleStage and GDR-Net infer intermediate geometric features and directly regress the 6DoF pose through the network.

[0007] 2. General object pose estimation method:

[0008] General object pose estimation models, trained on partial objects, can generalize to other arbitrary rigid objects. Specifically, given a target image, OVE6D and MegaPose retrieve the viewpoint most similar to the target image from a pre-rendered panoramic image database as a coarse pose estimate. Then, a custom neural network further refines this coarse pose estimate, and FoundationPose further constructs a pose ranking network to score each optimized pose hypothesis. Feature matching methods construct pixel-level or point cloud-level correspondences by learning feature descriptors and solve for the pose. In addition, FoundPose and FreeZe directly generate robust visual features from 2D images or partial 3D point clouds for feature matching.

[0009] However, general object pose estimation methods require dense view maps of the target object as a reference. These dense views need to be generated by taking a series of images of the target object or by scanning the CAD model of the target object and using image rendering, which is costly and time-consuming.

[0010] 3. Relative pose estimation method:

[0011] Relative object pose estimation uses only a single reference image to estimate the relative pose transformation of a general object. For example, RelPose and RelPose++ infer the distribution of relative rotation by utilizing an energy-based formula; furthermore, iFusion optimizes unknown relative poses by reversing a new-view synthetic diffusion model. In the field of relative 3DoF object pose estimation, given two RGB images, 3DHAV proposes a hypothesis-validation framework for scoring each relative pose hypothesis. Building upon this, DVMNet introduces a hypothesis-free process that computes relative poses through deep voxel matching.

[0012] However, existing relative pose estimation methods only use RGB modes to predict 3-DOF rotations, and have limited ability to predict relative translations and generalize beyond their training datasets.

[0013] In summary, existing object pose estimation methods suffer from high costs and long processing times in obtaining pose estimation reference maps, as well as poor generalization ability, which urgently need to be addressed. Summary of the Invention

[0014] This application provides a method and apparatus for object pose estimation based on a single-view reference template, in order to solve the problems of high cost, long time consumption and poor generalization ability in existing object pose estimation methods.

[0015] The first aspect of this application provides a method for object pose estimation based on a single-view reference template, comprising the following steps: acquiring a single reference RGB-D image and a target RGB-D image of the target to be measured, extracting feature information from the single reference RGB-D image, and generating a reference segmentation mask for the single reference RGB-D image based on the feature information; acquiring a target segmentation mask corresponding to the target RGB-D image, and performing a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target image corresponding to the single reference RGB-D image and the target RGB-D image. The reference point cloud and the target point cloud are coarsely matched based on a preset geometric encoder and color encoder to establish a first correspondence matrix between them. Coarse pose estimation information is then calculated using this first correspondence matrix. Based on the coarse pose estimation information, a fine point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between them. Fine pose estimation information is then obtained using this second correspondence matrix, and the final pose estimation result of the target is obtained based on the fine pose estimation information.

[0016] Optionally, in one embodiment of this application, the step of extracting feature information from the single reference RGB-D image and generating a reference segmentation mask for the single reference RGB-D image based on the feature information includes: extracting feature information from the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; generating multiple candidate mask proposals for the single reference RGB-D image, and calculating the feature similarity corresponding to the feature information, so as to score each candidate mask proposal in the multiple candidate mask proposals by the feature similarity, thereby obtaining a mask proposal score corresponding to each candidate mask proposal; and determining the reference segmentation mask for the single reference RGB-D image based on the mask proposal score.

[0017] Optionally, in one embodiment of this application, the step of performing a coarse point cloud matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and color encoder to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculating coarse pose estimation information through the first correspondence matrix, includes: transforming the reference point cloud and the target point cloud to a preset global reference frame, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; extracting the color features and geometric features of the point cloud sampling points, and concatenating the color features and the geometric features to obtain concatenated features; performing a geometric attention decoding operation on the concatenated features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; performing inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and performing SVD decomposition processing on the first correspondence matrix to obtain the coarse pose estimation information.

[0018] Optionally, in one embodiment of this application, the step of performing fine point cloud matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud includes: performing point cloud downsampling on the reference point cloud and the target point cloud to obtain the neighborhood point clouds corresponding to the reference point cloud and the target point cloud respectively.

[0019] A local pose robust reference frame is constructed, and the neighborhood point cloud is transformed into the local pose robust reference frame. The neighborhood point cloud in the local pose robust reference frame is encoded to generate a geometric position code, and a geometric attention decoding operation is performed on the geometric position code, the color feature, and the geometric feature to generate second feature vectors corresponding to the reference point cloud and the target point cloud, respectively. The second feature vectors corresponding to the reference point cloud and the target point cloud are processed by inner product processing to obtain the second correspondence matrix.

[0020] A second aspect of this application provides a method for object pose estimation based on a single-view reference template, comprising: a segmentation mask module, configured to acquire a single reference RGB-D image and a target RGB-D image of the target to be measured, extract feature information from the single reference RGB-D image, and generate a reference segmentation mask for the single reference RGB-D image based on the feature information; and a back-projection module, configured to acquire a target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image. The system comprises: a coarse estimation module, used to perform coarse point cloud matching operations on the reference point cloud and the target point cloud based on a preset geometric encoder and color encoder, to establish a first correspondence matrix between the reference point cloud and the target point cloud, and to calculate coarse pose estimation information through the first correspondence matrix; and a fine estimation module, used to perform fine point cloud matching operations on the reference point cloud and the target point cloud based on the coarse pose estimation information, to establish a second correspondence matrix between the reference point cloud and the target point cloud, to obtain fine pose estimation information through the second correspondence matrix, and to obtain the final pose estimation result of the target under test based on the fine pose estimation information.

[0021] Optionally, in one embodiment of this application, the segmentation mask module includes: an extraction unit, configured to extract feature information of the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; a scoring unit, configured to generate multiple candidate mask proposals for the single reference RGB-D image, and calculate the feature similarity corresponding to the feature information, so as to score each candidate mask proposal among the multiple candidate mask proposals based on the feature similarity, thereby obtaining a mask proposal score corresponding to each candidate mask proposal; and a determination unit, configured to determine a reference segmentation mask for the single reference RGB-D image based on the mask proposal score.

[0022] Optionally, in one embodiment of this application, the coarse estimation module includes: a sampling unit, configured to transform the reference point cloud and the target point cloud to a preset global reference frame, and sample the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; a stitching unit, configured to extract the color features and geometric features of the point cloud sampling points, and stitch the color features and the geometric features to obtain stitched features; a decoding unit, configured to perform geometric attention decoding on the stitched features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; and a first inner product unit, configured to perform inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain a first correspondence matrix, and perform SVD decomposition processing on the first correspondence matrix to obtain the coarse pose estimation information.

[0023] Optionally, in one embodiment of this application, the fine estimation module includes: a downsampling unit, configured to downsample the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively; a construction unit, configured to construct a local pose robust reference system and transform the neighborhood point cloud into the local pose robust reference system; an encoding unit, configured to encode the neighborhood point cloud in the local pose robust reference system to generate a geometric position code, and perform geometric attention decoding operations on the geometric position code, the color feature, and the geometric feature to generate second feature vectors corresponding to the reference point cloud and the target point cloud, respectively; and a second inner product unit, configured to perform inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

[0024] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object pose estimation method based on a single-view reference template as described in the above embodiments.

[0025] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described object pose estimation method based on a single-view reference template.

[0026] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described object pose estimation method based on a single-view reference template.

[0027] Therefore, the embodiments of this application have the following beneficial effects:

[0028] The embodiments of this application can acquire a single reference RGB-D image and a target RGB-D image of the target to be tested, extract feature information from the single reference RGB-D image, and generate a reference segmentation mask for the single reference RGB-D image based on the feature information; acquire the target segmentation mask corresponding to the target RGB-D image, and perform a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain the reference point cloud and target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; perform a coarse point cloud matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and color encoder to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first correspondence matrix; perform a fine point cloud matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information to establish a second correspondence matrix between the reference point cloud and the target point cloud, obtain fine pose estimation information through the second correspondence matrix, and obtain the final pose estimation result of the target to be tested based on the fine pose estimation information. This application fully utilizes RGB-D modal prediction to determine 6-DOF relative pose, while enhancing the network's generalization ability to novel objects and environments. Furthermore, by incorporating depth information, it can more accurately capture the 3D shape and position information of objects, improving the accuracy and robustness of pose estimation. Thus, it solves the problems of high cost, long time consumption, and poor generalization ability in existing object pose estimation methods.

[0029] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0030] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0031] Figure 1 This is a flowchart of an object pose estimation method based on a single-view reference template provided according to an embodiment of this application;

[0032] Figure 2 A comparative schematic diagram of reference images of this application and conventional methods is provided for one embodiment of this application;

[0033] Figure 3 A schematic diagram of the logical architecture of a two-stage point cloud matching paradigm provided for one embodiment of this application;

[0034] Figure 4 This is an example diagram of an object pose estimation device based on a single-view reference template according to an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0036] Among them, 10-object pose estimation device based on single-view reference template; 100-segmentation mask module, 200-back projection module, 300-coarse estimation module, 400-fine estimation module; 501-memory, 502-processor, 503-communication interface. Detailed Implementation

[0037] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0038] The following description, with reference to the accompanying drawings, describes an object pose estimation method and apparatus based on a single-view reference template according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides an object pose estimation method based on a single-view reference template. In this method, a single reference RGB-D image and a target RGB-D image of the target are acquired, and feature information of the single reference RGB-D image is extracted. Based on the feature information, a reference segmentation mask for the single reference RGB-D image is generated. A target segmentation mask corresponding to the target RGB-D image is obtained, and a back-projection operation is performed on the reference segmentation mask and the target segmentation mask to obtain the single reference RGB-D image and the target RGB-D image. The method involves creating corresponding reference and target point clouds. Based on a pre-defined geometric encoder and color encoder, a coarse point cloud matching operation is performed on the reference and target point clouds to establish a first correspondence matrix. Coarse pose estimation information is then calculated using this first correspondence matrix. Based on this coarse pose estimation information, a fine point cloud matching operation is performed on the reference and target point clouds to establish a second correspondence matrix. Fine pose estimation information is obtained using this second correspondence matrix, and the final pose estimation result of the target is obtained based on this fine pose estimation information. This application fully utilizes RGB-D modal prediction to predict 6-DOF relative poses, while enhancing the network's generalization ability to novel objects and environments. Furthermore, by combining depth information, this application can more accurately capture the 3D shape and position information of objects, improving the accuracy and robustness of pose estimation. This solves the problems of high cost, long time consumption, and poor generalization ability in existing object pose estimation methods.

[0039] Specifically, Figure 1 This is a flowchart illustrating an object pose estimation method based on a single-view reference template, provided in an embodiment of this application.

[0040] like Figure 1 As shown, the object pose estimation method based on a single-view reference template includes the following steps:

[0041] In step S101, a single reference RGB-D image and a target RGB-D image of the target to be tested are acquired, and feature information of the single reference RGB-D image is extracted. Based on the feature information, a reference segmentation mask of the single reference RGB-D image is generated.

[0042] In this embodiment, feature information of a single reference RGB-D image of the target to be tested is first extracted, and then a reference segmentation mask of the single reference RGB-D image is generated by combining it with a preset SAM algorithm. Thus, even with only a single reference image, the target object can be effectively segmented from a complex background, providing accurate input for subsequent pose estimation steps.

[0043] Optionally, in one embodiment of this application, extracting feature information from a single reference RGB-D image and generating a reference segmentation mask for the single reference RGB-D image based on the feature information includes: extracting feature information from the single reference RGB-D image, wherein the feature information includes global feature information and local feature information; generating multiple candidate mask proposals for the single reference RGB-D image and calculating the feature similarity corresponding to the feature information, so as to score each candidate mask proposal in the multiple candidate mask proposals by feature similarity to obtain a mask proposal score corresponding to each candidate mask proposal; and determining the reference segmentation mask for the single reference RGB-D image based on the mask proposal score.

[0044] Those skilled in the art should understand that segmenting a query object from a cluttered background benefits from the powerful generalization ability of the visual basic model, thus enabling the effective segmentation of novel objects using CAD models.

[0045] Unlike traditional methods that generate multiple descriptors from multiple rendered views, this application embodiment fully utilizes global and local descriptors in DINOv2 to score the mask proposal for SAM when only a single reference image is accessible.

[0046] Specifically, embodiments of this application can use the DINOv2 model to extract rich feature information from a reference image, including global features (describing the overall properties of the entire object) and local features (focusing on the specific details of the object); then, embodiments of this application can combine the feature information with candidate mask proposals generated by the SAM algorithm, and score each mask proposal by calculating feature similarity or matching degree, with the higher the score, the greater the probability that the mask proposal will become the target object for segmentation.

[0047] Therefore, the embodiments of this application realize the segmentation of unseen objects and 6-DOF pose estimation using a single RGB-D reference image for the first time. This not only expands the application scope of object recognition and pose estimation, but also significantly improves the ability to handle unseen objects in complex scenes, and has high technical advancement and practicality.

[0048] In step S102, the target segmentation mask corresponding to the target RGB-D image is obtained, and the reference segmentation mask and the target segmentation mask are back-projected to obtain the reference point cloud and target point cloud corresponding to a single reference RGB-D image and the target RGB-D image.

[0049] In step S103, based on the preset geometric encoder and color encoder, a coarse point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix.

[0050] Given reference and target RGB-D images and their segmentation masks, embodiments of this application can further backproject depth into the point cloud space to obtain target point clouds and reference point clouds; subsequently, embodiments of this application can utilize point cloud registration methods to establish a correspondence matrix between the target point cloud and the reference point cloud, such as... Figure 2 As shown, the relative pose transformation between the target point cloud and the reference point cloud is estimated.

[0051] Optionally, in one embodiment of this application, based on a preset geometric encoder and color encoder, a coarse point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix. This includes: transforming the reference point cloud and the target point cloud into a preset global reference frame, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; extracting the color features and geometric features of the point cloud sampling points, and concatenating the color features and geometric features to obtain concatenated features; performing a geometric attention decoding operation on the concatenated features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; performing inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and performing SVD decomposition processing on the first correspondence matrix to obtain coarse pose estimation information.

[0052] Understandably, given only a single reference image, the relative pose to be predicted is arbitrary in the entire SE(3) space (i.e., the three-dimensional Euclidean transformation group containing rotations and translations), which poses a significant challenge to predicting robust correspondences.

[0053] Therefore, embodiments of this application can use a pose-robust global reference frame transformation strategy to transform any point cloud to that reference frame.

[0054] Specifically, in this embodiment, the origin of the global reference system can be set at the center of the point cloud, the maximum radius of the point cloud can be normalized to 1, and a rotation matrix related to the point cloud distribution can be designed as the pointing of the three coordinate axes of the reference system. After transforming the point cloud to this reference system, this embodiment can sample some points from the original point cloud to obtain point cloud sampling points. Color features and geometric features are extracted from the point cloud sampling points, and the stitched features obtained are input to a preset geometric attention decoding module, such as... Figure 3 As shown, the feature vectors of the reference point cloud and the target point cloud are output respectively; the inner product of the feature vectors can form the correspondence matrix (i.e., the first correspondence matrix), and the matrix is ​​decomposed by SVD to obtain the initial pose estimation of the coarse matching stage (i.e., coarse pose estimation information).

[0055] In step S104, based on the coarse pose estimation information, a fine point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud. Fine pose estimation information is obtained through the second correspondence matrix, and the final pose estimation result of the target to be measured is obtained based on the fine pose estimation information.

[0056] It should be noted that in point cloud registration, the embodiments of this application can use color and geometric cues to construct a correspondence and follow the widely used two-stage matching paradigm of coarse matching stage and fine matching stage, thereby solving the problem of novel object pose estimation based on a single frame RGB-D reference image, which has broad application prospects in factory object assembly and robotic arm operation.

[0057] Therefore, the embodiments of this application can design a novel and extensive benchmark system, which is specifically designed to evaluate the performance of algorithms for segmenting and estimating unseen objects using a single reference image. This benchmark not only provides researchers in related fields with a unified evaluation standard, but also promotes fair comparison and continuous progress of technologies by including evaluations of various traditional methods and learning-based methods.

[0058] Optionally, in one embodiment of this application, based on coarse pose estimation information, a fine point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud. This includes: downsampling the reference point cloud and the target point cloud to obtain the neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively; constructing a local pose robust reference system and transforming the neighborhood point clouds into the local pose robust reference system; encoding the neighborhood point clouds in the local pose robust reference system to generate geometric position codes, and performing geometric attention decoding operations on the geometric position codes, color features, and geometric features to generate second feature vectors corresponding to the reference point cloud and the target point cloud, respectively; and performing inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

[0059] In actual implementation, the fine-grained matching phase differs from the coarse-grained matching phase in the following two ways:

[0060] 1. The fine-grained matching stage requires sampling more points to establish a correspondence matrix;

[0061] 2. In this embodiment, a local pose robust reference system is established for each point's neighborhood, and a local reference system transformation is performed on the point cloud. The neighborhood point cloud is encoded by the encoder to form a geometric position code, which is sent to the subsequent decoding module along with the color features and geometric features mentioned in the coarse matching stage. This module outputs the feature vectors of the reference point cloud and the target point cloud respectively. The inner product of the feature vectors forms the correspondence matrix (i.e., the second correspondence matrix). The matrix is ​​decomposed by SVD to obtain the final pose estimate of the fine matching stage.

[0062] Therefore, the embodiments of this application make full use of RGB-D modes to predict 6-DOF relative poses, while enhancing the network's ability to generalize to novel objects and environments; in addition, by combining depth information, the embodiments of this application can more accurately capture the 3D shape and position information of objects, thereby improving the accuracy and robustness of pose estimation.

[0063] In summary, since traditional methods mostly select the most similar reference view as the pose anchor point to significantly reduce the pose prediction space, however, the relative pose to be predicted in the embodiments of this application may be spread throughout the entire SE(3) space; in addition, in part-to-part object matching, factors such as occlusion, sensor noise and extreme geometry may seriously interfere with the matching process, and traditional techniques are difficult to use a single RGB-D reference image to estimate the pose of novel objects.

[0064] Therefore, this application provides a novel method and benchmark for estimating the pose of unseen objects based on a single reference image. First, it leverages the powerful generalization ability of a visual base model to effectively segment the unseen object. Second, it employs a coarse-to-fine correspondence matching paradigm to estimate the relative pose between the reference object and the query object. To establish reliable correspondences in part-to-part object matching, this application utilizes an overlap predictor to identify overlapping regions between objects.

[0065] It is understood that the embodiments of this application introduce a local and global pose-invariant reference frame, which can eliminate the influence of changes in object pose and size on object representation, thereby achieving a consistent and standardized description of object features. It also utilizes a two-stage point cloud matching paradigm to learn a reliable correspondence between reference objects and target objects, achieving efficient and accurate pose estimation. In addition, the embodiments of this application creatively incorporate an overlap predictor, enabling the network to automatically adjust the confidence of each correspondence, further improving the robustness and accuracy of pose estimation, and demonstrating great potential and application value in solving the problem of pose estimation for unseen objects.

[0066] According to the object pose estimation method based on a single-view reference template proposed in this application, a single reference RGB-D image and a target RGB-D image of the target to be tested are acquired, and feature information of the single reference RGB-D image is extracted. Based on the feature information, a reference segmentation mask of the single reference RGB-D image is generated. The target segmentation mask corresponding to the target RGB-D image is acquired, and a back-projection operation is performed on the reference segmentation mask and the target segmentation mask to obtain the reference point cloud and target point cloud corresponding to the single reference RGB-D image and the target RGB-D image. Based on a preset geometric encoder and color encoder, a coarse point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a first correspondence matrix between the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence matrix. Based on the coarse pose estimation information, a fine point cloud matching operation is performed on the reference point cloud and the target point cloud to establish a second correspondence matrix between the reference point cloud and the target point cloud, and fine pose estimation information is obtained through the second correspondence matrix. The final pose estimation result of the target to be tested is obtained based on the fine pose estimation information. This application fully utilizes RGB-D modal prediction to predict 6-DOF relative pose, while enhancing the network's ability to generalize to novel objects and environments. Furthermore, by incorporating depth information, this application can more accurately capture the 3D shape and position information of objects, thereby improving the accuracy and robustness of pose estimation.

[0067] Secondly, with reference to the accompanying drawings, an object pose estimation device based on a single-view reference template according to an embodiment of this application is described.

[0068] Figure 4 This is a block diagram of an object pose estimation device based on a single-view reference template according to an embodiment of this application.

[0069] like Figure 4 As shown, the object pose estimation device 10 based on a single-view reference template includes: a segmentation mask module 100, a back projection module 200, a coarse estimation module 300, and a fine estimation module 400.

[0070] The segmentation mask module 100 is used to acquire a single reference RGB-D image and a target RGB-D image, extract feature information from the single reference RGB-D image, and generate a reference segmentation mask for the single reference RGB-D image based on the feature information.

[0071] The back projection module 200 is used to obtain the target segmentation mask corresponding to the target RGB-D image, and to perform back projection operation on the reference segmentation mask and the target segmentation mask to obtain the reference point cloud and target point cloud corresponding to a single reference RGB-D image and the target RGB-D image.

[0072] The coarse estimation module 300 is used to perform coarse point cloud matching operation on the reference point cloud and the target point cloud based on the preset geometric encoder and color encoder, so as to establish a first correspondence matrix between the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first correspondence matrix.

[0073] The fine estimation module 400 is used to perform fine point cloud matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information, so as to establish a second correspondence matrix between the reference point cloud and the target point cloud, obtain fine pose estimation information through the second correspondence matrix, and obtain the final pose estimation result of the target under test based on the fine pose estimation information.

[0074] Optionally, in one embodiment of this application, the segmentation mask module 100 includes: an extraction unit, a scoring unit, and a determination unit.

[0075] The extraction unit is used to extract feature information from a single reference RGB-D image, where the feature information includes global feature information and local feature information.

[0076] The scoring unit is used to generate multiple candidate mask proposals for a single reference RGB-D image and calculate the feature similarity corresponding to the feature information. The feature similarity is used to score each candidate mask proposal among the multiple candidate mask proposals to obtain the mask proposal score corresponding to each candidate mask proposal.

[0077] A determination unit is used to determine the reference segmentation mask for a single reference RGB-D image based on the mask proposal score.

[0078] Optionally, in one embodiment of this application, the coarse estimation module 300 includes: a sampling unit, a splicing unit, a decoding unit, and a first inner product unit.

[0079] The sampling unit is used to transform the reference point cloud and the target point cloud into a preset global reference system, and to sample the reference point cloud and the target point cloud respectively to obtain the point cloud sampling points corresponding to the reference point cloud and the target point cloud.

[0080] The stitching unit is used to extract the color and geometric features of the point cloud sampling points and stitch the color and geometric features together to obtain the stitched features.

[0081] The decoding unit is used to perform geometric attention decoding on the spliced ​​features to generate the first feature vectors corresponding to the reference point cloud and the target point cloud, respectively.

[0082] The first inner product unit is used to perform inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence matrix, and then perform SVD decomposition processing on the first correspondence matrix to obtain coarse pose estimation information.

[0083] Optionally, in one embodiment of this application, the fine estimation module 400 includes: a downsampling unit, a construction unit, an encoding unit, and a second inner product unit.

[0084] The downsampling unit is used to downsample the reference point cloud and the target point cloud to obtain the neighborhood point clouds corresponding to the reference point cloud and the target point cloud, respectively.

[0085] The building unit is used to construct a local pose robust reference frame and transform the neighborhood point cloud into the local pose robust reference frame.

[0086] The encoding unit is used to encode the neighborhood point cloud in the local pose robust reference frame to generate a geometric position code, and to perform geometric attention decoding operations on the geometric position code, color features and geometric features to generate the second feature vectors corresponding to the reference point cloud and the target point cloud, respectively.

[0087] The second inner product unit is used to perform inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

[0088] It should be noted that the foregoing explanation of the object pose estimation method based on a single-view reference template also applies to the object pose estimation device based on a single-view reference template in this embodiment, and will not be repeated here.

[0089] The object pose estimation device based on a single-view reference template proposed in this application includes a segmentation mask module for acquiring a single reference RGB-D image and a target RGB-D image of the target to be measured, extracting feature information from the single reference RGB-D image, and generating a reference segmentation mask for the single reference RGB-D image based on the feature information; and a back-projection module for acquiring a target segmentation mask corresponding to the target RGB-D image, and performing a back-projection operation on the reference segmentation mask and the target segmentation mask to obtain the reference point cloud and the target RGB-D image corresponding to the single reference RGB-D image and the target RGB-D image. The system comprises a target point cloud and a coarse estimation module. The coarse estimation module performs point cloud coarse matching on the reference and target point clouds based on a preset geometric encoder and color encoder to establish a first correspondence matrix between the reference and target point clouds, and calculates coarse pose estimation information using this first correspondence matrix. The fine estimation module performs point cloud fine matching on the reference and target point clouds based on the coarse pose estimation information to establish a second correspondence matrix between the reference and target point clouds, obtains fine pose estimation information using this second correspondence matrix, and obtains the final pose estimation result of the target object based on the fine pose estimation information. This application fully utilizes RGB-D modal prediction to predict 6-DOF relative pose, while enhancing the network's generalization ability to novel objects and environments. Furthermore, by combining depth information, this application can more accurately capture the 3D shape and position information of objects, improving the accuracy and robustness of pose estimation.

[0090] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0091] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0092] When the processor 502 executes the program, it implements the object pose estimation method based on a single-view reference template provided in the above embodiments.

[0093] Furthermore, electronic devices also include:

[0094] Communication interface 503 is used for communication between memory 501 and processor 502.

[0095] The memory 501 is used to store computer programs that can run on the processor 502.

[0096] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0097] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0098] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0099] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0100] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described object pose estimation method based on a single-view reference template.

[0101] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described object pose estimation method based on a single-view reference template.

[0102] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0103] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0104] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0106] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0107] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0108] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0109] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A single-view reference template based object pose estimation method, characterized in that, The method comprises the following steps: obtaining a single reference RGB-D image and a target RGB-D image of a to-be-tested target, extracting feature information of the single reference RGB-D image, and generating a reference segmentation mask of the single reference RGB-D image based on the feature information; obtaining a target segmentation mask corresponding to the target RGB-D image, and performing back projection operation on the reference segmentation mask and the target segmentation mask to obtain a reference point cloud and a target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; based on a preset geometric encoder and a color encoder, performing point cloud coarse matching operation on the reference point cloud and the target point cloud to establish a first correspondence relationship matrix of the reference point cloud and the target point cloud, and calculating coarse pose estimation information through the first correspondence relationship matrix; based on the coarse pose estimation information, performing point cloud fine matching operation on the reference point cloud and the target point cloud to establish a second correspondence relationship matrix of the reference point cloud and the target point cloud, and obtaining fine pose estimation information through the second correspondence relationship matrix, and obtaining a final pose estimation result of the to-be-tested target according to the fine pose estimation information; wherein, based on the preset geometric encoder and color encoder, the reference point cloud and the target point cloud are subjected to point cloud coarse matching operation to establish a first correspondence relationship matrix of the reference point cloud and the target point cloud, and coarse pose estimation information is calculated through the first correspondence relationship matrix, comprising: transforming the reference point cloud and the target point cloud into a preset global reference system, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain spliced features; performing geometric attention decoding operation on the spliced features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; performing inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first correspondence relationship matrix, and performing SVD decomposition processing on the first correspondence relationship matrix to obtain the coarse pose estimation information; based on the coarse pose estimation information, the reference point cloud and the target point cloud are subjected to point cloud fine matching operation to establish a second correspondence relationship matrix of the reference point cloud and the target point cloud, comprising: performing point cloud downsampling on the reference point cloud and the target point cloud to obtain neighborhood point clouds corresponding to the reference point cloud and the target point cloud respectively; constructing a local pose robust reference system and transforming the neighborhood point clouds into the local pose robust reference system; encoding the neighborhood point clouds in the local pose robust reference system to generate geometric position encodings, and performing geometric attention decoding operation on the geometric position encodings, the color features and the geometric features to generate second feature vectors corresponding to the reference point cloud and the target point cloud respectively; The second corresponding relation matrix is obtained by performing inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud.

2. The method of claim 1, wherein, The feature information of the single reference RGB-D image is extracted, and a reference segmentation mask of the single reference RGB-D image is generated based on the feature information. The feature information of the single reference RGB-D image is extracted, wherein the feature information includes global feature information and local feature information. A plurality of candidate mask proposals of the single reference RGB-D image are generated, and a feature similarity corresponding to the feature information is calculated to score each candidate mask proposal in the plurality of candidate mask proposals by using the feature similarity, so as to obtain a mask proposal score corresponding to each candidate mask proposal. The reference segmentation mask of the single reference RGB-D image is determined according to the mask proposal score.

3. An object pose estimation device based on a single-view reference template, characterized in that, It comprises: The segmentation mask module is used for acquiring a single reference RGB-D image and a target RGB-D image of a to-be-measured target, extracting feature information of the single reference RGB-D image, and generating a reference segmentation mask of the single reference RGB-D image based on the feature information; The back projection module is used for acquiring a target segmentation mask corresponding to the target RGB-D image, and performing back projection operation on the reference segmentation mask and the target segmentation mask to obtain reference point cloud and target point cloud corresponding to the single reference RGB-D image and the target RGB-D image; The coarse estimation module is used for performing point cloud coarse matching operation on the reference point cloud and the target point cloud based on a preset geometric encoder and a color encoder, to establish a first corresponding relation matrix of the reference point cloud and the target point cloud, and calculate coarse pose estimation information through the first corresponding relation matrix; The fine estimation module is used for performing point cloud fine matching operation on the reference point cloud and the target point cloud based on the coarse pose estimation information, to establish a second corresponding relation matrix of the reference point cloud and the target point cloud, and obtain fine pose estimation information through the second corresponding relation matrix, and obtain the final pose estimation result of the to-be-measured target according to the fine pose estimation information; The coarse estimation module comprises: The sampling unit is used for transforming the reference point cloud and the target point cloud into a preset global reference system, and sampling the reference point cloud and the target point cloud respectively to obtain point cloud sampling points corresponding to the reference point cloud and the target point cloud; The splicing unit is used for extracting color features and geometric features of the point cloud sampling points, and splicing the color features and the geometric features to obtain spliced features; The decoding unit is used for performing geometric attention decoding operation on the spliced features to generate first feature vectors corresponding to the reference point cloud and the target point cloud respectively; The first inner product unit is used for performing inner product processing on the first feature vectors corresponding to the reference point cloud and the target point cloud to obtain the first corresponding relation matrix, and performing SVD decomposition processing on the first corresponding relation matrix to obtain the coarse pose estimation information; The fine estimation module comprises: a downsampling unit, configured to perform point cloud downsampling on the reference point cloud and the target point cloud to obtain a neighborhood point cloud corresponding to the reference point cloud and the target point cloud respectively; a constructing unit, configured to construct a local pose robust reference system and transform the neighborhood point cloud into the local pose robust reference system; an encoding unit, configured to encode the neighborhood point cloud in the local pose robust reference system to generate a geometric position encoding, and perform a geometric attention decoding operation on the geometric position encoding, the color feature and the geometric feature to generate a second feature vector corresponding to the reference point cloud and the target point cloud respectively; a second inner product unit, configured to perform inner product processing on the second feature vectors corresponding to the reference point cloud and the target point cloud to obtain the second correspondence matrix.

4. The apparatus of claim 3, wherein, The segmentation mask module comprises: an extraction unit configured to extract feature information of the single reference RGB-D image, wherein the feature information comprises global feature information and local feature information; a scoring unit configured to generate a plurality of candidate mask proposals of the single reference RGB-D image, calculate feature similarities corresponding to the feature information, and score each candidate mask proposal in the plurality of candidate mask proposals by using the feature similarities to obtain a mask proposal score corresponding to each candidate mask proposal; a determination unit configured to determine a reference segmentation mask of the single reference RGB-D image according to the mask proposal score.

5. An electronic device, comprising: comprise: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the object pose estimation method based on a single-view reference template according to any one of claims 1-2.

6. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the object pose estimation method based on a single-view reference template according to any one of claims 1-2.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the object pose estimation method based on a single-view reference template according to any one of claims 1-2.

Images