Methods and systems for object pose estimation

Abstract

A computer-implemented method for relative pose estimation comprising: (S10) extracting visual features (130) from a query object view (110) and from reference object views (120) of reference objects pertaining to the same class of the query object; (S20) matching visual features of the query object view with the features of the reference object views; wherein the matched visual features are assigned with reference keypoints in the reference object views and query keypoints in the query object view; (S30) selecting a reference view (125) among the reference object views; (S40) forming a 3D representation (170) of the selected reference view (125), wherein the reference keypoints are lifted to 3D coordinates; (S50) estimating a relative pose (190) of the query object with respect to the selected reference view (125) by aligning a 2D projection (180) of the reference keypoints of the 3D representation (170) with the query keypoints.

Description

METHODS AND SYSTEMS FOR OBJECT POSE ESTIMATIONBACKGROUND OF THE INVENTION1. Field of the Invention

[0001] The invention relates to the field of pose estimation. In particular, the invention relates to category-level object pose estimation.2. Description of Related Art

[0002] The task of 6D pose estimation, which relates to determining the 3D position and orientation of objects in space, is important for various applications such as robotic and augmented reality.

[0003] In the specific field of category-level pose estimation, models learn to estimate the pose of a previously unseen object by using training data based on many different variants of objects of the same category. In particular, the models learn to generalize the pose estimation capabilities to unseen variants of objects of a known category (see [REF 1]). However, such methods require extensive training data which limits their applicability (see [REF 2]).

[0004] In this context, some methods rather explore ways to estimate poses of unseen objects via an online optimization process by transforming the (absolute) pose estimation task to a relative pose estimation task of a query object view with respect to a single reference object, see [REF 3] & [REF 4].

[0005] However, such methods of the prior art generally rely on 3D scanning of the reference object ([REF 3]) or require input images comprising depth information ([REF 4]), which constrains their application.

[0006] The following references disclose various methods relating to object pose estimation.[REF 1] TIAN, Meng, ANG, Marcelo H., et LEE, Gim Hee. Shape prior deformation for categorical 6d object pose and size estimation. In : Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16. Springer International Publishing, 2020. p. 530-546.[REF 2] LI, Xiaolong, WANG, He, YI, Li, et al. Category-level articulated object pose estimation. In : Proceedings of the IEEE / CVF conference on computer vision and patternrecognition. 2020. p. 3706-3715.[REF 3] GOODWIN, Walter, HAVOUTIS, loannis, et POSNER, Ingmar. You only look at one: Category-level object representations for pose estimation from a single example. arXiv preprint arXiv:2305.12626, 2023.[REF 4] GOODWIN, Walter, VAZE, Sagar, HAVOUTIS, loannis, et al. Zero-shot category-level object pose estimation. In : European Conference on Computer Vision. Cham : Springer Nature Switzerland, 2022. p. 516-532.[REF 5] REIZENSTEIN, Jeremy, SHAPOVALOV, Roman, HENZLER, Philipp, et al. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In : Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021. p. 10901-10911.[REF 6] CARON, Mathilde, TOUVRON, Hugo, MISRA, Ishan, et al. Emerging properties in self-supervised vision transformers. In : Proceedings of the IEEE / CVF international conference on computer vision. 2021. p. 9650-9660.[REF 7] OQUAB, Maxime, DARCET, Timothee, MOUTAKANNI, Theo, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.[REF 8] ZHOU, Yi, BARNES, Connelly, LU, Jingwan, et al. On the continuity of rotation representations in neural networks. In : Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019. p. 5745-5753.[REF 9] KINGMA, Diederik P. et BA, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.SUMMARY OF THE INVENTION

[0007] According to aspects of the present disclosure, a computer-implemented method for relative pose estimation is provided. The method comprises steps of:- extracting visual features from a query object view and from a plurality of reference object views of reference objects pertaining to the same class of the query object;- matching visual features extracted from the query object view with visual featuresextracted from the reference object views; wherein the matched visual features are assigned with reference keypoints in the reference object views and query keypoints in the query object view;- selecting a reference view among the reference object views;- forming a 3D representation of the selected reference view, wherein the reference keypoints are lifted to 3D coordinates; and- estimating a relative pose of the query object with respect to the selected reference view by aligning a 2D projection of the reference keypoints of the 3D representation with the query keypoints of the query object view.

[0008] In the present description, the visual features are features that encode points pertaining to an object in a particular way that allows for distinguishing the object from other objects. Further, the visual features of the present description are adapted for category-level pose estimation so that the same features are used for the same parts for all variants in an object category. Therefore, a single variant of the object in the category may be used as a unique reference and allow for matching to another variant.

[0009] In the present description, the 3D representation of the selected reference view is understood as a 3D representation of a 3D object that is observed under a certain pose in the selected reference view. Therefore, when forming the 3D representation, the 2D keypoints of the selected reference view are lifted to 3D coordinates (thereby forming 3D keypoints).

[0010] With the present method, it is possible to estimate the relative pose of an object that is not known (no prior knowledge on the object is required) from a single query image which does not comprise depth information. Therefore, data requirements are substantially reduced compared to methods of the prior art, and the applicability of the method is broadened.

[0011] Further, in the present method, an online optimization of the pose is advantageously implemented by exploiting feature correspondences between the query image and reference images of objects pertaining to the same class as the query object (comprised in the query image). Advantageously, the present method can even be based on the use of reference images of a single reference object (of the same category of the query object.

[0012] Further, in embodiments of the present method, no depth information onthe reference views is systematically required, which further reduces the data requirements compared to other methods.

[0013] According to embodiments, the step of selecting a reference view among the reference object views comprises selecting the reference view based on the quality of the matched visual features between the query object view and the reference object views.

[0014] According to embodiments, the step of selecting a reference view among the reference object views comprises ranking the matched visual features based on a cyclical distance between visual features of the query object view and visual features of the reference object views, wherein such ranking indicates a level a quality of the matched features.

[0015] According to embodiments, the step of forming a 3D representation of the selected reference view comprises using depth information extracted from the selected reference view.

[0016] According to embodiments, the depth information of the selected reference view is directly known from the fact that the reference object views are depth images comprising depth information.

[0017] According to embodiments, the depth information of the selected reference view is obtained by processing the selected reference view with a monocular depth estimation algorithm.

[0018] According to embodiments, the depth information of the selected reference view is obtained by processing the reference object views view with a multi- view depth estimation algorithm.

[0019] According to embodiments, the depth information of the selected reference view is obtained by processing the reference object views with a structure from motion algorithm.

[0020] According to embodiments, the selected reference view is the best among the reference object views in terms of quality of the matched features.

[0021] According to embodiments, the method further comprises:- iteratively optimizing the relative pose by minimizing residual visual features distances between visual features of projected reference keypoints obtained from a 2D projection of a 3D representation of the selected reference view under the previously obtained relative pose and the visual features of query keypoints of the query object view.

[0022] With such embodiments, the relative pose may therefore be obtained after a plurality of optimization steps. In embodiments, the initial relative pose used for the initial step of the optimization may be the reference pose of the selected reference view. Such reference pose of the selected view may be obtained, for example, through triangulation based on the reference views.

[0023] According to embodiments, the method further comprises optimizing the relative pose for both rotation and translation using first-order stochastic gradients.

[0024] The present disclosure is further related to a data processing system comprising a processing unit configured for executing the steps of the above method.

[0025] The present disclosure is further related to a computer program including instructions for executing the steps of the above method when said program is executed by a computer. This program can use any programming language and take the form of source code, object code or a code intermediate between source code and object code, such as a partially compiled form, or any other desirable form.

[0026] The present disclosure is further related to a non-transitory recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of the above described method. The recording medium can be any entity or device capable of storing the program. For example, For example, the medium can include a mass memory device, such as hard drive. In general, mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of nonvolatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; and magneto-optical disks.

[0027] Alternatively, the recording medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to execute the method in question or to be used in its execution.BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Features, advantages, and technical and industrial significance of exemplary embodiments of the invention will be described below with reference to the accompanying drawings, in which like signs denote like elements, and wherein:FIG. 1 is a flowchart representing aspects of a method for relative pose estimation according to the present disclosure;FIG. 2 is a block diagram representing aspects of a method for relative pose estimation according to the present disclosure;FIGS. 3-4 are tables illustrating performances of the method according to the present disclosure compared to other methods.DETAILED DESCRIPTION OF EMBODIMENTS

[0029] Methods and systems for relative pose estimation, which constitute exemplary embodiments of the present disclosure, are now going to be presented in relation with FIGS.1-4.

[0030] In reference to FIGS.1-2, embodiments of the method for relative pose estimation comprises the following steps:- S10 extracting visual features 130 from a query object view 110 and from a plurality of reference object views 120 of reference objects pertaining to the same class of the query object;- S20 matching visual features of the query object view with the features of the reference object views; wherein the matched visual features are assigned with reference keypoints in the reference object views and query keypoints in the query object view;- S30 selecting a reference view 125 among the reference object views;- S40 forming a 3D representation 170 of the selected reference view 125, wherein the reference keypoints are lifted to 3D coordinates; and- S50 estimating a relative pose 190 of the query object with respect to the selected reference view 125 by aligning a 2D projection 180 of the reference keypoints of the 3D representation 170 with the query keypoints of the query object view 110.

[0031] The present method estimates the 6D pose of an object from a single query object view 110 which comprises only RGB information and no depth information. The relative pose estimation task may be expressed as follows.(R, t) = T (Iq, Kq,J , (Equation 1)

[0032] In Equation 1, J =E N) . Further, R E S0(3) , and, t E IR3, are the 3D translation and 3D rotation, respectively; IqE ]RHX KX3is the singleRGB query input (query object view). The ordered reference set comprises, Ii G n^HxM / x3^ and, DtG IRHx K, as the z-th RGB-D pair of the reference object; Ri. ti are one of the N extrinsic parameters of the view I intrinsic parameters are assumed to be known for both query object view 110 and the reference object views 120 and denoted as Kq, Kt. The relative pose estimation task implemented by the present method differs from most tasks of the prior art as it does not use depth information on the query object view 110. Instead depth is required only for the reference views 120. Therefore, only an individual reference object of each category of interest (among the possible categories to which the query object may pertain) needs to be captured with an RGB-D camera or reconstructed from multiple reference views 120.

[0033] In a first step, S10, visual features such as semantic features are extracted from the query object view 110 and from the reference object views 120. Various foundation models 130 may be used in order to extract reliable features that encode semantic and spatial information across various inputs, relating even sketches with natural images. In particular, the extracted features are used as semantic keypoint descriptors that transfer across different instances pertaining to the same category. These particular features are used as they provide features that are similar throughout semantically similar structure within a category, thereby providing stable correspondences for such similar structure (for example for different looking object pertaining to the same class).

[0034] The features may comprise 2D coordinates of a keypoint and a semantic descriptor associated to said keypoint.

[0035] In embodiments, a foundation model 130 is used to map a RGB query object view It G [^>HxM / x3jn(0 afeature map, Ftwhich can be expressed as follows.(Equation 2)

[0036] In Equation 2, , represents the feature extraction operation; H and W are the height and width of the original image, while H' and W' are the sizes of the feature map, and C the number of channels embedding semantic information of the query object. In particular, H' = [H / a] and W' = [W / a], where a is the side of one patch of the image corresponding to one pixel of the feature map. The extracted features may be referred to as semantically enriched features.

[0037] In embodiments, the used foundation model 130 is a feature extractormodel based on a DinoVl algorithm (see [REF 6]) or a DinoV2 algorithm (see [REF 7]).

[0038] In embodiments, the features extracted from the object views may be comprise latent features of generative image-to-image models such as Stable Diffusion models.

[0039] In a second step, S20, the features extracted from the query object view 110 and the features extracted from the reference object views 120 are matched in order to establish features to features correspondences 140. In embodiments, correspondences 140 are established between the query object view 110 and all the reference object views 120.

[0040] Such establishing of correspondences 140 may be implemented using a soft nearest neighbor operator, which can be expressed as follows. (Equation s)are the 2D coordinates of the top K correspondences that are extracted.

[0042] In step, S30, a reference view 125 is selected among the reference views 120 based on the matched features. In embodiments, the selected reference view 125 is the best in terms of quality of the matched features.

[0043] In particular, it is possible to rank feature matches of features ftJG FLand fqG Fqof points x and xqlby their cyclical distance (see [REF 4]). In order to calculate the cyclic distance, the closest feature match to fqis first found in Ftin order to establish a cyclic point of Ftas follows. j *= argminy\\fk- ftJ||2(Equation 4) (Equation 5)

[0044] Then, the cyclic distance is used to rank the cyclic points by ascending value of distance, d, which can be expressed as follows. (Equation 6)

[0045] The K best correspondences between the query object view and the images of the reference set (the reference object views) may further be utilized to select the best reference object view from the reference set, as expressed as follows. i *= argmin(Equation 7) EN

[0046] In Equation 7, the top K feature matches have the lowest cumulativecyclic distance.

[0047] In step S40, the information from the selected reference object view (possibly the best reference view) is lifted to 3D by unprojecting the point, xtthrough a reverse projection operator. Such operation may be expressed as follows.(Equation 8)

[0048] In Equation 8, ? is the projection operator for a pinhole camera. The “unprojection” operation relates in particular to the fact of inverting the “projection” operation wherein a 3D point is projected to a 2D point in a camera image. The unprojection operation may comprise obtaining of a viewing ray of the 2D image point, Xt, via the pinhole camera defined by intrinsic parameters, Kt. Further, the viewing ray may be scaled, if depth information is available, to the original point in the 3D space.

[0049] Further, in step, S40, 3D positions of the keypoints may be obtained throught triangulation from a plurality of reference views.

[0050] In step, S50, once the keypoints and their semantic description have been lifted to 3D, the features correspondences 140 (matches) are used to optimize for a pose. In the present method, a differentiable pose optimizer is designed based on an iterative online procedure. Formally, the problem is to optimize for a translation, tq, and rotation, Rqgiven the correspondences X-, xq.

[0051] The translation may be represented as a vector in Euclidean space. Further, since the rotation cannot be parametrized in a Euclidean space of dimension three, a 6-dimensional representation of the rotation may be used instead (see [REF 8]).

[0052] Rotation and translation are optimized in terms of reprojection residuals for matched feature points. Such procedure may be expressed as follows. (Equation 9)

[0053] In Equation 9, Rq, and, tq, are the resulting query rotation and translation, respectively. This means that they represent the change of orientation of the best reference object view 125 that is required in order to align the features obtained from the best reference object view 125 onto the features of the query object view 110.

[0054] In particular, the reprojection operation described by Equation 9 may be considered as aiming at finding of correspondence pairs between the 3D keypoints in the reference view 125 and the 2D keypoints in the query view 110 based on the featuredistances. Therefore, each 3D point from the reference view 125 is projected to the query view 110 using a current pose estimate and camera intrinsics in order to compute the euclidean distance between the 2D projections of the 3D keypoints and the 2D keypoints of the query object view. The sum of the residual features distances forms an optimization cost that is used to optimize jointly the relative rotation and translation between the reference view 125 and the query view 110.

[0055] At each iteration of the optimization process, the estimated relative pose 160 (obtained from the previously matched features) is used to recalculate 201 a new 2D projection of the 3D keypoints corresponding to the selected reference view 125 onto the query object view.

[0056] The optimization process may be implemented iteratively using first-order stochastic-gradients calculated with an Adam optimizer (see [REF 9]). This allows for iteratively aligning feature keypoints while optimizing for pose parameters (translation and rotation).

[0057] Results of embodiments of the present method when applied to relative pose estimation tasks on known datasets and comparison with known methods are presented in reference to Figs.3-4. In particular, the results in Figs. 3-4 are obtained when applying methods to the “Common Objects in 3D” (CO3D) dataset (see [REF 5]).

[0058] Experiments are carried out using feature extractors based on the DinoV 1 foundation model (see [REF 6]) or the DinoV2 foundation model (see [REF 7]). Adam optimizer is used with a 3.10' learning rate for a maximum 5000 iterations (see for example [REF 9]). In embodiments, early stopping may be applied to reduce inference time when the standard deviation of the loss drops below 10' in a considered window of 50 iterations.

[0059] When applying the methods, for n available reference object views, t images are chosen by uniform sampling. Single query object views are also randomly selected. Further, feature extraction and feature matching are performed in parallel using a graphical processing unit (GPU).

[0060] Performances of the methods for relative pose estimation tasks are evaluated with respect to the median error (med.err.), the accuracy at 15° (accl5°), and the accuracy at 30° (acc30°). For accuracy at 15° (respectively 30°), a sample rate is calculated for which the relative pose is estimated closer than 15 degrees (respectively 30°). In thetables, the second column indicates the query setup (q. setup) that is used, either comprising an image without depth data (RGB) or comprising an image with depth data (RGB-D).

[0061] In Fig. 3, the present method is compared to the ZSP model disclosed in [REF 4]. In order to fairly compare the two methods, a variant of the ZSP model is used that is only taking RGB data as inputs (depth data is not fed to the method). Five reference object views are used for every experiment used to obtain the results.

[0062] The present method and the variant of the ZSP model to which it is compared are both implemented using a feature extractor based on the DinoV 1 foundation model. In this setting of ZSP, after finding correspondences between features provided by the DinoV 1 model, the essential matrix linking the two views (from which the features are extracted) is obtained.

[0063] The results shown in Fig.3 show that the present method outperforms the ZSP model on all three metrics.

[0064] In Fig.4, the influence of the number of query views on performance of the present method is observed, and compared with the ZSP model.

[0065] In this analysis, a variant of the ZSP model is used, which is fed with depth image data (RGB-D) and is upgraded in order to use the DinoV2 foundation model (which is known to be intrinsically superior to the DinoV 1 model).

[0066] The results in Fig.4 show that the present method provides performances close to the ZSP model while not using any depth data of the query object view. This shows the performance of the present method for relative pose estimation.

[0067] Although the present disclosure refers to specific exemplary embodiments, modifications may be provided to these examples without departing from the general scope of the invention as defined by the claims. In particular, individual characteristics of the different illustrated / mentioned embodiments may be combined in additional embodiments. Therefore, the description and the drawings should be considered in an illustrative rather than in a restrictive sense.

Claims

CLAIMS1. A computer-implemented method for relative pose estimation comprising:- (S10) extracting visual features from a query object view (110) and from a plurality of reference object views (120) of reference objects pertaining to the same class of the query object;- (S20) matching visual features of the query object view (110) with the features of the reference object views (120); wherein the matched visual features (140) are assigned with reference keypoints in the reference object views (120) and query keypoints in the query object view (110);- (S30) selecting a reference view (125) among the reference object views (120);- (S40) forming a 3D representation (170) of the selected reference view (125), wherein the reference keypoints are lifted to 3D coordinates; and- (S50) estimating a relative pose (190) of the query object with respect to the selected reference view (125) by aligning a 2D projection (180) of the reference keypoints of the 3D representation (170) with the query keypoints of the query object view (110).

2. The method of claim 1, wherein the step of (S30) selecting a reference view (125) among the reference object views (120) comprises:- selecting the reference view (125) based on the quality of the matched visual features (140) between the query object view (110) and the reference object views (120).

3. The method of claim 2, wherein the step of (S30) selecting a reference view (125) among the reference object views (120) further comprises:- ranking the matched visual features (140) based on a cyclical distance between visual features of the query object view (110) and visual features of the reference object views (120), wherein such ranking indicates a level a quality of the matched features.

4. The method of any of claims 1 to 3, wherein the step of (S40) forming a 3D representation (170) of the selected reference view (125) comprises:- using depth information extracted from the selected reference view (125).

5. The method of claim 4, wherein the depth information of the selected reference view (125) is directly known from the fact that the reference object views (120) are depth images comprising depth information.

6. The method of claim 4, wherein the depth information of the selected reference view (125) is obtained by processing the selected reference view (125) with a monocular depth estimation algorithm.

7. The method of claim 4, wherein the depth information of the selected reference view (125) is obtained by processing the reference object views (120) view with a multi-view depth estimation algorithm.

8. The method of claim 4, wherein the depth information of the selected reference view (125) is obtained by processing the reference object views (120) with a structure from motion algorithm.

9. The method of any of claims 2 to 8, wherein the selected reference view (125) is the best among the reference object views (120) in terms of quality of the matched features.

10. The method of any of claims 1 to 9, wherein the visual features are extracted using a feature extractor model (130) based on a DinoVl model or a DinoV2 model.

11. The method of any of claims 1 to 10, wherein the method further comprises:- iteratively optimizing the relative pose (190) by minimizing residual visual features distances between visual features of projected reference keypoints obtained from a 2D projection (180) of a 3D representation (170) of the selected reference view under the previously obtained relative pose, and the visual features of query keypoints of the query object view (110).

12. The method of claim 11, wherein the method further comprises:- optimizing the relative pose for both rotation and translation using first-order stochastic gradients.

13. A data processing system comprising a processing unit configured for executing the steps of the method according to any of claims 1 to 12.

14. A computer program including instructions for executing the steps of the method according to any of claims 1 to 12 when said program is executed by a computer.

15. A non-transitory recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of the method according to any of claims 1 to 12.

Images