Multi-stage object pose estimation

Abstract

The invention relates to determination of a multi-dimensional pose of an object. A corresponding method for estimating a multi-dimensional pose mDOF of an object OBJ utilizes a 2D image IMA of the object OBJ and a plurality of 2D templates TEMPL (i) generated from a 3D model MOD of the object OBJ in a rendering process from different known virtual viewpoints vVIEW (i). In a template matching stage S2, a template TEMPL (J) that optimally matches the image IMA is identified. In a stage S3 of correspondence determination, a representation repTEMPL (J) of the identified template TEMPL (J) is compared with a representation repIMA of the image IMA to determine a 2D-3D correspondence, 2D 3D, between pixels in the representation repIMA of the image IMA and voxels of the 3D model MOD of the object OBJ. In the stage S4 of attitude estimation, the 2D-3D correspondence is further processed to estimate the multi-dimensional attitude mDOF based on the 2D-3D correspondence (2D-3D).

Description

Technical Field

[0001] The present invention relates to determination of the multi-dimensional pose of an object. Background Art

[0002] Object detection and multi-dimensional pose estimation of detected objects are frequently solved problems in computer vision because they are applicable in a wide range of applications in different fields. Just for example, autonomous driving, augmented reality, robotics, and several medical applications would be nearly impossible without fast and accurate object localization in multiple dimensions.

[0003] In recent years, several computer vision tasks such as detection and segmentation have experienced tremendous progress due to the development of deep learning. Successful 2D object detection alone is insufficient for real-world applications that require 6D object pose information, such as in the context of augmented reality, robotic manipulation, autonomous driving, etc. Therefore, the ability to recover the pose of objects in a 3D environment - also known as the 6D pose estimation problem - is essential for such applications.

[0004] Pose estimation typically applies one or more pre-trained artificial neural networks ANN (for the sake of brevity, "artificial neural network ANN" will sometimes be named "network ANN" below) to estimate the pose of an object in a scene based on images of the object from different perspectives and based on a comprehensive database, such as described in EP3511904B1, WO2020 / 156836A1, EP3905194A1. In addition, US2019 / 026917A1 describes a method for aligning a model of an object with an image of the object. Finally, the alignment produces discrete pose parameters that are propagated to the object.

[0005] Therefore, a rapid development of high-quality 6D pose estimation methods can be observed due to advances in data preparation, evaluation tools, and the methods themselves. However, in pursuit of high accuracy, the top-performing methods often ignore aspects such as scalability and generalization to new objects that have not been detected before. Most of them involve lengthy annotations of real data or rendering of synthetic training data. In addition, a necessary step is to train the detector on a specific object to make the method object-specific.

[0006] Only a few works try to generalize to new objects, but they mainly focus on objects of the same category. Other works like "CorNet" and "LSE" train networks to estimate local shape descriptors for each object pixel. They focus on 3D corners and objects that share similar geometry, such as those belonging to the "T-Less" dataset. Other methods extend to generalize to unseen objects, but they still use standard 2D detectors for object localization and estimate the rotation of new unseen objects through generalized template matching. Summary of the invention

[0007] Therefore, a method and system are required which serve the need to solve the above mentioned problems and enable multi-dimensional object pose estimation even for newly seen objects. This is solved by the method suggested in claim 1 and the system suggested in claim 13. The dependent claims describe further advantageous embodiments.

[0008] A method for estimating a multi-dimensional posture mDOF of an object OBJ based on an image IMA of the object OBJ comprises a preparation stage, wherein an image IMA depicting the object OBJ and a plurality of templates TEMPL(i) are provided. The template TEMPL(i) is generated from a 3D model MOD of the object OBJ during a rendering process, wherein different templates TEMPL(i), TEMPL(j) of the plurality of templates are generated from the model MOD by rendering from different corresponding known virtual viewpoints vVIEW(i), vVIEW(j) on the model MOD, wherein i≠j. For example, the known virtual viewpoints vVIEW(i) may be evenly distributed on a sphere of the model MOD surrounding the object OBJ, so that it may be assumed that the posture of the object model in the corresponding template TEMPL(i) is known. In the template matching stage S2, at least one template TEMPL(j) that best matches the image IMA is identified and selected from the plurality of templates TEMPL(i), respectively. The inventive concept presented herein comprises further stages S3 and S4. In a subsequent stage S3 of correspondence determination, the representation repTEMPL(J) of the identified template TEMPL(J), such as the identified template TEMPL(J) itself, or preferably, the feature map of the identified template TEMPL(J), is compared with the representation repIMA of the image IMA, such as the image IMA itself, or a segment of the corresponding image IMA containing pixels belonging to the object OBJ, or preferably, the image IMA or the feature map of such a segment of the image IMA, to finally determine the 2D-3D correspondence 2D3D between the pixels in the representation repIMA of the image IMA belonging to the object OBJ and the voxels of the 3D model MOD of the object OBJ. In the stage S4 of pose estimation, the multidimensional pose mDOF is estimated based on the 2D-3D correspondence 2D3D. The fourth stage S4 of pose estimation can apply a PnP+RANSAC process to estimate the multidimensional pose mDOF from the 2D-3D correspondence 2D3D.

[0009] For the sake of clarity, the term "stage" in the context of the methods presented herein has the meaning of a method step.

[0010] Preferably, in a stage S1 of object detection, which is preferably before a second stage S2 of template matching and is performed by a trained network ANN1, a binary segmentation mask SEGM is generated from the image IMA, wherein the mask SEGM respectively identifies those pixels of the image IMA that belong to the object OBJ and which shows the location of the visible parts of the object OBJ in the image IMA.

[0011] In the above and in the following, the expression "pixels belonging to" the object OBJ or, as the case may be, the model MOD" means those pixels of the respective image IMA or TEMPL(i) representing the respective visible part of the object OBJ or the model MOD.

[0012] To generate the segmentation mask SEGM, the features of the image IMA are combined with the object descriptor tensor o representing multiple templates TEMPL(i) k Dense matching is performed and semantic segmentation is performed by ANN1.

[0013] Furthermore, the first stage S1 of object detection includes a plurality of steps S1.1, S1.2, S1.4. In the first step S1.1, the feature extractor from the template TEMPL(i) is used by the trained network ANN1. To calculate the object descriptor tensor of the model MOD of the object OBJ The object descriptor tensor o k All templates TEMPL(i) rendered from virtual viewpoints vVIEW(i) around the model MOD of the object are collected and represented respectively. In a second step S1.2 which may be performed before, after or in parallel with the first step S1.1, the feature map f k =F FE (IMA) where k = 1, ..., K is generated by the network ANN1 using the feature extractor F of the image IMA FE In a further step S1.4, the network ANN1 calculates the correlation tensor c k Compute the binary segmentation mask SEGM, the correlation tensor c k By the object descriptor tensor o k And the feature map f of image IMA k A comparison of the image features of the representation is produced.

[0014] In another step S1.4, the feature map f of the image IMA is calculated k and from the object descriptor tensor o k The per-pixel correlation between the features of the image IMA is k Each pixel in is associated with the entire object descriptor o k matches - this produces the correlation tensor c k , where the specific correlation tensor value for a specific pixel (h, w) of the image IMA is defined as wherein corr denotes a correlation function, preferably, but not necessarily, according to Pearson correlation.

[0015] The third stage S3 of correspondence determination comprises at least steps S3.1 and S3.2, performed by the trained network ANN3. In a first step S3.1, a 2D-2D correspondence 2D2D between the representation repTEMPL(J) of the identified template TEMPL(J) and the representation repIMA of the image IMA is calculated. In a second step S3.2, the 2D-2D correspondence 2D2D calculated in the first step S3.1 is further processed taking into account a known virtual viewpoint vVIEW(J) to provide a 2D-3D correspondence 2D3D between pixels in the representation repIMA of the image IMA belonging to the object OBJ and voxels of the 3D model MOD of the object OBJ.

[0016] This exploits the fact that due to the known virtual camera pose vVIEW(i) corresponding to the template TEMPL(i), the object OBJ pose is available for all templates TEMPL(i), including the identified template TEMPL(J). Therefore, for each pixel on the template TEMPL(i), it is known to which 3D point or "voxel" of the model MOD it corresponds. Therefore, for all templates TEMPL(i) from the beginning, the 2D-3D correspondence between the template TEMPL(i) and the model MOD is known from the beginning, as long as the template TEMPL(i) has been generated for the corresponding virtual viewpoint vVIEW(i). Therefore, when the 2D-2D correspondence 2D2D between repIMA and the template TEMPL(J) is known according to step S3.1, the corresponding 2D-3D correspondence 2D3D between the image representation repIMA and the model MOD of the object can be easily derived.

[0017] In a first step S3.1, in order to calculate the 2D-2D correspondence 2D2D, the representation repIMA of the image IMA and the representation repTEMPL(J) of the identified template TEMPL(J) are correlated and based on the correlation result the 2D-2D correspondence 2D2D is calculated by the trained artificial neural network ANN3.

[0018] Therein, in the first step S3.1, in order to calculate the 2D-2D correspondence 2D2D, the representation repIMA of the image IMA is an image feature map repIMAfm of a segment of the image IMA including at least all pixels belonging to the object OBJ, and the representation repTEMPL(J) of the identified template TEMPL(J) is the template feature map repTEMPL(J)fm of the identified template TEMPL(J).

[0019] Furthermore, in a first step S3.1 for the calculation of the 2D-2D correspondence 2D2D, a 2D-2D correlation tensor c is calculated by matching each pixel of one of the feature maps repIMAfm, TEMPL(J)fm with all pixels of the corresponding other feature map TEMPL(J)fm, repIMAfm. k , the correlation tensor c k is a measure of the similarity between repIMA and TEMPL(J), i.e., each pixel of the image feature map repIMAfm matches all pixels of the template feature map TEMPL(J)fm. The 2D-2D correlation tensor c is then further processed by the trained artificial neural network ANN3. k , to determine the 2D-2D correspondence relationship 2D2D.

[0020] The second stage S2 of template matching for identifying the template TEMPL(J) that best matches the image IMA from a plurality of templates TEMPL(i) comprises at least steps S2.1, S2.2 and S2.3 performed by the network ANN2. In a first step S2.1, for each template TEMPL(i), at least for the template foreground segment of the corresponding template TEMPL(i) containing pixels belonging to the model MOD, or for the entire template TEMPL(i), a template feature map TEMPL(i)fm is calculated. In a second step S2.2, a feature map IMAfm is calculated at least for the image foreground segment of the image IMA containing pixels belonging to the object OBJ, or for the entire image IMA. Steps S2.1 and S2.2 may be performed in any desired order. In the third step S2.3, for each template TEMPL(i) and each i, the similarity sim(i) is calculated for the corresponding template feature map TEMPL(i)fm and the image feature map IMAfm, where the template TEMPL(J) with the highest similarity sim(j) determined in the third step S2.3 is selected as the identified template TEMPL(J).

[0021] Therein, an image foreground segment is cropped from the image IMA by utilizing the segmentation mask SEGM generated earlier. The template foreground segment of the template TEMPL(i) is known in nature, since the template TEMPL(i) depicts the object OBJ in a known pose anyway. Therefore, for each template TEMPL(i), the ground truth segmentation mask can be used to crop the template foreground segment from the template.

[0022] A corresponding system for estimating a multi-dimensional pose mDOF of an object OBJ based on a representation IMA of the object OBJ, comprising a computer-implemented control system, is configured to perform the above-described method.

[0023] The system comprises a sensor for capturing a representation IMA of the object OBJ, wherein the sensor is connected to the control system for providing the representation IMA to the control system for further processing.

[0024] Preferably, the sensor is a camera for capturing an optical image, preferably an RGB image.

[0025] In summary, the proposed method introduces a multi-stage single-shot object detector and a multi-dimensional pose estimation framework that does not require training on the target object, but only requires images and models of the object and thus generalizes to new unseen objects. The proposed method uses an input image IMA of an object OBJ and a 3D texture model MOD of the object OBJ as input at test time. The core aspect is to utilize multiple 2D templates TEMPL(i) rendered, for example in the form of RGB images, from different viewpoints to represent the 3D model MOD, for example on a sphere surrounding the model MOD of the object OBJ, where for example the model MOD is in the center of the sphere. This enables CNN-based features to be extracted directly from the image IMA and jointly and densely matched with the descriptors of all templates TEMPL(i). Then, an approximate object orientation is estimated via template matching. Subsequently, a dense 2D-2D correspondence is established between the detected object OBJ and the matching template of the previous step, for which the virtual viewpoint and the pose therewith are known. This allows to establish dense 2D-3D correspondences between pixels of the image IMA of the object OBJ and the corresponding object model MOD.The final pose can be computed based on the 2D-3D correspondences, for example, by a PnP+RANSAC algorithm (described below).

[0026] Advantageously, our proposed method is a scalable and fast single-shot multi-dimensional pose estimation method that generalizes fully to new unseen objects and scenes and does not require real data for training.

[0027] The proposed multi-stage object pose estimation method can be applied in, for example, robotic manipulation, augmented reality, visual servoing, etc. to help machines or intelligent mechanisms better understand 3D environments. Multi-dimensional object pose estimation is one of the core capabilities that support machines operating in the real world. It is a bridge for these applications from 2D images to the 3D world. By accurately locating objects and estimating their poses, it can improve action efficiency and mission success rate.

[0028] The proposed method has key cost and time advantages, namely, no complex and time-consuming data annotation steps are required, and there is no need to retrain the network for each specific object based on deep neural networks. These features make the method highly relevant to various applications in augmented reality, robotics, robotic grasping, etc., allowing those methods to operate in more dynamic scenarios involving various objects.

[0029] Many more usage scenarios of the solution proposed herein are conceivable and the specifically presented examples must not be understood as limiting the scope of the invention.

[0030] It will be understood that the elements and features recited in the appended claims may be combined in different ways to create new claims that also fall within the scope of the invention. Thus, where the dependent claims appended below are only dependent upon a single independent or dependent claim, it will be understood that these dependent claims may alternatively be made dependent upon alternatives from any preceding or following claim, whether independent or dependent, and such new combinations will be understood to form part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In the following, possible embodiments of different aspects of the present invention are described in more detail with reference to the accompanying drawings. The purpose of the present embodiment and further advantages will become clearer and easier to understand from the following description of the preferred embodiment considered in conjunction with the accompanying drawings, in which:

[0032] Figure 1 An image IMA of a real-world scene SCN with several objects, including an object of interest OBJ, is shown.

[0033] Figure 2 shows the object OBJ,

[0034] Figure 3 A system for pose estimation is shown,

[0035] Figure 4 shows a scene with a known virtual camera viewpoint used to generate a template TEMPL(i),

[0036] Figure 5 Flowchart showing the method MET for attitude estimation,

[0037] Figure 6 A flow chart showing the first stage S1 of the method MET,

[0038] Figure 7 Flow chart showing the second stage S2 of the method MET,

[0039] Figure 8A flow chart showing a third stage S3 and a fourth stage S4 of the method MET is shown. DETAILED DESCRIPTION

[0040] Figure 1 An image IMA of an exemplary real-world scene SCN with multiple objects is shown, including an object of interest OBJ for which a multi-dimensional pose is to be determined. For clarity of what the object of interest is in this exemplary case, Figure 2 An object OBJ is also depicted.

[0041] Figure 3 A system 100 is shown that includes a sensor 110 for capturing a representation IMA of an object OBJ. For example, the sensor 110 may be a camera, and the representation IMA may be an image, such as a two-dimensional RGB image. In addition, the system 100 includes a control system 130, for example implemented on and executed by a computer 120 of the system 100 by means of corresponding software. The camera 110 is connected to the computer 120 so that the captured image IMA can be transmitted to the computer 120 for further processing by the control system 130 as described below. Such a connection may be a wired or wireless connection. The image IMA transmitted to the computer 120 may be stored in a memory 140 of the computer 120 so that the control system 130 can access the image IMA from the memory 140.

[0042] For example, Figure 1 The scene SCN of the scene shown in can be imaged by a camera 110 so as to produce an RGB image IMA, which will be further processed by a control system 130 as described below to estimate the multi-dimensional pose of an object OBJ in the scene SCN. The camera 110 is arranged and configured to capture the image IMA so that it depicts at least the object of interest OBJ, and typically but not necessarily a portion of its environment, which may include additional objects. As indicated, Figure 1 The presentation in may actually correspond to the appearance of such an RGB image IMA captured by camera 110.

[0043] The captured images IMA of the object OBJ are respectively basis and input of a method MET for estimating the multidimensional pose mDOF of the object OBJ. Furthermore, the method MET utilizes as input data a three-dimensional model MOD of the object OBJ, for example a CAD model, which may be stored in and provided by the memory 140 .

[0044] For the following, the 3D model MOD of the object OBJ is represented by a large number of 2D query templates TEMPL(i), where i=1, ..., NT and NT represents the number of such templates, for example NT=100. Therefore, the idea is to describe the 3D model MOD of the object OBJ by multiple viewpoint-based templates TEMPL(i), which are obtained by rendering the object model MOD at different virtual rotations and viewing angles, respectively. This makes the problem closer to the standard 2D single-shot method.

[0045] The individual templates TEMPL(i) are obtained by rendering the model MOD from various known virtual camera viewpoints vVIEW(i) placed on a sphere SPH surrounding the model MOD of the object OBJ, for example where the model MOD is at the center of the sphere SPH. Such rendering is known per se and will not be described in detail in this document. Figure 4 It is shown as an example in FIG. Figure 4 Each point in represents a known virtual camera viewpoint vVIEW(i) for which and from which a corresponding template TEMPL(i) is rendered on the model MOD. Preferably, the points and virtual camera viewpoints are evenly distributed on the surface of the sphere SPH, so that the resulting total template TEMPL(i) represents the view on the model MOD from each viewpoint without a preferred viewing direction.

[0046] The control system 130 is configured to perform the method MET as described hereinafter. Figure 5 A high-level flow chart of the method MET is shown, which is essentially a multi-stage method comprising four stages S1-S4. The detailed processes in the individual stages S1-S4 will be described in conjunction with the subsequent figures. However, it may already be mentioned here that the method MET is exemplarily implemented such that each stage of the method MET applies a separate artificial neural network ANN, in the first stage S1 for binary segmentation of the image IMA, in the second stage S2 for template matching and initial viewpoint estimation, and in the third stage S3 for 2D-2D matching.

[0047] As indicated, the input to the method MET and to the first stage S1 are respectively the RGB image IMA and the 3D model MOD of the object OBJ, wherein the 3D model MOD is actually represented by a plurality of templates TEMPL(i).

[0048] In the first “object detection” stage S1, object segmentation conditioned on the 3D model MOD is performed, or in other words, the object OBJ represented by its textured 3D model MOD in the RGB image is detected. To this end, the features of the image IMA are compared with the object descriptor o kDense matching to perform semantic segmentation, such as combining Figure 6 The object descriptor o k is represented by a sparse set of templates TEMPL(i). The output of the first stage S1 is a binary segmentation mask SEGM showing the locations of the visible parts of the object OBJ in the input image IMA. Accordingly, the segmentation mask has the same size as the image IMA.

[0049] Semantic segmentation is preferably single-shot semantic segmentation: "Single-shot" detectors are a family of 2D object detection methods that generalize to new unseen objects. Such single-shot detectors use two disjoint sets of object classes for training and testing. This eliminates the need for manual annotation of training data for the classes used during testing, and the need to retrain the detectors on them. During inference, they detect new objects represented by a single template in the input target image. These methods typically use a "Siamese" network to match pre-computed features from the template to the target image.

[0050] After the first stage S1, the object OBJ is located in the image IMA, but its pose is still unknown.

[0051] The binary segmentation mask SEGM generated in the first stage S1 of the method MET is used as input to the second stage S2 of the method MET. Additionally, the second stage S2 again uses the template TEMPL(i) and the image IMA as input data.

[0052] In the second "template matching" stage S2, an initial rotation estimation is performed via a template matching method on a dense set of templates TEMPL(i) covering the entire virtual pose space constrained to the sphere SPH. Thereafter, the second stage S2 selects one template TEMPL(J) with the highest similarity to the image IMA from the multiple templates TEMPL(i), where 1≤J≤NT.

[0053] The template TEMPL(J) selected in the second stage S2 is further processed in a third "correspondence determination" stage S3 of the method MET together with the image IMA as additional input. The second stage S2 consists in determining dense 2D-3D correspondences 2D3D between the image IMA and the 3D object model MOD by matching 2D-2D correspondences of the image IMA with the selected template TEMPL(J), which represents the object OBJ in a known pose. The output of the third stage S3 is therefore a 2D-3D correspondence 2D3D.

[0054] In the fourth "pose estimation" stage S4, a multidimensional pose mDOF, preferably a 6D pose, is estimated based on the 2D-3D correspondence 2D3D determined in the third stage S3. This estimation can apply a known pose estimation process, like "PnP" (Vincent Lepetit, Francesc Moreno Noguer, Pascal Fua: "Epnp: An accurate o(n) solution to the pnpproblem"; IJCV; 2009) combined with "RANSAC" (Martin A Fischler, Robert C Bolles: "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography"; Communications of the ACM; 1981) or Kabsch (Paul J Besl, Neil D McKay: "Method for registration of 3-d shapes"; Sensor Fusion IV: Control Paradigms and Data Structures: 1992). In other words, a "PnP+RANSAC" process is applied to estimate the multidimensional pose mDOF from the 2D-3D correspondence 2D3D.

[0055] In the following, the feature maps computed for various input datasets are presented and further processed. The overall approach for feature extraction to generate feature maps foresees two feature extractors: and They are actually implemented by the same network ANN1. Assume that “I” denotes an image of size W×H, such as image IMA or template TEMPL(i), which implicitly depends on the object model MOD and its pose T∈SE(3). The first feature extractor Use the pre-trained network ANN1 to calculate the depth dimension D based on the image I k The feature map of For extracting features from such an image in all stages of the method MET. In practice, precomputed feature maps from several depth levels of the network ANN 1 can be used, the precomputed feature maps being indexed by k∈N (1≤k≤N); where N is the number of convolutional layers of the network ANN1. In other words, assuming that the network ANN1 is a convolutional neural network (CNN) with several layers, each layer k or at least more than one layer produces its own feature map. Due to the larger field of view, the later layers produce more global features, but have a lower spatial resolution. The feature maps from the earlier layers have a higher spatial resolution but a smaller field of view. represents a feature extractor that is expanded by spatially averaging the height and width along each depth dimension of the feature map to produce a single vector of length D k Specifically, in the first stage S1 of the method MET, is used to extract features from the template TEMPL(i), and in the second S2 and third stages S3, Used to extract features from template TEMPL(i). For image IMA, it is applied in all stages of feature extraction

[0056] Back to the first stage S1, Figure 6 A flow chart of the process under the first stage S1 is shown.As mentioned, the first stage S1 of the proposed method MET is responsible for single-shot object localization of the object OBJ in the image IMA via binary semantic segmentation and generating a binary segmentation map SEGM.

[0057] In the first step S1.1 of the first stage S1, the 4D single descriptor tensor o is calculated from the pre-rendered template TEMPL(i) by the trained network ANN1. k , so that the input image IMA can then be matched with all templates TEMPL(i) in a single shot, in contrast to standard single-shot object detection where each query template is processed independently. 4D descriptor tensor All templates TEMPL(i) rendered from a virtual viewpoint vVIEW(i) on a sphere SPH surrounding the object model MOD are collected and represented respectively, where the first two dimensions (X, Y) represent the virtual camera position relative to the object coordinate system, such as polar coordinates, and the third dimension Z represents the in-plane rotation. The fourth dimension D k Descriptor tensor o refers to the feature map extracted from each template TEMPL(i) at multiple depth levels k=1,...,K (e.g. K=3) of the neural network ANN1. k Each element of is a viewpoint template TEMPL(i), which is represented by the corresponding feature map of the corresponding query template TEMPL(i). It is defined as where I(R(x, y, z) MOD) represents the template TEMPL(i), and R is the rotation matrix representing the virtual viewpoint on the sphere SPH surrounding the model MOD of the object OBJ, for example, where the model M is at the center of the sphere. In practice, when combining the features from each template TEMPL(i) to generate the tensor o k Previously, using feature extractors The features are precomputed.

[0058] In the second step S1.2 of the first stage S1, which can be performed before, after or in parallel with the first step S1.1, the network ANN1 calculates a feature map for the image IMA Where k=1,...

[0059] In the third step S1.3 of the first phase, the network ANN1 receives the descriptor tensor o k And the feature map f of the previous steps S1.1 and S1.2 k As input, and based on the feature map of image IMA and the 4D descriptor tensor of the model MOD of the object OBJ Comparison of image features expressed to predict the correlation tensor c k This builds on the idea of ​​dense feature matching and further extends it to the task of object detection.

[0060] The core principle of the comparison in step S1.3 is to calculate the feature map f of the image IMA k and from the object descriptor o k The per-pixel correlation between the features of the input image IMA. k Each pixel in is associated with the entire object descriptor o k This results in the correlation tensor For a particular pixel (h, w) of the input image IMA, the correlation tensor value is defined as For example, "corr" might indicate the "Pearson" correlation introduced by Karl Pearson.

[0061] In step S1.4, the correlation tensor c is further processed k To finally generate the segmentation mask SEGM. First, the correlation tensor c k is flattened into a 3D tensor In this way, the input image IMA feature map f k Each pixel gets its feature vector and descriptor o k Afterwards, a standard convolutional layer further processes the 3D tensor.

[0062] The flattened correlation tensor c can be used in two ways: k′ First, flatten the correlation tensor c k The calculated correlation of ′ is directly used as input to the network ANN1 to produce the binary segmentation mask SEGM. To this end, the flattened tensor c is further processed by a 1×1 convolutional layer. k′ , to change the dimension from (X k Y k Z k ) is reduced to Lk The resulting tensor c k″ It is directly used as the input feature to the subsequent convolutional layer of the network ANN1. Since the template TEMPL(i) is in the tensor o k are always ordered in the same way, so the network ANN1 can learn from the computed correlations during training and then generalize to unseen objects at test time.

[0063] The second use of correlation is to calculate the feature map f of the image IMA k The pixel-wise attention allows the original image IMA features to be effectively incorporated into the stacked tensor (described below) and c k″ The feature tensor created by CNNs can be used for more accurate segmentation.

[0064] The raw pixel-wise attention at feature map level k is simply defined as all (X k Y k Z k ) is the sum of the correlations, as

[0065] Since simple attention can be very noisy in early layers compared to later ones, we propose to condition the attention on the attention from the last level, which tends to be more precise but has lower resolution, and threshold the attention value to be below the average:

[0066]

[0067] in, From the last level To A k Bilinear upsampling of size attention. It is thresholded but not subject to any condition. All values ​​are scaled to be between 0 and 1. With the attention map, the feature map f of the image IMA k is transformed as follows:

[0068]

[0069] Subsequently, the features of interest are processed by a 1×1 convolutional layer of ANN1 to reduce the dimensionality. and c kare used jointly by subsequent layers. Overall, this segment of the first stage is similar to the "UNet" approach (Olaf Ronneberger, Philipp Fischer, Thomas Brox: "Unet: Convolutional networks for biomedical image segmentation"; In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015) where the feature map is upsampled followed by a convolutional layer until the initial image size is reached. The core difference from this approach is that the network ANN1 uses stacked and c k Instead of skip connections.

[0070] Finally, the trained network ANN1 predicts the per-pixel probability that a pixel of the image IMA contains a visible part of the object OBJ, for example, embodied as a binary segmentation mask SEGM. For example, a per-pixel "Dice" loss can be used to train the network and handle unbalanced class distributions. The determination of the segmentation mask SEGM is equivalent to the detection of the object OBJ in the image IMA.

[0071] Therefore, in training time, the network ANN1 is trained by maximizing the predicted probability of pixels with object OBJ and minimizing the probability of pixels without object OBJ. In test time, pixels with a predicted probability greater than 0.5, for example, are considered to be visible parts of object OBJ.

[0072] Back to the second stage S2, Figure 7 A flow chart of the process under the second stage S2 is shown. The second stage S2 applies the second network ANN2 and receives and processes the predicted segmentation mask SEGM, the image IMA and the template TEMPL(i) of the first stage S1 as input data to select one template TEMPL(J) that best matches the image IMA from multiple templates TEMPL(i). The selected template TEMPL(J) and the corresponding known virtual camera position and viewpoint represent the initial viewpoint estimation, respectively.

[0073] For this initial viewpoint estimation, template matching via “manifold learning” can be applied by ANN2, which scales well to a large number of objects and generalizes to unseen objects. For this task, the same feature extractor F as in the first stage S1 is applied FE However, only features from the last layer, i.e., k = K, are adopted. In addition, a 1×1 convolutional layer can be added to reduce the dimension from H×W×D to H×W×D^′.

[0074] In step S2.1 of the second stage S2, for each template TEMPL(i), ANN2 calculates a feature map from the foreground segment in the corresponding template TEMPL(i). The foreground segment represents the object OBJ in the template TEMPL(i).

[0075] In step 2.2 of the second stage S2, which may be performed before, after or in parallel with the first step S2.1, the feature map Such foreground segments are computed by the ANN2 using the segmentation mask SEGM predicted in the first stage S1 from the foreground segments in the image IMA representing the object OBJ to identify such foreground segments.

[0076] In each case, a "foreground segment" of an image - for example a template TEMPL(i) in step S2.1 and an image IMA in step S2.2 - identifies a segment of the respective image containing pixels belonging to the object of interest OBJ. Such a foreground segment can be found by applying a corresponding segmentation mask. With respect to the image IMA, a segmentation mask SEGM has been generated in a first stage S1. In any case the template TEMPL(i) depicts the object OBJ in a known pose. Therefore, for each template TEMPL(i), a ground truth segmentation mask is available. In the following, such "foreground segments" are sometimes denoted "patches".

[0077] Similar to the first stage S1, in step S2.3 of the second stage S2, the similarity of two patches is estimated by ANN2 by computing the per-pixel correlation between the feature map f of the foreground segment of image IMA and the feature map t(i) of the foreground segment of the corresponding template TEMPL(i), respectively, for each i and each template TEMPL(i), using

[0078] sim(i)=sim(f,t(i))=∑ h，w corr(f h，w , t(i) h，w ).

[0079] As mentioned above, "corr" may denote the Pearson correlation. Where f and t(i) are tensors, the correlation is compared individually for each "pixel" (h, w) and then summed.

[0080] The template TEMPL(J) corresponding to the template feature map t(J) for which the best correlation, ie the highest similarity to the detected object OBJ in the image IMA, is selected as the match to be further processed in the third stage S3.

[0081] As mentioned earlier, the virtual camera viewpoint and pose for each template TEMPL(i) are known, so that the matched, selected template TEMPL(J) or at least the index or identifier ID(J) of this template TEMPL(J) gives an initial rotation estimate of the detected object and allows conclusions about the pose of the object OBJ. In any case, the output of the second stage S2 is a representation rTEMPL(J) of the selected template TEMPL(J), such as the selected template TEMPL(J) itself, the identifier ID(J) of the selected template TEMPL(J), or a virtual viewpoint to the corresponding virtual camera pose that generated TEMPL(J).

[0082] The training of the network ANN2 aims to increase the similarity of patches depicting objects with very similar rotations, and at the same time penalize the similarity for distant rotations. The specific query template TEMPL(J) with the highest similarity to the detected object in the target image IMA is selected as the best match at test time. A modified triplet loss with dynamic margin is exploited by optimizing

[0083]

[0084] where m is set to the angle between the rotation of the object in the anchor and the puller patch. Using terminology from Sergey Zakharov, Wadim Kehl, Benjamin Planche, Andreas Hutter, Slobodan Ilic, “3D object instance recognition and pose estimation using triplet loss with dynamic margin” in IRO S 2017, f anchor is the descriptor of a randomly selected object patch. + corresponds to the puller - a template whose pose is very similar to that in the anchor, and f - Corresponds to a pusher with a dissimilar pose.

[0085] During the training of ANN2, the rotation matrix is ​​transformed from the egocentric coordinate system to the non-egocentric coordinate system. This transformation ensures that the visual appearance of the object depends exclusively on the rotation component of the SE (3) pose.

[0086] Back to the third stage S3, Figure 8A flow chart of the process under the third stage S3 is shown. The third stage S3 applies a third network ANN3 to process input data including the representation rTEMPL(J) and the image IMA from the second stage S2 to generate 2D-2D correspondences 2D2D of TEMPL(J) and IMA, and to generate 2D-3D correspondences 2D3D from those 2D-2D correspondences 2D2D.

[0087] Therefore, the goal of the third stage S3 is to provide the necessary data to enable multi-dimensional pose estimation in the fourth stage S4, which can be based on 2D-3D correspondences 2D3D.

[0088] In more detail, after the initial viewpoint estimation of the second stage S2 leading to the identification of the matched, selected template TEMPL(J), the image IMA with the detected object OBJ of unknown pose and the matching template TEMPL(J) of known pose are available. The trained network ANN3 calculates S3 dense 2D-2D correspondences 2D2D between the image and its representation repIMA and the selected template TEMPL(J), respectively, in the first step S3.1 of the third stage. Therein, the representation repIMA of the image IMA can be the image IMA itself, or preferably a segment of the corresponding image IMA containing pixels belonging to the object OBJ, i.e. a "patch" in the wording introduced above. By utilizing the segmentation mask SEGM, patches and representation repIMA can be generated respectively from the entire image IMA.

[0089] A 2D-2D correspondence 2D2D is calculated based on the input repIMA and TEMPL(J). Preferably, this essentially means that the corresponding feature maps repIMAfm, TEMPL(J)fm of repIMA and TEMPL(J) are calculated and correlated as described above, and the network ANN3 predicts the 2D-2D correspondence 2D2D of repIMA and TEMPL(J) based on the correlation of those feature maps repIMAfm, TEMPL(J)fm.

[0090] The architecture of 2D-2D matching for determining 2D-2D correspondence 2D2D follows the general idea of ​​dense feature matching. The feature map f of the image representation repIMA k = The template feature map t of each pixel of repIMAfm and the selected template TEMPL(J) k =TEMPL(J)fm matches all pixels to form a 2D-2D correlation tensor c k . 2D-2D correlation tensor c kIt does not yet represent the 2D-2D correspondence 2D2D, but it is a measure of the similarity between repIMA and TEMPL(J). Then, it is trained to get the 2D-2D correlation tensor c k The network ANN3 predicts the 2D-2D correspondences and computes the 2D-2D correspondences 2D2D. More specifically, the network ANN3 processes the 2D-2D correlation tensor c k Three values ​​for each pixel of the predicted image repIMA, i.e., the binary foreground / background segmentation mask, are required to show which points of the object are visible in the template TEMPL(J) and the 2D coordinates of the corresponding pixels on the template TEMPL(J), or in other words, the probability that a specific pixel is visible in the template TEMPL(J) and the two coordinate values ​​of the corresponding point on the template TEMPL(J).

[0091] Based on the 2D-2D correspondence 2D2D, the network ANN3 can provide a 2D-3D correspondence 2D3D between the object pixels in repIMA and the corresponding voxels of the 3D model MOD of the object OBJ in the second step S3.2 of the third stage S3. This takes advantage of the fact that due to the known virtual camera pose vVIEW(i) corresponding to the template TEMPL(i), the object OBJ pose is available for all templates TEMPL(i) - including the selected template TEMPL(J). Therefore, for each pixel on the template TEMPL(i), it is known which 3D point or "voxel" on the model MOD it corresponds to. Therefore, for all templates TEMPL(i), the 2D-3D correspondence between the template TEMPL(i) and the model MOD is known. Therefore, when the 2D-2D correspondence 2D2D between repIMA and the template TEMPL(J) is known, the corresponding 2D-3D correspondence 2D3D between the image representation repIMA and the model MOD of the object can be easily derived.

[0092] The network ANN3 is trained in a fully supervised manner. During training, random object crop images I are sampled from the synthetic dataset obj and its associated posture T obj ∈SE(3). Then, the random template I tmp With its posture T tmp ∈SE(3) are picked together, so that T obj and T tmp Relatively close. The availability of the object pose allows the computation of a per-pixel 2D-3D correspondence map of the two patches. Let C: Marked at a given posture T obj ∈SE(3) . The correspondence is computed as “normalized object coordinates” (NOCS). Its inverse C -1Recalculate the correspondences with respect to the unnormalized object coordinates that correspond to the actual 3D object coordinates. It allows to define 2D correspondences to distances in the 3D coordinate space of the model:

[0093] d(p, p′)=||C -1 (I obj ) p -C -1 (I tmp ) p′ ‖2

[0094] where p and p′ are the pixel coordinates in the image and template patch, respectively. The ground truth dense 2D-2D correspondence is established by simply matching the pixel pairs corresponding to the closest points in the model’s 3D coordinate system. More formally, for a point p∈I obj , whose corresponding template point is calculated as

[0095]

[0096] Due to the random sampling, outlier-aware rejection is adopted for 2D-2D correspondences with large spatial variance.

[0097] Similar to single-shot localization, the segmentation loss is defined as a per-pixel “Dice” loss. On top of that, the network predicts discrete 2D coordinates using a standard cross-entropy classification loss, since dense correspondence predictions tend to work better if correspondence estimation is posed as a classification problem.

[0098] Back to the fourth stage S4, Figure 8 A flow chart of the process under the fourth stage S4 is also shown. The fourth stage S4 applies a fourth network ANN4 to process the 2D-3D correspondence 2D3D generated in the third stage S3 to estimate the multidimensional pose mDOF of the object OBJ, such as six-dimensional 6DOF, in step S4.1 of the fourth stage S4. To this end, the network ANN4 applies known techniques such as PnP+RANSAC.

[0099] Specifically, the template TEMPL(i) can be rendered at 25 frames per second, with a resolution of 128x128 and the model downsampled to 5K faces on a standard NVIDIA Geforce RTX 2080TI. Using standard equipment, it takes about an hour to render the 9OK template for the second stage S2 of the network. However, additional ablation studies show that using a 5K template instead of 90K leads to similar performance. It takes 200 seconds to render the 5K template. This process only needs to be done once for each new object.

[0100] Although the present invention has been described above with reference to various embodiments, it will be appreciated that many changes and modifications may be made to the described embodiments. Therefore, the foregoing description is intended to be considered illustrative rather than restrictive, and it will be appreciated that all equivalents and / or combinations of the embodiments are intended to be included in this description. Therefore, the present invention is not limited to the embodiments described above, but variations may be derived by those skilled in the art without departing from the scope of the invention.

Claims

1. A method for estimating the multi-dimensional pose mDOF of an object OBJ based on an image IMA of the object OBJ, wherein - in a preparation phase, an image IMA depicting an object OBJ and a plurality of templates TEMPL(i) are provided, wherein the templates TEMPL(i) are generated from a 3D model MOD of the object OBJ during a rendering process, wherein different templates TEMPL(i), TEMPL(j) of the plurality of templates are generated by rendering from different known virtual viewpoints vVIEW(i), vVIEW(j) on the model MOD, where i≠j, - in a template matching phase S2, identifying from said plurality of templates TEMPL(i) at least one template TEMPL(J) which best matches the image IMA, Features - in a correspondence determination phase S3, the representation repTEMPL(J) of the identified template TEMPL(J) is compared with the representation repIMA of the image IMA to determine a 2D-3D correspondence 2D3D between pixels in the image IMA and voxels of the 3D model MOD of the object OBJ, and - A pose estimation stage S4, in which a multi-dimensional pose mDOF is estimated based on the 2D-3D correspondences 2D3D.

2. The method according to claim 1, wherein: In a stage S1 of object detection, which is preferably performed before a second stage S2 of template matching, a segmentation mask SEGM is generated from the image IMA identifying those pixels of the image IMA that belong to the object OBJ.

3. The method according to claim 2, wherein: To generate the segmentation mask SEGM, the features of the image IMA are combined with the object descriptor tensor o representing the multiple templates TEMPL(i) k Dense matching is performed to perform semantic segmentation.

4. The method according to any one of claims 2 to 3, wherein: In the object detection stage S1 - In a first step S1.1, using the feature extractor from the template TEMPL(i) Calculate the object descriptor tensor of the model MOD of the object OBJ The object descriptor tensor o k represents all templates TEMPL(i), - In a second step S1.2, using the feature extractor F of the image IMA FE To calculate the feature map f k =F FE (IMA), - In a further step S1.4, based on the correlation tensor c k Compute the binary segmentation mask SEGM, the correlation tensor c k By the object descriptor tensor o k And the feature map f of image IMA k A comparison of the image features of the representation is produced.

5. The method according to claim 4, wherein: In a further step S1.4, the feature map f of the image IMA is calculated k and from the object descriptor tensor o k The per-pixel correlation between the features of the image IMA is k Each pixel in is associated with an object descriptor o k Matching, which produces the correlation tensor c k , where the specific correlation tensor value for a specific pixel (h, w) of the image IMA is defined as where corr denotes a correlation function, preferably according to the Pearson correlation.

6. The method according to any one of claims 1 to 5, wherein: The pose estimation stage S4 applies a PnP+RANSAC process to estimate the multi-dimensional pose mDOF from the 2D-3D correspondences 2D3D.

7. The method according to any one of claims 1 to 6, wherein: In the phase S3 where the correspondence is determined - in a first step S3.1, a 2D-2D correspondence 2D2D between the representation repTEMPL(J) of the identified template TEMPL(J) and the representation repIMA of the image IMA is calculated by the trained network ANN3, In a second step S3.2, the 2D-2D correspondence 2D2D is further processed to provide a 2D-3D correspondence 2D3D between pixels in the image IMA belonging to the object OBJ and voxels of the 3D model MOD of the object OBJ, taking into account a known virtual viewpoint vVIEW(J).

8. The method according to claim 7, wherein: In a first step S3.1 for the calculation of the 2D-2D correspondence 2D2D, the representation repIMA of the image IMA and the representation repTEMPL(J) of the identified template TEMPL(J) are correlated and based on the correlation result the 2D-2D correspondence 2D2D is calculated.

9. The method according to any one of claims 7 to 8, wherein: In the first step S3.1 for calculating the 2D-2D correspondence 2D2D the representation repIMA of the image IMA is an image feature map repIMAfm of a segment of the image IMA comprising at least pixels belonging to the object OBJ, and -The representation repTEMPL(J) of the identified template TEMPL(J) is the template feature map repTEMPL(J)fm of the identified template TEMPL(J).

10. The method according to claim 9, wherein: In a first step S3.1 for calculation of the 2D-2D correspondence 2D2D, - Compute a 2D-2D correlation tensor c by matching each pixel of one of the feature maps repIMAfm, TEMPL(J)fm with all pixels of the corresponding other feature map TEMPL(J)fm, repIMAfm k , - Further processing of the 2D-2D correlation tensor c k To determine the 2D-2D correspondence relationship 2D2D.

11. The method according to any one of claims 1 to 10, wherein: In a template matching stage S2 for identifying a template TEMPL(J) that best matches the image IMA from among the plurality of templates TEMPL(i), - In the first step S2.1 For each template TEMPL(i), a template feature map TEMPL(i)fm is calculated for at least the template foreground segment of the corresponding template TEMPL(i) containing pixels belonging to the model MOD, - In the second step S2.2 calculating a feature map IMAfm for at least an image foreground segment of the image IMA containing pixels belonging to the object OBJ, - In the third step S2.3 For each template TEMPL(i), the similarity sim(i) is calculated for the corresponding template feature map TEMPL(i)fm and image feature map IMAfm. The template TEMPL(J) for which the highest similarity sim(j) was determined in the third step S2.3 is selected as the identified template TEMPL(J).

12. The method according to claim 11, wherein: An image foreground region is cropped from the image IMA by using a segmentation mask SEGM generated by a method according to any one of claims 2 to 5 .

13. A system for estimating a multi-dimensional pose mDOF of an object OBJ based on a representation IMA of the object OBJ, comprising a control system (130) implemented by a computer (120) and configured to execute the method according to any one of claims 1 to 12.

14. The system according to claim 13, comprising a sensor (110) for capturing the representation IMA of the object OBJ, wherein the sensor (110) is connected to the control system (130) for providing the representation IMA to the control system (130) for further processing.

15. The system of claim 14, wherein: The sensor (110) is a camera for capturing optical images, preferably RGB images.

Images