Model-free six-dimensional object pose estimation

Abstract

A composite pose-estimation algorithm includes a video-object segmentation sub-algorithm (311) configured to determine a mask of a visual object in an image, and an object-pose tracking sub-algorithm (312) configured to track a pose of a visual object over multiple depth-video frames, wherein the pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the video-object segmentation sub-algorithm, which determines respective object masks to be used by the object-pose tracking sub-algorithm alongside the depth video. A method of tracking a pose of a physical object comprises: obtaining a depth video depicting a physical object in a plurality of poses from an input interface (330); forming a storable data item representing the physical object by applying the pose-estimation algorithm to the depth video; and tracking the physical object or a copy thereof using an instance of the pose-estimation algorithm which has been initialized by means of the storable data item.

Description

MODEL-FREE SIX-DIMENSIONAL OBJECT POSE ESTIMATIONTECHNICAL FIELD

[0001] The present disclosure relates to the field of computer vision and in particular to (six-dimensional) pose estimation or pose tracking of a physical object. A pose-tracking method disclosed herein is suitable for assisting kinesthetic robot programming.BACKGROUND

[0002] Emerging markets for robotic applications, such as laboratory automation and food preparation, are characterized by the presence of less structured environments and users with no prior programming experience. To provide the necessary programs for these novel applications, the demand for programming interfaces that are relatively easy to use and / or to start using is expected to increase. This includes programming interfaces adapted for demonstration-based teaching.

[0003] In many existing and novel robotic applications, robot manipulation is performed in relation to an object in the environment whose position might not be defined a priori and might change over time. Moreover, some tasks might require manipulating specific physical objects which have been designed for that task alone, and for which no prior knowledge can be drawn on. Therefore, the ease of use of the learning framework heavily relies on the availability of a flexible vision solution that allows users to intuitively and time-efficiently identify new objects.

[0004] Existing state-of-the-art 6D pose estimation algorithms, including algorithms that use an underlying deep neural network (DNN), can oftentimes be inadequate for this task, namely, since they would require onerous retraining when deployed to work with new objects; the retraining consumes significant computational resources or requires a new round of data collection, or both. For example, in implementations using the CosyPose algorithm (see Y. Labbe et al., “CosyPose: Consistent multi-view multi-object 6D pose estimation”, arXiv:2008.08465 [cs.CV], retrieved from arXiv.org), a new object is learned based on a mixture of real and synthetic annotated images, including CAD models of the new object. The requirements for a CAD model and the training costs translate into several days of setup and code execution if conducted by a non-expert user.

[0005] Further, the introduction of a new object could sometimes necessitate a readaptation of the 2D or 3D vision systems used, especially in an unstructured environment (e.g., handling of a tube rack inside a medical container, opening / closing lid of a tube sample centrifuge machine). This setup time contributes further to the overall cost of introducing a new object.

[0006] To offer an attractive robot teaching framework that meets the user community’s expectations of ease of use, it would be desirable to propose a vision solution that is capable of providing 6D pose estimation of new objects with a typical setup and learning time of no more than a few minutes.SUMMARY

[0007] One objective of the present disclosure is to make available a method and a computer system for tracking a pose of a physical object. A further objective is to enable single-shot pose estimation of a known physical object with no other preparation than analyzing a depth video of the object (wherein the depth video is possibly associated with an operator’s indication of an initial object mask).Preferably, the depth video is unlabeled in the sense that it can be successfully used without annotations or labels with correct poses to be learned. Further preferably, the operator is not required to provide a CAD model of the physical object; this maybe described as model-free pose estimation. A further objective is to separate (in time and / or in space) a pose-estimation algorithm’s analysis of the depth video and its use during pose tracking. Particularly, it would be desirable to store the relevant outputs of the depth-video analysis as a data item, which can be retrieved and provided to the pose-estimation algorithm at a later point in time, or transferred without delay to a different instance of the pose-estimation algorithm. A still further objective is to provide a pose-estimation algorithm which does not require any training of a neural network when a new physical object is to be tracked; to the extent the poseestimation algorithm includes a trainable neural network, the neural network is universally applicable to all physical objects (e.g., to all physical objects in an intended category of objects), whereas the pose-estimation algorithm is configured for a particular type of physical object by means of the data in the data item introduced above.

[0008] At least some of these objectives are achieved by the invention, as defined by the independent claims appended hereto. The dependent claims relate to currently preferred embodiments of the invention.

[0009] In a first aspect of the present disclosure, there is provided a method of tracking a pose of a physical object, which includes the use of a composite poseestimation algorithm. The composite pose-estimation algorithm includes- a video-object segmentation (VOS) sub-algorithm configured to determine a mask of a visual object in an image, and- an object-pose tracking (OPT) sub-algorithm configured to track a pose of a visual object over multiple depth-video frames.The pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm alongside the depth video. The method comprises recording a depth video depicting a physical object in a plurality of poses; forming a storable data item representing the physical object by applying the pose-estimation algorithm to the depth video; and tracking the physical object or a copy thereof using an instance of the pose-estimation algorithm which has been initialized by means of the storable data item.

[0010] The instance of the pose-estimation algorithm which has been initialized by means of the storable data item can coincide with the instance of the same algorithm that has previously been applied to the depth video. This means that a learning phase, in which the composite pose-estimation algorithm learns a new physical object, can be separated in time from a production phase, in which the composite pose-estimation algorithm estimates the pose of the familiar physical object or a copy thereof. Accordingly, the production phase can start more or less instantly, in as little time as it takes to load the data item into a working memory. There is normally no need to repeat the learning phase, or re-analyze the depth video. The initialization of the composite pose-estimation algorithm by means of the storable data item could also be used for backup purposes, e.g., to resume productive operation after one instance of the composite pose-estimation algorithm has stopped working.[oon] Alternatively, the instance of the pose-estimation algorithm which has been initialized by means of the storable data item is a different instance of the same algorithm than the one which has previously been applied to the depth video. This means that the outcome of the learning phase can be shared with a recipient in the form of a storable data item, such that the recipient can benefit from starting a production phase for a new physical more or less instantly, without having carried out a learning phase for that physical object.

[0012] In a second aspect of the present disclosure, there is provided a computer system configured to track a pose of a physical object. The computer system comprises first processing circuitry and second processing circuitry, each configured with a composite pose-estimation algorithm including the VOS sub-algorithm and the OPT sub-algorithm introduced above. The computer system further comprises an interface for inputting a depth video depicting a physical object in a plurality of poses and a memory. According to the second aspect, the first processing circuitry is further configured to apply the pose-estimation algorithm to the depth video, for thereby storing a data item representing the physical object in the memory. Moreover, the second processing circuitry is further configured to retrieve the data item from the memory and use it to initialize an instance of the pose-estimation algorithm.

[0013] The present disclosure further relates to a computer program containing instructions for causing a computer or computer system to carry out the above method.

[0014] The present disclosure further relates to storable data item obtained by applying the above-outlined pose-estimation algorithm to a depth video depicting a physical object in a plurality of poses. The data item can be used for initializing an instance of the pose-estimation algorithm such that it can begin estimating poses of the physical object without undergoing learning.

[0015] The computer program or the data item may be stored or distributed on a data carrier. As used herein, a “data carrier” may be a transitory data carrier, such as modulated electromagnetic or optical waves, or a non-transitory data carrier. Non- transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storage media of magnetic, optical or solid-state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable.

[0016] In the terminology of the present disclosure, a “depth video” is a sequence of images which include a depth component, such as a red-green-blue (RGB) depth (D) format.

[0017] A “mask” or “object mask” of a visual object in an image may include a vector-based or bitmap-based representation from which it is derivable what regions of the image correspond to the object. In a depth image (e.g., depth-video frame), the mask may be expressed in the plane of the image, that is, independently of the depth component. This to say, the mask generally does not specify the extent of the physical object in the depth direction.

[0018] A “pose” of a physical object may indicate an orientation of the physical object, e.g., in terms of a set of angles, such as yaw, pitch, roll. Optionally, a “pose” additionally includes a position of the physical object, so that the pose has a total of six degrees of freedom (6D pose).

[0019] In the terminology of the present disclosure, an “industrial robot” is a device with a physical manipulator, which is designed for or suitable for manufacturing, processing, material handling, destruction and similar tasks in an industrial context. The term industrial robot covers the full range from lightweight robots designed to replace human manual work (including the emerging market use cases mentioned initially), over collaborative robots for supporting a human worker, all the way up to heavy-duty robots.

[0020] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a / an / the element, apparatus, component, means, step, etc.” are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, on which: figure 1 is a flowchart of a method of tracking a pose of a physical object;figure 2 is a flowchart of a method of programming an industrial robot, which includes executing the method according to figure i; figure 3 shows a computer system comprising two sets of processing circuitry, each configured with a composite pose-estimation algorithm, and further comprising a shared memory; figure 4 shows a computer system with a single set of processing circuitry which is configured with a composite pose-estimation algorithm; figure 5 is a functional block diagram of data processing taking place during an execution of a composite pose-estimation algorithm, wherein preceding frame information, object segmentation data and memory data to be included in the storable data item have been indicated using frames of different graphical styles; figure 6 illustrates a process of extracting runtime data from an instance of a composite pose-estimation algorithm, which has been applied to a depth video of a physical object, and forming a storable data item representing the physical object, wherein the same graphical styles as in figure 5 have been used; figure 7 illustrates a process of loading runtime data from a data item into an instance of a composite pose-estimation algorithm, wherein the data item corresponds to a physical object and the algorithm thus initialized is to be used for pose tracking of the physical object or a copy thereof; and figure 8 shows a work area of an industrial robot, in which a vision system associated executes a composite pose-estimation algorithm according to the present disclosure is active, for thereby facilitating programming of the industrial robot.DETAILED DESCRIPTION

[0022] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, on which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.System overview

[0023] Figure 8 shows an industrial robot 800 made up of a robot manipulator 810 and a robot controller 820. The robot manipulator 810 and the robot controller 820 are joined by a wired or wireless bidirectional data connection, as suggested by the solid line below these units, which conveys control signals, sensor data etc.

[0024] The robot manipulator 810 includes an arm, which extends from a base 812 and is made up of structural elements 814 and at least one linear or rotary joint 813. The arm may further carry tools 811 that allow it to interact with various workpieces 830, which are present in a work area 850 of the robot manipulator 810. The workpieces are subject to manufacturing operations, processing or handling by the robot manipulator 810. The work area 850 may further include additional objects 840, such as containers, fixtures, separators, insulators, supports etc.; these objects 840 maybe generic or maybe specifically adapted to the workpieces 830 handled by the robot manipulator 810. The arm of the robot manipulator 810 is movable by action of internal motors, drives or actuators (not shown), and it includes transducers, sensors and other measuring equipment (not shown), from which the manipulator’s 810 current position, pose, technical condition, load etc. can be derived, to some degree of accuracy. The position, pose etc. of the robot manipulator 810 may in particular refer to a point on the arm, particularly to a tool-center point (TCP).

[0025] The positions, poses etc. maybe expressed with respect to one of multiple possible reference frames, including a fixed reference frame (O850) with its origin in a point in the work area 850, a fixed reference frame (not shown) with its origin in the base 812, a moving reference frame (O830a) with its origin in a workpiece or a moving reference frame (O811) with its origin at the TCP and oriented parallel to the tool 811 at all times.

[0026] The robot controller 820 comprises processing circuitry 821, a memory 822 and a communication interface 823. Example content of the memory 822 during operation may includeS: an operating system, basic settings, software implementing generic movements, sensing, self-monitoring, generically useful functionalities and services (all typically contributed by an original manufacturer), task- or role-specificconfigurations, configuration templates (typically contributed by a robot system integrator), and site-specific settings (typically contributed by an end user);C: robot programs or projects causing the robot manipulator 810 to perform useful or intended tasks in its work area 850.Processes that execute the robot programs may do so in accordance with the program-independent memory content S, e.g., by making calls to available functionalities, libraries, routines or parameter values therein. It is recalled that the memory 822 and processing circuitry 821 of the robot controller 820 maybe distributed and / or contain networked resources having a different physical localization than figure 1 suggests.

[0027] Each of the programs C may contain a plurality of movement instructions relating to locations such as points, poses, paths as well as modulated paths. A program maybe a compiled executable (binary) or a script. A movement instruction relating to a modulated path maybe expressed as - or may include - a process-on- path instruction. The programs C may be created by an operator with the aid of a robot programming device (programming station) or a general-purpose computer, or they maybe created directly at the robot controller 820 if it has an operator interface (not shown). In the first two cases, versions of the programs may be downloaded to the robot controller 820 over a wired or wireless connection or by being temporarily stored on a portable memory. As will be described below, a robot program C may further be created by means of kinesthetic programming supplemented with image data, including depth image data and object poses derived from such data.

[0028] A dedicated programming device 860 is shown in the right-hand portion of figure 1. From the programming device 860, the created programs C can be transmitted to the communication interface 823 of the robot controller 820 and then stored in the memory 822 where they are available for execution. It is noted that the programming device 860 maybe implemented as a set of collaborating components within the robot controller 820. In fact, the components can be shared with the robot controller 820, e.g., by using the processing circuitry 821 for the dual purposes of robot control and programming and / or using the memory 822 for the same dual purposes. In other words, the programming device 860 may constitute a portion of a multi-purpose device; it need not be a standalone device or a device with programming as its sole or main purpose.

[0029] The programming device 860 - which is shown as a standalone device in the non-limiting example of figure 8 - comprises processing circuitry 861, memory 862 and at least one communication interface 863. To perform pose tracking, the programming device 860 utilizes a composite pose-estimation algorithm of the type to be described in the next section. Instructions 864 which implement the composite pose-estimation algorithm when executed by the processing circuitry 861 may be stored in the memory 862. The programming device 860 is shown connected, via the at least one communication interface 863, to an external memory 870, which maybe shared with one or more further devices, such as one or more further programming devices (not shown). The external memory 870 maybe used for storing or sharing data items which have been obtained by applying the pose-estimation algorithm to a depth video depicting a physical object in a plurality of poses; the data items can be used for initializing the pose-estimation algorithm to handle a new physical object.

[0030] The programming device 860 further has access to an imaging device 865, by which images of the work area 850 can be captured. The imaging device may be a camera or video camera. In particular, the imaging device maybe a depth camera or depth video camera, which in addition to the two-dimensional appearance of a physical object also determines a depth coordinate of the object, e.g., by time-of-flight measurements, triangulation, radar, lidar or other per se known techniques. An RGB-D camera is an example of a depth camera. The imaging device 865 maybe a part of the programming device 860, or it maybe integrated in the industrial robot 800 and optionally be used for other tasks as well.

[0031] During a kinesthetic programming session, the programming device 860 may have access to position data representing an actual position, a recorded position or recorded movements of the robot manipulator 810. The position data may for example be obtained through the intermediary of the robot controller 820, which monitors position data in the normal course of its operation. Alternatively, the programming device 860 is granted access to signals from the transducers, sensors or other measuring equipment (not shown) in the robot manipulator 810.Composite pose-estimation algorithm

[0032] The method and computer systems disclosed herein will make reference to a class of composite pose-estimation algorithms with certain characteristics. The composite pose-estimation algorithm includes:- a video-object segmentation (VOS) sub-algorithm configured to determine a mask of a visual object in an image, and- an object-pose tracking (OPT) sub-algorithm configured to track a pose of a visual object over multiple depth-video frames.When the algorithm is given a depth video as input and extracts frames from the video, the two sub-algorithms may cooperate as follows. The extracted frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm. The OPT sub-algorithm then performs the pose tracking of the visual object over the extracted video frames in accordance with the object masks. It is recalled that an object mask of a visual object may include a vector-based or bitmap-based representation, from which the OPT sub-algorithm is able to derive what regions of the image (i.e., the video frame) correspond to the object. In particular, the OPT sub-algorithm maybe configured to disregard those regions of the video frame which are outside the object mask.

[0033] The VOS sub-algorithm may include one or more of the following operations:- derive a working memory entry from a preceding video frame;- derive a long-term memory entry from multiple preceding video frames, optionally by means of a memory-potentiation process;- process a current video frame on the basis of the concatenation of a working memory entry with a long-term memory entry.With respect to the first and second operations, it is noted that the preceding video frame maybe an immediately preceding video frame or a video frame separated by one or more intervening frames. The working memory entry and the long-term memory entry maybe part of a unified feature memory architecture in the VOS subalgorithm. In some implementations of the VOS sub-algorithm, including XMem++ (see below), the working memory is subdivided into permanent working memory and temporary working memory. It is noteworthy that the (permanent and temporary) working memory is separate from any trainable neural network in the VOS subalgorithm; in other words, the working memory has a conventional way of functioning which is not controlled by or used for artificial intelligence (machine learning).

[0034] The VOS sub-algorithm may for example include elements of one or more of the XMem architecture (see H. K. Cheng et al., “XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model”, arXiv: 2207.07115 [cs.CV], retrieved from arXiv.org), the XMem++ architecture (see M. Bekuzarov et al., “XMem++: Production-level Video Segmentation From Few Annotated Frames”, arXiv:23O7.i5958 [cs.CV], retrieved from arXiv.org), the Track Anything architecture (see J. Yang et al., “Track Anything: Segment Anything Meets Videos”, arXiv:23O4.H968 [cs.CV], retrieved from arXiv.org), the Cutie architecture (see H. K. Cheng et al., “Putting the Object Back into Video Object Segmentation”, arXiv:23io.12982 [cs.CV], retrieved from arXiv.org), or it maybe an algorithm using reference-guided mask propagation (see S. W. Oh et al., “Fast Video Object Segmentation by Reference-Guided Mask Propagation”, Proceedings of the IEEE conference on computer vision and pattern recognition 2018, pp. 7376-7385) or an algorithm using a space-time memory network (see S. W. Oh et al., “Video Object Segmentation using Space-Time Memory Networks”, Proceedings of the IEEE conference on computer vision and pattern recognition 2018, pp. 9226-9235; and H. K. Cheng et al., “Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation”, arXiv: 2106.05210 [cs.CV], retrieved from arXiv.org).

[0035] Some implementations of the OPT sub-algorithm make reference to a collection of keyframes, where each keyframe represents an observation of the physical object. The OPT may then include one or more of the following operations:- maintaining the collection of keyframes in the form of a double-ended queue;- determine a keyframe for a current video frame based on a pose graph optimization using a plurality of keyframes from the stored collection of keyframes;- using a keyframe for a preceding video frame for the purpose of initializing a pose of a current video frame.With respect to the third operation, it is noted that the preceding video frame may be an immediately preceding video frame or a video frame separated by one or more intervening frames.

[0036] The OPT sub-algorithm may for example include elements of the BundleTrack architecture (see B. Wen et al., “BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models”, Proceedings of 2021 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS)) or BundleSDF architecture (see B. Wen et al., “BundleSDF: Neural 6-D0F Tracking and 3D Reconstruction of Unknown Objects”, arXiv: 2303.14158 [cs.CV], retrieved from arXiv.org).

[0037] Example implementations which fall within the class of composite poseestimation algorithms under consideration have been described in the art; see the research paper M. Sun et al., “Instance-Agnostic Geometry and Contact Dynamics Learning”, arXiv:23O9.05832 [cs.CV], retrieved from arXiv.org. As far as can be understood from the patent application published as CN115187633A, it discloses a further example implementation.

[0038] Some data processing operations within the pose-estimation algorithm will now be reviewed with reference to the functional block diagram in figure 5, in which implementations of the VOS sub-algorithm and OPT sub-algorithm are designated by reference numbers 311 and 312, respectively. The implementations largely correspond to the XMem and BundleTrack architectures; for further details on each architecture, reference is made to the corresponding publications identified above.

[0039] The VOS sub-algorithm 311 receives input in the form of RGB images 501, which maybe frames in a video sequence, from an interface 330. It is optional for the frames to have a depth component, which will not be used in the processing within this implementation of the VOS sub-algorithm 311. The implementation consists of three end-to-end trainable convolutional networks: a query encoder 502 which extracts query-specific image features 503, a decoder 506 which takes the output of a memory reading step to generate an object mask 508, and a value encoder 519 which combines the image (frame) with the object mask to extract new features. There is further provided a long-term memory (comprising a sub-memory for keys 511 and a sub-memory for values 513), a working memory (comprising a sub-memory for keys 510 and a sub-memory for values 512) and a sensory memory 507. For each new frame, the VOS sub-algorithm 311 performs attention-based memory reading from these memories, and the readout features are used to generate a segmentation mask. The memories are updated at different frequencies, e.g., the sensory memory 507 isupdated every frame using a gated recurrent unit (GRU), and features are inserted into the working memory 510, 512 every rthframe (r > 2). When the working memory reaches a preconfigured maximum of Tmaxframes, those features are consolidated from the working memory 510, 512 into the long-term memory 511, 513 in compact form. When the long-term memory is also full (e.g., after processing of approximately one thousand frames), obsolete features maybe discarded.

[0040] Concerning the memory reading and mask reading operation, it is seen in figure 5 that the mask is computed via the decoder 506, which receives as input a feature vectorF = vW(k, q~), where q is the extracted query 503, k is a key 516 which is formed by concatenating 514 data from the working and long-term key memories 510-511, W (k, q) is an affinity matrix determined by the key 516 and the query 503, and v is a value 517 which is formed by concatenating 515 data from the working and long-term value memories 512-513. The decoder 506 further receives preceding frame information from the sensory memory 507, and it has skip connections 509 to the query encoder 502.

[0041] The output of the value encoder 519 is used for updating the working memory 510, 512. Every rthframe, the query 503 is copied into memory 520 as a new key, and a new value 521 is generated by feeding the current frame and a predicted mask into the value encoder 519. The new key and value are appended 522 to the working memory and are later used in memory reading for subsequent frames. The new key 520 and value 521 are also used to provide a deep update 518 of a current item in the sensory memory 507. This maybe performed every rthframe in addition to the per-frame updating by the GRU.

[0042] The input to the OPT sub-algorithm 312 includes time-consecutive RGB-depth frames 531, 532 provided from the interface 330. Each frame corresponds to a snapshot of the physical object, whose pose is to be estimated.

[0043] A block 533 applies the object mask 508 to the respective RGB-D frames 531, 532, whereby mask-annotated RGB-D frames 536, 537 are obtained. The processing steps further downstream can thus infer the extent of the object in each frame 536, 537.

[0044] The target object regions in both frames are cropped, resized and sent to a keypoint detection network 538, 539 to compute keypoints and feature descriptors 540, 541. Each keypoint detection network 538, 539 maybe an LF-Net (see Y. Ono et al., “LF-Net: Learning local features from images,” Proceedings of the NeurlPS, 2018, pp. 6234-6244). A data association process consisting of feature matching and outlier pruning identifies feature correspondences. The correspondences between the thus annotated frames 542, 543 are illustrated by vertical connecting lines. Based on these correspondences, a registration between the current and preceding frames can be solved in closed form, and the result is then used to provide a coarse estimate of the transform (rotation) between the two snapshots (frames). The keypoints and feature descriptors 540 and the frame 542 annotated therewith constitute preceding- frame information, which guides the processing of the current frame.

[0045] The estimated transform is used to initialize a current node 552 in a pose graph optimization step. The graph optimization may be configured to find the minimum H-subgraph of an edge-weighted graph problem. To define the rest of the nodes 545, 546, 547 of the pose graph, no more than K keyframes are selected from a keyframe memory pool 553 to participate in the optimization. The value assigned to the number K is chosen with a view to balance efficiency and accuracy, such as K = 15. Pose graph edges 551 may represent both feature correspondences and geometric correspondences, which are computed in parallel. Given this information, the pose graph optimization step outputs online the optimized pose for the current timestep. If the latest frame corresponds to a novel view of the physical object, then that view is also included in the memory pool 553 together with the keyframes 545-550 already present therein.

[0046] As suggested by the solid arrows between the boldface rectangles with rounded corners in figure 5, the two sub-algorithms cooperate within the composite pose-estimation algorithm. The cooperation includes that data describing the object masks 508 determined by the VOS sub-algorithm 311 is fed to the OPT sub-algorithm 312 for use in determining the mask-annotated RGB-D frames 536, 537.Pose tracking method

[0047] With reference now to the flowchart in figure 1, a method 100 of tracking a pose of a physical object will be described. The method 100 maybe implemented in a computer. In particular, the method 100 maybe executed in a computer system 300of the type illustrated in figures 3 and 4, which may constitute or form part of a robot controller 820.

[0048] In figure 3, the computer system 300 comprises first processing circuitry 310 and second processing circuitry 320, which are separate (distinct) from each other. Each of these is configured for executing the above-described composite poseestimation algorithm. Accordingly, the first processing circuitry 310 is configured with instances of the VOS sub-algorithm 311 and the OPT sub-algorithm 312, and the second processing circuitry 320 is independently configured with instances of the VOS sub-algorithm 321 and the OPT sub-algorithm 322. The computer system 300 further comprises an input interface 330 and a memory 340 connected via a data bus 360. The input interface 330 is suitable for receiving image or video input, for example by being connected to a camera 350 which images a physical object 390 in different poses. The memory 340 is suitable for storing a data item 341 obtained by applying the above-described composite pose-estimation algorithm to a depth video depicting the physical object 390 in a plurality of poses. The memory 340 may furthermore be used for storing computer readable instructions for performing the method 100 illustrated in figure 1. The computer system 300 in figure 3 thus benefits from the ability to spatially separate the learning phase, in which the composite poseestimation algorithm learns a new physical object, from the production phase, in which the composite pose-estimation algorithm estimates the pose of the familiar physical object or a copy thereof.

[0049] In figure 4, the computer system 300 has a single set of processing circuitry which executes the instance of the composite pose-estimation algorithm which is used in the learning phase and the instance of the composite poseestimation algorithm which is used in the production phase. The two instances of the composite pose-estimation algorithm maybe said to coincide. To emphasize this double role, the single set of processing circuitry is designated in figure 4 by both reference numbers 310, 320 (i.e., the first processing circuitry 310 and second processing circuitry 320 coincide), and similarly the VOS sub-algorithm is designated by 311, 321 and the OPT sub-algorithm is designated by 312, 322. The computer system 300 in figure 4 thus benefits from the ability to separate the learning phase from the production phase in time, e.g., for purposes such as swifter initialization ofthe production phase, data backup and the like. The computer system 300 comprises an input interface 330 and a memory 340, which are not shown in figure 4.

[0050] Returning to figure 1, a first step no of the method 100 includes defining a composite pose-estimation algorithm with at least the following features. The poseestimation algorithm includes:- a VOS sub-algorithm 311, 321 configured to determine a mask of a visual object in an image, and- an OPT sub-algorithm 312, 322 configured to track a pose of a visual object over multiple depth-video frames, wherein the composite pose-estimation algorithm is configured to extract frames from a depth video and feed such frames to the VOS sub-algorithm 311, 321, which determines respective object masks to be used by the OPT sub-algorithm 312, 322 in addition to the depth video.

[0051] Within the first step no, “defining” the composite pose-estimation algorithm may include providing executable program code in accordance with one of the implementations reviewed above. The program code may be generated by the entity executing the method 100, or it may be received from another entity. Additionally or alternatively, the first step no may include creating a data structure, reserving memory space in a working memory of the processing circuitry 310, instantiating an object, or the like. Further still, the first step 110 may include performing training of a neural network, with the proviso that the training is universally applicable to all physical objects, as opposed to such forms of training that would specialize the algorithm to process images of a particular physical object.

[0052] In a second step 111, a depth video depicting a physical object 390 in a plurality of poses is recorded, such as by means of the camera 350.

[0053] In a third step 112 (learning phase), the composite pose-estimation algorithm is applied to the depth video, whereby the various memories 507, 510-513, 520, 521, 553 of the sub-algorithms are populated with data originating from the depth video.

[0054] In some embodiments of the method 100, the storable data item 341 can be formed from at least a snapshot of a concatenation of a working memory entry with a long-term memory entry in the VOS sub-algorithm 311 (i.e., in a runtimememory used by the executing instance of the VOS sub-algorithm 311), wherein each memory entry relates to the physical object 390. A “snapshot” in this sense may refer to a current content of a memory, e.g., a static copy of that content. (As already explained, the working memory may be subdivided into permanent working memory and temporary working memory in some implementations of the VOS sub-algorithm 311-)

[0055] In such implementations where each memory entry is a memory key or a memory value, or a combination of these, the storable data item 341 is formed by including at least: a snapshot of a concatenation of a working memory key with a long-term memory key in the VOS sub-algorithm, and / or a snapshot of a concatenation of a working memory value with a long-term memory value in the VOS sub-algorithm 311.

[0056] A further option is to form the storable data item 341 by serializing one or more memory objects which represent the (permanent or temporary) working memory entry and the long-term memory entry.

[0057] Additionally or alternatively to the above, the storable data item 341 may be formed by including at least a snapshot of a stored collection of keyframes in the OPT sub-algorithm 312. Here, each keyframe representing an observation of the physical object 390. The keyframe maybe a spatial arrangement (or projected image) of detected keypoints, such as semantic keypoints. The physical object’s 390 orientation and position are, in some implementations, derivable from the keyframe. For example, the storable data item 341 maybe formed by:- identifying files to be included in the storable data item such that each file contains one keyframe;- establishing a first and a second data structure in the storable data item; and- including the filenames of the identified files in the first data structure and including the identified files in the second data structure.

[0058] Additionally or alternatively, further, the storable data item 341 may be formed by including at least values of configuration parameters of the VOS subalgorithm 311 and / or configuration parameters of the OPT sub-algorithm 312.

[0059] As shown in figure 5, a storable data item 341 can for example be formed on the basis of data extracted from the working and long-term memories 510-513and the keyframe memory pool 553. After this data has been extracted, the forming of the data item 341 may include gathering the data, e.g., in accordance with a prespecified format. Optionally, the gathered data is serialized into a data stream or linear data record, to allow storage in a certain type of memory or transmission over a certain data medium.

[0060] The XMem implementation will be discussed as an illustrative example. The Python module “pickle” may be used to serialize the object structure of memory object of type “MemoryManager”, which manages the memory in the XMem implementation, within the “InterferenceCore” class; the serialized (pickled) file is stored to the memory 340. Optionally, the configuration parameters for XMem can be saved as well, at least to the extent that they have been modified from their default values; within the “InferenceCore” class, then, the object variable “config” of type dictionary can be serialized and stored to disk.

[0061] In the BundleTrack architecture, the memory is managed by the “Bundler” class in the “bundler.cpp” file. The “Memory Object” is specifically a list of keyframes that are stored in a public variable from the “Bundler” class called “_keyframes”. This variable is an array of objects (double ended queue to be more specific) from the class “Frame” in the “Frame.cpp” file. Each of the “Frame” objects holds information about a particular image frame from the video (e.g., color image, depth image, mask, filename, pose, status). The keyframes are selected and stored by the “checkAndAddKeyframe” method in the “Bundler” class. After the BundleTrack algorithm has been applied to the depth video, a memory-storing method that stores to disk the frame objects in the “_keyframes” object variable is executed. One way to achieve this is to store the filenames of the keyframes, and later copy the depth, color, object-mask and object-pose files to a separate folder to hold the memory components and build the “Database” object. A file (e.g., YAML file) containing the configuration that was used to run the BundleTrack algorithm is saved.

[0062] The data extraction process is also illustrated in figure 6, where some details have been deliberately omitted to improve the overall visibility. The hollow solid rectangles with rounded corners (same style as used in figure 5) represent preceding frame information in each of the sub-algorithms.

[0063] In a fourth step 113 (production phase) of the method 100, the same or a different instance of the composite pose-estimation algorithm is initialized(substep 113.1) by means of the storable data item 341 and is used for tracking the physical object 390 or a copy thereof. The thus initialized instance of the composite pose-estimation algorithm maybe expected to be fit for this purpose as long as the copy of the physical object 390 is in adequate visual agreement with the physical object 390, e.g. agreeing with respect to shape, size, color, reflectivity, whereas properties such as mass, conductivity etc. are unimportant. The physical object 390 and its copy may for example be two samples of a serially manufactured object.

[0064] The initialization 113.1 may proceed as shown in figure 7, namely, by loading the respective portions of the data item 341 into the working and long-term memories 510-513 and the keyframe memory pool 553, which the dashed shapes represent. This may include deserializing the storable data item 341 into one or more memory objects which represent a (temporary or permanent) working memory entry and the long-term memory entry in the VOS sub-algorithm 311. The instance of the OPT sub-algorithm 312 maybe initialized such that the pose of the first video frame to be processed in the production phase (step 113) shall be initialized by that key- frame - from among the keyframes in the stored collection of keyframes (keyframe memory pool 553) - which has the maximum number of matching points relative to the first video frame. For the avoidance of doubt, it is recalled that the storable data item 341 maybe loaded into the working and long-term memories 510-513 and the keyframe memory pool 553 of the instance of the VOS sub-algorithm and OPT subalgorithm which formed the storable data item 341, or the storable data item 341 may be loaded into the working and long-term memories 510-513 and the keyframe memory pool 553 of a different instance of the VOS sub-algorithm and OPT subalgorithm than the one which formed the storable data item 341.

[0065] In the XMem implementation, a new “XMem” object is created from the “network.py” module, wherein default configuration parameters or saved non-default configuration parameters are used. Next, a new “InferenceCore” object is created from the “inference_core.py” module, again using default configuration parameters or saved non-default configuration parameters. Then, the stored serialized (pickled) file is opened and deserialized, such that the “MemoryManager” object is recovered. The object variable “memory” in the “InterferenceCore” object is updated to refer to the recovered “MemoryManager” object. The previous-frame information is cancelled, e.g., by populating the sensory memory 507 with a neutral value, such as zero.

[0066] As to the BundleTrack part, the configuration file (e.g., YAML file) that was stored in the memory storing process is loaded. A new class is written in the “DataLoader.cpp” file that allows creating a “Frame” object from the color, depth, mask and pose files. See “DataLoaderNOCS::getFrameByIndex” method for an example (pose loading is not included in this method). A new “Bundler” object is created using as parameters the parameters stored in the configuration file (e.g., YAML file) and the new “DataLoader” class just mentioned. A new method is written in the “Bundler” class to load the keyframes from disk and to modify the object variable “_keyframes”. For this, the new DataLoader class is used to create the necessary “Frame” objects using the list of filenames of the keyframes. Then, the feature detection pipeline is run for each of the new frame objects; this may involve use of the “detectFeature” method in the FeatureManager.cpp file.

[0067] The initialization 113.1 of the BundleTrack instance may include the following substeps. Compared to normal operation of BundleTrack, the “processNewFrame” method in the “Bundler” class is reworked, and some of the parameters in the configuration file are modified. In particular, the following configuration parameters can be modified:- feature_corres — max_dist_no_neighbor (linear feature matching tolerance parameter) and- feature_corres — max_normal_no_neighbor (angular feature matching tolerance parameter).Both parameters refer to feature matching tolerances used in the feature matching algorithm between two frames. These parameters are used in scenarios when the images that are being compared are not adjacent frames in the video. Their default values (intended for regular uses of the algorithm) are too low to allow matching between frames where the objects are far apart from each other. However, setting feature_corres — max_dist_no_neighbor to a high placeholder value (e.g., at least one meter or several meters) and leaving feature_corres — max_normal_no_neighbor at the default 45 degrees value is enough to initialize the BundleTrack sub-algorithm for an immediate start of the production phase. In this context, the default values may be of the order of one centimeter or a few centimeters, whereas a high placeholder value is significantly greater than the default values. For some illustrative implementations of the algorithm, the high placeholder value maybe set to at least one meter or at least several meters. The parameter feature_corres — max_normal_no_neighbor may alternatively be set to an angle greater than 45 degrees.

[0068] The “processNewFrame” method in the “Bundler” class may be reworked as follows to achieve the initialization 113.1:- Instead of using the previous frame pose to initialize the pose of the current frame, a heuristic based on keyframe information is required. A strategy which performs feature matching between the current frame and all the keyframes can be used; the keyframe that has the highest number of matching points is used to initialize the pose of the current frame.- The keyframe update process to memory needs to be disabled to not modify the loaded “MemoryObject” object.

[0069] The data loading process is shown in figure 7 as well. Figure 7 particularly shows that, during initialization, the previous-frame information will not be used in the regular way.

[0070] The present method 100 may allow significant efficiency gains. According to prototype testing by the inventors, a representative operator spent approximately 10 minutes to record a 2-minute video of the physical object 390 and process it with the VOS sub-algorithm 311 and OPT sub-algorithm 312 using consumer grade GPUs (e.g., Nvidia GTX1080). After this the vision system with the composite poseestimation algorithm was ready for use at runtime to perform object segmentation and pose estimation.

[0071] In an envisioned further development of the pose-tracking method 100, step 111 is repeated so as to record a further depth video depicting a further physical object in a plurality of poses. Then, step 112 is repeated such that the pose-estimation algorithm is applied to the further depth video, for thereby obtaining data representing the physical object to be included in a storable data item 341. Such data may enable an initialization of a further instance of the composite pose-estimation algorithm such that the further instance becomes capable of estimating the pose of the further physical object. If two instances of the composite pose-estimation algorithm are active in parallel, pose estimation of a physical object - whether it hasthe appearance of the initially described physical object 390 or of the further physical object - is possible.Robot programming method

[0072] To illustrate the usefulness of the pose-tracking method 100, a robotprogramming method 200 will be described by way of an envisioned use case. The method 200 is concerned with an industrial robot 800 of the type illustrated in figure 8, namely one which includes a robot manipulator 810 and a robot controller 820. The robot programming method 200 is machine-implemented, in particular computer-implemented, and may for example be executed by the robot programming device 860, which has been described above with reference to figure 8.

[0073] In a first step 210 of the robot programming method 200, movements of the robot manipulator 810 are defined during a programming session, which includes handling a workpiece. The movements may be defined by user inputs or by loading a script provided by a programmer. Alternatively, the movements of the robot manipulator 810 maybe defined by being sensed during a kinesthetic programming session. It is recalled that the term kinesthetic programming - by allusion to the robot’s proprioceptive ability to sense its own position, pose and movements - is a programming approach in which the user performs a demonstration by physically moving the robot manipulator through the desired motions. The terms kinesthetic programming and lead-through programming are synonymous or at least partially overlapping in meaning. The state of the robot during a kinesthetic programming session is typically recorded by means of the robot’s onboard sensors, e.g., joint angles and torques. As used herein, kinesthetic programming also includes teleoperation as a special case, where the movement of the robot manipulator is controlled by an external input to the robot through a joystick, graphical user interface or other input means; kinesthetic programming by teleoperation does not require the user to be present in the work area of the robot.

[0074] In a second step 211, a depth video of the programming session is recorded, and poses of the workpiece are estimated by means of an implementation of the method 100 described in the preceding subsection. Poses in this sense may include positions and orientations.

[0075] The estimated poses may be used to determine the reference frames in which the robot program C shall express the movements of a tool 811 carried by the robot manipulator 810, and particularly to determine relationships between different reference frames. This way, even if a physical object which defines a reference frame moves between two executions of a certain trajectory, the trajectory can be replicated with respect to the programming session (in particular, with respect to the user demonstration) so that the relative motion between the TCP and the reference frame stays the same. A further possible use of the estimated poses of the workpiece 830 is for the purpose of annotating the robot trajectory. The estimated poses may appear explicitly in the annotations, e.g., to allow a verification that the robot program is executing in the way expected, so that irregularities can be discovered early. Alternatively, the estimated poses of the workpiece 830 may have a merely indirect influence, such as preferable ways of gripping or lifting the workpiece 830 using the tool 811 carried by the robot manipulator 810.

[0076] In a third step 212, then, a robot program to be executed by the robot controller 820 is generated on the basis of the defined movements and estimated workpiece poses.

[0077] The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

Claims

CLAIMS1. A method (100) of tracking a pose of a physical object (390), comprising: defining (no) a composite pose-estimation algorithm including a video-object segmentation, VOS, sub-algorithm (311, 312) configured to determine a mask of a visual object in an image, and an object-pose tracking, OPT, sub-algorithm (312, 322) configured to track a pose of a visual object over multiple depth-video frames, wherein the pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm alongside the depth video; recording (111) a depth video depicting a physical object in a plurality of poses; forming (112) a storable data item representing the physical object by applying the pose-estimation algorithm to the depth video; and tracking (113) the physical object or a copy thereof using an instance of the poseestimation algorithm which has been initialized (113.1) by means of the storable data item.

2. The method (100) of claim 1, wherein the storable data item is formed (112) by including at least a snapshot of a concatenation of a working memory entry with a long-term memory entry in the VOS sub-algorithm, each memory entry relating to the physical object.

3. The method (100) of claim 2, wherein each memory entry is a memory key and / or a memory value, whereby the storable data item is formed (112) by including at least: a snapshot of a concatenation of a working memory key with a long-term memory key in the VOS sub-algorithm, and / or a snapshot of a concatenation of a working memory value with a long-term memory value in the VOS sub-algorithm.

4. The method (100) of claim 2 or 3, wherein the VOS sub-algorithm includes one or more of the following operations:derive the working memory entry from a preceding video frame; derive the long-term memory entry from multiple preceding video frames, optionally by means of a memory-potentiation process; process a current video frame on the basis of the concatenation of the working memory entry with the long-term memory entry.

5. The method (100) of any of claims 2 to 4, wherein the storable data item is formed (112) by serializing one or more memory objects which represent the working memory entry and the long-term memory entry.

6. The method (100) of any of the preceding claims, wherein the working memory entry and the long-term memory entry are part of a unified feature memory architecture in the VOS sub-algorithm.

7. The method (100) of any of the preceding claims, wherein the working memory entry and the long-term memory entry are separate from any trainable neural network in the VOS sub-algorithm.

8. The method (100) of any of the preceding claims, wherein the storable data item is formed (112) by including at least a snapshot of a stored collection of keyframes in the OPT sub-algorithm, each keyframe representing an observation of the physical object.

9. The method (100) of claim 8, wherein the OPT sub-algorithm includes one or more of the following operations: maintaining the collection of keyframes in the form of a double-ended queue; determining a keyframe for a current video frame based on a pose graph optimization using a plurality of keyframes from the stored collection of keyframes.

10. The method (100) of claim 8 or 9, wherein the storable data item is formed (112) by: identifying files to be included in the storable data item such that each file contains one keyframe; establishing a first and a second data structure in the storable data item; and including the filenames of the identified files in the first data structure and including the identified files in the second data structure.

11. The method (100) of claim 8, 9 or 10, wherein: the OPT sub-algorithm includes using a keyframe for a preceding video frame to initialize a pose of a current video frame; and the instance of the pose-estimation algorithm is initialized (113.1) such that the pose of the first video frame to be processed during said tracking (113) shall be initialized by that keyframe, in the stored collection of keyframes, which has the maximum number of matching points relative to the first video frame.

12. The method (100) of any of the preceding claims, wherein the storable data item is formed (112) by including at least values of configuration parameters of the VOS sub-algorithm and / or configuration parameters of the OPT sub-algorithm.

13. The method (100) of any of the preceding claims, wherein initializing (113.1) the instance of the pose-estimation algorithm to be used for tracking (113) includes deserializing the storable data item into one or more memory objects which represent a working memory entry and the long-term memory entry in the VOS sub-algorithm.

14. The method (100) of any of the preceding claims, wherein the instance of the pose-estimation algorithm to be used for tracking (113) is initialized (113.1) such that a linear feature matching tolerance parameter in the OPT sub-algorithm has a high placeholder value and an angular feature matching tolerance parameter in the OPT sub-algorithm is greater than or equal to 45 degrees.

15. The method (100) of any of the preceding claims, wherein the VOS subalgorithm is chosen from a group consisting of XMem, XMem++, Track Anything, Cutie, an algorithm using reference-guided mask propagation, an algorithm using a space-time memory network.

16. The method (100) of any of the preceding claims, wherein the OPT subalgorithm is chosen from a group consisting of BundleTrack, BundleSDF.

17. The method (100) of any of the preceding claims, further comprising: recording (111) a further depth video depicting a further physical object in a plurality of poses; and applying the pose-estimation algorithm to the further depth video, for thereby obtaining data representing the physical object to be included in a storable data item.

18. A method (200) of programming an industrial robot (800), which comprises a robot manipulator (810) and a robot controller (820), the method comprising: defining (210) movements of the robot manipulator during a programming session, during which a workpiece is handled; recording (211) a depth video of the programming session and estimating poses of the workpiece using the method (100) of any of the preceding claims; and on the basis of the sensed movements and estimated workpiece poses, generating (212) a robot program to be executed by the robot controller.

19. The method (200) of claim 19, wherein the movements are defined (210) by being sensed during a kinesthetic programming session.

20. A computer system (300) configured to track a pose of a physical object, the computer system comprising: first processing circuitry (310) and second processing circuitry (320), each processing circuitry configured with a composite pose-estimation algorithm including a video-object segmentation, VOS, sub-algorithm (311, 321) configured to determine a mask of a visual object in an image, and an object-pose tracking, OPT, sub-algorithm (312, 322) configured to track a pose of a visual object over multiple depth-video frames, wherein the pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm alongside the depth video; an interface (330) for inputting a depth video depicting a physical object in a plurality of poses; and a memory (340), characterized in that the first processing circuitry is further configured to apply the pose-estimation algorithm to the depth video, for thereby storing, in said memory, a data item (341) representing the physical object; and in that the second processing circuitry is further configured to retrieve the data item from the memory and use it to initialize an instance of the pose-estimation algorithm.

21. A computer program (342) comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 1 to 19.

22. A storable data item (341) obtained by applying a pose-estimation algorithm to a depth video depicting a physical object in a plurality of poses, wherein: the pose-estimation algorithm is a composite pose-estimation algorithm including a video-object segmentation, VOS, sub-algorithm (311, 321) configured to determine a mask of a visual object in an image, and an object-pose tracking, OPT, sub-algorithm (312, 322) configured to track a pose of a visual object over multiple depth-video frames, the pose-estimation algorithm is configured to input a depth video, from which frames are extracted and fed to the VOS sub-algorithm, which determines respective object masks to be used by the OPT sub-algorithm alongside the depth video.

23. A data carrier (340) carrying the computer program (342) of claim 21 or the storable data item (341) of claim 22.

Images