Discriminative 3D Shape Modeling for Few-Shot Instance Segmentation

Abstract

An imaging controller for segmenting an instance from a depth image containing an object to be manipulated by a robot is provided. The imaging controller includes an input interface configured to receive a depth image containing the object, a memory configured to store instructions and a neural network trained to segment the instance from the object in the depth image, and a processor coupled to the memory and configured to execute the instructions to segment the pickable instance using the trained neural network. The instructions include selecting a highest point in the depth image, defining a region with a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled point, submitting the depth geodesic curve to the neural network to segment the pickable instance among the instances of the object in the depth image, and an output interface configured to output geometric features of the pickable instance to a manipulator controller of the robot.

Description

[Technical field]

[0001] This disclosure generally relates to automatically segmenting multiple instances of objects in depth images, where a segmentation model is trained in a few-shot setting using only a few annotated training examples. [Background technology]

[0002] Instance segmentation is an essential step in various autonomous tasks: some examples are (i) a robotic arm that needs to pick similar products moving on a conveyor belt in a manufacturing environment, (ii) a supermarket robot that needs to pick and place fruits from bins, (iii) a library assistant robot that needs to pick books from boxes and hand them over to a person, (iv) identifying instances of cars on the road in a self-driving environment, or (v) identifying instances of body tissues in medical images for automated diagnosis.

[0003] Standard deep learning solutions, such as Mask-RCNN and variants, to solve such instance segmentation tasks typically require large training sets to train the underlying neural network, but creating such training sets and annotating the training examples can be expensive and involve significant physical effort. Thus, creating such datasets may not be feasible in many situations. The instance segmentation task may also be viewed as one of applying standard clustering schemes to depth image point clouds. For example, segmentation can be derived from applying K-means, Gaussian mixture modeling, or spectral clustering to the point cloud. However, such clustering algorithms usually make assumptions on the point cloud distribution and therefore may not be flexible to segment real-world objects whose shapes may vary. There are also recent approaches, such as InSeGAN, that can generate instance segmentations in an unsupervised setting. However, InSeGAN requires a large unlabeled training set with diversity in instance placements, which may also be difficult to obtain in real-world conditions.

[0004] Therefore, there is a need to design an approach for instance segmentation that can automatically segment multiple instances of an object in a depth image and requires very few training examples to train the segmentation model. Summary of the Invention

[0005] This disclosure generally relates to automatically segmenting multiple instances of objects in depth images, where a segmentation model is trained in a few-shot setting using only a few annotated training examples.

[0006] Some embodiments of the present disclosure provide a method for segmenting object instances in depth images. In contrast to conventional methods that typically use deep neural networks for segmentation, the present disclosure includes a custom-built instance segmentation pipeline that does not require any 3D CAD model of the object, but instead requires only a few annotated examples of the object instance in a small number of depth images.

[0007] The embodiments of the present disclosure are configured for automatically segmenting multiple instances of nearly convex 3D objects in depth images in a few-shot setting, a novel approach in the computer vision domain. The embodiments of the present disclosure can be configured to perform for a number of applications, including robotic assistance tasks, such as picking an object from a bin containing multiple instances where the instances need to be accurately segmented for a robot to plan a grasp.

[0008] Some embodiments of the present disclosure include an architecture for instance segmentation that follows some aspects of deep neural networks, but standard neural networks require a large training set of annotated instances in depth images for training. A key insight in this disclosure is to not directly use instances to train a neural network, but instead train the neural network to classify the depth of surface geodesics that connect pairs of 3D points in a depth image. Because there are quadratically many such pairs of points possible relative to the number of pixels in a depth image, a small number of annotated instances can generate a large set of annotated surface geodesics, which can then be used to train a neural network. The method first selects a random sample of spatial locations in a given training depth image and computes pairwise surface geodesics between each and every pair.

[0009] At least one realization that emerges from the experiments is that the surface geodesic curves computed between pairs of depth points can be of arbitrary length and therefore can be difficult to use to train neural networks that expect fixed-sized inputs. To this end, the presented model discretizes each geodesic curve into a fixed-dimensional vector. In this disclosure, each geodesic curve is divided into equal segments and each dimension of the discretized geodesic curve vector is filled with the depth of the pixel at the end point of the respective geodesic curve segment.

[0010] Another realization for the present disclosure comes from the theoretical insight that for any two points on a convex surface, the surface geodesic curve connecting these two points (which is the shortest curve) will have a convex shape. However, if the two points belong to different instances and each instance is convex, the surface geodesic curve connecting the two points will have a non-convex shape or discontinuity at the depth point where the two object surfaces intersect. These two types of curves (convex and non-convex) are used to train a neural network for classification purposes. The neural network, when sufficiently trained, will classify each geodesic curve as either entirely within an instance segment (convex geodesic curve) or intersecting across multiple instances (non-convex geodesic curve).

[0011] A realization that emerges from experiments is that in the discretization of surface geodesics, convex / non-convex discontinuities in the geodesics may be lost if the two instances for which the curves are computed are very close. For example, consider two cubes touching each other. In the depth image, where the cubes touch, the curves may be nearly continuous, and the discontinuities may be skipped by the discretization step if the number of dimensions in the fixed-dimensional geodesic vector is not so large as to capture the discontinuities. A similar problem may arise when there is noise in the depth image, such that standard denoising and hole-filling algorithms typically used in smoothing depth images may over-smooth the boundaries of instance segments, and thus suppress ground truth geodesic discontinuities. To avoid this problem, training annotations for the instances are deployed. The annotations guide the neural network for classification even when the discretized geodesic vectors suppress object crossing discontinuities, thereby requiring the neural network to learn other surface features that characterize the surface of the 3D object.

[0012] Another realization from the experiment is that randomly selecting pairs of points from a depth image to construct geodesics to train a neural network may be suboptimal. For example, there may be geodesics that are very easy for a neural network to learn to segment, but there may be other geodesics that are difficult to classify. Since easy examples may sometimes significantly outnumber difficult examples, neural network training may be biased and may not produce correct segmentations in difficult segmentation cases. Another case is when the size of the object to be segmented is small, but there are many instances in a given depth image. This results in a large set of geodesics that go through multiple instances, but only a few geodesics that are completely within the instances. To solve these problems, the present disclosure proposes to create an initial segmentation of a depth image using a Watershed algorithm (WA). The method in WA generates two outputs: (i) a set of peaks that are points in the depth image that are said to correspond to the centroids of the instances, and (ii) a segmentation of each instance. However, since the WA method is unsupervised, the peaks or segmentations it generates may be inaccurate. The key insight in this disclosure is to improve the WA segmentation using ground truth training annotations and a geodesic classification neural network. More specifically, one end of the geodesic is obtained from the peak generated by WA, and the other end of the geodesic is constrained to be within the respective segment generated by WA for its corresponding selected peak. The geodesic is then given a label using ground truth annotations as to whether it belongs to the exact same instance. If WA makes a mistake in segmenting an instance, the ground truth instance labeling of the geodesic will allow the neural network to correct this mistake, leading to better instance segmentation.At least one benefit of selecting geodesics from within the instances generated by WA is that it allows for creating a better training set for the neural network with more geodesics that belong to the same instance and difficult examples generated by WA's incorrect segmentation. The geodesics that belong to the same instance allow for learning a better implicit 3D model of the object shape within the neural network, while the difficult examples allow for learning a discriminative network that works well in the case of difficult segmentations.

[0013] For example, when using a trained model for segmentation, the architecture takes as input a depth image on which depth geodesics are computed between pairs of selected locations in the depth image. The locations are selected based on a predefined practical criterion. One embodiment of the criterion is based on the subsequent task for which the segmentation will be used. For example, when using the instance segmentation generated by the present approach to pick instances with a robotic arm, it is useful if the top instance of the stack of instances is segmented first. Such instances that are desired to be segmented first are called pickable instances.

[0014] The present disclosure proposes to select pairs of spatial locations on a depth image to compute depth geodesics on an input test image, where the selection is based on the selectable instances in a given test depth image. Depth geodesics are computed between all pairs of selected locations. One end of each geodesic is fixed to the highest depth point on the selectable instance. This point is called the seed point. The other end of the geodesic is selected from a region within a predefined radius around the highest depth point. Each geodesic is discretized as described in the training phase. The pre-trained neural network then classifies each geodesic as belonging entirely to one instance surface or spanning multiple instances. The convex hull of all points classified as belonging to the same instance is then used to define the instance segment. This process is repeated for various selected locations in the image to generate multiple such instance segmentations.

[0015] A realization from experiments for segmenting a given test image is that randomly selecting depth points from around a seed point may not be computationally efficient because it may not cover the instance in all directions. This insight suggests a systematic selection of points around the seed point. An embodiment of this concept in the present disclosure is to create a sample set of points along angularly equally spaced radial lines on concentric circles around the seed point. Such a selection allows for uniform coverage of a circular region around the seed point to construct a geodesic curve.

[0016] A realization that arises from the manner in which the test geodesics are constructed is that these geodesics are always directed from higher depth points to lower points. Therefore, it may be inefficient to include in the training set geodesics that do not respect this property. To correct for this, the depth points used to train the neural network are first sorted in ascending order of their depth, and the geodesics for training are calculated from points with lower depths (or highest points) to points with higher depths.

[0017] An embodiment of the present disclosure provides an imaging controller for segmenting an instance from a depth image including an object to be manipulated by a robot. The imaging controller may include an input interface configured to receive a depth image including the object; a memory configured to store instructions and a neural network trained to segment the instance from the object in the depth image; and a processor coupled to the memory and configured to execute the instructions to segment the pickable instance using the trained neural network, where the instructions may include selecting a highest point of all instances in the depth image, defining a region with a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled point, submitting the depth geodesic curve to the neural network to segment the pickable instance among the instances of the object in the depth image, and an output interface configured to output a geometric feature of the pickable instance to a manipulator controller of the robot.

[0018] Further, in accordance with some embodiments of the present invention, there is provided a computer-implemented method for training a neural network to segment instances in a depth image. In this case, the method uses a processor coupled with stored instructions for implementing the method, which instructions, when executed by the processor, perform steps of the method, including the steps of selecting a depth image from a set of depth images, and determining points at xy spatial locations on a two-dimensional image grid and corresponding depth points for the selected depth image, where the points on the two-dimensional image grid are each annotated with a ground truth instance label, and the steps further include the steps of computing geodesic lines between pairs of annotated, determined points, generating depth geodesic lines by projecting the geodesic lines onto the depth image, and discretizing each of the depth geodesic lines to create discretized vectors, where each discretized vector corresponds to one of the depth geodesic lines between pairs of annotated, determined points, and the steps further include the step of submitting the discretized vectors and the corresponding annotated labels of the discretized vectors to a neural network, and the steps from selecting to providing are performed repeatedly until all remaining of the set of depth images have been used.

[0019] Further, in accordance with some embodiments of the present invention, there is provided a bin picking system for picking objects from a bin. The bin picking system may include: an end tool configured to pick an object from among the objects; a robotic arm including the end tool, the robotic arm being driven by control signals including an end tool signal and configured to pick an object from the bin using the end tool; an interface configured to transmit and receive the control signals, a sensor signal of a sensor disposed on the robotic arm, and an imaging signal of at least one imaging device; and a memory configured to store robot control program instructions, a classifier, and a trained neural network for segmenting instances from objects in a depth image, the trained neural network being trained by the computer-implemented method described above; and the bin picking system may further include: an interface configured to transmit and receive the control signals, a sensor signal of a sensor disposed on the robotic arm, and an imaging signal of at least one imaging device; a memory configured to store instructions of a robot control program, a classifier, and a trained neural network for segmenting instances from objects in a depth image, the trained neural network being trained by the computer-implemented method described above; the bin picking system may further include: an interface configured to transmit and receive the control signals, a sensor signal of a sensor disposed on the robotic arm, and an imaging signal of at least one imaging device; and ... The method may include a processor configured to segment the possible instances and generate control signals to drive the robotic arm and end tool, the steps of the instructions including selecting the highest point in the depth image, defining a region using a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled points, submitting the depth geodesic curve to a neural network to segment selectable instances among the instances of the object in the depth image, generating geometric features and control signals for the selectable instances based on the imaging signal, and sending the generated geometric features and generated control signals to the robotic arm to cause the end tool to use the end tool to pick an object corresponding to the selectable instance from a bin.

[0020] Thus, the present system and method as a whole cannot be viewed in an abstract sense as merely implementing aspects of the human mind, nor as gathering (collecting) data, processing data, analyzing data, and displaying specific results. In contrast, the present system and method can generalize to cases where annotation is difficult or operates on complex scenes with less cost and computational expense, and can be competitive and beneficial to participants in the computer vision industry and other industries.

[0021] The presently disclosed embodiments are further described with reference to the accompanying drawings, in which: The drawings shown are not necessarily to scale, with emphasis generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief description of the drawings]

[0022] [Figure 1] 1 is an illustrative example of depth image segmentation of nearly identical chicken nugget instances within a bin, according to an embodiment of the present invention. [Diagram 2] (A) shows an example illustrating a depth image containing multiple instances of an object according to an embodiment of the present invention, and (B) shows a surface curve between two arbitrary points p1 and p2 of Figure 2(A) according to an embodiment of the present invention. [Diagram 3] 1 illustrates a pipeline for training a neural network to learn an implicit shape model of a single instance of an object using surface geodesics, according to an embodiment of the invention. [Figure 4] 4A shows an example illustrating selectable instances in a depth point cloud according to an embodiment of the present invention, and (B) shows a segment instance at the top of the bin in FIG. 4A according to an embodiment of the present invention. [Diagram 5] 1 illustrates an example of an inference pipeline for segmenting instances, according to an embodiment of the present invention. [Figure 6]FIG. 1A shows an example illustrating random selection of points around a chosen point for computing a geodesic curve according to an embodiment of the present invention; and FIG. 1B shows an example illustrating systematic selection of points around a chosen point for computing a geodesic curve according to an embodiment of the present invention. [Figure 7A] 1 illustrates an example of a Watershed transform based step for generating depth maps and instance segmentations according to an embodiment of the present invention. [Figure 7B] 1 illustrates an example of a Watershed transform based step for generating depth maps and instance segmentations according to an embodiment of the present invention. [Figure 7C] 1 illustrates an example of a Watershed transform based step for generating depth maps and instance segmentations according to an embodiment of the present invention. [Figure 7D] 1 illustrates an example of a Watershed transform based step for generating depth maps and instance segmentations according to an embodiment of the present invention. [Figure 8] We present a comparison of results obtained by the present invention and other approaches on the Fried Chicken dataset. [Figure 9] We show example images (depth images) of the four object classes used to create the dataset in the experiments. [Figure 10] 10 is an example showing segmentation results using the dataset of FIG. 9 according to an embodiment of the present invention. [Figure 11A] We show some examples of segmentations produced by our setup for instances of chopped carrots, chicken nuggets, taro, and fried foods, in accordance with an embodiment of the present invention. [Figure 11B] We show some examples of segmentations produced by our setup for shredded carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 11C]We show some examples of segmentations produced by our setup for shredded carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 11D] We show some examples of segmentations produced by our setup for shredded carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 12] FIG. 1 illustrates a hardware diagram illustrating some components that may be incorporated in some systems of the present disclosure, in accordance with some embodiments of the present invention. [Figure 13] 1 illustrates a robot configuration for a robot according to an embodiment of the present invention. [Figure 14] FIG. 1 is a schematic diagram illustrating, by way of non-limiting example, a computing device that may be used to implement some techniques according to embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Although the above-identified drawing figures set forth embodiments disclosed herein, other embodiments are contemplated as noted in the present discussion. This disclosure presents exemplary embodiments by way of representation and not limitation. Those skilled in the art may devise numerous other modifications and embodiments that fall within the scope and spirit of the principles of the disclosed embodiments.

[0024] Segmenting nearly identical object instances is a ubiquitous problem in various robotic bin picking applications. Some examples are (i) a robotic arm that needs to pick products moving on a conveyor belt in a manufacturing environment, (ii) a supermarket robot that needs to pick and place fruits from a bin, or (iii) a library assistant robot that needs to pick books from a box and hand them over to a person.

[0025] FIG. 1 is an illustrative example of depth image segmentation of nearly identical chicken nugget instances in bins according to some embodiments of the present invention. See FIG. 1 for an illustrative example of depth image segmentation of nearly identical chicken nugget instances in bins. A standard deep learning solution for this task would require a large training set and significant annotation effort. One can also think of this problem as one of applying a standard clustering scheme to the depth points. For example, the segmentation can be derived from applying, for example, K-means, EM, or spectral clustering to the point cloud. However, such clustering algorithms usually make assumptions on the shape of the distribution or involve hyperparameters. For example, K-means assumes that the distribution is spherical around the cluster centroids or elliptical as in EM. Spectral clustering requires specifying a proximity hyperparameter to define the edges of the underlying graph Laplacian. There are also some recent approaches for unsupervised instance segmentation. However, they require large unlabeled training sets with diversity in instance placements, which again can be difficult to obtain in real-world conditions. For example, some methods require thousands of unlabeled depth images, which requires a robotic arm to shake a bin of objects, a task that is not scalable and sometimes infeasible if the object of interest is fragile or could be damaged by such shaking.

[0026] In this disclosure, we consider this problem of instance segmentation of nearly identical convex object instances in depth images in a few-shot setting, where we assume that we have access to a limited set (less than 5) of annotated depth images, each of which has a few instances annotated with their segments. Our key concept is to create surface trajectories or geodesics on the 3D surface of the depth images with the goal of training a neural network to classify these trajectories as being within or crossing two ground truth instances, so that the network has the potential to learn an implicit 3D model of a single object instance within its parameters, even if trained using only a single geodesic trajectory. For a depth image with n pixels, n(n-1) / 2 such geodesics are potentially possible, which, if used judiciously, can provide a significantly larger dataset to train on. Our concept is to exploit this insight towards instance segmentation when the number of annotated images is very small. Specifically, our algorithm has the following steps: (i) For two randomly selected points on the depth image, we compute a surface geodesic, which is the projection onto the depth image of the straight line connecting those two points on the 2D RGB image grid. Assuming that our assumed object is convex (and the camera plane is assumed orthogonal to the object), this projection will be (approximately) the shortest path connecting two points of the object's depth on the surface, and thus will be the depth geodesic. (ii) We discretize this geodesic into a predefined (fixed) set of bins, where each bin will hold the value of the geodesic depth at that bin location. The bins are equally spaced on the straight line onto which the geodesic is projected. (iii) We then label the geodesic with the provided ground truth segments.Specifically, we give a discretized geodesic vector a label of 1 if the two ends of the geodesic belong to the same object instance, and zero otherwise. (iv) We train a neural network classifier on these discretized vectors and their labels.

[0027] At test time, given a depth image, we first select a seed location in the depth image to start the segmentation process. In a robot bin picking application, it is usually easier for the robot to pick the instance that is located on the top (i.e., closest to the robot). In some other cases, the most isolated instance may be preferred. We propose various heuristics to compute this initial seed. We then compute geodesics from this seed point to random spatial locations within a predefined radius around the seed point. We discretize these geodesics and use a pre-trained classifier to classify each geodesic as belonging to the same instance or not. For all points classified as being in the same instance, we compute the convex hull of these points and consider all pixels within this hull to correspond to the same instance, thus achieving instance segmentation. To create segmentations for multiple instances, we select another seed point from the depth image that is outside a predefined proximity of already segmented instances and repeat the above process until we obtain a suitable number of object instances for the task. Proposed method

[0028]

number

[0029]

number

[0030] We also refer to the geodesic curves as z-geodesic curves in the following description.To derive our method, we make the following assumptions in our problem formulation.

[0031] Assumption 1 (surface convexity)

[0032] We assume that the objects used in our setting are convex and that the depth patches associated with the instances form an approximately convex smooth surface.

[0033]

number

[0034] Proposition 1

[0035]

number

[0036] Assumption 2 (orthogonal projection)

[0037]

number

[0038]

number

[0039] Assumption 3 (static posture)

[0040] Furthermore, we assume that the camera position and pose, as well as the bins, are stationary when capturing all depth images.

[0041] We also assume that there are one or more instances of an object in a bin in the training images, and that all instances are of the same object. We make no assumptions on either the placement of the instances in the bins or the number of instances in the bins. We also assume that the ground truth annotations are reasonably accurate, and that at least one instance in each training image is associated with a ground truth annotation. Although one may have access to RGB images of the bins along with the depth images, our approach described in this work does not use these RGB images. Furthermore, our approach can also be easily extended to work with depth point clouds instead of depth images. In this case, the geodesic approximation using assumption 2 may not be directly applicable, since the XY points may no longer be described by a fixed image grid. Discriminative Shape Modeling

[0042] 2(A) and 2(B) show a depth image containing multiple instances of an object, and a surface curve between two arbitrary points p1 and p2. FIG. 2(B) plots this curve as a one-dimensional curve g(t) for varying t, where the X-axis of the plot shows t varying from t=0 corresponding to point p1 to t=1 corresponding to point p2 in equal steps of 100, and the Y-axis shows the geodesic depth in the depth image for the corresponding value of t. Our key insight for developing our approach is that if the two ends of this curve belong to different instances, then this curve will be non-convex or non-smooth at the points where the instances overlap. Our approach attempts to exploit this insight to learn an implicit object shape using a neural network. In the following, we explain each step in our setup in detail. Before proceeding, we need to discuss some practical choices that need to be considered from a computational and implementation point of view.

[0043] Geodesic discretization: From a practical perspective, directly applying the concept of using geodesics for instance segmentation is problematic since it would then be necessary to implicitly parameterize surface geodesics as continuous curves, which can be difficult for arbitrary curves and objects where there may not be any analytical form for such curves (e.g., surface geodesics on chicken nuggets?). Instead, to keep it computationally cheap, we discretize the curve using a fixed number of bins.

number

[0044] Instance Supervision: If discontinuities or non-convexity of surface geodesics are sufficient to find instance boundaries, why do we need instance annotations? This is because if two instances are very close, the discretization step above may skip discontinuities in the curve. For example, consider two cubes touching each other. In the depth image, where the cubes touch, the curve may be nearly continuous and discontinuities may be skipped by the discretization step. A similar problem may arise when there is noise in the depth image, where standard denoising and hole-filling algorithms may smooth the depth image and ground truth geodesic discontinuities may be suppressed. To avoid these problems, the method assumes that we have access to a ground truth instance mask. Instance Segmentation Training Pipeline

[0045] 3 shows an example of a training pipeline according to some embodiments of the invention for computing surface geodesics on a given training depth image and training a neural network. For a given training depth image D, the first step in our pipeline is to select a random set of M points 301 on the 2D image grid.

number

[0046]

number

[0047]

number

[0048] FIG. 4(A) shows an example illustrating selectable instances in a depth point cloud according to an embodiment of the present invention, and FIG. 4(B) shows a segment instance at the top of a bin in FIG. 4(A).

[0049] At test time, given a test depth image D, our goal is to repeat the process during the training phase for instance segmentation. Since our goal is ultimately to generate a segmentation for the instances in the bin that are likely to be most useful for a robotic arm to grasp and pick, we propose to segment the instances at the top of the bin (i.e., the instances closest to the camera) as shown in Figures 4(A) and 4(B), with the goal of first generating that instance segmentation mask. We call such instances pickable instances.

[0050] Figure 5 shows an example of an inference pipeline for segmenting an instance, according to some embodiments of the invention. First, we choose a seed point in the test depth image that corresponds to the highest point on a possible instance. We call this point H (501). We then use an approximate region around H within which the instance may be. We call the radius of this region the selection radius r. The region is a square box 502 of size 2r centered on H.

number

[0051] To create segmentations for different instances, we select another high point H' from the depth image such that the selection radius r around H' does not overlap with the selection radius around H. That is, we look for an instance whose depth geodesic does not overlap with an already segmented instance. Once we find point H', we apply the procedure described above. We do this process continuously, generating one instance segment at a time. Algorithm Extensions Debiasing the training set

[0052] As the astute reader may immediately notice, there is a difference between the way the geodesics are computed at training and testing time. The training samples in the above setting were randomly selected from the image grid, while the test samples are selected from the highest points in the depth map. Thus, in the latter, the initial dimension (closest to H) in the discretized geodesics v will have a tendency to rise (i.e., increase in depth), whereas in the training set this need not be the case, resulting in a bias in the training and testing distributions. To mitigate this issue, we sort the training points in ascending order of their depth using their depth values, and during training we always compute the geodesics from points with lower depth to points with higher depth. Systematic sampling of test geodesics

[0053] In the basic inference algorithm above, we randomly sampled the test points around the seed point. However, a more efficient approach would be to select the points systematically. To this end, we propose to define a circular region around the selection point H using a selection radius r, and then divide this region into equal sectors by dividing r into β equal parts and dividing the circle into ζ equal angles. This leads to βζ points to be considered to generate the surface geodesic, and these parameters can be adjusted depending on the shape of the base of the segment to be learned. Figure 6(A) shows an example illustrating the random selection of points around the selection point for computing the geodesic, and Figure 6(B) illustrates the systematic selection of points around the selection point for computing the geodesic, according to some embodiments of the present invention. Watershed Initialization a. So far, we have used randomly sampled points (albeit sorted) during the training phase. Such sampling does not distinguish between using easy geodesics versus hard geodesics for learning a classifier. For example, trivial discontinuities may be enough for a classifier to flag out-of-instance trajectories, but when such discontinuities do not occur, perhaps there are other subtle cues in the geodesics that the classifier should pay attention to? Such attention may be difficult to learn when they may be relatively less frequent in the flood of simple trajectories. To this end, we propose a hard-negative mining extension to our basic approach using the Watershed transform (WT). The Watershed algorithm is a classical unsupervised method for image segmentation that uses the analogy of blocking water that pours from a mountain peak (interior to the instance) and flows towards a valley (i.e., a segment boundary). If we use a "dam" to block the valley (via characterizing the edges using the image Laplacian), the pixels where the water is trapped form a segment. The challenge in getting the above approach to work properly lies in choosing where to build the dam so that the captured water corresponds to the ground truth segments. b. In WT, the points to start region growing (i.e., the location of pouring water) are found using a distance transform. That is, first, the distance transform is calculated on the image to find the regions where peaks (corresponding to the points farthest from the edge) exist, and then these points are selected for region growing. There are two advantages to using this concept in our setting: (i) points isolated from other instances may have such peaks, and such isolated instances may be useful for robot selection, as well as instances corresponding to the highest depth points; and (ii) no matter where there is clutter, i.e., where the peaks are higher (because water cannot be blocked by edges due to discontinuous / broken edges), the segments generated by WT may correspond to multiple instances that are wrongly segmented as a single instance by WT, and therefore our geodesic trajectory-based scheme may be useful to better correct with the provided supervision. Therefore, we propose to improve the selection of seed points for constructing geodesic trajectories via selecting peaks generated by the WT, and constraining the end points of the geodesic to within the segmentation mask generated by the WT for its respective peak point. Figures 7A, 7B, 7C, and 7D show examples of the depth map and Watershed transform based steps for generating instance segmentations according to an embodiment of the present invention.

[0054] experiment a. In this section, we provide experiments that demonstrate the empirical performance of our method for the task of instance segmentation. For this empirical study, we used a dataset consisting of several pieces of chicken nuggets in a bin. The images were of HD quality, but in our experiments we resized them to 320x240. We used only a single annotated depth image to train our setup, while the test set consisted of 17 images. The depth images were created with an Ensenso camera. For our systematic sampling of endpoints, we used k=14, and the number of angles depended on the selection radius (i.e., β=2πr / 3). The selection radius is chosen depending on the size of the object to be segmented, for example from the average radius of the instances in the provided ground truth segmentation. b. Neural Network: We used a discretization of the geodesic trajectory with 50 bins, therefore we used m=51 bins. Our neural network consists of 5 MLPs with respective output dimensions m,5m,m,m / 2,2, and using ReLU activation, and used Adam with default learning rate and other settings for optimization. We also experimented with other nonlinear classifier models (such as nonlinear SVM) for the proposed approach and present the results shortly. c. Evaluation: To evaluate our method, we sampled 1000 points from the ground truth and predicted instance segments and calculated the F1 score over this overlap, regardless of whether the classifier correctly predicted these samples. We calculate the performance for predicting various numbers of instance segments in depth images. One caveat of our sequential method of predicting segmentations is that sometimes the method will not return the required number of segmentations, since some instances partially overlap with other instances and excluding instances using a selection radius will not be able to find these overlapping instances. Therefore, we only evaluate for instances where selection points were identified.

[0055] experiment

[0056] Figure 8 shows a comparison of the results obtained by our and other approaches on the Fried Chicken dataset using neural networks against other alternatives such as linear SVM, non-linear SVM using RBF kernel, and geodesic K-means which directly classifies the discretized trajectories using K-means clustering. The depth images consisted of multiple instances of chicken nuggets. The number of instances varied from 3 to 17. In this experiment, we used 81K positive trajectories (i.e., those with label 1) and 30K negative trajectories by sampling the images using the algorithmic improvements described in the previous section. Appropriate weights were used during the training process to take into account the imbalance in the dataset. As is evident from the table, using our proposed neural variant performs significantly better than other alternatives.

[0057] Computational performance: Since our scheme consists of elementary calculations on images and depth maps, and trajectories are discretized into small dimensional vectors, our method is computationally very efficient, taking about 5 minutes to train on a 4-core CPU with 100K trajectories and about 0.05 seconds to segment instances during inference.

[0058] Versatility

[0059] FIG. 9 shows examples of grayscale images of four object classes used to create the dataset (depth images) in the experiments. Each depth image consists of multiple instances of the same object, but with pose variation, self-occlusion between the instances. The object classes correspond to objects of various shapes, sizes, and number of instances. The objects used are (i) different categories of chicken nuggets, (ii) fried foods, (iii) sliced ​​carrots, and (iv) taro. The chicken nuggets have several instances of fried chicken nuggets, there may be slight shape variations, and the shapes are not necessarily convex. The fried foods class has only a few instances, but those instances are larger than the other shapes, and are elliptical with large major axis and small minor axis, changing the geodesic in different directions. The sliced ​​carrot and taro classes have small objects, and the taro class has an approximately spherical shape, while the sliced ​​carrots are pyramidal. These classes exhibit significant inter-object and self-occlusion, which makes segmenting the instances in these classes difficult.

[0060] Figure 10 is an example showing segmentation results using the dataset of Figure 9 according to some embodiments of the present invention. The table in Figure 9 reports precision, recall, and F1 accuracy scores for segmenting single and multiple instances of objects in each of the categories listed in Figure 9 using the method of the present invention. The results show that the method is approximately 80-90% accurate in segmenting the instances.

[0061] Qualitative Segmentation Results

[0062] 11A, 11B, 11C, and 11D show some examples of segmentations generated by our setup for the instances of chopped carrots, chicken nuggets, taro, and fried foods. 11A, 11B, 11C, and 11D show a selectable instance selected for segmentation, which is highlighted and its highest point is marked as "x". A square region is selected around point "x", as indicated by the white square. A depth geodesic is calculated from the highest point "x" to a systematic set of points around "x" and inside the square. Points classified as belonging to the same instance are marked by small black circles. A convex hull is calculated for all these black circle points, which form the instance segmentation. Once a segment is found, it is removed from the depth image and another point "x" that is the highest point in the remaining point cloud and a region around it that does not overlap with any previously selected region by more than 50% is selected and the segmentation process is repeated.

[0063] FIG. 12 shows a hardware diagram illustrating some components that may be incorporated into the imaging controller 1200 of the present disclosure, according to some embodiments of the present disclosure. The imaging controller 1200 may include a processor 1220 configured to execute stored instructions and a memory 1240 storing instructions executable by the processor. The processor 1220 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1240 may include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. The processor 1220 is connected to one or more input / output devices via a bus 1206. The imaging controller 1200 may also include a storage device 1230 adapted to store a first module (pre-trained neural network 1231), a second module (seed point selection program module) 1232, and a third module (geodesic program module) 1233, for storing executable instructions for the processor 1220. In some cases, modules 1231, 1232 and 1233 may include the architecture of the pre-trained image segmentation neural network 505 or 304 shown in FIG. 5 or FIG. 3, a module for selecting seed points 301 shown in FIG. 5 or FIG. 3, and a module for creating depth geodesics 303, 504 in FIG. 3 and FIG. 5. The storage device 1230 may store data in data storage 1234. The storage device 1230 may be realized using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0064] The imaging controller 1200 may include an input interface for receiving external data 1295 including depth images from an imaging device including a camera or a set of training data sets. The input interface may include a human machine interface 1210 connecting the processor 1220 to a keyboard / measuring device 1211 and a pointing device 1212 within the imaging controller 1200, which may include a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, or a touch screen, among others. Alternatively, the input interface may include a network interface controller 1250 adapted to connect the imaging controller 1200 to a network 1290 through the bus 1206. Through the network 1290, the external data 1295 may be downloaded and stored in the storage system 1230 as training data and / or operational data 1234 for storage and / or further processing.

[0065] 12, the imaging controller 1200 can include an output interface linked through a bus 1206 and a display interface 1260 adapted to connect the imaging controller 1200 to a display device 1265, where the display device 1265 can include, among others, a computer monitor, a camera, a television, a projector, or a mobile device. For example, the imaging controller 1200 can be connected to a system interface 1270 adapted to connect a controlled system (robot controller) 1275, or to a different system 1275 to be controlled based on some aspect. Additionally or alternatively, the imaging controller 1200 can be connected to an application interface 1280 via a bus adapted to connect the imaging controller 1200 to an imaging device / camera (depth imaging camera) 1285 of the robot 150 of FIG. 13. In this case, the imaging controller 1200 is configured to obtain a depth image of an object to be manipulated by the robot 150 from the imaging device / camera 106 (FIG. 13), segment pickable instances among the instances of the object in the depth image, and generate geometric features of the pickable instances to a controller (manipulator controller) of the robot 150. In some cases, the geometric features generated from the segmented instances by the imaging controller 1200 may include an outer size of the instance, a pose of the instance, a shape of the instance, etc., such that the robot 150 picks the corresponding pickable object from among the objects in a bin or on a worktable.

[0066] FIG. 13 shows a robot configuration 100 of a robot 150 according to some embodiments. The robot 150 includes a controller and a robot arm 101 for performing an instance pick operation. The robot arm 101 includes a wrist 102 for ensuring multiple degrees of freedom to move an object. In some implementations, the wrist 102 is provided with an end tool 104 for holding an object 103 and / or for performing any other robot operation, such as a pick operation. The end tool 104 may be, for example, a gripper or other pick tool, such as a vacuum tool. In the following, "end tool" may represent "gripper" or other pick tool, such as a vacuum tool, or they may be used interchangeably. According to an embodiment, the objective of the pick operation is to pick a single instance of an object, typically the instance closest to the gripper.

[0067] The robot 150 is configured to perform a picking operation, e.g., pick a segmented object instance 103, along a trajectory while using an imaging device 106 connected to an imaging controller 1200, which can provide a depth image of an object to be manipulated by the robot arm 101. The imaging controller 1200 is connected to the controller of the robot 150 so that the controller of the robot 150 obtains and uses features of the segmented instance from the imaging controller 1200. Here, the trajectory corresponds to a path that defines the movement of the object 103 held by the gripper 104 to perform the picking operation. In a simple case, the trajectory may define only the vertical movement of the wrist 102. However, since the wrist 102 includes multiple degrees of freedom, the trajectory may include a motion profile that spans a multi-dimensional space.

[0068] The pose of an object refers to a combination of the object's position and orientation. The gripper 104 is movable in a start pose 111. The pose of the gripper 104 corresponding to the start pose 111 is called the start pose of the gripper 104. According to an embodiment, the objective of the picking operation is to pick a segmented instance object 103. The pose 115 of the object 112 may refer to the position and / or orientation of the object 112. The robot 150 is configured to move the gripper 104 along the trajectory 113 to pick the object 103 in a pose 114. The pose 114 of the object 103 of the object 112 is called the target pose. The pose of the gripper 104 corresponding to the target pose is called the target pose of the gripper 104.

[0069] The target pose of the gripper 104 is determined based on the position of the object 112. At the end of the successful execution of the picking operation, the pose of the gripper 104 of the robot arm 101 is considered to have reached the target pose of the gripper 104. Thus, achieving the target pose of the gripper 104 is equivalent to the successful execution of the picking operation. According to an embodiment, the trajectory 113 is defined according to the starting and target poses of the gripper 104 and the pose 115 of the object 112. Moreover, such a picking operation may be repeatedly executed by the robot 150.

[0070] FIG. 14 is a schematic diagram illustrating, by way of non-limiting example, a computing device 1400 that can be used to implement some techniques of the method and system according to an embodiment of the present disclosure. The computing device or device 1400 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing device 1400 can include a power supply 1408, a processor 1409, a memory 1410, and a storage device 1411, all connected to a bus 1450. In some cases, the storage device 1411 can include an architecture of a trained neural network 505 (304, 1231) based on the training pipeline shown in FIG. 3. Furthermore, the architecture of the trained neural network can be obtained, via a network, from an external data 1295 that stores the trained neural network. Additionally, a high-speed interface 1412, a low-speed interface 1413, a high-speed expansion port 1414, and a low-speed expansion port 1415 are connected to the bus 1450. In addition, a low speed connection port 1416 is connected to the bus 1450 .

[0071] Depending on the particular application, various component configurations are contemplated, by way of non-limiting example 1430, that may be mounted on a common motherboard. Additionally, input interface 1417 may be connected to external receiver 1406 and output interface 1418 via bus 1450. Receiver 1419 may be connected to external transmitter 1407 and transmitter 1420 via bus 1450. External memory 1404, external sensors 1403, machine 1402, and environment 1401 may also be connected to bus 1450. Additionally, one or more external input / output devices 1405 may be connected to bus 1450. Network interface controller (NIC) 1421 may be adapted to connect to network 1422 via bus 1450, and, among other things, data or other data may be rendered on a third-party display device, a third-party imaging device, and / or a third-party printing device external to computing device 1400.

[0072] 14, it is also contemplated that the memory 1410 can store instructions executable by the computer device 1400, historical data, and any data that can be utilized by the methods and systems of the present disclosure. The memory 1410 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. The memory 1410 can be one or more volatile memory units and / or one or more non-volatile memory units. The memory 1410 can also be another form of computer-readable medium, such as a magnetic or optical disk.

[0073] 14, the storage device 1411 may be adapted to store auxiliary data and / or software modules used by the computer device 1400. For example, the storage device 1411 may store historical data and other relevant data as described above with respect to the present disclosure. Additionally or alternatively, the storage device 1411 may store historical data similar to that described above with respect to the present disclosure. The storage device 1411 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Additionally, the storage device 1411 may include a computer readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, an array of devices including a flash memory or other similar solid state memory device, or a device in a storage area network or other configuration, and the like. The instructions may be stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., the processor 1409), perform one or more methods, such as those described above. The system can be linked via a bus 1450, optionally to a display or user interface (HMI) 1423 adapted to connect the system to a display device 1425 and a keyboard 1424, which can include, among others, a computer monitor, a camera, a television, a projector, or a mobile device.

[0074] 14, the computing device 1400 can include a user input interface 1417 adapted to a printer interface (not shown), which can also be connected via a bus 1450 and adapted to connect to a printing device (not shown), which can include a liquid inkjet printer, a solid ink printer, a large scale commercial printer, a thermal printer, a UV printer, or a dye sublimation printer, among others. The high-speed interface 1412 manages bandwidth-intensive operations for the computing device 1400, while the low-speed interface 1413 manages less bandwidth-intensive operations. Such an allocation of functions is merely an example. In some implementations, the high-speed interface 1412 can be coupled to the memory 1410, the user interface (HMI) 1423, the keyboard 1424 and the display 1425 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1414 that can accept various expansion cards (not shown) via the bus 1450. In this embodiment, low speed interface 1413 is coupled to storage device 1411 and low speed expansion port 1415 via bus 1450. Low speed expansion port 1415, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices 1405 and other devices including a keyboard 1424, a pointing device (not shown), a scanner (not shown), or a networking device such as a switch or router, for example via a network adapter.

[0075] 14, computing device 1400 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 1426, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer, such as a laptop computer 1427. It may also be implemented as part of a rack server system 1428. Alternatively, components from computing device 1400 may be combined with other components, such as the embodiment of FIG. 13. Each such device may include one or more of computing devices 1300 and device 1400, and the entire system may be composed of multiple computing devices in communication with each other.

[0076] Although the robot system described above illustrates a robot that can manipulate and assemble parts of a product, it can be applied to a robot that can be applied when many foods in a food processing plant are irregular shaped objects (such as cut vegetables, fried chicken, etc.). A robot system including a system for generating realistic images from real depth images and automatically segmenting multiple instances of rigid objects in depth images can be applied to food processing plants, automating industrial robots that can manipulate food. Furthermore, the above-mentioned robot system can be applied to a segmentation (method) system for food recognition. Segmentation is one of the most popular and important problems in image processing. For application in food processing plants, it is essential to have high segmentation accuracy and short both training and computation time.

[0077] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the claims.

[0078] Although the disclosure has been described with reference to certain preferred embodiments, it will be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the disclosure.

Claims

1. 1. An imaging controller for segmenting an instance from a depth image that includes an object to be manipulated by a robot, comprising: an input interface configured to receive a depth image including the object; a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image; a processor coupled to the memory and configured to execute the instructions to segment selectable instances using the trained neural network, the instructions comprising: selecting the highest point in the depth image; defining a region with a shape surrounding said highest point; sampling points in the region of the depth image; calculating a depth geodesic curve between the highest point and the sampled point; submitting the depth geodesic curves to the neural network to segment the selectable instances among the instances of the object in the depth image; and an output interface configured to output geometric characteristics of the pickable instances to a manipulator controller of the robot.

2. The imaging controller of claim 1 , wherein the depth image is acquired by a camera or a sensor.

3. The steps further include: selecting a next highest point from the depth image, a selection radius around the next highest point excluding overlap with the selectable instance, said step further comprising: defining a next region using the shape, the next region encompassing the next highest point, the step further comprising: sampling a next point in the next region of the depth image; calculating a next depth geodesic curve between the next highest point and the next sampled point; and submitting the depth geodesic curve to the neural network to segment a next selectable one of the instances of the object in the depth image.

4. The imaging controller of claim 3 , wherein the steps of selecting, defining, sampling, calculating, and submitting are continued until the steps have been performed for the remainder of the object in the depth image.

5. The imaging controller of claim 1 , wherein the certain shape is a square, a rectangle, a triangle, a circle, or an ellipse.

6. The imaging controller of claim 1 , wherein the neural network is trained to classify each feature vector as belonging to a same instance or a different instance.

7. The imaging controller of claim 1 , wherein the geodesic curve endpoints are initialized using peaks generated by a Watershed algorithm.

8. 1. A computer-implemented method for training a neural network to segment instances in a depth image, the method using a processor in conjunction with stored instructions implementing the method, the instructions, when executed by the processor, perform steps of the method, the steps including: selecting a depth image from a set of depth images; determining points at xy locations on a two-dimensional image grid and corresponding depth points for the selected depth image, the points on the two-dimensional image grid being annotated with ground truth instance labels, the step further comprising: calculating a geodesic line between the pair of annotated and determined points; generating a depth geodesic curve by projecting the geodesic line onto the depth image; and discretizing each of the depth geodesic curves to create discretized vectors, each discretized vector corresponding to one of the depth geodesic curves between the annotated determined pair of points, said step further comprising: A computer-implemented method comprising the step of submitting the discretized vector and corresponding annotated labels of the discretized vector to the neural network, wherein the steps from the selecting step to the providing step are performed iteratively until all remaining parts of the set of depth images have been used.

9. The method of claim 8 , wherein the neural network is trained to classify each feature vector as belonging to the same instance or to a different instance.

10. The method of claim 9 , further comprising the step of computing the convex hull of all end points of the geodesic curves classified as the geodesic curves belonging to the same instance as possible pickable points.

11. The method of claim 8 , wherein the generated depth geodesic curves are debiased such that the geodesic curves are created from points of lower depth to points of higher depth.

12. The method of claim 8 , wherein the geodesic curve endpoints are initialized using peaks generated by a Watershed algorithm.

13. The method of claim 12 , wherein the endpoints of the geodesic curve are determined using a systematic selection method.

14. 1. A bin picking system for picking objects from a bin, comprising: an end tool configured to select an object from the objects; and a robotic arm including the end tool, the robotic arm being configured to be driven by control signals including an end tool signal to pick the object from the bin using the end tool, the bin picking system further comprising: an interface configured to transmit and receive the control signal, a sensor signal of a sensor disposed on the robot arm, and an imaging signal of at least one imaging device; and a memory configured to store robot control program instructions, a classifier, and a trained neural network that segments instances from the objects in the depth image, the trained neural network being trained by the computer-implemented method of claim 8, the bin picking system further comprising: a processor coupled to the memory and configured to execute the instructions to use the trained neural network to segment selectable instances and generate the control signals to drive the robotic arm and the end tool, the instructions comprising: selecting the highest point in the depth image; defining a region with a shape surrounding said highest point; sampling points in the region of the depth image; calculating a depth geodesic curve between the highest point and the sampled point; submitting the depth geodesic curves to the neural network to segment the selectable instances among the instances of the object in the depth image; generating geometric characteristics of the selectable instances and the control signal based on the imaging signal; and transmitting the generated geometric features and the generated control signals to the robotic arm to cause the end tool to pick the object corresponding to the pickable instance from the bin using the end tool.

15. The bin picking system of claim 14 , wherein the depth image is acquired by a camera or sensor.

16. The steps further include: selecting a next highest point from the depth image, a selection radius around the next highest point excluding overlap with the selectable instance, said step further comprising: defining a next region using the shape, the next region encompassing the next highest point, the step further comprising: sampling a next point in the next region of the depth image; calculating a next depth geodesic curve between the next highest point and the next sampled point; and submitting the depth geodesic curve to the neural network to segment a next selectable instance of the instances of the object in the depth image.

17. 17. The bin selection system of claim 16, wherein the steps of selecting, defining, sampling, calculating, and submitting are continued until the steps have been performed for the remainder of the object in the depth image.

18. 15. The bin picking system of claim 14, wherein the certain shape of the certain region is a square, a rectangle, a triangle, a circle, or an oval.

19. The bin selection system of claim 14 , wherein the neural network is trained to classify each feature vector as belonging to the same instance or a different instance.

20. The bin selection system of claim 14 , wherein the geodesic curve endpoints are initialized using peaks generated by a Watershed algorithm.

Images