Discriminative 3D Shape Modeling for Few-Shot Instance Segmentation

Abstract

An imaging controller for segmenting an instance from a depth image containing an object to be manipulated by a robot is provided. The imaging controller includes an input interface configured to receive a depth image containing the object, a memory configured to store instructions and a neural network trained to segment the instance from the object in the depth image, and a processor coupled to the memory and configured to execute the instructions to segment the pickable instance using the trained neural network. The instructions include selecting a highest point in the depth image, defining a region with a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled point, submitting the depth geodesic curve to the neural network to segment the pickable instance among the instances of the object in the depth image, and an output interface configured to output geometric features of the pickable instance to a manipulator controller of the robot.

Description

[Technical Field]

[0001] This disclosure generally relates to automatically segmenting multiple instances of objects in depth images, where a segmentation model is trained in a few-shot setting using only a few annotated training examples. [Background technology]

[0002] Instance segmentation is an essential step in various autonomous tasks. Some examples include (i) a robotic arm that needs to pick similar products moving on a conveyor belt in a manufacturing environment, (ii) a supermarket robot that needs to pick and place fruit from a bin, (iii) a library assistant robot that needs to pick books from a box and hand them over to a person, (iv) identifying instances of cars on the road in a self-driving environment, or (v) identifying instances of body tissues in medical images for automated diagnosis.

[0003] Standard deep learning solutions, such as Mask-RCNN and its variants, for solving such instance segmentation tasks typically require large training sets to train the underlying neural network. However, creating such training sets and annotating the training examples can be expensive and involve significant physical effort. Therefore, creating such datasets may not be feasible in many situations. The instance segmentation task can also be viewed as one of applying standard clustering schemes to depth image point clouds. For example, segmentation can be derived from applying K-means, Gaussian mixture modeling, or spectral clustering to the point cloud. However, such clustering algorithms typically make assumptions on the point cloud distribution and therefore may not be flexible enough to segment real-world objects whose shapes may vary. There are also recent approaches, such as InSeGAN, that can generate instance segmentations in an unsupervised setting. However, InSeGAN requires a large unlabeled training set with diversity in instance placements, which can also be difficult to obtain in real-world conditions.

[0004] Therefore, there is a need to design an approach for instance segmentation that can automatically segment multiple instances of an object in a depth image and requires very few training examples to train the segmentation model. Summary of the Invention

[0005] This disclosure generally relates to automatically segmenting multiple instances of objects in depth images, where a segmentation model is trained in a few-shot setting using only a few annotated training examples.

[0006] Some embodiments of the present disclosure provide methods for segmenting object instances in depth images. In contrast to conventional methods that typically use deep neural networks for segmentation, the present disclosure includes a custom-built instance segmentation pipeline that does not require any 3D CAD models of the objects, but instead requires only a few annotated examples of object instances in a small number of depth images.

[0007] Embodiments of the present disclosure are configured for automatically segmenting multiple instances of nearly convex 3D objects in depth images in a few-shot setting, a novel approach in the computer vision domain. Embodiments of the present disclosure can be configured to perform for several applications, including robotic assistance tasks, such as picking an object from a bin containing multiple instances where the robot needs to accurately segment the instances to plan a grasp.

[0008] Some embodiments of the present disclosure include an architecture for instance segmentation that conforms to some aspects of deep neural networks, whereas standard neural networks require a large training set of annotated instances in depth images for training. A key insight in this disclosure is that the instances are not used directly to train the neural network, but instead, the neural network is trained to classify the depth of surface geodesics connecting pairs of three-dimensional points in the depth image. Because there are quadratically many possible pairs of such points relative to the number of pixels in the depth image, a small number of annotated instances can generate a large set of annotated surface geodesics, which can then be used to train the neural network. The method first selects a random sample of spatial locations in a given training depth image and calculates pairwise surface geodesics between each and every pair.

[0009] At least one realization from the experiments is that the surface geodesics calculated between pairs of depth points can be of arbitrary length and therefore can be difficult to use to train neural networks that expect fixed-size inputs. To this end, the presented model discretizes each geodesic into a fixed-dimensional vector. In this disclosure, each geodesic is divided into equal segments, and each dimension of the discretized geodesic vector is filled with the depth of the pixel at the end point of the respective geodesic segment.

[0010] Another realization for this disclosure comes from the theoretical insight that for any two points on a convex surface, the surface geodesic connecting these two points (which is the shortest curve) will have a convex shape. However, if the two points belong to different instances and each instance is convex, the surface geodesic connecting the two points will have a non-convex shape or discontinuity at the depth point where the two object surfaces intersect. These two types of curves (convex and non-convex) are used to train a neural network for classification purposes. Once sufficiently trained, the neural network will classify each geodesic as either entirely within an instance segment (convex geodesic) or intersecting across multiple instances (non-convex geodesic).

[0011] Experiments have shown that discretizing surface geodesics can miss convex / non-convex discontinuities in the geodesics when the two instances for which the curves are computed are very close. For example, consider two cubes touching each other. In a depth image, the curve may be nearly continuous where the cubes touch, and the discontinuity may be skipped by the discretization step if the dimensionality of the fixed-dimensional geodesic vector is not large enough to capture the discontinuity. A similar problem can arise when there is noise in the depth image; as a result, standard denoising and hole-filling algorithms typically used to smooth depth images can over-smooth the boundaries of instance segments, thus suppressing ground-truth geodesic discontinuities. To avoid this problem, training annotations for instances are developed. The annotations guide the neural network for classification even when the discretized geodesic vectors suppress object-crossing discontinuities, thereby requiring the neural network to learn other surface features that characterize the surface of the 3D object.

[0012] Another finding from experiments is that randomly selecting pairs of points from a depth image to construct geodesics for training a neural network may be suboptimal. For example, there may be geodesics that are very easy for a neural network to learn to segment, but there may be others that are difficult to classify. Because easy examples can sometimes significantly outnumber difficult examples, neural network training may be biased and may not produce correct segmentations in difficult segmentation cases. Another case is when the size of the object to be segmented is small, but there are many instances in a given depth image. This results in a large set of geodesics that span multiple instances, but only a few that lie entirely within the instances. To solve these problems, this disclosure proposes using a Watershed algorithm (WA) to create an initial segmentation of a depth image. The WA method generates two outputs: (i) a set of peaks in the depth image that are points that are believed to correspond to the centroids of the instances, and (ii) a segmentation for each instance. However, because the WA method is unsupervised, the peaks or segmentations it generates may be inaccurate. A key insight in this disclosure is to improve WA segmentation using ground truth training annotations and a geodesic classification neural network. More specifically, one end of a geodesic is obtained from a peak generated by WA, and the other end of the geodesic is constrained to be within the respective segment generated by WA for the corresponding selected peak. The geodesic is then labeled using ground truth annotations as to whether it belongs to the exact same instance. If WA makes a mistake in segmenting an instance, the ground truth instance labeling of the geodesic allows the neural network to correct this error, leading to better instance segmentation.At least one benefit of selecting geodesics from within the instances generated by WA is that it allows for the creation of a better training set for the neural network, with more geodesics belonging to the same instance and difficult examples generated by WA's incorrect segmentation. The geodesics belonging to the same instance allow for the neural network to learn a better implicit 3D model of the object shape, while the difficult examples allow for the learning of a discriminative network that performs well in cases of difficult segmentation.

[0013] For example, when using the trained model for segmentation, the architecture takes a depth image as input, on which depth geodesics are calculated between pairs of selected locations in the depth image. The locations are selected based on a predetermined practical criterion. One embodiment of this criterion is based on the subsequent task for which the segmentation will be used. For example, if the instance segmentation generated by this approach is used to pick instances with a robotic arm, it is useful if the top instance of the stack of instances is segmented first. Such instances that are desired to be segmented first are called pickable instances.

[0014] This disclosure proposes selecting pairs of spatial locations on a depth image to calculate depth geodesics on an input test image, where the selection is based on selectable instances in the given test depth image. Depth geodesics are calculated between all pairs of selected locations. One end of each geodesic is fixed to the highest depth point on the selectable instance. This point is called the seed point. The other end of the geodesic is selected from a region within a predefined radius around the highest depth point. Each geodesic is discretized as described in the training phase. A pre-trained neural network then classifies each geodesic as belonging entirely to one instance surface or spanning multiple instances. The convex hull of all points classified as belonging to the same instance is then used to define the instance segment. This process is repeated for various selected locations in the image to generate multiple such instance segmentations.

[0015] A finding from experiments for segmenting a given test image is that randomly selecting depth points from around a seed point may not be computationally efficient because it may not cover the instance in all directions. This insight suggests a systematic selection of points around the seed point. An embodiment of this concept in this disclosure is to create a sample set of points along angularly equally spaced radial lines on concentric circles around the seed point. Such a selection allows for uniform coverage of a circular region around the seed point, so as to construct a geodesic curve.

[0016] A realization arising from the manner in which test geodesics are constructed is that these geodesics always point from higher depth points to lower points. Therefore, it may be inefficient to include geodesics in the training set that do not respect this property. To correct for this, the depth points used to train the neural network are first sorted in ascending order of their depth, and geodesics for training are calculated from points with lower depths (or highest points) to points with higher depths.

[0017] An embodiment of the present disclosure provides an imaging controller for segmenting instances from a depth image including an object to be manipulated by a robot. The imaging controller may include an input interface configured to receive a depth image including the object; a memory configured to store instructions and a neural network trained to segment instances from objects in the depth image; and a processor coupled to the memory and configured to execute the instructions to segment pickable instances using the trained neural network, where the instructions may include selecting a highest point among all instances in the depth image, defining a region using a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled point, submitting the depth geodesic curve to the neural network to segment pickable instances among instances of the object in the depth image, and an output interface configured to output geometric characteristics of the pickable instances to a manipulator controller of the robot.

[0018] Further, in accordance with some embodiments of the present invention, there is provided a computer-implemented method for training a neural network to segment instances in depth images. In this case, the method uses a processor coupled with stored instructions that implement the method, which, when executed by the processor, perform the steps of the method, including selecting a depth image from a set of depth images and determining points at x-y spatial locations on a two-dimensional image grid and corresponding depth points for the selected depth image, where the points on the two-dimensional image grid are each annotated with a ground truth instance label, and further including calculating geodesic lines between pairs of annotated, determined points and generating depth geodesic lines by projecting the geodesic lines onto the depth image and discretizing each of the depth geodesic lines to create discretized vectors, where each discretized vector corresponds to one of the depth geodesic lines between pairs of annotated, determined points, and further including submitting the discretized vectors and their corresponding annotated labels to a neural network, and the steps from selecting to providing are performed repeatedly until all remaining depth images have been used.

[0019] Further, according to some embodiments of the present invention, there is provided a bin picking system for picking objects from bins. The bin picking system may include an end tool configured to pick an object from among the objects; a robotic arm including the end tool, the robotic arm being driven by control signals including the end tool signal and configured to pick the object from the bin using the end tool; and a memory configured to store instructions for a robot control program, a classifier, and a trained neural network that segments instances from objects in a depth image, the trained neural network being trained by the computer-implemented method described above. The bin picking system may further include: an interface configured to transmit and receive the control signals, sensor signals from a sensor disposed on the robotic arm, and imaging signals from at least one imaging device; and a memory configured to store instructions for a robot control program, a classifier, and a trained neural network that segments instances from objects in a depth image, the trained neural network being trained by the computer-implemented method described above. The method may include a processor configured to segment the possible instances and generate control signals to drive the robot arm and end tool, the steps of the instructions including selecting the highest point in the depth image, defining a region using a shape to surround the highest point, sampling points in the region of the depth image, calculating a depth geodesic curve between the highest point and the sampled point, submitting the depth geodesic curve to a neural network to segment selectable instances among the instances of the object in the depth image, generating geometric features and control signals for the selectable instances based on the imaging signal, and sending the generated geometric features and generated control signals to the robot arm to cause the end tool to use the end tool to select an object corresponding to the selectable instance from a bin.

[0020] Thus, the present system and method as a whole cannot be viewed in the abstract as simply implementing aspects of the human mind, nor as gathering (collecting) data, processing data, analyzing data, and displaying particular results. In contrast, the present system and method can generalize to cases where annotation is difficult or operates on complex scenes with less cost and computational expense, and can be competitive and beneficial to participants in the computer vision industry and other industries.

[0021] The presently disclosed embodiments are further described with reference to the accompanying drawings, in which: The drawings shown are not necessarily to scale, with emphasis generally being placed upon illustrating the principles of embodiments of the present disclosure. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is an illustrative example of depth image segmentation of nearly identical chicken nugget instances within a bin, according to an embodiment of the present invention. [Figure 2] (A) shows an example illustrating a depth image containing multiple instances of an object according to an embodiment of the present invention, and (B) shows a surface curve between two arbitrary points p1 and p2 of Figure 2(A) according to an embodiment of the present invention. [Figure 3] 1 illustrates a pipeline for training a neural network to learn an implicit shape model of a single instance of an object using surface geodesics, according to an embodiment of the present invention. [Figure 4] 4A shows an example illustrating selectable instances in a depth point cloud according to an embodiment of the present invention, and FIG. 4B shows a segment instance at the top of the bin in FIG. 4A according to an embodiment of the present invention. [Figure 5] 1 illustrates an example inference pipeline for segmenting instances, according to an embodiment of the present invention. [Figure 6](A) shows an example illustrating random selection of points around a selected point for computing geodesic curves according to an embodiment of the present invention; (B) shows an example illustrating systematic selection of points around a selected point for computing geodesic curves according to an embodiment of the present invention. [Figure 7A] 1 illustrates an example of a Watershed transform-based step for generating depth maps and instance segmentation according to an embodiment of the present invention. [Figure 7B] 1 illustrates an example of a Watershed transform-based step for generating depth maps and instance segmentation according to an embodiment of the present invention. [Figure 7C] 1 illustrates an example of a Watershed transform-based step for generating depth maps and instance segmentation according to an embodiment of the present invention. [Figure 7D] 1 illustrates an example of a Watershed transform-based step for generating depth maps and instance segmentation according to an embodiment of the present invention. [Figure 8] 1 shows a comparison of results obtained by the present invention and other approaches on the Fried Chicken dataset. [Figure 9] Shows example images (depth images) of four object classes used to create the dataset in the experiments. [Figure 10] 10 is an example showing segmentation results using the dataset of FIG. 9 according to an embodiment of the present invention. [Figure 11A] 10 shows some examples of segmentations produced by our setup for instances of chopped carrots, chicken nuggets, taro, and fried foods, according to embodiments of the present invention. [Figure 11B] 10 shows some examples of segmentations produced by our setup for chopped carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 11C]10 shows some examples of segmentations produced by our setup for chopped carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 11D] 10 shows some examples of segmentations produced by our setup for chopped carrots, chicken nuggets, taro, and fried food instances, according to embodiments of the present invention. [Figure 12] FIG. 1 shows a hardware diagram illustrating some components that may be incorporated into some systems of the present disclosure, in accordance with some embodiments of the present invention. [Figure 13] 1 illustrates a robot configuration for a robot according to an embodiment of the present invention. [Figure 14] 1 is a schematic diagram illustrating, by way of non-limiting example, a computing device that may be used to implement some techniques according to embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0023] While the above-identified drawings set forth embodiments disclosed herein, other embodiments are contemplated, as noted in the present discussion. The present disclosure presents exemplary embodiments by way of representation and not limitation. Those skilled in the art may devise numerous other modifications and embodiments that fall within the scope and spirit of the principles of the disclosed embodiments.

[0024] Segmenting nearly identical object instances is a ubiquitous problem in various robotic bin picking applications. Some examples include (i) a robotic arm that needs to pick products moving on a conveyor belt in a manufacturing environment, (ii) a supermarket robot that needs to pick and place fruit from a bin, or (iii) a library assistance robot that needs to pick books from a box and hand them over to a person.

[0025] FIG. 1 is an illustrative example of depth image segmentation of nearly identical chicken nugget instances in a bin, according to some embodiments of the present invention. See FIG. 1 for an illustrative example of depth image segmentation of nearly identical chicken nugget instances in a bin. A standard deep learning solution for this task would require a large training set and significant annotation effort. This problem can also be considered as one of applying a standard clustering scheme to the depth points. For example, segmentation can be derived from applying, e.g., K-means, EM, or spectral clustering to the point cloud. However, such clustering algorithms typically make assumptions on the shape of the distribution or involve hyperparameters. For example, K-means assumes that the distribution is spherical around the cluster centroid or elliptical, as in EM. Spectral clustering requires specifying proximity hyperparameters to define the edges of the underlying graph Laplacian. There are also several recent approaches for unsupervised instance segmentation. However, they require large unlabeled training sets with diversity in instance placements, which can also be difficult to obtain in real-world conditions. For example, some methods require thousands of unlabeled depth images, which requires a robotic arm to shake a bin of objects, a task that is not scalable and sometimes infeasible if the object of interest is fragile or could be damaged by such shaking.

[0026] In this disclosure, we consider this problem of instance segmentation of nearly identical convex object instances in depth images in a few-shot setting, where we assume we have access to a limited set (fewer than five) of annotated depth images, each annotated with a small number of instances along with their segments. Our key concept is to create surface trajectories, or geodesics, on the 3D surface of the depth image with the goal of training a neural network to classify these trajectories as being within or intersecting two ground truth instances. Thus, even if trained using only a single geodesic trajectory, the network has the potential to learn an implicit 3D model of a single object instance within its parameters. For a depth image with n pixels, n(n-1) / 2 such geodesics are potentially possible, which, if used judiciously, can provide a significantly larger dataset for training. Our concept is to leverage this insight toward instance segmentation when the number of annotated images is very small. Specifically, our algorithm has the following steps: (i) For two randomly selected points on the depth image, we compute a surface geodesic, which is the projection onto the depth image of the line connecting those two points on the 2D RGB image grid. Given that our assumed object is convex (and the camera plane is assumed orthogonal to the object), this projection will be (approximately) the shortest path connecting the two points of the object's depth on the surface, and thus will be the depth geodesic. (ii) We discretize this geodesic into a predefined (fixed) set of bins, each of which will hold the geodesic depth value at that bin location. The bins are equally spaced along the line onto which the geodesic is projected. (iii) We then label the geodesic using the provided ground truth segments.Specifically, if the two ends of the geodesic belong to the same object instance, we give the discretized geodesic vector a label of 1, otherwise we give it a label of zero. (iv) We train a neural network classifier on these discrete vectors and their labels.

[0027] At test time, given a depth image, we first select a seed location in the depth image to begin the segmentation process. In robotic bin-picking applications, it is usually easier for the robot to pick the instance that is located at the top (i.e., closest to the robot). In some other cases, the most isolated instance may be preferred. We propose various heuristics to calculate this initial seed. We then calculate geodesics from this seed point to random spatial locations within a predefined radius around the seed point. We discretize these geodesics and use a pre-trained classifier to classify each geodesic as belonging to the same instance or not. For all points classified as being within the same instance, we calculate the convex hull of these points and consider all pixels within this hull to correspond to the same instance, thus achieving instance segmentation. To create segmentations for multiple instances, we select another seed point from the depth image that is outside the predefined proximity of already segmented instances and repeat the above process until we obtain a suitable number of object instances for the task. Suggested method

[0028]

number

[0029]

number

[0030] We also refer to geodesics as z-geodesics in the following description.To derive our method, we make the following assumptions in our problem formulation:

[0031] Assumption 1 (surface convexity)

[0032] We assume that the objects used in our setting are convex and that the depth patches associated with the instances form an approximately convex smooth surface.

[0033]

number

[0034] Proposition 1

[0035]

number

[0036] Assumption 2 (orthogonal projection)

[0037]

number

[0038]

number

[0039] Assumption 3 (static posture)

[0040] Furthermore, we assume that the camera position and pose, as well as the bins, are stationary when capturing all depth images.

[0041] We also assume that there are one or more instances of an object in a bin in the training images, and that all instances are of the same object. We make no assumptions on either the placement of instances in the bins or the number of instances in a bin. We also assume that the ground truth annotations are reasonably accurate and that at least one instance in each training image is associated with a ground truth annotation. While one may have access to RGB images of the bins along with the depth images, our approach described in this work does not use these RGB images. Furthermore, our approach can easily be extended to work with depth point clouds instead of depth images. In this case, geodesic approximations using Assumption 2 may not be directly applicable because the XY points may no longer be described by a fixed image grid. Discriminative Shape Modeling

[0042] Figures 2(A) and 2(B) show a depth image containing multiple instances of an object and a surface curve between two arbitrary points p1 and p2. Figure 2(B) plots this curve as a one-dimensional curve g(t) for varying t, where the X-axis of the plot represents t, which varies from t=0 corresponding to point p1 to t=1 corresponding to point p2 in equal steps of 100, and the Y-axis represents the geodesic depth in the depth image for the corresponding value of t. Our key insight for developing our approach is that if the two ends of this curve belong to different instances, the curve will be non-convex or non-smooth at the points where the instances overlap. Our approach attempts to leverage this insight to learn an implicit object shape using a neural network. Below, we describe each step in our setup in detail. Before proceeding, we need to discuss some practical choices that need to be considered from a computational and implementation perspective.

[0043] Geodesic Discretization: From a practical standpoint, directly applying the concept of using geodesics for instance segmentation is problematic because it would require implicitly parameterizing surface geodesics as continuous curves, which can be difficult for arbitrary curves and objects where there may not be any analytical form for such curves (e.g., surface geodesics on chicken nuggets?). Instead, to keep it computationally cheap, we discretize the curve using a fixed number of bins.

number

[0044] Instance Supervision: If discontinuities or non-convexity of surface geodesics are sufficient to find instance boundaries, why are instance annotations necessary? This is because the discretization step described above may skip discontinuities in the curves when two instances are very close. For example, consider two cubes touching each other. In the depth image, where the cubes touch, the curve may be nearly continuous, and discontinuities may be skipped by the discretization step. A similar problem can arise when there is noise in the depth image; standard denoising and hole-filling algorithms may smooth the depth image, suppressing ground truth geodesic discontinuities. To avoid these problems, the method assumes access to a ground truth instance mask. Instance Segmentation Training Pipeline

[0045] Figure 3 shows an example of a training pipeline according to some embodiments of the present invention for computing surface geodesics on a given training depth image and training a neural network. For a given training depth image D, the first step in our pipeline is to select a random set of M points 301 on the 2D image grid.

number

[0046]

number

[0047]

number

[0048] FIG. 4(A) shows an example illustrating selectable instances in a depth point cloud according to an embodiment of the present invention, and FIG. 4(B) shows a segment instance at the top of a bin in FIG. 4(A).

[0049] At test time, given a test depth image D, our goal is to repeat the process during the training phase for instance segmentation. Because our goal is ultimately to generate a segmentation for the instances in the bin that are likely to be most useful for a robotic arm to grasp and pick, we propose to segment the instances at the top of the bin (i.e., the instances closest to the camera) as shown in Figures 4(A) and 4(B) with the goal of first generating that instance segmentation mask. We refer to such instances as pickable instances.

[0050] Figure 5 shows an example inference pipeline for segmenting instances, according to some embodiments of the present invention. First, select a seed point in the test depth image that corresponds to the highest point on a possible instance. Call this point H (501). Next, use an approximate region around H within which the instance may lie. Call the radius of this region the selection radius r. The region is a square box 502 of size 2r centered at H.

number

[0051] To create segmentations for different instances, we select another high point H' from the depth image, such that the selection radius r around H' does not overlap with the selection radius around H. That is, we look for an instance whose depth geodesic does not overlap with an already segmented instance. Once we find point H', we apply the procedure described above. We perform this process continuously, generating one instance segment at a time. Algorithm Extensions Debiasing the training set

[0052] As the astute reader may immediately notice, there is a difference between the way geodesics are computed during training and testing. While training samples in the above setup were randomly selected from the image grid, test samples are selected from the highest points in the depth map. Thus, in the latter case, the initial dimension (closest to H) in the discretized geodesic v will tend to increase (i.e., depth increases), whereas this need not be the case for those in the training set, introducing bias into the training and test distributions. To alleviate this issue, we sort the training points in ascending order of their depth using their depth values, and during training, always compute geodesics from points with lower depth to points with higher depth. Systematic sampling of test geodesics

[0053] In the basic inference algorithm described above, we randomly sampled test points around the seed point. However, a more efficient approach would be to select points systematically. To this end, we propose defining a circular region around the selection point H using a selection radius r. We then divide this region into equal sectors by dividing r into β equal parts and dividing the circle into ζ equal angles. This results in βζ points to consider for generating the surface geodesic curves; these parameters can be adjusted depending on the underlying shape of the segment to be trained. Figure 6(A) shows an example illustrating the random selection of points around the selection point for computing geodesic curves, while Figure 6(B) illustrates the systematic selection of points around the selection point for computing geodesic curves, according to some embodiments of the present invention. Watershed Initialization a. So far, we have used randomly sampled points (albeit sorted) during the training phase. Such sampling does not distinguish between using easy geodesics versus hard ones for training a classifier. For example, trivial discontinuities may be enough for a classifier to flag out-of-instance trajectories, but if such discontinuities do not occur, perhaps there are other subtle cues in the geodesics that the classifier should pay attention to. Such attention can be difficult to learn when they may be relatively infrequent in the deluge of simple trajectories. To this end, we propose a hard-negative mining extension to our basic approach using the Watershed Transform (WT). The Watershed algorithm is a classic unsupervised method for image segmentation that uses the analogy of blocking water pouring from a mountain peak (interior of the instance) and flowing toward a valley (i.e., a segment boundary). When we use a "dam" to block a valley (via characterizing edges using the image Laplacian), the pixels where the water is trapped form a segment. The challenge to getting the above approach to work properly lies in choosing where to build the dam so that the captured water corresponds to the ground truth segments. b. In WT, the starting point for region growing (i.e., the location of pouring water) is found using a distance transform. That is, first, a distance transform is calculated on the image to find the region where the peak (corresponding to the point farthest from the edge) exists, and then these points are selected for region growing. Using this concept in our setting has two advantages: (i) points isolated from other instances may have such peaks, and such isolated instances may be useful for robot selection, as well as instances corresponding to the highest depth points; and (ii) no matter where there is clutter, i.e., where the peak is higher (because water cannot be blocked by edges due to discontinuous / broken edges), the segments generated by WT may correspond to multiple instances that are incorrectly segmented as a single instance by WT; therefore, our geodesic trajectory-based scheme may be useful to better correct with the provided supervision. Therefore, we propose to improve the selection of seed points for constructing geodesic trajectories through selecting peaks generated by the WT and constraining the endpoints of the geodesic trajectories within the segmentation masks generated by the WT for their respective peak points. Figures 7A, 7B, 7C, and 7D show examples of the depth map and Watershed transform-based steps for generating instance segmentations according to embodiments of the present invention.

[0054] experiment In this section, we provide experiments demonstrating the empirical performance of our method for the task of instance segmentation. For this empirical study, we used a dataset consisting of several pieces of chicken nuggets in a bin. The images were HD quality, but in our experiments, we resized them to 320 × 240. We used only a single annotated depth image to train our setup, while the test set consisted of 17 images. The depth images were created using an Ensenso camera. For our systematic sampling of endpoints, we used k = 14, and the number of angles depended on the selection radius (i.e., β = 2πr / 3). The selection radius was selected depending on the size of the object to be segmented, e.g., from the average radius of the instances in the provided ground truth segmentation. b. Neural Network: We used a discretization of the geodesic trajectory with 50 bins, therefore we used m = 51 bins. Our neural network consists of five MLPs with respective output dimensions m, 5m, m, m / 2, 2, using ReLU activation, and used Adam with default learning rate and other settings for optimization. We also experimented with other nonlinear classifier models (such as nonlinear SVM) for the proposed approach, and will present the results shortly. c. Evaluation: To evaluate our method, we sampled 1000 points from the ground truth and predicted instance segments and calculated the F1 score across this overlap, regardless of whether the classifier correctly predicted these samples. We calculate performance for predicting various numbers of instance segments in depth images. One caveat of our sequential method of predicting segmentations is that sometimes our method will not return the required number of segmentations because some instances partially overlap with other instances, and excluding instances using a selection radius will fail to find these overlapping instances. Therefore, we only evaluate for instances where selection points were identified.

[0055] experiment

[0056] Figure 8 shows a comparison of the results obtained by our and other approaches on the fried chicken dataset using neural networks against other options, such as linear SVM, nonlinear SVM using an RBF kernel, and geodesic k-means clustering, which directly classifies discretized trajectories using k-means clustering. Depth images consisted of multiple instances of chicken nuggets. The number of instances varied from 3 to 17. In this experiment, we used 81K positive trajectories (i.e., those with label 1) and 30K negative trajectories by sampling images using the algorithmic improvements described in the previous section. Appropriate weights were used during the training process to account for imbalances in the dataset. As is clear from the table, using our proposed neural variant performs significantly better than other alternatives.

[0057] Computational Performance: Because our scheme consists of basic calculations on images and depth maps, and trajectories are discretized into small-dimensional vectors, our method is computationally very efficient, taking about 5 minutes to train on a 4-core CPU with 100K trajectories and about 0.05 seconds to segment instances during inference.

[0058] Versatility

[0059] Figure 9 shows example grayscale images of four object classes used to create the dataset (depth images) in the experiment. Each depth image consists of multiple instances of the same object, but there are pose variations and self-occlusions between the instances. The object classes correspond to objects of various shapes, sizes, and instance numbers. The objects used are (i) different categories of chicken nuggets, (ii) fried foods, (iii) sliced ​​carrots, and (iv) taro. The chicken nuggets have several instances of fried chicken nuggets, and there may be slight shape variations, and the shapes are not necessarily convex. The fried foods class has only a few instances, but these instances are larger than the other shapes and are elliptical with large major and small minor axes, changing the geodesic curves in different directions. The sliced ​​carrot and taro classes have small objects; the taro class has an approximately spherical shape, while the sliced ​​carrots are pyramidal. These classes exhibit significant inter-object and self-occlusions, which makes segmenting instances in these classes difficult.

[0060] Figure 10 is an example showing segmentation results using the dataset of Figure 9, according to some embodiments of the present invention. The table in Figure 9 reports precision, recall, and F1 accuracy scores for segmenting single and multiple instances of objects in each of the categories listed in Figure 9 using the method of the present invention. The results show that the method is approximately 80-90% accurate in segmenting the instances.

[0061] Qualitative Segmentation Results

[0062] Figures 11A, 11B, 11C, and 11D show some examples of segmentations generated by our setup for the instances of chopped carrots, chicken nuggets, taro, and fried foods. Figures 11A, 11B, 11C, and 11D show a selectable instance selected for segmentation; the instance is highlighted and its highest point is marked as an "x." A square region is selected around point "x," as indicated by a white square. A depth geodesic is calculated from the highest point "x" to a systematic set of points around "x" and inside the square. Points classified as belonging to the same instance are marked by small black circles. A convex hull is calculated for all these black circle points, which form the instance segmentation. Once a segment is found, it is removed from the depth image and another point "x" that is the highest point in the remaining point cloud and a region around it that does not overlap with any previously selected region by more than 50% is selected and the segmentation process is repeated.

[0063] FIG. 12 shows a hardware diagram illustrating some components that may be incorporated into an imaging controller 1200 of the present disclosure, according to some embodiments of the present disclosure. The imaging controller 1200 may include a processor 1220 configured to execute stored instructions and a memory 1240 that stores instructions executable by the processor. The processor 1220 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1240 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 1220 is connected to one or more input / output devices via a bus 1206. The imaging controller 1200 may also include a storage device 1230 adapted to store a first module (a pre-trained neural network 1231), a second module (a seed point selection program module) 1232, and a third module (a geodesic program module) 1233, for storing executable instructions for the processor 1220. In some cases, modules 1231, 1232, and 1233 may include the architecture of the pre-trained image segmentation neural network 505 or 304 shown in FIG. 5 or 3, a module for selecting seed points 301 shown in FIG. 5 or 3, and a module for creating depth geodesics 303, 504 in FIG. 3 and 5. The storage device 1230 may store data in data storage 1234. The storage device 1230 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0064] The imaging controller 1200 may include an input interface for receiving external data 1295, including depth images from an imaging device including a camera, or a set of training data sets. The input interface may include a human-machine interface 1210 connecting the processor 1220 to a keyboard / measuring device 1211 and a pointing device 1212 within the imaging controller 1200, which may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, among others. Alternatively, the input interface may include a network interface controller 1250 adapted to connect the imaging controller 1200 to a network 1290 through the bus 1206. Through the network 1290, the external data 1295 may be downloaded and stored in the storage system 1230 as training data and / or operational data 1234 for storage and / or further processing.

[0065] 12 , the imaging controller 1200 can include an output interface linked through a bus 1206 and a display interface 1260 adapted to connect the imaging controller 1200 to a display device 1265, where the display device 1265 can include, among other things, a computer monitor, a camera, a television, a projector, or a mobile device. For example, the imaging controller 1200 can be connected to a system interface 1270 adapted to connect a controlled system (robot controller) 1275, or to a different system 1275 to be controlled based on some aspect. Additionally or alternatively, the imaging controller 1200 can be connected to an application interface 1280 via a bus adapted to connect the imaging controller 1200 to an imager / camera (depth imaging camera) 1285 of the robot 150 of FIG. 13 . In this case, the imaging controller 1200 is configured to obtain a depth image of an object to be manipulated by the robot 150 from the imaging device / camera 106 (FIG. 13), segment pickable instances among the instances of the object in the depth image, and generate geometric features of the pickable instances for a controller (manipulator controller) of the robot 150. In some cases, the geometric features generated from the segmented instances by the imaging controller 1200 may include an outer size of the instance, a pose of the instance, a shape of the instance, etc., so that the robot 150 can pick the corresponding pickable object from among the objects in a bin or on a workbench.

[0066] FIG. 13 illustrates a robot configuration 100 of a robot 150 according to some embodiments. The robot 150 includes a controller and a robot arm 101 for performing an instance pick operation. The robot arm 101 includes a wrist 102 for multiple degrees of freedom for moving an object. In some implementations, the wrist 102 is provided with an end tool 104 for holding an object 103 and / or for performing any other robotic operation, such as a pick operation. The end tool 104 may be, for example, a gripper or other pick tool, such as a vacuum tool. In the following, "end tool" may refer to a "gripper" or other pick tool, such as a vacuum tool, or they may be used interchangeably. According to embodiments, the goal of the pick operation is to pick a single instance of an object, typically the instance closest to the gripper.

[0067] The robot 150 is configured to perform a picking operation, e.g., pick a segmented object instance 103, along a trajectory while using an imaging device 106 connected to an imaging controller 1200, which can provide a depth image of an object to be manipulated by the robot arm 101. The imaging controller 1200 is connected to the controller of the robot 150 such that the controller of the robot 150 obtains and uses features of the segmented instance from the imaging controller 1200. Here, the trajectory corresponds to a path that defines the movement of the object 103 held by the gripper 104 to perform the picking operation. In a simple case, the trajectory may define only the vertical movement of the wrist 102. However, because the wrist 102 includes multiple degrees of freedom, the trajectory may include a motion profile that spans a multi-dimensional space.

[0068] The pose of an object refers to a combination of the object's position and orientation. The gripper 104 is movable in a start pose 111. The pose of the gripper 104 corresponding to the start pose 111 is referred to as the start pose of the gripper 104. According to an embodiment, the goal of the picking operation is to pick a segmented instance object 103. The pose 115 of the object 112 may refer to the position and / or orientation of the object 112. The robot 150 is configured to move the gripper 104 along the trajectory 113 to pick the object 103 in a pose 114. The pose 114 of the object 103 of the object 112 is referred to as the target pose. The pose of the gripper 104 corresponding to the target pose is referred to as the target pose of the gripper 104.

[0069] The target pose of the gripper 104 is determined based on the position of the object 112. At the end of the successful execution of the picking operation, the pose of the gripper 104 of the robot arm 101 is considered to have reached the target pose of the gripper 104. Therefore, achieving the target pose of the gripper 104 is equivalent to the successful execution of the picking operation. According to an embodiment, the trajectory 113 is defined according to the starting pose and the target pose of the gripper 104 and the pose 115 of the object 112. Moreover, such a picking operation may be repeatedly executed by the robot 150.

[0070] FIG. 14 is a schematic diagram illustrating, by way of non-limiting example, a computing device 1400 that can be used to implement some techniques of the methods and systems according to embodiments of the present disclosure. The computing device or device 1400 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing device 1400 may include a power supply 1408, a processor 1409, memory 1410, and storage 1411, all connected to a bus 1450. In some cases, the storage 1411 may include the architecture of the trained neural network 505 (304, 1231) based on the training pipeline shown in FIG. 3. Furthermore, the architecture of the trained neural network may be obtained over a network from an external data source 1295 that stores the trained neural network. Additionally, a high-speed interface 1412, a low-speed interface 1413, a high-speed expansion port 1414, and a low-speed expansion port 1415 are connected to the bus 1450. Additionally, a low speed connection port 1416 is connected to the bus 1450 .

[0071] Depending on the particular application, by way of non-limiting example 1430, various component configurations are contemplated that may be mounted on a common motherboard. Additionally, input interface 1417 may be connected to external receiver 1406 and output interface 1418 via bus 1450. Receiver 1419 may be connected to external transmitter 1407 and transmitter 1420 via bus 1450. External memory 1404, external sensors 1403, machine 1402, and environment 1401 may also be connected to bus 1450. Additionally, one or more external input / output devices 1405 may be connected to bus 1450. Network interface controller (NIC) 1421 may be adapted to connect to network 1422 via bus 1450, so that, among other things, data or other data may be rendered on a third-party display device, a third-party imaging device, and / or a third-party printing device external to computing device 1400.

[0072] 14 , it is also contemplated that memory 1410 may store instructions executable by computer device 1400, historical data, and any data that may be utilized by the methods and systems of the present disclosure. Memory 1410 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 1410 may be one or more volatile memory units and / or one or more non-volatile memory units. Memory 1410 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0073] 14 , the storage device 1411 may be adapted to store auxiliary data and / or software modules used by the computing device 1400. For example, the storage device 1411 may store historical data and other relevant data as described above with respect to this disclosure. Additionally or alternatively, the storage device 1411 may store historical data similar to that described above with respect to this disclosure. The storage device 1411 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Furthermore, the storage device 1411 may include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, an array of devices including a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration, or the like. The instructions may be stored on an information carrier. When executed by one or more processing devices (e.g., the processor 1409), the instructions perform one or more methods, such as those described above. The system can be linked via a bus 1450, optionally to a display or user interface (HMI) 1423 adapted to connect the system to a display device 1425 and a keyboard 1424, which can include, among others, a computer monitor, a camera, a television, a projector, or a mobile device.

[0074] 14 , computing device 1400 may include a user input interface 1417 adapted to a printer interface (not shown), which may also be connected via bus 1450 and adapted to connect to a printing device (not shown), which may include, among others, a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye sublimation printer. High-speed interface 1412 manages bandwidth-intensive operations for computing device 1400, while low-speed interface 1413 manages less bandwidth-intensive operations. This allocation of functionality is merely exemplary. In some implementations, high-speed interface 1412 may be coupled to memory 1410, a user interface (HMI) 1423, a keyboard 1424 and a display 1425 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1414 capable of accepting various expansion cards (not shown) via bus 1450. In this embodiment, low-speed interface 1413 is coupled to storage device 1411 and low-speed expansion port 1415 via bus 1450. Low-speed expansion port 1415, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices 1405 and other devices, including a keyboard 1424, a pointing device (not shown), a scanner (not shown), or a networking device such as a switch or router, for example, via a network adapter.

[0075] 14, computing device 1400, as shown, may be implemented in several different forms. For example, it may be implemented as a standard server 1426, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer, such as a laptop computer 1427. It may also be implemented as part of a rack server system 1428. Alternatively, components from computing device 1400 may be combined with other components, such as the embodiment of FIG. 13. Each such device may include one or more of computing devices 1300 and 1400, and the entire system may be composed of multiple computing devices in communication with each other.

[0076] The robotic system described above illustrates a robot capable of manipulating and assembling product parts. However, it can also be applied to robots used in food processing plants where many foods are irregularly shaped objects (e.g., chopped vegetables, fried chicken, etc.). A robotic system including a system for generating realistic images from real depth images and automatically segmenting multiple instances of rigid objects in depth images can be applied to the automation of food processing plants and industrial robots capable of manipulating food. Furthermore, the robotic system described above can be applied to a segmentation system for food recognition. Segmentation is one of the most popular and important problems in image processing. For application in food processing plants, high segmentation accuracy and short training and computation times are essential.

[0077] The above description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the claims.

[0078] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is therefore the object of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the disclosure.

Claims

1. 1. An imaging controller for segmenting an instance from a depth image containing an object to be manipulated by a robot, the imaging controller comprising: an input interface configured to receive a depth image including the object; a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image; a processor coupled to the memory and configured to execute the instructions to segment selectable instances using the trained neural network, the instructions comprising: selecting the highest point in the depth image; defining a region using a shape to surround the highest point; sampling points in the region of the depth image; calculating a depth geodesic curve between the highest point and the sampled point; submitting the depth geodesic curves to the neural network to segment the selectable instances among the object instances in the depth image; an output interface configured to output geometric characteristics of the pickable instance to a manipulator controller of the robot.

2. The imaging controller of claim 1 , wherein the depth image is acquired by a camera or a sensor.

3. The step further comprises: selecting a next highest point from the depth image, a selection radius around the next highest point excluding overlap with the selectable instance, said step further comprising: defining a next region using the shape, the next region encompassing the next highest point, said step further comprising: sampling a next point in the next region of the depth image; calculating a next depth geodesic curve between the next highest point and the next sampled point; and submitting the depth geodesic curve to the neural network to segment a next possible one of the instances of the object in the depth image.

4. The imaging controller of claim 3 , wherein the steps of selecting, defining, sampling, calculating, and submitting continue until the steps have been performed for the remainder of the object in the depth image.

5. The imaging controller according to claim 1 , wherein the certain shape is a square, a rectangle, a triangle, a circle, or an ellipse.

6. The imaging controller of claim 1 , wherein the neural network is trained to classify each feature vector as belonging to the same instance or a different instance.

7. The endpoints of the geodesic curves are first determined using the peaks generated by the Watershed algorithm. The imaging controller of claim 1 , wherein the imaging controller is synchronized.

8. 1. A computer-implemented method for training a neural network to segment instances in a depth image, the method using a processor coupled with stored instructions implementing the method, the instructions, when executed by the processor, perform steps of the method, the steps comprising: selecting a depth image from a set of depth images; determining points at xy locations on a two-dimensional image grid and corresponding depth points for the selected depth image, wherein the points on the two-dimensional image grid are each annotated with a ground truth instance label, the step further comprising: calculating geodesic lines between the pairs of annotated and determined points; generating a depth geodesic curve by projecting the geodesic line onto the depth image; and discretizing each of the depth geodesic curves to create discretized vectors, each discretized vector corresponding to one of the depth geodesic curves between the annotated determined point pairs, said step further comprising: A computer-implemented method comprising the step of submitting the discretized vectors and corresponding annotated labels of the discretized vectors to the neural network, wherein the steps from the selecting step to the submitting step are performed iteratively until all remaining parts of the set of depth images have been used.

9. The method of claim 8 , wherein the neural network is trained to classify each feature vector as belonging to the same instance or a different instance.

10. The method of claim 9 , further comprising the step of calculating the convex hull of all endpoints of the geodesic lines classified as the geodesic lines belonging to the same instance as the selectable points.

11. The method of claim 8 , wherein the generated depth geodesic curves are debiased such that the geodesic curves are created from points of lower depth to points of higher depth.

12. The endpoints of the geodesic curves are first determined using the peaks generated by the Watershed algorithm. The method of claim 8 , wherein the

13. The method of claim 12 , wherein the endpoints of the geodesic curve are determined using a systematic selection method.

14. 1. A bin picking system for picking objects from a bin, comprising: an end tool configured to select an object from the objects; a robotic arm including the end tool, the robotic arm configured to be driven by control signals including an end tool signal to pick the object from the bin using the end tool, the bin picking system further comprising: an interface configured to transmit and receive the control signal, a sensor signal from a sensor disposed on the robot arm, and an imaging signal from at least one imaging device; and a memory configured to store robotic control program instructions, a classifier, and a trained neural network that segments instances from the objects in the depth image, wherein the trained neural network is trained by the computer-implemented method of claim 8, the bin picking system further comprising: a processor coupled to the memory and configured to execute the instructions to use the trained neural network to segment selectable instances and generate the control signals to drive the robotic arm and the end tool, the steps of the instructions comprising: selecting the highest point in the depth image; defining a region using a shape to surround the highest point; sampling points in the region of the depth image; calculating a depth geodesic curve between the highest point and the sampled point; submitting the depth geodesic curves to the neural network to segment the selectable instances among the object instances in the depth image; generating geometric characteristics of the selectable instances and the control signal based on the imaging signal; and transmitting the generated geometric features and the generated control signals to the robot arm to cause the end tool to pick the object corresponding to the pickable instance from the bin using the end tool.

15. The bin picking system of claim 14 , wherein the depth image is acquired by a camera or sensor.

16. The step further comprises: selecting a next highest point from the depth image, a selection radius around the next highest point excluding overlap with the selectable instance, said step further comprising: defining a next region using the shape, the next region encompassing the next highest point, said step further comprising: sampling a next point in the next region of the depth image; calculating a next depth geodesic curve between the next highest point and the next sampled point; and submitting the depth geodesic curves to the neural network to segment a next selectable one of the instances of the object in the depth image.

17. 17. The bin-picking system of claim 16, wherein the selecting, defining, sampling, calculating, and submitting steps continue until the steps have been performed for the remainder of the object in the depth image.

18. 15. The bin picking system of claim 14, wherein the certain shape of the certain region is a square, a rectangle, a triangle, a circle, or an oval.

19. The bin selection system of claim 14 , wherein the neural network is trained to classify each feature vector as belonging to the same instance or a different instance.

20. The endpoints of the geodesic curves are first determined using the peaks generated by the Watershed algorithm.

15. The bin picking system of claim 14, wherein the bin picking system is synchronized.

Images