Discriminative 3d shape modeling for small sample instance segmentation

Abstract

An imaging controller for segmenting instances from a depth image including objects to be manipulated by a robot is provided. The imaging controller includes an input interface configured to receive a depth image including objects, a memory configured to store instructions and a neural network trained to segment instances from the objects in the depth image, and a processor coupled with the memory configured to execute the instructions to segment a pickable instance using the trained neural network. The instructions include the steps of selecting a highest point in the depth image, defining a region using a shape such that the region surrounds the highest point, sampling points in the region of the depth image, computing a depth geodesic between the highest point and the sampled points, submitting the depth geodesics to the neural network to segment the pickable instance in an instance of the objects in the depth image, and an output interface configured to output geometric features of the pickable instance to a manipulator controller of the robot.

Description

Technical Field

[0001] This disclosure generally relates to the automatic segmentation of multiple instances of objects in depth images, with the segmentation model trained in a few-sample environment using only a few annotated training examples. Background Technology

[0002] Instance segmentation is a necessary step in a variety of autonomous tasks. Some examples include: (i) robotic arms that need to pick up similar products moving on a conveyor belt in a manufacturing environment, (ii) supermarket robots that need to pick up and place fruit from containers, (iii) library-assisted robots that need to pick up books from boxes and hand them to people, (iv) identifying instances of cars on a road in an autonomous driving environment, or (v) identifying instances of body tissue in medical images for automated diagnosis.

[0003] Standard deep learning solutions (e.g., Mask-RCNN and variants for solving such instance segmentation tasks) typically require large training sets to train the underlying neural networks; however, creating such training sets and annotating training examples can be expensive and involve significant manual labor. Therefore, creating such datasets may be infeasible in many cases. Instance segmentation can also be viewed as applying standard clustering schemes to point clouds of deep images. For example, segmentation can be derived by applying K-means, Gaussian mixture modeling, or spectral clustering to the point cloud. However, such clustering algorithms typically make assumptions about the distribution of the point cloud and may therefore be inflexible for segmenting real-world objects whose shapes may vary. Recent methods for instance segmentation in unsupervised environments also exist, such as InSeGAN. However, InSeGAN requires large, unlabeled training sets with diversity in instance arrangements, which may also be difficult to obtain under real-world conditions.

[0004] Therefore, there is a need to design a method for instance segmentation that can automatically segment multiple instances of an object in a depth image and requires only a few training examples to train the segmentation model. Summary of the Invention

[0005] This disclosure generally relates to the automatic segmentation of multiple instances of objects in depth images, where the segmentation model is trained in a few-sample environment using only a small number of annotated training examples.

[0006] Some embodiments of this disclosure provide a method for segmenting object instances in a depth image. Compared to existing methods that typically use deep neural networks for segmentation, this disclosure includes a uniquely created instance segmentation pipeline that requires no 3D CAD model of the object, but only a few annotated examples of object instances from a small number of depth images.

[0007] The embodiments of this disclosure are configured to automatically segment multiple instances of approximately convex 3D objects in depth images in a few-sample environment, representing a novel approach in the field of computer vision. The embodiments of this disclosure can be configured to perform multiple applications, including robot-assisted tasks. For example, in picking objects from a container containing multiple instances, the robot needs to correctly segment the instances for planned grasping.

[0008] Some embodiments of this disclosure include architectures for instance segmentation that follow aspects of deep neural networks; however, standard neural networks require a large training set of annotated instances in a depth image for training. A unique insight in this disclosure is that instead of directly using instances to train the neural network, it trains the network to classify depth surface geodesics that connect pairs of 3D points in a depth image. Since there can be many such point pairs for a given number of pixels in a depth image, a small number of annotated instances can generate a large set of annotated surface geodesics, which can then be used to train the neural network. The method first selects random samples of spatial locations in a given training depth image and computes pairwise surface geodesics between each pair.

[0009] At least one insight gained from the experiments is that surface geodesics computed between depth point pairs can have arbitrary lengths, and therefore may be difficult to use for training neural networks that expect fixed-size inputs. To address this, the presented model discretizes each geodesic into a fixed-dimensional vector. In this disclosure, each geodesic is divided into equal segments, and each dimension of the discrete geodesic vector is filled with the depth of a pixel at the endpoints of the corresponding geodesic segment.

[0010] Another insight derived from this disclosure is that for any two points on a convex surface, the surface geodesic connecting these two points (which is the shortest curve) will have a convex shape. However, if the two points belong to different instances, and each instance is convex, then the surface geodesic connecting the two points will have a non-convex shape or discontinuity at the depth point where the two object surfaces intersect. These two types of curves (convex and non-convex) are used to train a neural network for classifying objects. When well-trained, the neural network classifies each geodesic as either entirely within the instance segment (convex geodesic) or spanning multiple instances (non-convex geodesic).

[0011] The insights gained from the experiments are that by discretizing surface geodesics, convex / non-convex discontinuities in the geodesics can be lost if the curve spans very close two instances in which it is computed. For example, consider two cubes in contact with each other. In the depth image, at the point where the cubes touch, the curve can be almost continuous, and discontinuities can be skipped by the discretization step when the number of dimensions in the fixed-dimensional geodesic vector is not large enough to capture them. A similar problem can occur when noise is present in the depth image; therefore, standard noise removal and hole-filling algorithms typically used to smooth depth images may over-smooth the boundaries of instance segments and thus suppress ground-based geodesic discontinuities. To avoid this problem, training annotations are deployed for the instances. Even when the discretized geodesic vectors have already suppressed object intersection discontinuities, the annotations guide the neural network for classification, requiring the network to learn additional surface features that characterize the surfaces of the 3D objects.

[0012] Another insight gained from the experiments is that randomly selecting point pairs from a depth image to construct geodesics for training a neural network can be suboptimal. For example, there may be geodesics that the neural network can learn to segment very easily, while others may be difficult to classify. Since easy examples can sometimes be significantly more numerous than difficult examples, neural network training can be biased and may not produce correct segmentations for difficult segmentation scenarios. Another scenario is when the size of the object to be segmented is small, yet there are many instances in a given depth image. This results in a large number of geodesics spanning multiple instances, while only a few geodesics are entirely within the instances. To address these issues, this disclosure proposes using the Watershed Algorithm (WA) to create an initial segmentation of the depth image. The method in WA produces two outputs: (i) a set of peaks, which are points in the depth image that are claimed to correspond to the centroids of instances, and (ii) the segmentation of the corresponding instances. However, since the WA method is unsupervised, the peaks or segmentations it produces may be inaccurate. The unique insight in this disclosure is to improve WA segmentation by using a ground-based training annotation and geodesic classification neural network. More clearly, one end of the geodesic is taken from the peak generated by the WA, and the other end is constrained within the corresponding segment generated by the WA for that selected peak. Next, the geodesic is labeled using ground-real-world annotations regarding whether it belongs entirely to the same instance. If the WA makes an error in segmenting instances, labeling the ground-real-world instance of the geodesic will allow the neural network to correct the error, resulting in better instance segmentation. At least one benefit of selecting geodesics from instances generated by the WA is that it allows for the creation of a better training set for the neural network, containing more geodesics belonging to the same instance as well as difficult examples resulting from incorrect segmentation by the WA. Geodesics belonging to the same instance allow for learning a better implicit 3D model of object shape within the neural network, while difficult examples allow for learning a discriminative network that works well in challenging segmentation scenarios.

[0013] When using a trained model for, for example, segmentation, this architecture takes a depth image as input and computes depth-geodesics between selected pairs of locations in the depth image. The locations are selected based on pre-determined practical criteria. One implementation of these criteria is based on the subsequent task the segmentation will be used for. For example, if the instance segmentation produced by this method is used to pick up instances by a robotic arm, it would be useful to segment the topmost instance in the stack first. Such instances, which are expected to be segmented first, are referred to as pickable instances.

[0014] This disclosure proposes selecting pairs of spatial locations on a depth image to compute depth geodesics on an input test image, wherein the selection is based on pickable instances in a given test depth image. Depth geodesics are computed across all selected pairs of locations. One end of each geodesic is fixed at the highest depth point on the pickable instance. This point is called the seed point. The other end of the geodesic is selected from a region surrounding the highest depth point and within a predefined radius. Each geodesic is discretized, as described in the training phase. A pre-trained neural network then classifies each geodesic as belonging entirely to a single instance surface or spanning more than one instance. Instance segmentation is then defined using the convex hull of all points classified as belonging to the same instance. The process is repeated for each selected location in the image to produce multiple such instance segmentations.

[0015] Experiments used to segment a given test image have shown that randomly selected depth points from around a seed point may not be computationally efficient because they may not cover instances in all directions. This insight suggests a systematic selection of points around the seed point. An implementation of this idea in this disclosure involves creating a sample set of points around the seed point on concentric circles along radial lines spaced at angles. This selection allows for uniform coverage of the circular region around the seed point to construct geodesics.

[0016] The underlying principle behind constructing test geodesics is that these geodesics always point from higher depth points to lower depth points. Therefore, including geodesics in the training set that do not adhere to this property can be inefficient. To address this issue, the depth points used to train the neural network are first sorted in ascending order of their depths, and geodesics for training are computed from points with lower depths (or the highest point) to points with higher depths.

[0017] Embodiments of this disclosure provide an imaging controller for segmenting instances from a depth image including an object to be manipulated by a robot. The imaging controller may include: an input interface configured to receive a depth image including the object; a memory configured to store instructions and a neural network trained to segment instances of the object from the depth image; and a processor coupled to the memory, configured to execute the instructions to segment pickable instances using the trained neural network, wherein the instructions include the steps of: selecting a highest point among all instances in the depth image; defining a region using a shape such that the region surrounds the highest point; sampling points in the region of the depth image; calculating a depth geodesic between the highest point and the sampled points; submitting the depth geodesic to the neural network to segment the pickable instances among instances of the object in the depth image; and an output interface configured to output geometric features of the pickable instances to a manipulator controller of the robot.

[0018] Furthermore, according to some embodiments of the present invention, a computer-implemented method for training a neural network to segment instances in a depth image is provided. In this case, the method uses a processor coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, which include the following steps: selecting a depth image from a set of depth images; determining points at xy-space locations on a two-dimensional image grid and corresponding depth points with respect to the selected depth image, wherein the points on the two-dimensional image grid are respectively annotated with ground truth instance labels; calculating geodesic lines between the determined annotated point pairs; generating depth geodesics by projecting the geodesic lines onto the depth images; discretizing each depth geodesic to create a discretized vector, wherein each discretized vector corresponds to one of the depth geodesics between the determined annotated point pairs; and submitting the discretized vector and its corresponding annotated label to the neural network, wherein the steps from selection to submission are repeated until the remainder of the set of depth images is used.

[0019] Furthermore, according to some embodiments of the present invention, a container picking system for picking up objects from a container is provided. The container picking system may include: an end tool configured to pick up objects from an object; a robotic arm including the end tool, wherein the robotic arm is configured to be driven by control signals including end tool signals to pick up objects from a container using the end tool; an interface configured to send and receive control signals, sensor signals from sensors arranged on the robotic arm, and imaging signals from at least one imaging device; a memory configured to store instructions for a robot control program, a classifier for segmenting instances of objects from a depth image, and a trained neural network, the trained neural network having been trained by the aforementioned computer-implemented method; and a processor coupled to the memory, the processor being... The system is configured to execute instructions to segment pickable instances using a trained neural network and generate control signals to drive a robotic arm and an end effector. The instructions include the following steps: selecting a highest point in a depth image; defining a region around the highest point using a shape; sampling points in the region of the depth image; calculating a depth geodesic between the highest point and the sampled points; submitting the depth geodesic to the neural network to segment pickable instances among instances of objects in the depth image; generating geometric features and control signals for the pickable instances based on the imaging signals; and sending the generated geometric features and control signals to the robotic arm, causing the end effector to pick up an object corresponding to a pickable instance from a container.

[0020] Therefore, systems and methods as a whole should not be viewed merely as aspects of executing human thought, nor simply as collecting (aggregating), processing, analyzing, and displaying data in an abstract manner. Instead, systems and methods can summarize when it is difficult to annotate, or when they can operate in complex scenarios with less cost and computational expense, and can be competitive and beneficial to participants in the computer vision industry and other industries. Attached Figure Description

[0021] The embodiments disclosed herein will be further explained with reference to the accompanying drawings. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the embodiments disclosed herein.

[0022] [ Figure 1A ]

[0023] Figure 1A This is an illustrative example of depth image segmentation of nearly identical chicken nugget instances in a container according to an embodiment of the present invention.

[0024] [ Figure 1B ]

[0025] Figure 1BThis is an illustrative example of depth image segmentation of nearly identical chicken nugget instances in a container according to an embodiment of the present invention.

[0026] [ Figure 2A ]

[0027] Figure 2A An example of a depth image containing multiple instances of an object, illustrating an embodiment of the present invention, is shown.

[0028] [ Figure 2B ]

[0029] Figure 2B An embodiment of the present invention is shown. Figure 2A The surface curve between two arbitrary points p1 and p2 in the graph.

[0030] [ Figure 3 ]

[0031] Figure 3 A pipeline for training a neural network using surface geodesics to learn an implicit shape model of a single instance of an object, according to an embodiment of the present invention, is illustrated.

[0032] [ Figure 4A ]

[0033] Figure 4A Examples illustrating pickable instances in a depth point cloud according to embodiments of the present invention are shown.

[0034] [ Figure 4B ]

[0035] Figure 4B An embodiment of the invention is shown in Figure 4A The split instance at the top of the container.

[0036] [ Figure 5 ]

[0037] Figure 5 An example of an inference pipeline for segmenting instances according to an embodiment of the present invention is shown.

[0038] [ Figure 6A ]

[0039] Figure 6A An example of random selection of points around a pick-up point for calculating a geodesic line is shown according to an embodiment of the present invention.

[0040] [ Figure 6B ]

[0041] Figure 6B An example of a system selection for calculating points around a pick-up point of a geodesic line according to an embodiment of the present invention is shown.

[0042] [ Figure 7A ]

[0043] Figure 7A An example of a depth map and steps for generating instance segmentation based on a watershed transform according to an embodiment of the present invention is shown.

[0044] [ Figure 7B ]

[0045] Figure 7B An example of a depth map and steps for generating instance segmentation based on a watershed transform according to an embodiment of the present invention is shown.

[0046] [ Figure 7C ]

[0047] Figure 7C An example of a depth map and steps for generating instance segmentation based on a watershed transform according to an embodiment of the present invention is shown.

[0048] [ Figure 7D ]

[0049] Figure 7D An example of a depth map and steps for generating instance segmentation based on a watershed transform according to an embodiment of the present invention is shown.

[0050] [ Figure 8 ]

[0051] Figure 8 A comparison of results obtained using the present invention and other methods for the fried chicken dataset is shown.

[0052] [ Figure 9 ]

[0053] Figure 9 Examples of images for four object categories are shown for use in creating the dataset (depth images) in the experiment.

[0054] [ Figure 10 ]

[0055] Figure 10 This illustrates the use according to an embodiment of the invention. Figure 9 An example of the segmentation results of the dataset.

[0056] [ Figure 11A ]

[0057] Figure 11A Several examples of segmentation produced by our arrangement of cut carrots, chicken nuggets, taro, and fried food examples according to embodiments of the present invention are shown.

[0058] [ Figure 11B ]

[0059] Figure 11B Several examples of segmentation produced by our arrangement of cut carrots, chicken nuggets, taro, and fried food examples according to embodiments of the present invention are shown.

[0060] [ Figure 11C ]

[0061] Figure 11C Several examples of segmentation produced by our arrangement of cut carrots, chicken nuggets, taro, and fried food examples according to embodiments of the present invention are shown.

[0062] [ Figure 11D ]

[0063] Figure 11D Several examples of segmentation produced by our arrangement of cut carrots, chicken nuggets, taro, and fried food examples according to embodiments of the present invention are shown.

[0064] [ Figure 12 ]

[0065] Figure 12 Hardware diagrams illustrating some components that can be incorporated into some systems according to some embodiments of the present disclosure are shown.

[0066] [ Figure 13 ]

[0067] Figure 13 The robot configuration of the robot according to an embodiment of the present invention is shown.

[0068] [ Figure 14 ]

[0069] Figure 14 The diagram illustrates, by way of non-limiting example, a computing device that can be used to implement some technologies according to embodiments of the present invention. Detailed Implementation

[0070] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated as noted in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art can devise many other modifications and embodiments falling within the scope and spirit of the principles of the embodiments currently disclosed.

[0071] Segmenting nearly identical object instances is a common problem in various robotic container picking applications. Some examples include: (i) robotic arms that need to pick up products moving on a conveyor belt in a manufacturing environment, (ii) supermarket robots that need to pick up and place fruit from containers, or (iii) library-assisted robots that need to pick up books from boxes and hand them to people.

[0072] Figure 1A and Figure 1B This is an illustrative example of depth image segmentation of nearly identical chicken nugget instances in a container according to some embodiments of the present invention. See also Figure 1A This is an illustrative example of depth image segmentation of nearly identical chicken nugget instances within a container. Standard deep learning solutions for this task would require large training sets and significant annotation effort. The problem can also be viewed as applying one of the standard clustering schemes to depth points. For example, segmentation can be derived by applying K-means, EM, or spectral clustering to the point cloud. However, such clustering algorithms often make assumptions about the shape of the distribution or involve hyperparameters. For example, K-means assumes the distribution is spherical around the cluster centroid, or elliptical as in EM. Spectral clustering requires specifying a proximity hyperparameter to define the edges of the base graph Laplacian operator. There are also some recent methods for unsupervised instance segmentation. However, they require large unlabeled training sets with diversity in instance arrangements, which can be difficult to obtain under real-world conditions. For example, some methods require thousands of unlabeled depth images, which requires a robotic arm to shake the container of the object—a non-scalable task that is sometimes infeasible if the object of interest is fragile or could be damaged by such shaking.

[0073] In this disclosure, we consider the problem of instance segmentation of nearly identical convex object instances in depth images in a few-sample environment, where we assume that a finite set (less than five) of annotated depth images is available, each with several instances annotated with its segments. Our key idea is to create surface trajectories, or geodesics, on the 3D surface of the depth image, with the goal of training a neural network to classify these trajectories as being within or across two ground reality instances; the network thus potentially learns an implicit 3D model of a single object instance within its parameters, even though it is trained using only a single geodesic trajectory. For a depth image with n pixels, there may potentially be n(n-1) / 2 such geodesics, which, if used carefully, can provide a significantly large dataset to train on. Our idea is to leverage this insight for instance segmentation when the number of annotated images is small. Specifically, our algorithm has the following steps: (i) For two randomly selected points on the depth image, we compute a surface geodesic, which is the projection of the straight line connecting the two points on a 2D RGB image grid onto the depth image. Given that we assume the object is convex (and that the camera plane is orthogonal to the object), the projection will (approximately) be the shortest path connecting two points on the depth surface of the object, and therefore will be a depth geodesic. (ii) We discretize this geodesic into a predetermined (fixed) set of containers, where each container will hold the depth value of the geodesic at the container's location. The containers are equidistant from the straight line from which the geodesic is projected. (iii) Next, we label the geodesic with the provided ground reality segments. Specifically, we assign label 1 to the discrete geodesic vector 1 if the two ends of the geodesic belong to the same object instance, otherwise it is zero. (iv) We train a neural network classifier on these discrete vectors and their labels.

[0074] During testing, given a depth image, we first select a seed location in the depth image to begin the segmentation process. In robotic container picking applications, the robot is generally more likely to pick up the instance located at the top (i.e., closest to the robot). In some other cases, the most distant instance may be preferred. We propose various heuristics to compute this initial seed. Next, we compute geodesics from this seed point to random spatial locations around the seed point within a predefined radius. We discretize these geodesics and use a pre-trained classifier to classify each geodesic as belonging to the same instance or not. For all points classified as belonging to the same instance, we compute the convex hull of these points and treat all pixels within that convex hull as corresponding to the same instance, thus achieving instance segmentation. To create segmentations for more than one instance, we select another seed point from the depth image that is outside the predefined proximity of already segmented instances and repeat the above process until we have obtained a suitable number of object instances for the task.

[0075] The proposed method

[0076] Suppose we are given a set of annotated depth images. Each of them Define an image grid with width W and height H pixels, such that D xy We maintain a nonnegative value corresponding to the depth of the scene at position (x, y) ∈ [H] × [W], where [Z] represents the set of indices {0, 1, ..., Z-1}. For a pixel (x, y) on the image grid, we assume it is labeled with instance label l. xy ∈[L D ]∪{L B} Comments, where L D L is the number of instances in the depth image D, and L B This corresponds to the background label (i.e., pixels that do not contain the depth of any object instances, such as the bottom of the container, the walls of the container, etc.). To introduce our method, we will need some background symbols, which we will describe below.

[0077] Surface geodesics and assumptions

[0078] For two distinct points (x1, y1) and (x2, y2) on the image grid, suppose γ(t) (for t∈[0, 1]) is a point that starts at... and end at The directional surface curve. That is to say, and Having all its points We define Let be the length of the curve γ, and let g be a curve (or set of curves) with the minimum length connecting two points. That is,

[0079]

[0080] In the following text, we will refer to geodesics as z-geodesics. To derive our method, we make the following assumptions about our problem setting.

[0081] Assumption 1 (Surface convexity)

[0082] We assume that the objects used in our setup are convex, and that the depth patches associated with the instances form an approximately convex smooth surface.

[0083] By a convex object surface, we mean that all one-dimensional curves γ(t) on the surface are convex with respect to t, such that any Assumption These are patches from depth image D, where... All elements in the image have the same instance label l. Then, for two distinct points (x1, y1) and (x2, y2) on the image grid, where... if It began and end at The geodesic, and if the label represents a point on the geodesic. Given the instance labels, we have the following proposal, which directly proves the use of the basic properties of convexity. To simplify our annotation, we will now omit information about... We will restore the subscript and superscript to it whenever needed.

[0084] Recommendation 1

[0085] if It is a convex depth map patch from depth map D, and if g(t) is from... arrive The geodesic, then

[0086] Assumption 2 (Orthogonal Projection)

[0087] The camera projection plane is appropriately positioned away from the instance such that the image XY-plane is approximately orthogonal to the velocity of any trajectory on the depth surface.

[0088] This assumption allows us to consider xy(t) = (1-t)(x1,y1) + t(x2,y2), t∈[0,1], through the straight line Connect 3D points and geodesic Parameterization. For simplicity, we will use e(xy(t)) to represent it. Furthermore, using this parameterization, we have a geodesic as g(e(xy(t))), where, instead of t, we now use points on the line to index the depth.

[0089] Assumption 3 (Fixed posture)

[0090] We also assume that the camera position and pose, as well as the container, are stationary when all depth images are captured.

[0091] We also assume that there are one or more instances of an object within the containers in the training images, and that all instances are of the same object. We make no assumptions about the arrangement of instances within the containers or the number of instances within a container. We also assume that the ground truth annotations are appropriately accurate, and that at least one instance in each training image is associated with a ground truth annotation. Although we have access to the RGB images of the containers next to the depth images, we do not use these RGB images in the method described in this work. Furthermore, the method can be easily extended to work with depth point clouds instead of depth images. In this case, the geodesic approximation using Assumption 2 may not be directly applicable, as the XY points may no longer be described by a fixed image grid.

[0092] Discriminative shape modeling

[0093] Figure 2A and Figure 2B The image shows a depth image containing multiple instances of an object, as well as a surface curve between two arbitrary points p1 and p2. Figure 2B The curve is plotted as a one-dimensional curve g(t) for varying t, where the X-axis of the plot shows t changing from t=0 corresponding to point p1 to t=1 corresponding to point p2 in 100 equal steps, and the Y-axis shows the depth of the geodesic in the depth image for the corresponding value of t. Our unique insight in developing our method is that if the two ends of the curve belong to different instances, the curve will be non-convex or non-smooth at the points where the instances overlap. Our method aims to leverage this insight to learn the implicit shape of objects using a neural network. Below, we describe each step in our setup in detail. Before we proceed, we need to discuss some practical choices that need to be considered from a computational and implementation perspective.

[0094] Geodesic Discretization: Practically speaking, directly applying the idea of ​​segmentation using geodesics is problematic because in that case, one would need to implicitly parameterize the surface geodesics as continuous curves, which can be difficult for arbitrary curves and objects for which no analytical form exists (e.g., the surface geodesics on chicken nuggets?). Instead, to keep computationally cheap, we discretize the curves using a fixed number of intervals (bins). Specifically, for the geodesic g(t), we use a fixed m-dimensional vector. Let v represent it, where the k-th dimension is v k = g((k-1) / m). Such discretized geodesics can be calculated very cheaply using assumption 2 of orthogonal projection onto the camera plane, because in that case, it is only necessary to first approximate the Euclidean geodesics. Divide into m parts, that is, To obtain the (x, y) 2D image grid position, this (x, y) 2D image grid position can then be used to directly index the depth map to obtain v. k =D xy(k) .

[0095] Instance Supervision: If the discontinuities or non-convexity of the surface geodesics are sufficient to find instance boundaries, why are instance annotations needed? This is because if two instances are very close, the discretization step described above can skip discontinuities in the curve. For example, consider two cubes in contact with each other. In the depth image, at the point where the cubes touch, the curve can be almost continuous, and discontinuities can be skipped by the discretization step. Similar problems can occur when noise is present in the depth image; standard noise removal and hole-filling algorithms can smooth the depth image where ground-based geodesic discontinuities might be suppressed. To avoid these problems, the method of this invention can access ground-based instance masks.

[0096] Instance segmentation training pipeline

[0097] Figure 3 An example of a training pipeline for computing surface geodesics on a given training depth image and training a neural network, according to some embodiments of the present invention, is shown. For a given training depth image D, the first step in our pipeline is to select a random set of M points 301 on a 2D image grid. Let's call this set... Next, for each pair of such points We calculate the Euclidean (geodetic) straight line 302 (A point like this and the straight lines from it to a pair of other points exist only when...) Figure 3 (As shown in the image). This step is followed by calculating the depth geodesics on the depth image by projecting these Euclidean geodesics onto the depth map. Then, each depth geodesic is discretized into a set of M vectors as described in the above sections. There are m intervals, each v corresponding to a discrete depth geodesic line 303. Assume... Is it from point (x) i ,y i ) to (x i ,y i The discretized vector corresponding to the depth geodesic of ) is then labeled with the tag. g Assigned to v as:

[0098]

[0099] Call l xy It is the instance label associated with the image point (x, y).

[0100] Our final step in the training pipeline is to use ensembles. The corresponding binary labels are used to train the neural network model parameterized by θ. Specifically, neural network 304 is a series of multilayer perceptrons (MLPs) and will draw from... We take a batch of samples as input and predict the labels for the corresponding samples. Then, we use a softmax-crossentropy loss of 306 to match this prediction with the ground truth binary labels of 305, and then use this to derive gradients to train the network parameters. In our experiments, we found that using Euclidean geodesics... The length of each vector is used to expand the length of the vector. (That is, using this length to add an additional (m+1)th dimension) improves the network's training and performance. This is because, for cases where there are no discontinuities in the geodesics that the network can distinguish, it can learn an approximate size of the basic shape used for classification.

[0101] Instance segmentation inference pipeline

[0102] Figure 4A Examples illustrating pickable instances in a depth point cloud according to embodiments of the present invention are shown, and Figure 4B It shows in Figure 4A The split instance at the top of the container.

[0103] During testing, given a test depth image D, our goal is to repeat this process during the training phase for instance segmentation. Since our ultimate objective is to generate segmentations for instances within the container, which are likely most useful for robotic arm grasping and picking, we propose segmenting instances located at the top of the container (i.e., those closest to the camera), such as... Figure 4A and Figure 4B As shown, the goal is to first generate its instance segmentation mask. We call such instances pickable instances.

[0104] Figure 5 An example of an inference pipeline for segmenting instances according to some embodiments of the present invention is shown. First, we select a seed point in the test depth image corresponding to the highest point on the pickable instance. Let's call this point H(501). Next, we use an approximate region around H in which instances might be located. We call the radius of this region the pick radius r. This region is a square box 502 centered on H with a size of 2r. Next, we uniformly sample m′ points {(x1,y1), (x2,y2), ..., (x...} around H. m′ y m′ )}, and create Euclidean geodesics (503). Then, these geodesics are mapped to discrete depth geodesics v (504), and the pre-trained f is used. θ (Corresponding to neural network 505 of trained neural network 304) is classified to represent the other endpoint of v (corresponding to the point (x) around H). i y i Whether it is within the instance segment. Points classified as within the segment are then fed into a robust convex hull calculation algorithm to produce the instance segment (506). Note that the convexity of the object is therefore important for this step to work correctly.

[0105] To create segments for different instances, we select another high point H′ from the depth image such that the pick radius r around H′ will not overlap with the pick radius around H. In other words, we search for instances whose depth geodesics will not overlap with instances we have already segmented. Once we find point H′, we apply the above process. We perform this process sequentially, generating one instance segment at a time.

[0106] Algorithm extension

[0107] Remove bias from the training set

[0108] Astute readers will immediately recognize the difference in how geodesics are computed during training and testing. While the training samples in the above setup are randomly selected from an image grid, the test samples are selected from the highest points in the depth map. Therefore, in the latter, the initial size (closest to H) of the discrete geodesics v will tend to increase (i.e., depth increases), which is unnecessary for those in the training set that introduce bias in the training and testing distributions. To mitigate this problem, we sort the training points by their depth values ​​in ascending order of their depth, and during training, geodesics are always computed from points with lower depths to points with higher depths.

[0109] System sampling for testing geodesics

[0110] In the basic inference algorithm described above, we randomly sample test points around the seed point. However, a more efficient approach is to systematically select points. To this end, we propose using a picking radius r to define a circular region around the picking point H; then, this region is divided into equal sectors by dividing r into β equal parts and the circle into ζ equal angles. This results in considering βζ points for generating surface geodesics, where these parameters can be adjusted based on the underlying shape of the segmentation we should learn. Figure 6A An example illustrating the random selection of points around the pick-up point for calculating geodesic lines is shown, and Figure 6BAn example illustrating the system selection for calculating points around a pick-up point of a geodesic line according to some embodiments of the present invention is shown.

[0111] Watershed initialization

[0112] a. So far, we have used random sampling points (albeit ordered) during the training phase. This sampling does not distinguish between simple and hard geodesics for learning a classifier. For example, trivial discontinuities may be sufficient for the classifier to label out-of-instance trajectories; however, if such discontinuities do not occur, are there other subtle cues in the geodesics that the classifier should pay attention to? Such attention may be difficult to learn when they might be relatively infrequent in a large number of simple trajectories. To address this, we propose a hard-negative mining extension to our basic method using the Watershed Transform (WT). The watershed algorithm is a classic unsupervised method for image segmentation that uses the analogy of blocking water flowing from a mountaintop (the interior of an instance) toward a valley (i.e., the segment boundary). If we use a “dam” (representing the edge by using the Laplacian operator of the image) to block the valley, the pixels that trap water within it form segments. The challenge of the above method working correctly lies in choosing where to construct the dam such that the trapped water corresponds to a real segment on the ground.

[0113] b. In WT, a distance transform is used to find the starting point for region growth (i.e., the location where water is poured). That is, a first distance transform is computed on the image to find the region containing the peak (which corresponds to the point furthest from the edge), and these points are then selected for region growth. Using this idea in our setup has two advantages: (i) points isolated from other instances can have such peaks, and such isolated instances are useful for robot picking, similar to instances corresponding to the highest depth point; and (ii) in cluttered areas, i.e., where peaks are high (because water cannot be blocked by edges due to discontinuous / broken edges), those segments generated by WT may correspond to multiple instances incorrectly segmented into single instances by WT, and thus can be used in a geodesic trajectory-based scheme that we can better correct using the provided supervision. Therefore, we propose improving seed point selection for constructing geodesic trajectories by selecting peaks generated by WT and restricting the endpoints of the geodesics to within the segmentation mask generated by WT for the corresponding peak points. Figure 7A , Figure 7B , Figure 7C and Figure 7D An example of a depth map and steps for generating instance segmentation based on watershed transform according to an embodiment of the present invention is shown.

[0114] experiment

[0115] a. In this section, we provide experiments demonstrating the empirical performance of our method for instance segmentation. For this empirical study, we use a dataset consisting of several chicken pieces in a container. However, for our experiments, the images are HD quality, and we resize them to 320×240. We train our setup using only a single annotated depth image, while the test set consists of 17 images. Depth images are created using an Ensenso camera. For systematic sampling of our endpoints, we use k=14 and the number of angles dependent on the pick radius (i.e., β=2πr / 3). The pick radius is chosen based on the size of the objects to be segmented, e.g., the average radius of instances from the provided ground reality segmentation.

[0116] b. Neural Network: We use a discretization of the geodesic trajectory with 50 intervals, and therefore use m = 51 intervals. Our neural network consists of 5 MLPs with corresponding output dimensions m, 5m, m, m / 2, and 2, and ReLU activation, and Adam is used for optimization with the default learning rate and other settings. We also experimented with the proposed method using other nonlinear classifier models (such as nonlinear SVM), and the results will be presented soon.

[0117] c. Evaluation: For the evaluation of this method, we sample 1000 points from the ground truth and the predicted instance segmentations, and calculate the F1 score on this overlap to determine whether the classifier correctly predicted these samples. We calculate the performance for predicting various numbers of instance segmentations in depth images. A caveat regarding the order of our predicted segmentation is that sometimes the method will not return the required number of segmentations because some instances will partially overlap with other instances, and instances excluding those using the pick radius will not be found in these overlapping instances. Therefore, we only evaluate for instances where pick points were identified.

[0118] experiment

[0119] Figure 8 The results obtained using neural networks through this invention and other methods for the fried chicken dataset are shown in comparison with other options such as linear SVM, nonlinear SVM using RBF kernels, and geodesic K-means (K-Mean) which directly classifies discretized trajectories using KMeans clustering. The depth images consist of multiple instances of chicken pieces. The number of instances varies from 3 to 17. For this experiment, we sampled the images using the algorithmic improvements described in the preceding sections, using 81K positive trajectories (i.e., those with label 1) and 30K negative trajectories. To address the imbalance in the dataset, we used appropriate weights during the training process. It is clear from the table that using our proposed neural variant performs significantly better than other alternatives.

[0120] Computational performance: Since our scheme consists of basic computations on the image and depth map, and the trajectories are discretized into small-dimensional vectors, our method is computationally very efficient, taking approximately 5 minutes to train on a 4-core CPU with 100K trajectories and approximately 0.05 seconds to segment instances during inference.

[0121] universality

[0122] Figure 9 Examples of grayscale images representing four object categories used in the experiment to create the dataset (depth images) are shown. Each depth image consists of multiple instances of the same object, however, there are pose variations (self-occlusion) between the instances. Object classes correspond to objects of different shapes, sizes, and numbers of instances. The following objects are used: (i) different categories of chicken nuggets, (ii) fried foods, (iii) cut carrots, and (iv) taro. The chicken nugget class has multiple instances of fried chicken nuggets, which may have slight shape variations and are not necessarily convex. The fried food class has only a few instances, however, the instances are larger than the other shapes and are elliptical with a large major axis and a small minor axis, resulting in different geodesics in different directions. The cut carrot and taro classes have small objects; the taro class has an approximately spherical shape, while the cut carrot is pyramidal. Significant inter-object occlusion and self-occlusion caused by these classes make segmenting instances within these classes challenging.

[0123] Figure 10 This illustrates the use of some embodiments of the invention. Figure 9 An example of the segmentation results of the dataset. Figure 9 The table in the document reports the segmentation using the method described in this invention. Figure 9 The results show that the method is accurate for single instances and multiple instances of objects in each category described in the figure, with invocation and F1 accuracy scores. The results also show that the method is accurate in segmenting instances at approximately 80%-90%.

[0124] Qualitative segmentation results

[0125] Figure 11A , Figure 11B , Figure 11C and Figure 11D Several examples of segmentation are shown, produced by our settings for cutting carrots, chicken nuggets, taro, and fried food. Figure 11A , Figure 11B , Figure 11C and Figure 11DThe selected pickable instance for segmentation is shown, highlighted with its highest point labeled 'x'. A square region is selected around point 'x', as indicated by the white square. Depth geodesics are calculated from the highest point 'x' to the system of points surrounding 'x' and inside the square. Points classified as belonging to the same instance are marked with small black circles. A convex hull is calculated for all these black-circled points, forming the instance segment. Once a segment is found to be removed from the depth image, another point 'x' is selected as the highest point in the remaining point cloud, and the surrounding region will not overlap with any previously selected region by more than 50%, and the segmentation process is repeated.

[0126] Figure 12 Hardware diagrams illustrating some components that may be incorporated into an imaging controller 1200 according to some embodiments of the present disclosure are shown. The imaging controller 1200 may include a processor 1220 configured to execute stored instructions, and a memory 1240 storing instructions executable by the processor. The processor 1220 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1240 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The processor 1220 is connected to one or more input and output devices via a bus 1206. The imaging controller 1200 may also include a storage device 1230 suitable for storing a first module (pre-trained neural network 1231), a second module (seed point selection program module) 1232, and a third module (geodesic program module) 1233 for storing instructions executable by the processor 1220. In some cases, modules 1231, 1232, and 1233 may include... Figure 5 or Figure 3 The architecture of the pre-trained image segmentation neural network 505 or 304 shown is selected. Figure 5 or Figure 3 The module for seed point 301 shown, and the creation Figure 3 and Figure 5 The module contains depth geodesics 303 and 504. Storage device 1230 can store data in data storage device 1234. Storage device 1230 can be implemented using a hard disk drive, optical disk drive, thumb drive, drive array, or any combination thereof.

[0127] Imaging controller 1200 may include an input interface for receiving depth images or external data 1295 including a set of training datasets from an imaging device including a camera. The input interface may include a human-machine interface 1210 within the imaging controller 1200, which connects processor 1220 to a keyboard / measurement device 1211 and a pointing device 1212, wherein the pointing device 1212 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc. Alternatively, the input interface may include a network interface controller 1250 adapted to connect imaging controller 1200 to a network 1290 via bus 1206. Through network 1290, external data 1295 may be downloaded and stored in storage system 1230 as training and / or operational data 1234 for storage and / or further processing.

[0128] Still referencing Figure 12 The imaging controller 1200 may include an output interface linked via a bus 1206 and a display interface 1260 adapted to connect the imaging controller 1200 to a display device 1265. The display device 1265 may include a computer monitor, camera, television, projector, or mobile device, etc. For example, the imaging controller 1200 may be connected to a system interface 1270 adapted to connect a controlled system (robot controller) 1275 or different systems 1275 controlled based on some aspects. Alternatively, the imaging controller 1200 may be connected via a bus to an application interface 1280, the bus being adapted to connect the imaging controller 1200 to... Figure 13 The robot 150 has an imaging device / camera (depth imaging camera) 1285. In this case, the imaging controller 1200 is configured to control the imaging device / camera 106 ( Figure 13 The robot 150 acquires a depth image of the object to be manipulated by the robot 150, segments pickable instances among the object instances in the depth image, and generates geometric features of the pickable instances for the robot 150's controller (manipulator controller). In some cases, the geometric features generated by the imaging controller 1200 from the segmented instances may include the external size of the instance, the orientation of the instance, the shape of the instance, etc., enabling the robot 150 to pick up the corresponding pickable object from objects in a container or on a worktable.

[0129] Figure 13A robot configuration 100 of a robot 150 according to some embodiments is shown. The robot 150 includes a controller and a robotic arm 101 for performing instance picking operations. The robotic arm 101 includes a wrist 102 for securing multiple degrees of freedom of a moving object. In some implementations, the wrist 102 is provided with an end tool 104 for holding an object 103 and / or for performing any other robotic operations, such as picking operations. The end tool 104 may be, for example, a gripper or other picking tool, such as a vacuum tool. Hereinafter, 'end tool' may refer to a 'grip' or other picking tool, such as a vacuum tool, or they may be used interchangeably. According to embodiments, the purpose of the picking operation is to pick up a single instance of an object, typically the instance closest to the gripper.

[0130] Robot 150 is configured to perform a pickup operation (e.g., pickup of segmented object instance 103) along a trajectory while using imaging device 106 connected to imaging controller 1200, which can provide a depth image of the object to be manipulated by robotic arm 101. Imaging controller 1200 is connected to controller of robot 150 such that controller of robot 150 acquires and uses features of the segmented instance from imaging controller 1200. As used herein, the trajectory corresponds to a path defining the motion of object 103 held by gripper 104 for performing the pickup operation. In simple scenarios, the trajectory may simply indicate the vertical movement of wrist 102. However, since wrist 102 includes multiple degrees of freedom, the trajectory may include a motion profile spanning multidimensional space.

[0131] The pose of an object refers to the combination of its position and orientation. The gripper 104 can move in a starting pose 111. The pose of the gripper 104 corresponding to the starting pose 111 is called the starting pose of the gripper 104. According to the embodiment, the purpose of the pick-up operation is to pick up the segmented instance object 103. The pose 115 of object 112 can refer to the position and / or orientation of object 112. The robot 150 is configured to move the gripper 104 along trajectory 113 to pick up object 103 in pose 114. The pose 114 of object 103 is called the target pose. The pose of the gripper 104 corresponding to the target pose is called the target pose of the gripper 104.

[0132] The target orientation of the gripper 104 is determined based on the position of the object 112. Upon successful completion of the pick-up operation, the orientation of the gripper 104 of the robot arm 101 is considered to have reached the target orientation. Therefore, achieving the target orientation of the gripper 104 is equivalent to successfully performing the pick-up operation. According to the embodiment, the trajectory 113 is defined based on the initial and target orientations of the gripper 104 and the orientation 115 of the object 112. Furthermore, such a pick-up operation can be repeatedly performed by the robot 150.

[0133] Figure 14 This is a schematic diagram illustrating, by way of non-limiting example, some techniques that can be used to implement methods and systems according to embodiments of the present disclosure. The computing device 1400 represents various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing device 1400 may include a power supply 1408, a processor 1409, a memory 1410, and a storage device 1411, all connected to a bus 1450. In some cases, the storage device 1411 may include a power supply 1408, a processor 1409, a memory 1410, and a storage device 1411, all connected to a bus 1450. Figure 3 The diagram shows the architecture of the trained neural network 505 (304, 1231) in the training pipeline. Furthermore, the architecture of the trained neural network can be obtained via the network from external data 1295 storing the trained neural network. Additionally, a high-speed interface 1412, a low-speed interface 1413, a high-speed expansion port 1414, and a low-speed connection port 1415 can be connected to the bus 1450. Furthermore, a low-speed expansion port 1416 is connected to the bus 1450.

[0134] Depending on the specific application, various component configurations that can be mounted on a common motherboard are considered, as a non-limiting example, 1430. Furthermore, input interface 1417 can be connected to external receiver 1406 and output interface 1418 via bus 1450. Receiver 1419 can be connected to external transmitter 1407 and transmitter 1420 via bus 1450. External memory 1404, external sensor 1403, machine 1402, and environment 1401 can also be connected to bus 1450. Additionally, one or more external input / output devices 1405 can be connected to bus 1450. Network interface controller (NIC) 1421 can be adapted to connect to network 1422 via bus 1450, wherein data or other data can be presented on third-party display devices, third-party imaging devices, and / or third-party printing devices external to computer device 1400.

[0135] Still referencing Figure 14 Furthermore, it is anticipated that memory 1410 can store instructions executable by computer device 1400, historical data, and any data that can be utilized by the methods and systems of this disclosure. Memory 1410 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 1410 may be one or more volatile memory cells and / or one or more non-volatile memory cells. Memory 1410 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0136] Still referencing Figure 14 Storage device 1411 may be adapted to store supplemental data and / or software modules used by computer device 1400. For example, storage device 1411 may store historical data and other relevant data as mentioned above with respect to this disclosure. Alternatively or additionally, storage device 1411 may store historical data similar to the data mentioned above with respect to this disclosure. Storage device 1411 may include hard disk drives, optical disk drives, thumb drives, drive arrays, or any combination thereof. Furthermore, storage device 1411 may contain computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, or magnetic tape devices, flash memory or other similar solid-state storage devices, or device arrays, including devices in storage area networks or other configurations. Instructions may be stored in an information carrier. When executed by one or more processing devices (e.g., processor 1409), the instructions perform one or more methods, such as those described above. The system can be optionally linked via bus 1450 to a display interface or user interface (HMI) 1423 suitable for connecting the system to display device 1425 and keyboard 1424, wherein display device 1425 may include computer monitor, camera, television, projector or mobile device, etc.

[0137] Still referencing Figure 14 The computer device 1400 may include a user input interface 1417 adapted for a printer interface (not shown), which can also be connected via bus 1450 and adapted to connect to a printing device (not shown), wherein the printing device may include a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer, etc. A high-speed interface 1412 manages bandwidth-intensive operations of the computing device 1400, while a low-speed interface 1413 manages lower bandwidth-intensive operations. This allocation of functions is merely an example. In some implementations, the high-speed interface 1412 may be connected to a memory 1410, a user interface (HMI) 1423, and a keyboard 1424 and a display 1425 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1414, which can accept various expansion cards (not shown) via bus 1450. In this implementation, the low-speed interface 1413 is connected via bus 1450 to a storage device 1411 and a low-speed expansion port 1415. The low-speed expansion port 1415, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected, for example, via a network adapter to one or more input / output devices 1405, as well as a keyboard 1424, an indicating device (not shown), a scanner (not shown), or other networked devices such as a switch or router.

[0138] Still referencing Figure 14The computing device 1400 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 1426, or multiple times in a set of such servers. Furthermore, it can be implemented in a personal computer such as a laptop computer 1427. It can also be implemented as part of a rack server system 1428. Alternatively, components from the computing device 1400 can be used with, for example,... Figure 13 Other component combinations of the implementation method. Each such device may include one or more computing devices 1300 and 1400, and the entire system may consist of multiple computing devices communicating with each other.

[0139] While the aforementioned robotic systems, for example, describe robots capable of manipulating / assembling product components, they can also be applied to robots used in food processing plants where large quantities of food are irregularly shaped objects (cut vegetables, fried chicken, etc.). Robotic systems, including those for generating similar images from real depth images and automatically segmenting multiple instances of rigid objects in depth images, can be applied to the automation of industrial robots capable of manipulating food in food processing plants. Furthermore, these robotic systems can be applied to segmentation (method) systems for food recognition. Segmentation is one of the most popular and important problems in image processing. High segmentation accuracy and short training and computation times are essential for its application in food processing plants.

[0140] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing one or more exemplary embodiments. Various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the subject matter set forth in the appended claims.

[0141] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other modifications and alterations may be made within the spirit and scope of this disclosure. Therefore, the aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. An imaging controller for segmenting instances from a depth image including an object to be manipulated by a robot, the imaging controller comprising: An input interface configured to receive a depth image including an object; A memory configured to store instructions and a neural network trained to segment instances of the object from the depth image; as well as A processor, coupled to the memory, configured to execute the instructions to segment pickable instances using a trained neural network. The steps of the instruction include: Select the highest point in the depth image; Use shapes to define regions such that the regions surround the highest point; Points in the region of the depth image are sampled; Calculate the depth geodesic between the highest point and the sampling point; Submit the depth geodesic to the neural network to segment pickable instances among instances of the object in the depth image; and An output interface configured to output the geometric features of the pickable instance to the robot's manipulator controller.

2. The imaging controller according to claim 1, wherein, The depth image is acquired by a camera or sensor.

3. The imaging controller according to claim 1, wherein, The steps also include: Select the next highest point from the depth image such that the pickup radius around the next highest point does not include overlap with the pickup instance; Define the next region using a shape, such that the next region surrounds the next highest point; Sample subsequent points in the next region of the depth image; Calculate the subsequent depth geodesic between the next highest point and the sampled subsequent points; and The depth geodesic is submitted to the neural network to segment the next pickable instance among instances of the object in the depth image.

4. The imaging controller according to claim 3, wherein, Continue with the steps of selection, limitation, sampling, calculation, and submission until the steps have been performed on the remaining objects in the depth image.

5. The imaging controller according to claim 1, wherein, The shape is a square, rectangle, triangle, circle, or ellipse.

6. The imaging controller according to claim 1, wherein, The neural network is trained to classify each feature vector as belonging to the same instance or different instances.

7. The imaging controller according to claim 1, wherein, The endpoints of the geodesic are initialized using peak values ​​generated by the watershed algorithm.

8. A computer-implemented method for training a neural network to segment instances in a depth image, wherein, The method uses a processor coupled with stored instructions for implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the following steps: Select a depth image from a set of depth images; Determine the points at xy positions on a two-dimensional image grid and their corresponding depth points with respect to a selected depth image, wherein the points on the two-dimensional image grid are annotated with ground fact instance labels; Calculate the geodesic straight lines between the identified annotated point pairs; A depth geodesic is generated by projecting the geodesic line onto the depth image; Each of the depth geodesics is discretized to create a discretized vector, wherein each discretized vector corresponds to one of the depth geodesics between the determined annotated pair of points; and The discretized vector and its corresponding annotation label are submitted to the neural network, wherein the steps from selection to provision are repeated until the remainder of the entire set of depth images has been used.

9. The method according to claim 8, wherein, The neural network is trained to classify each feature vector as belonging to the same instance or different instances.

10. The method according to claim 9, further comprising the following step: The convex hull of all endpoints of the geodesic is calculated as pickable points, and the geodesic is classified as belonging to the same instance.

11. The method according to claim 8, wherein, The generated depth geodesics are debiased so that the geodesics are created from points at lower depths to points at higher depths.

12. The method according to claim 8, wherein, The endpoints of the geodesic are initialized using peak values ​​generated by the watershed algorithm.

13. The method according to claim 12, wherein, The endpoints of the geodesic are determined using a system selection method.

14. A container picking system for picking up objects from a container, the container picking system comprising: An end tool, the end tool being configured to pick up an object from the object; A robotic arm, including the end-effector, wherein the robotic arm is configured to be driven by a control signal including an end-effector signal to pick up the object from the container using the end-effector; An interface configured to send and receive the control signals, sensor signals from sensors arranged on the robotic arm, and imaging signals from at least one imaging device; A memory configured to store instructions for a robot control program, and a trained neural network and classifier for segmenting instances of the object from the depth image, the trained neural network having been trained by the computer-implemented method according to claim 8; and A processor, coupled to the memory, is configured to execute instructions to segment pickable instances using the trained neural network and generate control signals to drive the robotic arm and the end effector, wherein the instructions include the following steps: Select the highest point in the depth image; Use shapes to define regions such that the regions surround the highest point; Points in the region of the depth image are sampled; Calculate the depth geodesic between the highest point and the sampling point; The depth geodesic is submitted to the neural network to segment pickable instances among instances of the object in the depth image; Based on the imaging signal, generate the geometric features and control signals of the pickable instance; and The generated geometric features and the generated control signals are sent to the robotic arm, causing the end-effector to pick up the object corresponding to the pickable instance from the container.

15. The container picking system according to claim 14, wherein, The depth image is acquired by a camera or sensor.

16. The container picking system according to claim 14, wherein, The steps also include: Select the next highest point from the depth image such that the pickup radius around the next highest point does not include overlap with the pickup instance; Define the next region using a shape, such that the next region surrounds the next highest point; Sample subsequent points in the next region of the depth image; Calculate the subsequent depth geodesic between the next highest point and the sampled subsequent points; and The depth geodesic is submitted to the neural network to segment the next pickable instance among instances of the object in the depth image.

17. The container picking system according to claim 16, wherein, Continue with the steps of selection, limitation, sampling, calculation, and submission until the steps have been performed on the remaining objects in the depth image.

18. The container picking system according to claim 14, wherein, The shape of the area is square, rectangle, triangle, circle, or ellipse.

19. The container picking system according to claim 14, wherein, The neural network is trained to classify each feature vector as belonging to the same instance or different instances.

20. The container picking system according to claim 14, wherein, The endpoints of the geodesic are initialized using peak values ​​generated by the watershed algorithm.

Images