Insegan: generative method for instance segmentation in depth images

Abstract

Systems and methods for generating photorealistic images from real depth images. A generative adversarial neural network (GAN) is trained by accessing test depth images having instances that are the same as instances of real depth images. The test depth images are input into a generator to generate estimated depth images representing implicit three-dimensional models of objects. The respective estimated depth images are input into a discriminator to obtain a loss and into a pose encoder to obtain a matching loss. The process is iteratively repeated until the loss is minimized to a threshold to end training. The trained GAN pose encoder is used to identify instances in real images to generate pose transformation matrices for respective instances in the real images. Pixels in a depth image corresponding to the instances of the real images are identified and the pixels of the depth image are merged to form an instance segmentation map of the real depth image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to generating realistic depth images of real depth images, and more particularly, to automatically segmenting multiple instances of rigid objects in depth images while unsupervised using a three-dimensional generative adversarial network. BACKGROUND

[0002] Instance segmentation is a fundamental step in a variety of autonomous tasks. For example, segmenting instances of objects (e.g., cars) in a scene is critical for automated driving, object counting is necessary for visual reasoning, and identifying instances of organs in medical images is important for automated diagnosis.

[0003] Conventional approaches to instance segmentation are based on object recognition models and semantic segmentation models. These conventional models are trained on annotated training sets consisting of bounding boxes or class segmentation masks of objects. However, such approaches are inductive when annotations are difficult to obtain, or such approaches are computationally too expensive for participants in the computer vision industry. For example, industrial robots designed to pick up arbitrary objects, or robotic assistants in food factories where food items to be processed change from week to week, thus often require large annotated training sets to reliably predict instances of objects, which can be expensive to obtain, or models trained for one shape of an object can not be used to detect other shapes, rendering the trained model inductive.

[0004] Some conventional methods solve instance segmentation problems for 2D images and 3D images in a supervised manner by inputting training image data, which consists of either a bounding box of an object or its class segmentation mask. However, these conventional methods are computationally too expensive when confronted with complex scenes. Another problem that accompanies these conventional methods and many other conventional instance segmentation methods is with the training data. For example, the training image data is generated in a way that fails in generating photorealistic images or fake images of real depth images. For example, the training image data is generated in a way that lacks any realistic quality of real images and results in a domain shift problem where there is a different data distribution in the target domain (photorealistic images or fake images) compared to the source domain (real images). Among many problems that accompany these conventional methods with the domain shift problem, at least one reason is because the instance segmentation model learns to identify objects based on identifying such objects that are fine renderings of synthetic objects that do not appear in reality. In fact, many conventional methods fail when trying to handle complex scenes such as in a food factory. For example, in a robotic pick and place food application at a food factory site, the type of food items to be handled by the robot are randomly put into bins and where the type of food items to be handled can change daily or weekly. These conventional methods cannot handle such complex scenes, resulting in being computationally too expensive for today's computer vision industry participants.

[0005] Today's conventional instance segmentation methods need to provide more efficient solutions to achieve the expectations required by today's computer vision industry participants, such as designing a method that includes: (a) a method that does not require segmentation labels or CAD models; (b) a generative learning model that does not require camera parameters; (c) a method that is reliable and stable when handling complex scenes; (d) a method that is flexible in learning to automatically segment multiple (same) instances of objects in bins in an unsupervised manner; and so on.

[0006] Therefore, there is a need to generate photorealistic images from real depth images and more particularly, to automatically segment multiple instances of rigid objects in depth images while being unsupervised using a three-dimensional generative adversarial network. SUMMARY

[0007] The present disclosure relates generally to generating photorealistic images of real depth images and more particularly, to automatically segment multiple instances of rigid objects in depth images while being unsupervised using a three-dimensional generative adversarial network.

[0008] Embodiments of the present disclosure provide a modified generative adversarial network (GAN) for segmenting instances of the same rigid object in a depth image. In contrast to existing methods, the present disclosure includes a uniquely created instance segmentation GAN (InSeGAN) method that does not require any segmentation labels, camera parameters, or CAD models for training. Rather, the InSeGAN method uses only a set of depth images, each depth image including multiple instances of an object.

[0009] Embodiments of the present disclosure are configured to automatically segment multiple instances of a rigid object in a depth image while being unsupervised, which is a novel approach in the field of computer vision activities. Embodiments of the present disclosure are configured to perform a variety of applications including robot-assisted tasks. For example, to pick an object from clutter, a robot needs instance segments to plan grasping correctly.

[0010] Some embodiments of the present disclosure include an InSeGAN architecture that follows some aspects of a generative adversarial framework, however, the InSeGAN is redesigned to include some unlikely features that a regular GAN designer would not think to incorporate into a GAN. At least one goal of the InSeGAN is to design a learning model that uses only training depth images with the same instances as the instances of the real depth images (without any labels). Thus, at test time, when given real depth images of multiple instances of a three-dimensional (3D) object, the learned model outputs segmentation masks associated with each instance in the real depth images.

[0011] At least one implementation obtained from experiments is that the training image data should be training depth images with the same instances as the instances of the real depth images of the object, because the training model learns to generate instances similar to the instances in the training images in an unsupervised manner. The InSeGAN allows learning instance segmentation without any human-generated labels. Furthermore, if multiple instance depth images of several depths are provided, our model can adapt to learn any shape of the instances, allowing the inductiveness of the method. Regular methods typically use supervised training, while some embodiments of the present disclosure are based on modifying aspects of a standard 3D GAN method.

[0012] Another realization from the experiments is that the systematic injection of random noise helps control various attributes in the image generation. This is because the GAN learns the distribution of the training depth image data and maps the random noise distribution to the distribution of the image data. We hypothesize that the attributes in the image constitute a sub-distribution of the image distribution and the architecture of the system design of the GAN can allow targeting the random noise distribution to affect only the sub-distribution of attributes. Thus, this targeted noise injection allows disentangling the attributes (e.g., pose) of the instances from other factors such as the shape or appearance of these instances. This disentangling thus allows only estimating the pose of the instances at inference time, which simplifies the task and thereby improves the results over conventional approaches. At least one reason is that conventional approaches do not approach the problem from this perspective, while some embodiments of the present disclosure use a 3DGAN architecture that is the earliest approach to disentangle pose from appearance for the task of generating single faces or single 3D objects, and further apply it to multi-instance segmentation in depth images.

[0013] Some embodiments include InSeGAN designed with a generator module that does not take a single noise vector as input (as in standard GANs), but rather takes multiple noise vectors and each noise vector has d dimensions from a standard normal distribution, and generates a multi-instance depth image as output, as mentioned above. In other words, the InSeGAN is configured to capture the latent 3D representation of the 3D object using an implicit template implemented as a parameter tensor; generate 3D pose transformation matrices from the noise vectors using a pose encoder implemented as a first neural network. Thus generate pseudo depth images using a generator implemented as a second neural network configured to apply multiple 3D transformation matrices on the implicit template, said multiple 3D transformation matrices are generated by the first neural network using multiple noise vectors. It should be noted that in some cases, the “pose encoder” can be referred to as “pose decoder” and the “instance encoder” can be referred to as “instance pose encoder”.

[0014] At least one benefit of taking multiple random noise vectors (equal to the number of instances) as input to generate instance-specific feature maps is that the InSeGAN framework implicitly forces the generator to capture the distribution of the single object instances when training the model in a generative adversarial setting. These are merged and rendered to generate multiple instance depth images. These images are then fed to the discriminator that distinguishes them from the distribution of the ground-truth data.

[0015] Some embodiments of the present disclosure include a discriminator module trained to distinguish whether an input is an image generated by the generator or a sample from the data distribution. By way of non-limiting example, the generator and discriminator modules can be trained in a minimax adversarial game, such that the generator module can learn to generate images that can fool the discriminator module, while the discriminator module, in turn, can be trained to distinguish whether its input is real or fake; optimization occurs when the discriminator module is unable to identify whether its input is from the generator module or from the discriminator module.

[0016] When the trained modules are used for instance segmentation, the image encoder takes the generated images as input and is trained to reproduce instance noise vectors; these vectors are decoded individually by the generator to form individual object instances. The encoder can be an encoder module that is an important factor in enabling instance segmentation. Specifically, the encoder module takes the generated depth images as input and outputs latent noise vectors. The essence of InSeGAN is to have the generator generate instance segments with depth images known implicitly, such that the encoder module can be trained on them to learn to disentangle instances. The encoder module will ultimately learn to perform instance segmentation on real images from the training depth images, in the limit that the generated distribution coincides with the real data distribution guided by the discriminator.

[0017] The proposed InSeGAN is a novel 3D GAN that solves the conventional problem of unsupervised instance segmentation by using a generative adversarial framework. By pairing the discriminator with a carefully designed generator, the model can reconstruct individual object instances even in cluttered and heavily occluded scenarios; the learned model can be used to enable instance segmentation when encoded. Based on extensive experimental testing of InSeGAN, the results enable instance segmentation performance of the prior art, exemplifying performance that is substantially better than existing methods.

[0018] Some main contributions of the novelty of embodiments of the present disclosure can include designing a 3D GAN architecture that can learn the same instances of objects in a segmentation box in an unsupervised manner. Among others, the InSeGAN framework takes multiple noise vectors as input. Further, a custom designed generator or single instance generator learns the appearance distribution of rigid objects from a cluttered scene composed of multiple same instances. Another aspect includes a depth renderer that takes the aggregated feature map of a single instance and renders a multi-instance depth image. A further aspect is a discriminator that takes a depth image as input and learns to discriminate whether the generated image is composed of the exact number of instances as in the ground truth image. Another aspect is a custom encoder that takes the generated multi-instance depth image as input and as output, generates the individual noise vectors that produced the image. Yet another aspect is a per-depth pooling scheme that aggregates single instance depth images, aggregates them by z-buffer, applies a threshold, and finds the instance segmentation mask. A further aspect is a pose encoder that takes a noise vector and generates an axis-angle representation of the 6D pose of the instance. Another aspect is a 3D feature map template that renders a single instance of an object according to an arbitrary 6D pose generated by the pose encoder.

[0019] According to embodiments of the present disclosure, a system for generating photorealistic images from real depth images is provided. The system includes a processor connected to a transceiver and a data storage device, and computer readable instructions that, when executed by the processor, cause the processor to receive, via the transceiver, real depth images having instances of objects. During a training period, a generative adversarial neural network (GAN) having a generator, a discriminator, and a pose encoder is trained. The training period includes accessing training depth images from the data storage device. The training images have the same instances as the instances of the real depth images. The training depth images are input into the generator to generate estimated depth images representing implicit three-dimensional models of the objects. The respective estimated depth images are simultaneously input into the discriminator and the pose encoder. The discriminator discriminates the respective estimated depth images against the real depth images to obtain a discriminator loss. The pose encoder generates a set of pose transformation matrices matching a set of sampled transformation matrices to obtain a matching loss. The process is iteratively repeated for the estimated depth images until the discriminator loss and the matching loss are minimized to a threshold to end the training period. The instances in the real images are identified by inputting the real images into the pose encoder of the trained GAN to generate pose transformation matrices for respective instances in the real images. The respective pose transformation matrices are used to generate depth images for the respective instances. Pixels in the depth images corresponding to the instances in the real images are identified by identifying a set of pixels in the respective depth images based on a predetermined pixel depth threshold. The set of pixels of the depth images are merged to form an instance segmentation map of the real depth images.

[0020] According to another embodiment of the disclosure, a method for generating 3D object instance segmentation by generating photorealistic images from real depth images is provided. The method comprises the steps of: receiving a real depth image having instances of objects. During a training period, a generative adversarial neural network (GAN) is trained, the GAN having a generator, a discriminator, and a pose encoder. During the training period, a training depth image is accessed from a data storage device. The training depth image has the same instances as the real depth image. The training depth image is input into the generator to generate an estimated depth image representing an implicit three-dimensional model of the objects. Each estimated depth image is simultaneously input into the discriminator and the pose encoder. The discriminator discriminates each estimated depth image against the training depth image to obtain a discriminator loss. The pose encoder generates a set of pose transformation matrices matching a set of sampled transformation matrices to obtain a matching loss. The process is iteratively repeated for the estimated depth images until the discriminator loss and the matching loss are minimized to a threshold to complete the training period. The instances in the real depth image are identified by inputting the real depth image into the pose encoder of the trained GAN to generate pose transformation matrices for each instance in the real depth image. Each pose transformation is used to generate a simulated depth image for each instance in the real depth image. Pixels in each simulated depth image corresponding to the instances in the real depth image are identified by identifying a set of pixels in each simulated depth image based on a predetermined pixel depth threshold. The set of pixels in the simulated depth image corresponding to the same instance are merged to generate an instance segmentation map of a photorealistic image of the real depth image.

[0021] According to another embodiment of the present disclosure, a system for object-instance segmentation of depth images composed of identical instances of 3D objects is provided. The system includes a processor and a memory having computer code instructions stored thereon, the processor and memory being configured with the computer code instructions to cause the system to: capture latent 3D representations of 3D objects using an implicit template implemented as a parameter tensor. Generate 3D pose transformation matrices from noise vectors using a pose encoder implemented as a first neural network. Generate pseudo depth images using a generator implemented as a second neural network configured to apply a plurality of 3D transformation matrices on the implicit template, the plurality of 3D transformation matrices being generated by the first neural network using a plurality of noise vectors. Discriminate between real depth images and pseudo depth images using a discriminator implemented as a third neural network, and output a discrimination result in accordance with the discrimination. Decompose the pseudo depth images into latent vectors using an instance encoder implemented as a fourth neural network. The instance encoder outputs a set of latent vectors matching a set of noise vectors used in the pose encoder. During a training period, perform iterative operations of the implicit template, the pose encoder, the generator, the discriminator, and the instance encoder. Train the discriminator and the generator in an adversarial relationship with each other. Train the generator, the implicit template, the pose encoder, and the instance encoder in a cooperative relationship with each other. Wherein, the instance encoder trains its fourth neural network at the end of the training period to generate latent vectors matching noise vectors corresponding to 3D poses of instances in the generated depth images, the latent vectors having an increased amount of accuracy than at the beginning of the training period. Wherein, after the training period, given a real depth image comprising a plurality of instances of 3D objects, the instance encoder generates latent vectors of individual 3D object instances. The latent vectors are iteratively used by the pose encoder to generate corresponding 3D transformation matrices to generate single instance depth images by the generator, the single instance depth images composed of only one instance of the 3D objects in each depth image. Wherein, pixels corresponding to valid depths in the generated single instance depth images are used as instance segmentation.

[0022] According to another embodiment of the present disclosure, there is provided a system comprising a processor and a memory having computer code instructions stored thereon, the processor and memory being configured with the computer code instructions to cause the system to: use an implicit template configured to capture a latent 3D representation of a three-dimensional (3D) object, and a pose encoder configured to generate a 3D pose transformation matrix from a noise vector. The generator is configured to apply a plurality of 3D transformation matrices using a plurality of noise vectors on the implicit template to generate a pseudo depth image. The pseudo depth image is output jointly to a discriminator and an instance encoder. Wherein the discriminator is configured to distinguish between a real depth image and the pseudo depth image. An outcome of the discrimination is output and the instance encoder is configured to output a set of latent vectors matching a set of noise vectors used in the pose encoder. Wherein during a training period, the discriminator and the generator are trained in an adversarial relationship to each other, and the generator, the implicit template, the pose encoder, and the instance encoder are trained in a cooperative relationship to each other. Wherein the trained instance encoder is configured to generate a latent vector for each 3D object instance. The latent vector is iteratively used by the trained pose encoder to generate a corresponding 3D transformation matrix to generate a single instance depth image by the trained generator, the single instance depth image including only one instance of the 3D object in each depth image. Wherein pixels corresponding to valid depths in the generated single instance depth image are used as instance segmentation.

[0023] Further, some embodiments of the present disclosure can provide a robotic system for manipulating objects. The robotic system can include: a robotic arm having a manipulator configured to manipulate objects, wherein the manipulator includes: a twist configured to ensure a plurality of degrees of freedom of moving objects; an end tool configured to hold objects; an imaging device configured to measure a starting pose and / or a target pose of the robotic arm for successfully manipulating objects; and a robot controller including: a processor connected to a transceiver and a data storage device, and computer readable instructions including a trainable generative adversarial neural network (GAN) having a trainable generator, a trainable discriminator, and a trainable pose encoder, which, when executed by the processor, cause the processor to: receive, via the transceiver, a real depth image having a single instance of an object; simultaneously input each estimated depth image into the discriminator and the pose encoder, which discriminates each estimated depth image against the real depth image to obtain a discriminator loss, and which generates a set of pose transformation matrices matching a set of sampled transformation matrices to obtain a matching loss; iteratively repeat the process on the estimated depth image until the discriminator loss and the matching loss are minimized to a threshold to end a training period; identify instances in the real image by inputting the real image into the pose encoder of the trained GAN to generate a pose transformation matrix for each instance in the real image, wherein each pose transformation matrix is used to generate a depth image for each instance; and identify a set of pixels in each depth image based on a predetermined pixel depth threshold to thereby identify pixels in the depth image corresponding to instances in the real image, and merge the set of pixels of the depth image to form an instance segmentation map of the real depth image.

[0024] Practical Applications

[0025] The present embodiment addresses the computer vision industry-specific technical problem of being able to generalize when annotation is difficult or when operating in complex scenes. Furthermore, some embodiments of the present disclosure can be applied to a robot (robotic system) comprising a manipulator configured to pick and place food items (products) at a food factory activity site, where the food items of this kind to be picked by the robot are randomly put into a bin, and where the type of food items to be processed can change daily or weekly. For example, due to the unique technical features of this solution, it results in industrial robots or robot assistants in food factories designed to pick arbitrary objects being an ideal scenario to use this system and method, i.e., conventional methods are unable to generalize when annotation is difficult or when operating in complex scenes. These conventional methods also require segmentation labels, camera parameters, or CAD models for training, which includes further expenses. However, embodiments of the present disclosure do not require any segmentation labels, camera parameters, or CAD models for training. Rather, embodiments of the present disclosure use the InSeGAN method that only uses a set of depth images, each depth image comprising multiple instances of objects. Another problem that accompanies conventional methods is the expense of gaining access to an annotated training set consisting of object bounding boxes or their class segmentation masks to train a deep learning model, and these conventional methods typically use supervised training. Thus, given the above comments, these conventional methods become too computationally expensive as a participant in the computer vision industry activity site.

[0026] Some of the technical features of the embodiments of the present disclosure include using training depth images with the same instances as instances of real depth images (without any labels). Thus, at test time, when given multiple instances of real depth images of a three-dimensional (3D) object, the learned model outputs segmentation masks associated with each instance in the real depth images. Specifically, the training image data includes training depth images with the same instances as instances of real depth images of an object, where the training model learns to generate instances similar to the instances in the training images in an unsupervised manner. InSeGAN allows learning instance segmentation without any human-generated labels (i.e., unsupervised). If multiple instance depth images of several depths are provided, the InSeGAN model is adapted to learn any shape of instances, allowing fast induction of the method, which is an advantage over conventional methods. Some benefits of this technical feature of the present disclosure are that the amount of computation time for induction is reduced when annotation is difficult or when operating in complex scenes (i.e., food factories, segmenting object instances (e.g., cars) in scenes for automated driving, object counting for visual reasoning, and identifying instances of organs in medical images for automated diagnosis). With reduced computational expenditure and total expenditure, it results in making the embodiments useful to participants in computer vision industry events, which will make participants more profitable and more competitive compared to conventional methods for the reasons mentioned above.

[0027] Another technical feature is that the systematic injection of random noise helps control various properties of the image generation aspect. This is because the GAN learns the distribution of the training depth image data and maps the random noise distribution to the distribution of the image data. We assume that the properties in the image constitute a sub-distribution of the image distribution and the architecture of the systematic design of the GAN can allow aiming the random noise distribution to only affect the sub-distribution of properties. Thus, this aimed noise injection allows disentangling the properties (e.g., pose) of the instances from other factors such as the shape or appearance of these instances. Therefore, at inference time, this disentangling allows only estimating the pose of the instances, which simplifies the task and thereby improves the results, outperforming the conventional methods. Another technical feature is that the InSeGAN is designed with a generator module that does not take a single noise vector as input (as in standard GANs), but rather takes multiple noise vectors and each noise vector has d dimensions from a standard normal distribution, and generates a multi-instance depth image as output, as mentioned above. In other words, the InSeGAN is configured to capture the latent 3D representation of the 3D object using an implicit template implemented as a parameter tensor; to generate 3D pose transformation matrices from the noise vectors using a pose encoder implemented as a first neural network. To generate pseudo depth images using a generator implemented as a second neural network configured to apply multiple 3D transformation matrices on the implicit template, said multiple 3D transformation matrices being generated by the first neural network using multiple noise vectors.

[0028] These additional limitations above reflect improvements in the technical field of computer vision systems that integrate features and aspects into practical applications, and these technical features (i.e., (1) using training depth images with the same instances as the instances of the real depth images (without any labels), (2) the systematic injection of random noise helps control various properties of the image generation aspect, etc., as mentioned above) and aspects are integrated into the technical solution of the technical problem and provide meaningful limitations thereto.

[0029] For example, some of the meaningful limitations include: (1) a reduced amount of computation time for induction when annotation is difficult or when operating in complex scenes; and (2) allowing disentangling the properties (e.g., pose) of the instances from other factors such as the shape or appearance of these instances. Thus, by way of non-limiting example, at inference time, this disentangling allows only estimating the pose of the instances, which simplifies the task and thereby improves the results, outperforming the conventional methods.

[0030] Thus, the system and method as a whole cannot be seen as merely performing aspects in the way a human would think, nor as merely gathering data, processing data, analyzing data, and displaying certain results in abstract aspects. Rather, the system and method can induce, and can be competitive and profitable for participants in the computer vision industry and other industries, at times when annotation is difficult, or when operating in complex scenes with less expense and computational expense. BRIEF DESCRIPTION OF DRAWINGS

[0031] The disclosed embodiments are further illustrated by the referenced drawings. The drawings presented are not necessarily to scale, with the emphasis generally being placed upon illustrating the principles of the disclosed embodiments.

[0032] Figure 1A is a schematic diagram illustrating an example of a training scheme in a three-dimensional (3D) generative adversarial network (GAN) (3D-GAN) referred to as instance segmentation (Se) generative adversarial network (GAN) or (InSeGAN) according to embodiments of the present disclosure.

[0033] Figure 1B is a schematic diagram illustrating an example of a first control flow in some method steps of Figure 1A showing a generation process of generating a multi-instance depth image according to some embodiments of the present disclosure.

[0034] Figure 1C is a schematic diagram illustrating an example of a second control flow in some method steps of Figure 1A showing a scheme of encoding the generated depth image into its instances according to some embodiments of the present disclosure.

[0035] Figure 1D is a schematic diagram illustrating an example of a third control flow in some method steps of Figure 1A showing a control flow of training an instance encoder via using the encoded latent vectors to recreate the already generated image according to some embodiments of the present disclosure.

[0036] Figure 2 shows a segmented picture.

[0037] Figure 3 shows a segmented picture.

[0038] Figure 4A schematic diagram illustrating some aspects of an exemplary InSeGAN inference pipeline is shown, which can include inputting a multi-instance depth image into an encoder module, which generates a set of latent vectors. According to some embodiments of the present disclosure, each latent vector in the set is individually input into a trained single-instance generator, and its output is rendered using a rendering module to form a single-instance depth image corresponding to the latent vector.

[0039] Figure 5 A picture illustrating qualitative results using InSeGAN on a nut from a training depth image is shown.

[0040] Figure 6 A schematic diagram illustrating some aspects of an architecture of an InSeGAN generator according to some embodiments of the present disclosure is shown.

[0041] Figure 7 A schematic diagram illustrating some aspects of an architecture of an encoder module according to some embodiments of the present disclosure is shown.

[0042] Figure 8 A schematic diagram illustrating some aspects of an architecture of a discriminator module according to some embodiments of the present disclosure is shown.

[0043] Figure 9A A plot illustrating a plot of convergence with respect to an object is shown.

[0044] Figure 9B A plot illustrating a plot of convergence with respect to an object is shown.

[0045] Figure 9C A plot illustrating a plot of convergence with respect to an object is shown.

[0046] Figure 10A A plot of mean intersection over union (mloU) against increasing dataset size according to some embodiments of the present disclosure is shown.

[0047] Figure 10B A plot of mean intersection over union (mloU) against increasing number of instances used in InSeGAN(n) (against 5 ground truth instances) according to some embodiments of the present disclosure is shown.

[0048] Figure 11 Table 1 illustrating results of an Ablative study with respect to various losses used in the mloU and InSeGAN generator implemented for two classes according to some embodiments of the present disclosure is shown.

[0049] Figure 12 Results from InSeGAN according to some embodiments of the present disclosure are shown.

[0050] Figure 13 A hardware diagram illustrating some components that can be incorporated in some systems according to some embodiments of the present disclosure is shown;

[0051] Figure 14 is a schematic diagram of a computing device illustrating some techniques that can be used to implement the method and system according to some embodiments of the present disclosure.

[0052] Figure 15A A robot configuration 100 of a robot 150 according to some embodiments of the present disclosure is shown.

[0053] Figure 15B An exemplary assembly operation according to an embodiment is shown.

[0054] Figure 15C A result of an assembly operation caused by an inaccurate determination of the pose of an object by an imaging device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0055] While the above figures set forth embodiments of the present disclosure, other embodiments can also be contemplated as discussed in the Background. The present disclosure presents exemplary embodiments by way of representative, not limiting, examples. Those skilled in the art will be able to design many other modifications and embodiments within the principles and scope of the presently disclosed embodiments.

[0056] Figures 1A-1D is a schematic diagram illustrating an example of a training scheme in a three-dimensional (3D) generative adversarial network (GAN) (3D-GAN) referred to as instance segmentation (Se) generative adversarial network (GAN) or (InSeGAN) according to some embodiments of the present disclosure.

[0057] As discussed further below, there are three distinct control flows in this framework. Figure 1B A first control flow illustrating a generative process that captures the generation of multiple-instance depth images is exemplified. Figure 1C A second control flow (black dashed line) illustrating a scheme showing encoding of the generated depth images into their instances is exemplified. Figure 1D A third control flow (black dotted line) illustrating a control flow showing training of the instance encoder to recreate the images that have been generated by using the encoded latent vectors is exemplified.

[0058] Some embodiments disclose a training dataset of n instances of a rigid object as where each of the n instances of the rigid object is represented by a training depth image are training depth images each consisting of n instances of a rigid object. To simplify the annotation, we will use The distribution of x is also characterized. We assume that n is known, and that is fixed. Technically, one aspect of the framework can work even if there are different n's for individual depth images. However, for simplicity of the exposition, the approach used is to assume that n is fixed and known for the entire dataset. There can be a way to infer n from the data of a given image using an additional learning module in the pipeline configured for some implementations.

[0059] Referring to Figure 1A , at least one goal in InSeGAN is to design a learning model using only (no labels of any kind) so that at test time, when given a depth image x, the learned model outputs a segmentation mask associated with the individual instances in the depth image.

[0060] The overview of InSeGAN starts with a discussion of the basic architecture of InSeGAN, which follows a standard generative adversarial framework, however with several non-trivial twists. It consists of a generator module G that, instead of taking a single noise vector as input (as in standard GANs), takes n noise vectors {z1, z2,..., zn} as input, each n} and outputs a multi-instance depth image, i.e., where, is used to denote the distribution of the generated depth images, and has the limit We represent the set of noise vectors as and the distribution of Z by the matrix Next, the discriminator module D is trained to distinguish between whether its input is an image generated by G or a sample from the data distribution. Modules G and D are trained in a minimax adversarial game, such that G learns to generate images that can fool D, while D in turn is trained to distinguish between whether its input data is real or fake; optimality occurs when D cannot tell whether its input is from G or In addition to the generator module and the discriminator module, we have the encoder module E that is key to implementing instance segmentation. Specifically, E takes as input a generated depth image and outputs latent noise vectors, i.e., The essence of InSeGAN is to have the generator G generate instance segments of depth images whose entanglement is known (by Z) so that E can train on them to learn to disentangle instances. Within the constraints of guided by the discriminator D, E will end up learning to instance segment real images from . Figure 1A , Figure 1B ​、 Figure 1C and Figure 1D An overview of the InSeGAN training procedure is shown. Next, we will describe in more detail the individual modules of the described modules.

[0061] With reference to Figure 1A 、 Figure 1B and Figure 1C , the InSeGAN generator is at least one key aspect of InSeGAN, enabling the generator G to jointly accomplish two tasks: (i) generate a depth image matching the input image distribution and (ii) identify individual object instances in the generated image

[0062] Driven by the above insight, at least one further implementation is to separate the generator G into two distinct modules: (i) an instance generator that generates feature maps for a single object instance, and (ii) a depth image rendering module that aggregates single instance features and renders a multi-instance depth image. Since instances are assumed to be of the same object, we propose to sample each noise vector z e Z from the same latent distribution z ~ N(0, I d ). Furthermore, we assume the existence of an implicit (and learned) 3D object model (template) that generates varying appearances of instances when subjected to geometric transformations.

[0063] Still with reference to Figure 1A 、 Figure 1B 、 Figure 1C and Figure 1D , where the first step in the generator procedure can be to generate a 6D geometric transformation that can be applied to the template to render an implicit instance of the object. For this, each noise vector z e Z is converted into an element in the special Euclidean group (SE(3)) using a pose encoder module (see Figure 1A and Fig. 1c), which is a fully connected neural network denoted as In detail, G p generates an axis-angle representation corresponding to a given noise vector z; this representation is next converted into an element in the special Euclidean group SE(3). We denote this operator by , i.e., Λ generates a rotation matrix R e SO(3) and translation vector in. The natural question in this context is why we do not directly sample the transformation matrix. This is because, as will become clear shortly, we need to match the output of the encoder module E to the representation of the instance; and it is computationally more efficient to work with rotation matrices lying on a non-linear geometric manifold using a similarity metric than with Euclidean noise vectors.

[0064] Next, we use the transformation matrix thus created, i.e., A(G p (z)), to geometrically transform the implicit shape tensor This parameter tensor is shared by all instances and will implicitly capture the shape and appearance of the object when trained (with other modules in the pipeline). A spatial transformer network (STN) is used to apply the geometric transformation to this implicit template. The transformed T is reshaped and projected from 3D to 2D using a single-instance projection module Gsto output The above steps can be formally written as:

[0065]

[0066] Still referring to Figure 1B , next we propose to combine them by average pooling these feature maps to render a multi-instance depth image using a rendering module Gr, as follows:

[0067] where, where, denotes the depth image generated by G. We represent the distribution of the generated image as Figure 1C This generation control flow is depicted in using black arrows.

[0068] Referring to the InSeGAN discriminator, note that as in standard GANs, the task of the discriminator D is to decide whether its input is from the distribution of natural multi-instance depth images (i.e., ) or synthesized by our generator G (i.e., ). Following the standard architecture, D is composed of several 2D convolutional layers, instance normalization layers, and Leaky RELU layers, and outputs a classification score, i.e., [0, 1]. The goal of training the discriminator and the generator, respectively, is to minimize the following losses:

[0069]

[0070]

[0071] The task of our discriminator is clearly different from previous works, as it has to learn: (i) to count whether the number of rendered instances matches the number of instances in the data distribution, (ii) to verify that the rendered 3D posed objects obtained via transformations of the object template T that is still to be learned, capture the individual appearance of the instances (which are also being learned), and (iii) that the layout of the rendered images is similar to the composition of instances in the data depth images. Fortunately, as with adversarial training of the generator, D can automatically achieve these desired behaviors as it accesses a large enough dataset.

[0072] Still referring to Figure 1B We can now introduce the InSeGAN encoder as an encoder module E, which is key to instance segmentation. The task of this module is to take the multi-instance depth image generated by G as input, and to reconstruct the individual noise vectors in the noise vector Z that were used to generate said image. Let us assume that the encoder output is a set of latent vectors. Indeed, when generating by aggregating n independently sampled instance appearances of an object, reversing this process is equivalent to disentangling into its respective instances, and when the generator is well trained, i.e., we will have finally learned the individual instances in the disentangled ground truth image. While this idea is simple in concept, it is not straightforward to implement in practice. There are four main difficulties: (a) the inputs Z of the generator and the outputs of E are unordered sets and need to be aligned before comparing them, (b) the average pooling operator in (2) aggregates several feature maps into one, which is an operation that loses the uniqueness of the individual instance feature maps in said instance feature maps, (c) the depth renderer G r can remove occluded parts of instances, causing ambiguities when mapping them back to noise vectors, and (d) the pose encoder G p projects its noise input to the space of rigid transformations, which is an inherently low-rank and non-linear operation. We tackle these challenges via imposing a loss on the encoder E to make it learn the inverse of the individual modules in the generator. Let us assume that the encoder E has two sub-modules: (i) an image un-renderer that takes the depth image and generates an image of feature maps, and (ii) an instance decoder that takes the feature maps from and generates

[0073] Alignment and reconstruction : To tackle our first difficulty, we propose to align the unordered sets Z and the set of noise Z and alignments. Specifically, we manage to find an alignment matrix where Π denotes the set of all such alignments (i.e., permutations) on its input that minimize the reconstruction loss:

[0074] where, where D denotes the pairwise distance between Z and the columns in and OT is some suitable matching scheme. We use the universal optimal transport (IPOT (imprecise proximal optimal transport). A fast proximal point method for computing exact wasserstein distance by Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha, PMLR, Uncertainty in Artificial Intelligence, 22-25 July 2019, Tel Aviv, Israel, Volume 115, Pages 433-453, <URL: http: / / proceedings.mlr.press / v115 / xie20b / xie20b.pdf) which returns a permutation matrix p * The permutation matrix p * is used to align the matrices before comparing them using the l2 distance. We use Figure 1D The solid arrow in

[0075] Intermediate reconstruction : Still referring to Figure 1C , to cope with difficulties (b) and (c) in the encoder design, which involves E learning to invert the deep renderer, we use the output from the de-renderer sub-module in E. Specifically, we force to reconstruct the average-pooled feature map in (2).Let us denote this loss by

[0076] Pose decoding : Although the above intermediate feature decoding strategy can be applied to the pose encoder G p as well, we do not output Comparing to the rigid transformation generated during the generation process would not be very efficient. This is because the human-generated geometry matrix involves a rotation matrix and therefore the optimization would ideally require a Riemannian optimization method in SO(3) space, which is not very suitable for standard optimization schemes. Moreover, there can be several different geometric transformations that can achieve the same output. To avoid this technicality, we propose to indirectly learn the rigid transformation by avoiding the exact reconstruction of the transformation and instead requiring it to have the desired outcome during the generation process. Specifically, we suggest using the geometry matrix generated by the encoder and use it as a noise matrix to generate a depth image This depth image is then compared to the depth image generated using Z in the previous pass. The following loss captures this idea:

[0077]

[0078] The above control flow is illustrated in Figure 1C by the dashed arrows from the noise vector to the pose encoder and all the way to the depth renderer (i.e., the output of G).

[0079] Encoder loss Referring still to Figure 1D When training the parameters of the encoder module, we combine the three losses described above (see Supplementary Materials for details about its architecture):

[0080]

[0081] where λ provides a weight to each type of loss. When backpropagating the gradients with respect to the encoder loss, we fix the generator parameters, otherwise they would co-adapt with the encoder parameters, making the training unstable.

[0082] Some embodiments can be configured to be used with components of a manufacturing kit, where the components are objects. For example, the objects of the components can be associated with a particular customer and identified by an identifier, where the processor can access stored instructions associated with the particular customer that, when executed by the processor, cause the processor to: (a) access a list of expected components of the manufacturing kit and confirm that the particular customer is a client based on the identifier; (b) compare the list to a customer database in response to a timely appropriate request to verify that a list of components expected to be ordered are to be delivered to the particular customer within a time period; and (c) identify a difference between the list of expected components and the first customer database of objects upon confirmation of the delivery time period. The customer database can include other data associated with the customer, including: other types of components, other manufacturing associated with other types of information.

[0083] It is also contemplated that some embodiments can be configured with a customer’s registration system. Depending on the needs of the customer, some embodiments can be configured to associate a customer’s identity with a bin containing objects. The bin can be labeled with an identifier such that the registration system can read the identifier to obtain information about the customer, the customer’s order, etc. The identifier is a common identifier in the manufacturing industry and can include one or a combination of a radio frequency identification (RFID) transceiver, a WiFi transceiver, a marker reader, etc.

[0084] Figure 2 Segmented images 2A, 2B, and 2C are shown according to some embodiments of the present disclosure, and A, B, C, D, and E are single instances unentangled by InSeGAN on a nut instance depth image from the training depth image dataset.

[0085] Figure 3 Segmented images 3A, 3B, and 3C are shown according to some embodiments of the present disclosure, and A, B, C, D, and E are single instances unentangled by InSeGAN on a cone instance depth image from the training depth image dataset.

[0086] Figure 4 A schematic diagram illustrating some aspects of the InSeGAN inference procedure is shown, which can include inputting a multi-instance depth image into an encoder module that generates a set of latent vectors. According to some embodiments of the present disclosure, each latent vector in the set is individually input into a trained single-instance generator, and the output thereof is rendered using a rendering module to form a single-instance depth image corresponding to the latent vector.

[0087] InSeGAN inference At inference time, we assume only a depth image consisting of multiple instances of rigid objects is given; the following formulation is only used at training time.

[0088] L = II Z - π(Z ) II2, where π* = arg min OT(π, D(Z, Z )) (4)

[0089] Our goal is to segment the instances and render each instance individually while generating an instance segmentation about the input. To do this, our inference procedure is similar to the generation process but with some important differences as illustrated in Figure 4 Specifically, for inference, we input a multi-instance depth image into an encoder module E that generates a set of latent vectors Each is individually input into a trained single-instance generator G s r ​rendered to form a single instance depth image corresponding to We emphasize that at this stage, when the renderer takes the aggregated feature tensor as input, the depth renderer is inside the image generation stage, as opposed to in the training setting. Once the single instances are rendered (as shown in Figure 4 ), we use depth-wise max pooling on these instance depth images for inter-instance occlusion reasoning, followed by thresholding of the instances, where the threshold is determined by the average depth in the image. Thresholding removes any bias introduced during depth rendering. To generate a pixel-wise instance segmentation, we use the index of the generated instance that is selected for a given pixel.

[0090] Still referring to Figure 4 , the inquisitive practitioner of computer vision might ask how a single can decode a single object instance? This is because of the way the generator-discriminator flow is trained. For example, let us assume for a moment that the single generates more than one (or in the extreme case, all) instances in the depth image. Since is randomly sampled from the distribution, it is unlikely that only some of the elements in said element can render an instance while some do not, assuming that the aggregation of all generated instances should match the number of instances in the input, the discriminator is required to eventually learn to validate in the generated image. Furthermore, assuming that object appearances are varied, it can be easier for the generator to learn to render the appearance of a single instance compared to the joint appearance distribution of all instances, which can be very large and diverse.

[0091] Training procedure : One aspect of the complete framework (including the InSeGAN generator G, the discriminator D, and the encoder E) minimizes the sum of all losses given by:

[0092]

[0093] Still referring to Figure 4 , the gradients of the various modules are computed using PyTorch autograd. We use Adam for training all our models with a learning rate of 0.0002, β1 = 0.5, and β2 = 0.99.

[0094] Figure 5Figures 5A, 5B, 5C, and 5D show pictures illustrating qualitative results of using InSeGAN on nuts from training depth images according to some embodiments of the present disclosure: Figure 5A shows input images into InSeGAN; Figure 5B shows generated and rendered images as synthesized images by InSeGAN corresponding to the true depth images in Figure 5A; Figures 5C and 5D show instance segmentation and single instance of one of the nuts in Figure 5A, which is rendered independently at test time by a single instance generator.

[0095] During the experiments, many test experiments demonstrate the empirical benefits of InSeGAN for instance segmentation tasks. We will first introduce the new dataset Insta-10 on which our experiments are based, then review the evaluation metrics, and provide performance comparisons against existing methods.

[0096] Insta-10 dataset While there are several real-world datasets (such as MSCOCO, CityScapes, etc.) used for instance segmentation, they usually involve background objects, and other materials that are not relevant to those and segmentation-related materials. There have been methods that use simulated scenes (e.g., the CLEVR dataset) for instance segmentation, however, this dataset has been introduced for visual reasoning tasks, and involves objects with different colors, shapes, etc., and the results can not be able to fully analyze the benefits of segmentation schemes. To this end, we introduce Insta-10, a large-scale dataset collected under controlled settings, which consists of depth images of multiple instances of CAD object models. Unlike CLEVR, we remove colors and textures from the instances to analyze the segmentation performance under minimal attributes.

[0097] To create the dataset, we were inspired by recent work by Wu et al. (Yuanwei Wu, Tim Marks, Anoop Cherian, Siheng Chen, Chen Feng, Guanghui Wang, and Alan Sullivan, “Unsupervised joint 3d object model learning and 6d pose estimation for depth-based instance segmentation,” in IEEE International Conference on Computer Vision Workshops, 2019, https: / / openaccess.the cvf.com / content_ICCVW_2019 / html / R6D / Wu_Unsupervised_Joint_3D_Object_Model_Learning_and_6D_Pose_Estimation_ICCVW_2019_paper.html, <URL: https: / / openaccess.thecvf.com / content_ICCVW_2019 / papers / R6D / Wu_Unsupervised_Joint_3D_Object_Model_Learning_and_6D_Pose_Estimation_ICCVW_2019_paper.pdf>) that proposed a task of simulating objects being thrown into a bin (e.g., for a subsequent robot to pick up from the bin). Specifically, their setup involved a simulated bin given a depth, into which several instances of an object were thrown. We used the PhysX physics simulator4and 10 CAD object models from the T-less dataset to generate multiple instance depth images.

[0098] Table 1. Mean IoU between predicted segmentation masks by the method for ground truth masks.

[0099] Figure 5A shows the object models used. We used 5 object instances per depth image, such that there is substantial inter-instance occlusion, while we also adjusted the bin width to make the segmentation target less hard. Next to the depth images, we also provide the point cloud associated with each image as well as the ground truth instance segmentation masks; these masks were only used for evaluation and not during training. We collected 10K images per object, for a total of 100K images across the dataset, and the size of each image is 224 x 224. Sample depth images are provided for each object in Figure 5A.

[0100] Evaluation metrics and experimental setup

[0101] To evaluate our approach, we use the mean intersection over union (mloU) metric, which is a standard metric for semantic segmentation tasks. For training and evaluation, we split the data subsets associated with individual classes into training, validation, and test sets. We use a random selection of 100 images in each class for validation, while we carefully select 100 images for testing. Since we have no control over how objects fall into bins, there is a possibility that instances can be well separated from each other, in which case instance segmentation can be reduced to trivial clustering of point clouds. To avoid such scenarios that can break the evaluation, we first run 5 clusters on the point cloud associated with each depth image. Next, we compute the mloU of the found clusters against the ground truth. We select 100 poorly performing depth images from the dataset. Figure 5 A shows samples from our test set. Apparently, most of the test images are uninteresting for standard clustering algorithms that perform instance segmentation.

[0102] As mentioned above, Figures 5B to 5D provide qualitative results for the objects given in Figure 5A using InSeGAN. Figure 5A is the input depth image. Figure 5B is the rendering of the input depth image by the multi-instance depth renderer. Figure 5C is the predicted segmentation. Figure 5D shows the five individual instances generated by InSeGAN, which when rendered and pooled lead to the segmentation in Figure 5C.

[0103] It is observed from extensive experiments that the novel 3D GAN solves the problem of unsupervised instance segmentation using a generative adversarial framework. By pairing the discriminator with a carefully designed generator, the model can reconstruct individual object instances even in cluttered and heavily occluded situations. The learned model can be used to implement instance segmentation when encoding. The novel approach is empirically analyzed using a new large-scale dataset, and the model achieves state-of-the-art results. Specifically, the novel framework is designed to be simple, which allows it to be configured for other applications and domains. For example, the novel framework of InSeGAN leads to better performance in instance segmentation when compared to the conventional approaches documented, which allows InSeGAN to perform in a variety of autonomous tasks. For example, segmenting object instances (e.g., cars) in a scene is critical for automated driving, object counting is necessary for visual reasoning, and identifying instances of organs in medical images is important for automated diagnosis.

[0104] Network architecture

[0105] Figure 6A schematic diagram showing some aspects of the architecture exemplifying the InSeGAN generator, according to some embodiments of the present disclosure, is shown. The InSeGAN generator has five sub-modules, including:

[0106] (i) Pose encoder which takes n random noise vectors where, in our setting, n = 5 and generates a 6-D vector which is assumed to be an axis-angle representation of rotation and translation (three dimensions of rotation and three dimensions of translation), which is then transformed into a rotation matrix and a translation vector to generate an element in the special Euclidean group (SE(3));

[0107] (ii) 3D implicit template generation module which takes a 4x4x4x64 dimensional tensor (representing the implicit 3D template of the object) as input, 3D upsamples it using ResNet blocks and 3D instance normalization layers to generate a 16x16x16x16 feature map;

[0108] (iii) Spatial transformer network which takes the 3D implicit template and the geometric transformation as input for each instance and transforms the template, resamples it, and generates a transformed feature map of the same size as its input;

[0109] (iv) Single instance feature generator module which reshapes the transformed template features and generates a single instance 2D feature map (each of size 16x16x128); and

[0110] (v) Depth renderer module which takes average pooling over the features for each instance and renders multiple instance depth images from the pooled feature maps.

[0111] As observed during some test experiments, the 3D implicit template does not use any random modules (via MLPs) to generate random components in the generated images (in this particular case, RGB) which are critical in some test frameworks. During some test experiments, we found that using noise vectors failed to achieve our setting and we lost the ability to disentangle instances. This can be because our data is synthetically generated and does not have any noise or texture.

[0112] Still referring to Figure 6 , an exemplified diagram of a physics simulator is used to render our synthetic dataset Insta-10. Therein, the same object is thrown into a simulated box, and a ground truth instance segmentation mask for each of the instances in said instances. We use the depth images associated with these instances for training the InSeGAN to recover these segmentation maps at inference time.

[0113] Figure 7 A schematic diagram illustrating some aspects of the architecture of an encoder module according to some embodiments of the present disclosure is shown.

[0114] Figure 8 Schematic diagrams illustrating some aspects of the architecture of a discriminator module according to some embodiments of the present disclosure are shown. For example, Figure 7 and Figure 8 It shows in Figure 7 encoder and Figure 8 The discriminator uses a neural network that loosely follows a similar architecture, except that the discriminator takes a 64×64 depth image (generated or from a real example) as input and generates a scalar score, while Figure 7 The encoder takes the generated depth image and produces n instance vectors as output. We use a 128-D noise vector when generating the image, and therefore expect... Figure 7 The encoder generates 128-D features as output (for each instance). Both the encoder and discriminator use 2D convolutional modules, LeakyReLu activation modules, and 2D instance normalization modules.

[0115] Implementation details and training settings :Reference Figure 6 Our InSeGAN module is implemented in PyTorch. As indirectly mentioned above, we use our simulator to generate 224×224 depth images; however, our Insta-GAN pipeline uses 64×64 images. To do this, the individual 224×224 images are rescaled to 64×64 and normalized via variance, using mean subtraction and variance. For training, we use horizontal and vertical image flipping for data augmentation. We do not use any other augmentation schemes.

[0116] Evaluation details We use the mean IoU metric between ground truth segments and predicted segments for our evaluation. Specifically, for each ground truth segment, we find the predicted segments that overlap with it the most and calculate their Intersection over Union (IoU); then we use the mean IoU to calculate the mean across all segments.

[0117] Training :Reference Figure 6 , Figure 7 as well as Figure 8 We used a single GPU to train our module for up to 1000 epochs; each epoch, for each object, took approximately 30 seconds on ~10,000 training samples. We used the Adam optimizer with a learning rate of 2×10⁻⁶. -4β1 = 0.5. We use 128-D noise samples from a normal distribution of the noise vector, and a batch size of 128 samples.

[0118] Ablation studies We analyzed and confirmed the importance of each choice in InSeGAN. We eliminated various losses in the InSeGAN generator, requiring the use of a 3D GAN instead of a 2D single-instance generator, and so on.

[0119] Is the 3D generator important? What was achieved during the experiments was that the 3D generator was a key alternative to the 2D generator in InSeGAN. During the experiments, the 2D generator was the standard image-based generator commonly used in conditional GANs. Specifically, for the 2D generator, we replaced the 3D modules in InSeGAN (i.e., the 3D implicit template, pose encoder, and STN) with 2D convolutional layers and upsampling layers, similar to those used in the encoder and discriminator. We conducted two experiments to analyze and validate our choice: (i) evaluating training stability and convergence, and (ii) performing instance segmentation on various objects.

[0120] Figure 9A , Figure 9B as well as Figure 9C The illustration shows an example of using InSeGAN with 3D modules (i.e., using a pose encoder, 3D instance templates, and a spatial transformer network (STN)) to map three objects (i.e., objects) from a training depth image, according to some embodiments of this disclosure. Figure 9A The object (Obj01) Figure 9B cone and Figure 9C The convergence curve of the connector. For example, Figures 9A-9C The convergence plots of the three objects illustrated use an InSeGAN with 3D modules (i.e., using a pose encoder, 3D instance templates, and STN), instead of replacing the 3D modules with a 2D GAN (i.e., replacing the 3D modules with 2D convolutional layers and upsampling layers, similar to the reverse encoder and discriminator). Figures 9A-9CIn the middle, mIoU is plotted against epochs. Apparently, using 3D GAN leads to better and more stable convergence. Note that in the cone (middle plot), the 2D generator is unstable and often diverging, and when this happens, we reset the optimizer. This is captured by the discontinuity in the blue plot. In contrast, using the 3D generator leads to very table training of the generator and discriminator and converges faster. It was later found that the three observations we made include plots of (i) 3D GAN converges significantly faster than 2D GAN, (ii) 3D GAN is more stable, and (iii) leads to better mIoU for instance segmentation.

[0121] Are all losses important? The InSeGAN generator exists of three different losses: (i) a computation alignment loss (ii) a on the intermediate feature maps, and (iii) a between the generated depth image and the depth image that is regenerated from the encoded latent vector. For (i), we compare the greedy choice of alignment against using optimal transport. We provide an ablation study on two object classes, Bolts and Obj01. As is clear from Table 1, we find that using greedy alignment leads to lower performance. Furthermore, we find that using E is empirically very important and leads to a 10-20% performance difference. Our analysis confirms the importance of all losses used in our architecture.

[0122] Figure 10A is a plot of mean intersection-over-union (mIoU) against increasing dataset size according to some embodiments of the disclosure.

[0123] Figure 10B is a plot of mean intersection-over-union (mIoU) against increasing number of instances used in InSeGAN (n) (against 5 ground truth instances) according to some embodiments of the disclosure.

[0124] Do we need all training samples? In Figure 10A , we plot the performance against increasing number of data samples. I.e., we use a random subset of 10K depth images and evaluate it on our test set. We use subsets with 500, 1000, 3000, 7000, and all 9800 samples. In Figure 10B , we plot this performance. Apparently, more training data is useful, although this increment seems to depend on the object class.

[0125] Figure 11 Table 1 shows results illustrating an ablation study on various losses used in the mloU and InSeGAN generator implemented for two classes, according to some embodiments of the present disclosure. For example, we show qualitative results of instance segmentation obtained for different training set sizes to gain insight into Figure 10A The performance reported in the middle can be explained why. The results show that with more than about 3000 samples, our method seems to start generating reasonably good quality instance segmentation, although the mloU performance improves with more data.

[0126] Instance / disentanglement number A key question for our framework is whether we really need the exact number of instances to sample the noise vector. We will consider what happens when we only have a rough estimate. In Figure 10B In the middle, we plot the performance against the increasing number of instances used in InSeGAN, i.e., for the number of noise vectors we sample for the generator, we increase n from 1 to 7. Recall that all our depth images consist of 5 instances.

[0127] Figure 10B The plots in the middle for two objects (bolt and Obj01) show that when the number of instances is approximately close to the ground truth, InSeGAN performs quite well. What is observed from the experiment (named A) is a plot of the qualitative results from these choices. Interestingly, we determine that using n = 1 fails to capture the shape of the object completely, while n = 4 learns the bilateral bolt, and n = 5 seems to capture the shape perfectly. While n > 5 seems to show some improvement, it is inconsistent across data classes. Overall, from this experiment, it seems that a rough estimate of the number of instances is enough to obtain reasonable performance.

[0128] Based on extensive experiments, InSeGAN generates more reasonable segmentation when compared to the reported segmentation of other conventional methods. Figure 12 Sample results of instance segmentation generated according to embodiments are shown. Figure 12 The input to the InSeGAN algorithm is shown on the left (images 12A-1, 12B-1, and 12C-1), and the output instance segmentation map is shown on the right (images 12A-2, 12B-2, and 12C-2).

[0129] Figure 13A hardware diagram illustrating some components that can be incorporated into some systems 1300 in accordance with some embodiments of the present disclosure is shown. The system 1300 can include a processor 1320 configured to execute stored instructions, and a memory 1340 that stores instructions that can be executed by the processor. The processor 1320 can be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 1340 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory systems. The processor 1320 connects to one or more input and output devices via bus 1306. The system 1300 can also include a storage device 1330 adapted to store a first module 1331, a second module 1332, a third module 1333 for storing executable instructions for the processor 1320. In some cases, the modules 1331, 1332, and 1333 can include: Figure 6 the illustrated architecture of the trained InSeGAN generator, Figure 7 the illustrated architecture of the encoder module, Figure 8 the illustrated discriminator module, and Figure 4 the illustrated trained single-instance generator, trained generator-discriminator. Further, the architecture of the trained InSeGAN generator can be obtained from external data 1395 via a network that stores a trained neural network of the InSeGAN. The storage device 1330 can store data in a data storage device 1334. The storage device 1330 can be implemented using a hard disk drive, an optical drive, a thumb drive, an array of drives, or any combination thereof.

[0130] The system 1300 can include an input interface for receiving external data 1395. The input interface can include a human-machine interface 1310 within the system 1300 that connects the processor 1320 to a keyboard / measuring device 1311 and a pointing device 1312, which can include a mouse, trackball, touchpad, joystick, stylus, pen, or touch screen, among others. Alternatively, the input interface can include a network interface controller 1350 adapted to connect the system 1300 to a network 1390 via the bus 1306. Through the network 1390, external data 1395 can be downloaded and stored within the storage system 1330 as training and / or operational data 1334 for storage and / or further processing.

[0131] Still referring to Figure 13The system 1300 can include an output interface linked through a bus 1306, and a display interface 1360 suitable for connecting the system 1300 to a display device 1365. The display device 1365 can include a computer monitor, a video camera, a television, a projector, or a mobile device, among others. For example, the system 1300 can be connected to a system interface 1370 suitable for connecting the radar imaging system to different systems 1375 based on some aspect control. Additionally or alternatively, the system 1300 can be connected through the bus 1306 to an application interface 1380 suitable for connecting the system 1300 to an application device 1385 that can operate based on the results of the image reconstruction.

[0132] Figure 14 is a schematic diagram of a computing device 1400 that can be used to implement some of the techniques of the method and system according to embodiments of the present disclosure by way of non-limiting example. The computing device or apparatus 1400 represents various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computing device 1400 can include a power source 1408, a processor 1409, a memory 1410, a storage 1411, all connected to a bus 1450. In some cases, the storage 1411 can include Figure 6 the architecture of the trained InSeGAN generator shown in FIG. 1, Figure 7 the architecture of the encoder module shown in FIG. 2, Figure 8 the discriminator module shown in FIG. 3, and Figure 4 the trained single-instance generator, the trained generator-discriminator shown in FIG. 4. Moreover, the architecture of the trained InSeGAN generator can be obtained from an external data 1395 via a network that stores a trained neural network of the InSeGAN. Furthermore, a high-speed interface 1412, a low-speed interface 1413, a high-speed expansion port 1414, and a low-speed connection port 1415 can be connected to the bus 1450. Also, a low-speed expansion port 1416 is connected to the bus 1450.

[0133] By way of non-limiting example, various component configurations are contemplated that can be installed on a common motherboard according to specific applications. Still further, input interface 1417 can be connected to external receiver 1406 and output interface 1418 via bus 1450. Receiver 1419 can be connected to external transmitter 1407 and transmitter 1420 via bus 1450. Also connected to bus 1450 can be external memory 1404, external sensors 1403, machine 1402, and environment 1401. Additionally, one or more external input / output devices 1405 can be connected to bus 1450. Network interface controller (NIC) 1421 can be adapted to connect to network 1422 via bus 1450, where data or other data can be presented to third party display devices, third party imaging devices, and / or third party printing devices external to computer device 1400, among others.

[0134] Still referring to Figure 14 It is also contemplated that memory 1410 can store instructions executable by computer device 1400, historical data, and any data that can be utilized by the methods and systems of the present disclosure. Memory 1410 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. Memory 1410 can be one or more volatile memory units and / or one or more non-volatile memory units. Memory 1410 can also be another form of computer-readable media such as a magnetic or optical disk.

[0135] Still referring to Figure 14 Storage device 1411 can be adapted to store supplemental data and / or software modules used by computer device 1400. For example, storage device 1411 can store historical data and other related data as mentioned above with respect to the present disclosure. Additionally or alternatively, storage device 1411 can store historical data similar to that mentioned above with respect to the present disclosure. Storage device 1411 can include a hard disk drive, optical drive, thumb drive, array of drives, or any combination thereof. Further, storage device 1411 can contain a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in the information carrier. The instructions, when executed by one or more processing devices (for example, processor 1409), perform one or more methods, such as those described above. The system can be optionally linked through bus 1450 to a display interface or human-machine interface (HMI) 1423 adapted to connect the system to a display device 1425 and a keyboard 1424, where display device 1425 can include a computer monitor, video camera, television, projector, or mobile device, among others.

[0136] Still referring to Figure 14 The computer device 1400 can include a user input interface 1417 suitable for a printer interface (not shown) that can also be connected through the bus 1450 and suitable for connection to a printing device (not shown), which can include a liquid inkjet printer, a solid ink printer, a large format commercial printer, a thermal printer, a UV printer, or a thermal sublimation printer, among others. The high-speed interface 1412 manages bandwidth-intensive operations for the computer device 1400, while the low-speed interface 1413 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In some implementations, the high-speed interface 1412 is coupled to memory 1410, user interface (HMI) 1423, and keyboard 1424 and display 1425 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 1414, which can accept various expansion cards (not shown) via the bus 1450. In implementations, the low-speed interface 1413 is coupled to storage device 1411 and low-speed expansion port 1415 via the bus 1450. The low-speed expansion port 1415, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices 1405, as well as other devices (keyboard 1424, pointing device (not shown), scanner (not shown), or networking devices such as switches or routers), for example, through a network adapter.

[0137] Still referring to Figure 14 The computer device 1400 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 1426, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer such as a laptop computer 1427. It can also be implemented as part of a rack server system 1428. Alternatively, components from the computer device 1400 can be combined with other components in a Figure 13 embodiment, such as the implementation 1300. Each of such devices can contain one or more of the computer device 1300 and the device 1400, and an entire system can be made up of multiple computer devices communicating with each other.

[0138] Features

[0139] According to the system embodiments recited in the claims (i.e., claims 1, 13, and 14), when combined with each of the above independent claims, the following aspects are each intended to include another embodiment, or one or a combination of these aspects is intended to include embodiments having / according to the above independent claims.

[0140] For example, implementations can include a system independent of claim 1, or claim 13 or claim 14 based on one or a combination of aspects recited below. For example, an aspect can include: the real depth image is obtained from at least one sensor comprising one of: (a) a time-of-flight depth camera, (b) a structured light depth camera, (c) a stereo depth camera as at least two infrared cameras, an infrared projector, or a color camera. Wherein the at least one sensor is positioned to sense objects in one or more bins. Another aspect can be: the objects are components of a manufacturing kit or a shipment-to-be, and the objects are identified by an identifier, when the object is identified, the processor accesses stored instructions that, when executed by the processor, cause the processor to: (a) retrieve, based on the identifier, a list of expected components of the manufacturing kit or the shipment-to-be assigned to the first customer; (b) compare the list to a customer database to verify that the list of expected components is to be delivered to the first customer within a time period; and (c) identify a discrepancy between the list of expected components and the first customer database of objects upon confirmation of the delivery time period. A further aspect can also include: a registration system configured to associate an identity of the customer with one or more bins containing the objects, such that the registration system comprises one or a combination of: a radio frequency identification (RFID) transceiver, a WiFi transceiver, a reading device, a wireless transceiver device, or some other transceiver device or another reading device.

[0141] One aspect can be that the trained GAN is configured to learn to segment the same instances of objects randomly placed in a bin in an unsupervised manner. Another aspect can be that the generator is a single instance generator that learns the appearance distribution of rigid objects from a cluttered or complex scene comprising multiple same instances. Yet another aspect can be that the pose encoder takes as input an estimated depth image and generates as output a latent noise vector. Another aspect can be that the estimated depth image comprises a noise vector that is input into the pose encoder and as output generates an axis-angle representation of the 6D pose of the instance. It can be an aspect that the discriminator is configured to learn to count: whether the number of rendered instances matches the number of instances in the real depth image and whether the layout of the estimated depth image is similar to the composition of instances in the real depth image. Such that one aspect can be that the generator and the discriminator are trained in a minimax adversarial game such that the generator learns to generate images that can fool the discriminator while the discriminator in turn trains to distinguish whether the input data into the discriminator is real or fake such that optimization occurs when the discriminator is unable to identify whether the input data in the discriminator is from the generator or from a real depth image. Another aspect is that the training of the GAN is configured to implicitly force the generator to capture the distribution of single object instances and when the trained GAN is used for instance segmentation, the pose encoder takes as input an estimated depth image and is trained to reproduce the instance noise vector such that each instance noise vector is separately decoded by the generator to form a single object instance. Finally, another aspect is that the generator and the discriminator are trained in a minimax adversarial game such that the generator learns to generate images that can fool the discriminator while the discriminator in turn trains to distinguish whether the input data into the discriminator is real or fake such that optimization occurs when the discriminator is unable to identify whether the input data in the discriminator is from the generator or from a real depth image.

[0142] One aspect includes that the trained discriminator is configured to learn (a) to count whether the number of rendered instances matches the number of instances in the data distribution of real images; (b) to provide that the rendered instances in the estimated depth image look the same as instances in the data distribution of real images at various viewpoints; and (c) to estimate whether the layout of the estimated depth image is similar to the composition of instances in real depth images. Another aspect is that the estimated depth image is input into the trained GAN and the estimated depth image is segmented into its instances by encoding the instances using an instance encoder to generate latent vectors. The individual latent vectors are then independently translated into single instance depth images, which are subsequently thresholded in depth to find pixels associated with instances, which are merged to form single instance segments, which are merged to generate an instance segmentation of the real depth image input. One aspect is that the generator uses noise vectors from the training depth images to generate the estimated depth image and the encoder is configured to input the estimated depth image and as output, reconstructs individual ones of the noise vectors at training time.

[0143] Another aspect further includes that the individual noise vector outputs of the encoder match the noise vectors used by the generator such that the matching uses optimal transport loss; wherein the output noise vectors of the encoder are used to generate intermediate features that match the intermediate features generated by the generator using the noise vectors such that the matching uses Euclidean loss; wherein the noise vectors of the encoder are reused in the generator to generate new estimated depth images, the noise vectors configured to match previously generated estimated depth images such that the matching uses LI loss; and wherein the encoder is trained via backpropagation on a loss computed by adding the optimal transport loss, the Euclidean loss, and the LI loss.

[0144] One aspect is that the trained GAN is configured to learn to segment the same instances of objects placed randomly in a bin in an unsupervised method. Another aspect is that the generator is a single instance generator that learns the appearance distribution of rigid objects from a cluttered or complex scene comprising multiple same instances. Yet another aspect is that the pose encoder takes as input an estimated depth image and generates as output a latent noise vector. Still another aspect is that the estimated depth image comprises a noise vector that is input into the pose encoder and as output, generates an axis-angle representation of a 6D pose of an instance. Also, another aspect is that the discriminator is configured to learn to count whether the number of rendered instances matches the number of instances in real depth images and whether the layout of the estimated depth image is similar to the composition of instances in real depth images.

[0145] Figure 15AA robot configuration 100 of a robot 150 according to some embodiments is shown. The robot 150 comprises a robot arm 101 for performing assembly operations. The robot arm 101 comprises a wrist 102 for ensuring multiple degrees of freedom of movement of an object. In some implementations, the wrist 102 is provided with an end tool 104 for holding an object 103 and / or for performing any other robotic operation such as an assembly operation. The end tool 104 can for example be a gripper. In the following, "end tool" and "gripper" can be used interchangeably. According to embodiments, the purpose of the assembly operation is to place two components together, typically in close contact. For example, inserting an object into another object along a trajectory to assemble a product.

[0146] Figure 15B An exemplary assembly operation according to embodiments is shown. The assembly operation is explained in relation to the robot 150 shown. Figure 15A Figure 15B .

[0147] The robot 150 is configured to perform an assembly operation, for example, inserting an object 103 into another object along a trajectory. As used herein, the trajectory corresponds to a path that defines the motion of the object 103 held by the gripper 104 for performing the assembly operation. In a simple scenario, the trajectory can only indicate vertical motion of the wrist 102. However, since the wrist 102 comprises multiple degrees of freedom, the trajectory can comprise a motion profile spanning in a multi-dimensional space.

[0148] The pose of an object refers to the combination of the position and orientation of the object. The gripper 104 initially holds a movable object 103 (e.g., a peg) in a start pose 111. The pose of the gripper 104 corresponding to the start pose 111 is referred to as the start pose of the gripper 104. According to embodiments, the purpose of the insertion operation is to insert the movable object 103 into an immovable object 112 having a pose 115, where the object 112 comprises a hole that receives the object 103. The pose 115 of the object 112 can refer to the position and / or orientation of the object 112. The robot 150 is configured to move the gripper 104 along a trajectory 113 to insert and place the object 103 in the hole of the object 112 in a pose 114. The pose 114 of the object 103 in the hole of the object 112 is referred to as the target pose. The pose of the gripper 104 corresponding to the target pose is referred to as the target pose of the gripper 104.

[0149] ​The target pose of the gripper 104 is determined based on the position of the object 112. At the end of the successful execution of the insertion operation, the pose of the gripper 104 of the robot arm 101 is considered to have reached the target pose of the gripper 104. Thus, achieving the target pose of the gripper 104 is equivalent to the successful execution of the insertion operation. According to embodiments, the trajectory 113 is defined according to the start pose and the target pose of the gripper 104 and the pose 115 of the object 112. Moreover, such assembly operation can be repeatedly performed by the robot 150.

[0150] Some embodiments are based on the recognition that the pose of the object 103 involved in the assembly operation and the pose of the object 112 can change between repetitions of the assembly operation due to one or a combination of the objects 103 and 112 being positioned in different poses. For example, in the case of the object 112 arriving at a moving conveyor belt, it is not possible for the object 112 to arrive at the moving conveyor belt in a specific pose (e.g., the pose 115) every time. Thus, the object 112 can end in a different pose. To this end, the change in the pose (orientation and position) of the object 112 involved in the assembly operation results in at least one or a combination of a new start pose and a new target pose, which is referred to as an end pose modification. Since the trajectory is defined according to the start pose and the target pose of the gripper 104 and the pose 115 of the object 112, the trajectory 113 cannot be used for a different assembly operation involving other poses than the aforementioned poses. In such cases, various measuring devices are used to determine the poses of the objects 103 and 112. According to some embodiments, these measuring devices determine the new start pose and the new target pose of the gripper 104. The measuring devices include imaging devices 106 such as industrial vision cameras. In some implementations, a single imaging device can be used. In some cases, one or more measuring devices 106 Figure 13 1311, Figure 14 1403) in the system for generating photorealistic images from real depth images can be disposed on a fixed platform (e.g., a wall, a table, a frame, or a fixed structure) separate from the robot arm / robot, or on the robot arm / robot, or on another moving platform (or mobile cart).

[0151] However, the accuracy of the poses of object 103 and object 112 determined by such a camera may be insufficient for assembly operations. For example, unless expensive imaging devices are used, the error in pose determination by industrial vision cameras is approximately 1 mm to 2 mm. Such an error is at least an order of magnitude larger than the tolerance required for precise insertion operations (which can be approximately 0.1 mm). Therefore, due to the significant inaccuracy of the determined poses of objects 103 and 112, the object to be inserted (e.g., 103) may partially collide with another object (e.g., 112) involved in the assembly operation.

[0152] Figure 15C The results of the assembly operation, according to some embodiments, are shown due to the imaging device's inaccurate determination of the orientation of object 103. (Combined with...) Figure 15A and Figure 15B The robot 150 shown is used to explain Figure 15C For example, object 112 ( Figure 15B The pose 115 of the object 112 (as shown) can be changed, and the imaging device 106 can determine that the pose 115 has been changed to pose 116. Specifically, the imaging device 106 can determine that the object 112 is at position 116. As the position 115 of the object 112 changes to position 116, the target pose 114 (as shown)... Figure 15B (As shown) can be changed to the target pose 118. Trajectory 113 is transformed into trajectory 117 based on pose 116 and target pose 118. However, if the true position of object 112 is not accurately determined and moves away from the determined position 116 by a certain distance 119, the trajectory of trajectory 117 will not lead to correct insertion, and a collision may occur between object 103 and a portion of object 112 (e.g., edge 120). As a result, object 103 is displaced, and object 103 may still maintain an incorrect pose 121. Furthermore, due to this collision, the gripper 104 of the robotic arm 101 may experience forces specific to pose 121.

[0153] To this end, some embodiments are based on the recognition that the pose determined by the imaging device 106 alone is not sufficient to successfully perform the assembly operation. To overcome this limitation of the imaging device 106, an adaptive assembly strategy (AAS) 107 is used. The AAS 107 is based on the recognition that the forces experienced in the assembly operation can be used to correct inaccuracies in the pose determination by the imaging device 106. To this end, the robot 150 is equipped with a force sensor. For example, a force sensor 105 is operably connected to the wrist 102 or the end tool of the robot arm 101. The force sensor 105 is configured to generate force and / or torque measurements 108 (also referred to as force sensor measurements 108) experienced by the end tool (gripper 104) of the robot 150 during the assembly operation. In some implementations, the robot 150 is equipped with a torque sensor for measuring the torque experienced by the end tool 104. Some embodiments are based on the recognition that the force sensor measurements 108 can be used to correct the trajectory 117 such that the target pose 118 is achieved.

[0154] To this end, a non-linear mapping 109 is determined for the trajectory 113. The non-linear mapping maps the force sensor measurements 108 in a non-linear way onto a correction to the trajectory 117. In other words, the non-linear mapping provides a correction to the trajectory 117 of the robot 150 during the assembly operation along the trajectory 117. The correction can include a displacement to the object 103 which allows for a new target pose of the robot (robot arm) to be achieved. To this end, the non-linear mapping provides a mapping between the force and the displacement. In alternative embodiments, the correction can correspond to a pose and / or velocity correction. The trajectory 113 is referred to as the “original trajectory”. As explained below, the original trajectory is the trajectory for which the non-linear mapping is determined.

[0155] Some embodiments are based on the understanding that for a particular assembly operation along a particular trajectory (e.g., the trajectory 113), a non-linear mapping can be determined and can be repeated any number of times for the same assembly operation of the same robot 150. However, when the starting pose and / or the target pose involved in the assembly operation changes, the original trajectory 113 is transformed accordingly to generate a transformed trajectory. Subsequently, the non-linear mapping determined for the original trajectory 113 can need to be modified in accordance with the transformed trajectory (e.g., the trajectory 117).

[0156] However, some embodiments are based on the realization that if the original trajectory 113 is transformed according to changes in the start pose and / or the target pose, the non-linear mapping determined for the original trajectory 113 is valid for the transformed trajectory without any additional adaptation 110. For example, this realization is true because the sign and magnitude of the force depends entirely on the relative position of the two objects (e.g., object 103 and object 112) and not on their absolute position in space. Thus, if one of the objects 103 and 112 moves to a different position while the other object approaches it along a similarly transformed trajectory, the same force can be generated.

[0157] Accordingly, such a realization allows some embodiments to determine an original trajectory (e.g., trajectory 113) and a non-linear mapping of the original trajectory offline (i.e., in advance) and to transform the original trajectory online (i.e., during the assembly operation) to accommodate changes in the start pose and / or the target pose and to control the robot 150 according to the transformed trajectory and the non-linear mapping determined for the original trajectory. In this way, various embodiments can accommodate changes in the start pose and / or the target pose measured with an imaging device 106, such as a video camera, that has less precision than the precision of the assembly operation. As a result, it allows the use of an economical video camera in the assembly operation. Moreover, it minimizes task-specific robot programming since the non-linear mapping determined for the original trajectory can be preserved for the transformed trajectory.

[0158] The non-linear mapping can be determined via training. For example, a supervised machine learning algorithm can be used to learn the mapping between the force and the displacement caused by the force. This mapping is learned offline. The mapping from displacement to force is typically many-to-one, i.e., multiple displacements can sometimes result in the same force. During online, i.e., in real-time assembly operations, the inverse mapping of this mapping can be used for correction in the assembly operation. However, the inverse mapping can be one-to-many, i.e., a measured force can be mapped to multiple displacements, which is not a function that can be learned by means of machine learning. This ambiguity in the mapping challenges the possibility of learning a non-linear mapping. Some embodiments are based on the realization that adaptive compliant control learning can be used in the AAS to eliminate the ambiguity in the mapping of the non-linear compliant controller.

[0159] In some cases, a robot can include: a robot arm including an end tool having motion in a plurality of degrees of freedom, wherein, during operation of the robot, a force sensor is arranged to generate measurements indicative of forces experienced by the end tool of the robot arm during operation; a plurality of motors configured to change the motion of the robot arm in accordance with commands generated in dependence on a control law; at least one processor; and a memory having instructions stored thereon that, when executed by the at least one processor, cause the processor, upon receiving data indicative of an original trajectory of the motion of the robot arm, to cause a pose of the end tool to transition from a start pose to a target pose to perform an assembly operation to: determine a control law for controlling the plurality of motors to move the robot arm in accordance with the original trajectory; perform a self-exploration procedure that explores a space of the original trajectory by such an item that the plurality of motors are controlled in accordance with the control law while different displacements from the original trajectory are introduced into a state of the robot arm, and upon detecting a force experienced by the end tool of the robot arm, register a pair of the measurements of the force sensor and a corresponding value of the displacement from the original trajectory to generate training data indicative of the space of the original trajectory; learn a non-linear compliant control law using the training data, the non-linear compliant control law including a non-linear mapping that maps the measurements of the force sensor to a correction direction of the original trajectory that defines the control law; wherein the instructions executed by the at least one processor further cause the processor, upon receiving an end pose modification of the original trajectory including at least one or a combination of a new start pose of the end tool of the robot arm and a new target pose of the end tool measured with less accuracy than an accuracy of the assembly operation, to: transform the original trajectory in accordance with the end pose modification to generate a transformed trajectory; update the control law in accordance with the transformed trajectory to generate an updated control law; and command the plurality of motors to control the robot arm in accordance with the updated control law, the updated control law being corrected with the non-linear compliant control law learned for the original trajectory.

[0160] Although the above-described robot system represents a robot that can handle / assemble parts of a product as an example, the robot system can be applied to a robot that can be applied to a case where a large amount of food in a food processing factory is an irregularly shaped object (cut vegetables, fried chicken, etc.). The robot system including a system for generating a realistic image from a real depth image and automatically segmenting a plurality of instances of a rigid object in the depth image can be applied to automation of a food processing factory, an industrial robot that can handle food. Furthermore, the above-described robot system can be applied to a segmentation (method) system for food recognition. Segmentation is one of the most popular and important problems in image processing. It is essential to make the segmentation accuracy high and the training and computation time short for application to a food processing factory.

[0161] The above description is merely exemplary in nature of the inventive subject matter, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description of how the various exemplary embodiments can be implemented, and the intended scope of the disclosure is set out in the claims that follow. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter.

[0162] While the disclosure has been described with reference to certain preferred embodiments thereof, a worker skilled in the art will recognize from the disclosure that various changes can be made and equivalents employed without departing from the true spirit and scope of the disclosure. Accordingly, it is intended that all such changes and modifications be included within the true spirit and scope of the disclosure.

Claims

1. A system for generating realistic images from real depth images to generate 3D object instance segmentation maps, the system comprising: A processor connected to a transceiver and a data storage device storing computer-readable instructions including a trained generative adversarial neural network (GAN), the trained GAN having a trained generator, a trained discriminator, and a trained pose encoder, the computer-readable instructions, when executed by the processor, causing the processor to: Receive a true depth image of an instance containing an object via the transceiver; The training depth image accessed from the data storage device is input into the generator to generate an estimated depth image representing an implicit 3D model of the object; Simultaneously, each estimated depth image is input into the discriminator and the pose encoder. The discriminator identifies each estimated depth image by comparing it with the real depth image to obtain the discriminator loss, and the pose encoder generates a set of pose transformation matrices that match a set of sampled transformation matrices to obtain the matching loss. The process is repeated iteratively for each estimated depth image until the discriminator loss and the matching loss are minimized to a threshold, thus ending the training period; The instances in the real depth image are identified by inputting the real depth image into the pose encoder of the trained GAN, thereby generating pose transformation matrices for each instance in the real depth image, wherein each pose transformation matrix is ​​used to generate a depth image for each instance. as well as By identifying a group of pixels in each depth image based on a predetermined pixel depth threshold, pixels in the depth image corresponding to the instance in the real depth image are identified, and the group of pixels in the depth image is merged to form an instance segmentation map of a realistic image of the real depth image.

2. The system according to claim 1, wherein, The trained discriminator is configured to learn (a) to count whether the number of rendered instances matches the number of instances in the data distribution of the real depth image; and (b) to provide that the rendered instances in the estimated depth image appear the same as the instances in the data distribution of the real image from various viewpoints. (c) Whether the layout of the estimated depth image is similar to the composition of the instance in the real depth image.

3. The system according to claim 1, wherein, The estimated depth image is input into the trained GAN, and the estimated depth image is segmented into its instances by encoding the instances using an instance encoder to generate latent vectors. Each latent vector is then independently translated into a single instance depth image, which is subsequently depth-thresholded to find pixels associated with the instance. The pixels are merged to form the single instance segments, which are then merged to generate the instance segmentation based on the input real depth image.

4. The system according to claim 1, wherein, The generator uses noise vectors from the training depth image to generate the estimated depth image, and the encoder is configured to take the estimated depth image as input and, as output, reconstruct the individual noise vectors in the noise vectors during training time.

5. The system according to claim 1, further comprising: The noise vectors output by the encoder are matched with the noise vectors used by the generator, such that the matching uses optimal transmission loss. The encoder's output noise vector is used to generate intermediate features that match the intermediate features generated by the generator using the noise vector, such that the matching uses Euclidean loss. The noise vector from the encoder is reused in the generator to generate a new estimated depth image. The noise vector is configured to match the previously generated estimated depth image, such that the matching uses L1 loss. The encoder is trained by backpropagation on a loss calculated by adding the optimal transmission loss, the Euclidean loss, and the L1 loss.

6. The system according to claim 1, wherein, The true depth image is obtained from at least one sensor including one of the following: (a) a time-of-flight depth camera, (b) a structured light depth camera, and (c) a stereo depth camera that is one of at least two infrared cameras, infrared projectors, or color cameras.

7. The system according to claim 6, wherein, The at least one sensor is positioned to sense objects in one or more boxes.

8. The system according to claim 1, wherein, The object is a component of a manufacturing kit or an object to be shipped, and the object is identified by an identifier. When the object is identified, the processor accesses stored instructions that, when executed by the processor, cause the processor to: (a) retrieve a list of expected components of the manufacturing kit or object to be shipped assigned to a first customer based on the identifier; and (b) compare the list with a customer database to verify that the list of expected components is to be delivered to the first customer within a certain period of time. (c) When confirming the delivery period, identify the difference between the list of expected components and the first customer database of the object.

9. The system according to claim 1, wherein, The generator is configured to generate the estimated depth image based on the real depth image, and is configured to generate target segmentation and target attributes once the training period is completed.

10. The system according to claim 9, wherein, The estimated depth image is a translated version of the true depth image.

11. The system according to claim 9, wherein, The generator is also configured to generate the estimated depth image based on the target segmentation, the target attributes, and the latent vector.

12. The system according to claim 1, further comprising: A registration system configured to associate a customer’s identity with one or more boxes containing objects, such that the registration system includes one or a combination of an RFID transceiver, a WiFi transceiver, a reader, a wireless transceiver device, or another transceiver device or another reader.

13. The system according to claim 1, wherein, During the training period, the GAN, including the trained generator, the trained discriminator, and the trained pose encoder, has been trained by accessing training depth images from the data storage device, the training images having instances identical to the instances of the real depth images, and the training depth images are input into the generator to generate estimated depth images representing an implicit 3D model of the object.

14. A system for object-instance segmentation of a depth image composed of identical instances of 3D objects, the system comprising: A processor and a memory, wherein the memory stores computer code instructions, and the processor and the memory are configured using the computer code instructions to cause the system to: The latent 3D representation of the 3D object is captured using an implicit template implemented as a parametric tensor. A pose encoder, implemented as a first neural network, is used to generate a 3D pose transformation matrix from a noise vector. A generator implemented as a second neural network is used to generate a pseudo-depth image. The second neural network is configured to apply multiple 3D transformation matrices to the implicit template. The multiple 3D transformation matrices are generated by the first neural network using multiple noise vectors. A discriminator implemented as a third neural network is used to distinguish between the real depth image and the pseudo depth image, and an identification result is output based on the distinction. The pseudo-depth image is decomposed into latent vectors using an instance encoder implemented as a fourth neural network, the instance encoder outputting a set of latent vectors that match a set of noise vectors used in the pose encoder. as well as During the training period, iterative operations are performed on the implicit template, the pose encoder, the generator, the discriminator, and the instance encoder, such that the discriminator and the generator are trained in an adversarial relationship, and the generator, the implicit template, the pose encoder, and the instance encoder are trained in a cooperative relationship, wherein the instance encoder trains its fourth neural network at the end of the training period to generate a latent vector that matches the noise vector corresponding to the 3D pose of the instance in the generated depth image, the latent vector having an increased accuracy compared to the beginning of the training period. Wherein, after the training period, given real depth images including multiple instances of the 3D object, the instance encoder generates latent vectors for each 3D object instance. These latent vectors are iteratively used by the pose encoder to generate corresponding 3D transformation matrices, which are then used by the generator to generate a single instance depth image. This single instance depth image consists of only one instance of the 3D object from each of the depth images. Pixels corresponding to the effective depth in the generated single instance depth image are used as the instance segmentation.

15. The system according to claim 14, wherein, The training of the GAN is configured to implicitly enable the generator to capture the distribution of individual object instances, and when the trained GAN is used for instance segmentation, the pose encoder takes the estimated depth image as input and is trained to reproduce the instance noise vector, such that each instance noise vector is individually decoded by the generator to form a single object instance.

16. The system according to claim 14, wherein, The generator and the discriminator are trained using a min-max adversarial game, such that the generator learns to generate images that can fool the discriminator, while the discriminator is trained in turn to distinguish whether the input data entering the discriminator is real or fake, so that optimization occurs when the discriminator cannot identify whether the input data in the discriminator comes from the generator or from the real depth image.

17. The system according to claim 14, wherein, The trained GAN is configured to learn to segment identical instances of objects randomly placed in a box using an unsupervised method.

18. The system according to claim 14, wherein, The generator is a single instance generator that learns the appearance distribution of rigid objects from a cluttered or complex scene that includes multiple identical instances.

19. The system according to claim 14, wherein, The pose encoder takes the estimated depth image as input and generates a potential noise vector as output.

20. The system according to claim 14, wherein, The estimated depth image includes a noise vector input to the pose encoder, and as output, generates an axial-angle representation of the instance's 6D pose.

21. The system according to claim 14, wherein, The discriminator is configured to learn whether the number of rendered instances matches the number of instances in the real depth image and whether the layout of the estimated depth image is similar to the composition of the instances in the real depth image.

22. A method for generating a 3D object instance segmentation map by generating a realistic image from a real depth image, the method comprising the steps of: Receive a true depth image of an instance containing an object; The training depth image accessed from the data storage device is input into the trained generator to generate an estimated depth image representing an implicit 3D model of the object. Simultaneously, each estimated depth image is input into a trained discriminator and a trained pose encoder. The discriminator identifies each estimated depth image against the trained depth image to obtain a discriminator loss, and the pose encoder generates a set of pose transformation matrices that match a set of sampled transformation matrices to obtain a matching loss. The process is repeated iteratively on the estimated depth image until the discriminator loss and the matching loss are minimized to a threshold to complete the training period; The instances in the real depth image are identified by inputting the real depth image into the pose encoder of the trained GAN to generate pose transformation matrices for each instance in the real depth image, wherein each pose transformation is used to generate a simulated depth image for each instance in the real depth image. By identifying a set of pixels in each simulated depth image based on a predetermined pixel depth threshold, the pixels in the simulated depth image corresponding to the instance in the real depth image are identified; and The set of pixels corresponding to the same instance in the simulated depth image is merged to generate an instance segmentation map that is a realistic image of the real depth image.

23. The method according to claim 22, wherein, The GAN, trained during the training period, having the trained generator, trained discriminator, and trained pose encoder, is included in the Generative Adversarial Neural Network (GAN), wherein the GAN has been trained by accessing training depth images from a data storage device, the training depth images having instances identical to the instances of the real depth images.

24. A system for object-instance segmentation of a depth image composed of identical instances of 3D objects, the system comprising: A processor and a memory, wherein the memory stores computer code instructions, the processor and the memory being configured by the computer code instructions to cause the system to: Using an implicit template configured to capture the latent 3D representation of a 3D object, and a pose encoder configured to generate a 3D pose transformation matrix from a noise vector; as well as A generator is configured to apply multiple 3D transformation matrices using multiple noise vectors to the implicit template to generate a pseudo-depth image, and to jointly output the pseudo-depth image to a discriminator and an instance encoder, wherein the discriminator is configured to distinguish the real depth image from the pseudo-depth image and output a discrimination result based on the distinction, and the instance encoder is configured to output a set of latent vectors that match a set of noise vectors used in the pose encoder. During the training period, iterative operations train the discriminator and generator in an adversarial relationship, and train the generator, implicit template, pose encoder, and instance encoder in a cooperative relationship. The trained instance encoder is configured to generate latent vectors for each 3D object instance, which are iteratively used by the trained pose encoder to generate corresponding 3D transformation matrices to generate a single instance depth image by the trained generator. The single instance depth image includes only one instance of the 3D object in each depth image, and the pixels corresponding to the effective depth in the generated single instance depth image are used for instance segmentation.

25. A robotic system for manipulating an object, the robotic system comprising: A robotic arm having a manipulator configured to manipulate the object, wherein the manipulator includes: A torsion section, the torsion section being configured to ensure multiple degrees of freedom for moving the object; An end-effector configured to hold the object; An imaging device configured to measure the initial and / or target pose of the robotic arm for successful manipulation of the object; and Robot controller, the robot controller comprising: A processor connected to a transceiver and a data storage device storing computer-readable instructions including a trainable generative adversarial neural network (GAN), the trainable GAN having a trainable generator, a trained discriminator, and a trainable pose encoder, the computer-readable instructions, when executed by the processor, causing the processor to: Receive a true depth image of an instance containing an object via the transceiver; The training depth image accessed from the data storage device is input into the generator to generate an estimated depth image representing an implicit 3D model of the object; Simultaneously, each estimated depth image is input into the discriminator and the pose encoder. The discriminator identifies each estimated depth image by comparing it with the real depth image to obtain the discriminator loss, and the pose encoder generates a set of pose transformation matrices that match a set of sampled transformation matrices to obtain the matching loss. The process is performed iteratively on each estimated depth image until the discriminator loss and the matching loss are minimized to a threshold, at which point the training period ends. The instances in the real depth image are identified by inputting the real depth image into the pose encoder of the trained GAN, thereby generating pose transformation matrices for each instance in the real depth image, wherein each pose transformation matrix is ​​used to generate a depth image for each instance; and By identifying a set of pixels in each real depth image based on a predetermined pixel depth threshold, pixels in the depth image corresponding to the instance in the real depth image are identified, and the set of pixels in the real depth image is merged to form an instance segmentation map of the real depth image.

Images