Heuristic-based robotic grasping

Abstract

In some cases, the image and depth map can define bins with randomly configured objects. Among other drawbacks, the current approach of training a deep neural network to perform grasping calculations lacks in capabilities and efficiency, and it is understood that the resulting grasping calculations and grasps can be inaccurate or complex. The synthetic depth image can be labeled with grasping annotations generated based on heuristic-based analysis to define an annotated synthetic dataset. The annotated synthetic dataset can be used to train a neural network to determine the best grasping positions for various objects placed at various positions relative to each other.

Description

Technical Field

[0001] Background Art Autonomous operations such as gripping and manipulation by robots in unknown or dynamic environments pose various technical challenges. Autonomous operations in dynamic environments can be applied to mass customization (e.g., small-batch production of multiple varieties), on-demand flexible manufacturing processes in smart factories, automation of warehouses in smart stores, automatic delivery from logistics centers in smart logistics, and the like. In order to perform autonomous operations such as gripping and manipulation, in some cases, robots can learn skills using machine learning, particularly deep neural networks or reinforcement learning.

[0002] Traditional robot gripping or picking generally relies on teaching-based methods, and an operator can pre-define the posture of the robot at the pick position and the drop position. With the progress of computer vision and deep learning, flexible bin picking by robots has become possible, and in some cases, prior teaching of the pick position and the drop position is no longer necessary. For example, camera systems such as red, green, blue, depth (RGB-D) cameras can collect color photos or images and depth maps or point clouds, and neural networks can perform gripping calculations based on the images or depth maps. In some cases, such images and depth maps can define bins with randomly configured objects. However, various technical challenges remain regarding gripping calculations and the execution of gripping for such objects. For example, among other drawbacks, current methods of training deep neural networks to perform gripping calculations lack capabilities and efficiency, and it is understood that the resulting gripping calculations and gripping may become inaccurate or complex.

[0003] Summary of the Invention Embodiments of the present invention address and overcome one or more of the drawbacks described herein by providing methods, systems, and apparatus for generating synthetic datasets for training a grasping neural network. For example, in various embodiments, a synthetic depth image can be labeled with grasping annotations generated based on heuristic analysis to define an annotated synthetic dataset. The annotated synthetic dataset can be used to train a neural network to determine the best grasping position for various objects positioned at different relative positions to one another.

[0004] In an exemplary embodiment, the system includes a robot that defines an end effector configured to grasp objects. The system may further include a processor and memory, the memory storing instructions that, when executed by the processor, configure the system to perform various actions. For example, the system may acquire a depth image that defines a plurality of objects, each positioned at multiple locations. To identify exposed objects, the system may identify a set of objects from the plurality of objects, each defining a surface exposed to the end effector of the robot configured to grasp the plurality of objects. The system may further determine a grasping position at each of the exposed objects. The grasping position can define the area of ​​the exposed object that the end effector contacts to grasp the exposed object. In various examples, the system generates a probability or grasping label map containing a grasping annotation for each grasping position of the exposed objects. The depth image and the probability or grasping label map may define an annotated synthetic dataset containing multiple grasping annotations. A neural network may be trained on the annotated synthetic dataset to determine grasping positions on objects positioned at multiple locations.

[0005] In some examples, the system compares exposed objects to a graspability criterion based on an end effector and determines candidate regions of exposed objects that meet or exceed the graspability criterion. In an example where the end effector defines a vacuum-type gripper, the system can evaluate the candidate regions and determine a planar score for each candidate region. The planar score can represent the curvature defined by each candidate region. Furthermore, the system can compare each planar score to a predetermined threshold. In particular, based on the comparison, the system can determine a grasp annotation for each exposed object. Thus, according to various embodiments, a neural network can be trained on a generated annotated synthetic dataset. After training, for example, the neural network can determine grasp positions on various objects arranged in a container so as to be at least partially stacked on top of each other. The various objects may include objects that define different shapes and sizes from each other.

[0006] The aforementioned and other embodiments of the present invention will be best understood by considering the following detailed description in conjunction with the accompanying drawings. While preferred embodiments at present are shown in the drawings for illustrative purposes, it is understood that the present invention is not limited to the specific means disclosed. The drawings include the following figures: [Brief explanation of the drawing]

[0007] [Figure 1] This illustrates an exemplary autonomous system in an exemplary physical environment, including a bin capable of containing various objects according to an exemplary embodiment. [Figure 2] This is a flowchart illustrating the operations that a computing system according to an exemplary embodiment, such as the system in Figure 1, can perform. [Figure 3] This shows a neural network model that may be included in a computing system according to an exemplary embodiment, such as the system shown in Figure 1. [Figure 4] This shows an exemplary synthetic dataset that may include annotated gripping labels on objects, which can be generated by a computing system, such as the system shown in Figure 1, to train various neural networks (e.g., the neural network model in Figure 3) according to various exemplary embodiments. [Figure 5] This document describes a computing environment in which embodiments of this disclosure may be implemented.

[0008] Modes for carrying out the invention As a first problem, it is understood that various technical challenges in training neural networks relate more specifically to preparing and annotating appropriate datasets that are sufficiently generalized for diverse use cases. Furthermore, it is understood that collecting real-world datasets related to object and robotic grasping can be an expensive process due to various factors such as different types of objects, different shapes of objects, characteristics of camera sensors, and experimental costs. Moreover, these real-world datasets are often not sufficiently generalized due to assumptions made regarding specific cameras, physical models of grasping devices, object shapes, etc. In some cases, to train neural networks better and / or more efficiently, composite images (or datasets) of RGB-D images can be input into a deep learning algorithm to learn grasping policies for invisible environments. Here, it is understood that such deep learning algorithms and neural networks may rely on the depth channel of the camera frame, and that the composite dataset requires annotation for training. The embodiments described herein address, among other things, the challenges of annotating positive / negative grasping in such composite datasets that can be translated into real-world cases.

[0009] In particular, in one example, a gripping dataset is generated that can be generalized with respect to various object shapes, camera distances, gripping tool sizes, and bin environments. In various examples, the observation space is defined by depth images, and annotations are created in the same space (depth image) instead of creating annotations in 3D object space and mapping those annotations to 2D space. Current methods for vacuum suction gripping are understood to often use analytical models for gripping calculation directions in 3D object space. For example, in some cases, a compliant suction contact model can quantify seal formation using a quasi-static spring system. However, it is further understood that such methods require each 3D point of a single object to be sampled separately in world space and therefore may not be able to represent static interactions between the objects themselves. Thus, for example, when objects are in contact with each other, current methods may not be able to successfully pick up a given object that defines a surface area for calculating a good seal with a suction cup end effector, for example, when an object is buried beneath another object. Furthermore, these current methods may result in datasets containing cropped depth images centered around specific grasp candidates, which is understood to unnecessarily increase the size of the dataset (e.g., due to over-sampling) while failing to account for static relationships between objects (e.g., overlaps).

[0010] The embodiments described herein can address such drawbacks, in particular, by annotating in 2D space (e.g., depth images) instead of 3D space. Furthermore, according to various embodiments, a grasp probability map is generated that represents the spatial information of objects in a given scene, eliminating the need for a sampling-based grasp policy. Such a map, e.g., a grasp probability or grasp label map 403 (see Figure 4), can define a grasp label for each pixel of an image. Moreover, the map can train a fully convolutional neural network more efficiently and can be deployed in faster execution times compared to using annotated 3D images. For example, since a grasp label can be generated for each pixel of a given image, the image and labels can define the same dimensions to each other so that a fully convolutional network can be trained. It is understood that instead of training a fully convolutional network, a network with a single output channel can be trained with a sampling-based grasp policy. Therefore, in one example, the system can determine the state of a given bin, learn static interactions between objects in the scene, identify objects that are too close to the boundary, identify objects that are crowded and difficult to grasp, identify objects that are easier to grasp compared to other objects, and so on. In particular, a grasp probability map (e.g., map 403) can be generated for a depth image (e.g., depth image 401), and such probability maps and depth images can be used to train various grasp neural networks (e.g., fully convolutional networks) to perform efficient and accurate grasp calculations.

[0011] Referring here to Figure 1, an exemplary industrial or physical environment or workspace 100 is shown. As used herein, a physical environment or workspace can refer to any unknown industrial environment or dynamic industrial environment. Unless otherwise specified herein, physical environments and workspaces can be used interchangeably without limitation. A reproduction or model can define a virtual representation of the physical environment or workspace 100 or one or more objects 106 within the physical environment 100. For illustrative purposes, an object 106 can be placed in a bin or container, for example, a bin 107, so as to be positioned for gripping. Unless otherwise specified herein, bins, containers, trays, boxes, etc., can be used interchangeably without limitation. For example, an object 106 can be removed from the bin 107 by one or more robots and transported or placed, for example, to another location outside the bin 107. Although the exemplary object 106 is shown as a rectangular object such as a box, it will be understood that object 106 may be of a different shape or have a different structure as desired, and all such objects are considered to be within the scope of this disclosure.

[0012] The physical environment 100 may include a computerized autonomous system 102 configured to perform one or more manufacturing operations, such as assembly and transportation. The autonomous system 102 may include one or more robotic devices or autonomous machines, e.g., an autonomous machine or robotic device 104, configured to perform one or more industrial tasks, such as bin picking and gripping. The system 102 may include one or more computing processors configured to process information and control the operation of the system 102, in particular the autonomous machine 104. The autonomous machine 104 may include one or more processors, e.g., a processor 108, configured to process information and / or control various operations related to the autonomous machine 104. The autonomous system for operating the autonomous machine within the physical environment may further include memory for storing modules. Processors may be further configured to execute modules to process information and generate models based on that information. It will be understood that the illustrated environment 100 and system 102 are simplified for illustrative purposes. The environment 100 and system 102 may vary as desired, and all such systems and environments are considered to be within the scope of this disclosure.

[0013] Referring further to Figure 1, the autonomous machine 104 may further include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 may include wheels 114 or be configured to move in other ways within the physical environment 100. The autonomous machine 104 may further include an end effector 116 attached to the robotic manipulator 110. The end effector 116 may include one or more tools configured to grasp and / or move an object 106. An exemplary end effector 116 includes a finger-type gripper or a vacuum-type gripper. The robotic manipulator 110 may be configured to move to change the position of the end effector 116, for example, to position or move an object 106 within the physical environment 100. System 102 may further include one or more cameras or sensors, such as a depth camera or a three-dimensional (3D) point cloud camera 118, configured to detect or record objects 106 in the physical environment 100. Camera 118 can be mounted on a robotic manipulator 110 or otherwise configured to generate a 3D point cloud of a given scene, such as the physical environment 100. Alternatively, or in addition, one or more cameras in System 102 may include one or more standard two-dimensional (2D) cameras capable of recording or capturing images from various viewpoints (e.g., RGB images or depth images). These images can be used to construct a 3D image. For example, a 2D camera can be mounted on the robotic manipulator 110 to capture images from viewpoints along a given trajectory defined by the manipulator 110.

[0014] Referring further to Figure 1, the camera 118 can be configured to capture images of the bin 107, and therefore the object 106, along a first direction or transverse direction 120. In some cases, the deep neural network is trained on a set of objects. Based on that training, the deep neural network can calculate a grip score for each region of a given object, such as the object in the bin 107. For example, the robotic apparatus 104 and / or system 102 can define one or more neural networks configured to learn various objects to identify the pose, grip point (or position), and / or affordances of various objects that may be found in various industrial environments. An exemplary system or neural network model can be configured to learn objects and grip positions based on images, for example, according to various exemplary embodiments. After training the neural network, for example, images of objects can be transmitted to the neural network by the robotic apparatus 104 for classification, particularly for classification of grip position or affordance.

[0015] Referring again to Figure 1, camera 118 can be defined as a depth camera configured to capture depth images of the workspace 100 from a viewpoint along the transverse direction 120. For example, bin 107 can have an upper end 109 and a lower end 111 opposite the upper end 109 in the transverse direction 120. Bin 107 can further have a first side 113 and a second side 115 opposite the first side 113 in a second direction or transverse direction 122 substantially perpendicular to the transverse direction 120. Furthermore, bin 107 can have a front end 117 and a rear end 119 opposite the front end 117 in a third direction or longitudinal direction 124 substantially perpendicular to both the transverse direction 120 and the transverse direction 122. While the illustrated bin 107 defines a rectangular shape, it will be understood that bins or containers may have other shapes or sizes, and all such bins or containers are considered to be within the scope of this disclosure.

[0016] See also Figure 2, which illustrates an exemplary operation 200 that can be performed by a system 102 that may include one or more neural networks, such as exemplary neural network 300 (see Figure 3). In 202, the system 102, in particular the camera 118, can capture a depth image, for example, depth image 401 (see Figure 4). Alternatively, depth image 401 may define a composite depth image, and in 202, the system 102 can acquire the composite depth image. A composite depth image may define an image containing artificially generated objects or information rather than being generated by real-world events (e.g., captured by a camera). In one example, composite depth image 401 may contain multiple objects 106. In 204, the system 102 can generate labels for the depth image. A composite image, for example, composite depth image 401, can be acquired using various open-source or physics engines (e.g., pybullet, gazebo, mujoco) and open-source rendering tools (e.g., pyrender). In such an engine, for example, random objects can be loaded using the engine's mesh model. The objects can be placed in or dropped into artificial bins to define a simulated scene. A rendering tool can then take a snapshot of the simulated scene to generate a composite dataset, such as a composite depth image 401. In particular, for example, system 102 can capture a snapshot of the simulated scene, such as an RGB-D image, containing object 106 in bin 107. As further described herein, system 102 can use the composite RGB-D image, such as depth image 401, to create a grasp label map (e.g., map 403) as annotations.

[0017] As an example, also referring to Figure 4, the system 102 can generate annotated grip labels 402 associated with a given depth image 401 in order to define an annotated depth image or an annotated composite data image 400. The grip labels 402 can indicate the position on the object 106 that can be gripped by the end effector 116. The grip labels 402 can be generated based on the type of end effector 116 (e.g., finger gripper, vacuum gripper, etc.) or the geometry of the end effector 116 (e.g., size, shape). As an example, the grip labels 402 may differ for a vacuum gripper (e.g., suction cup) that defines a first radius compared to a vacuum gripper that defines a second radius larger than the first radius, and may represent different positions, for example. As a further example, the gripping label 402 may differ for the vacuum-type end effector 116 compared to the finger-type end effector 116 to represent different gripping positions. The gripping label 402 may further be based on the interaction of objects 106, which may be called object interaction. Object interaction can indicate the position of an object relative to other objects. For example, a given label 402 may indicate object interaction that refers to the validity of the given label 402 on a particular object in the scene when that particular object is touching another object. In one example, a sufficiently isolated or separated object 106 is given a positive label (as it is grippable), and no positive gripping label is associated with another object 106 that is hidden or obscured, as it cannot be gripped. In one example, each gripping label 402 may define one or more pixels on the map 403 having a specific color to distinguish a gripping position on an object from other positions on the object, but the gripping labels may also be shaped or configured to identify a gripping position on an object, and it will be understood that all such alternatively configured annotated gripping labels are within the scope of this disclosure.

[0018] Continuing to refer to Figure 2, in order to generate annotated gripping labels 402, system 102 can identify exposed objects 106 in 206. In one example, system 102 defines a rendering tool configured to generate a segmentation mask for each object of interest among a plurality of objects 106. The segmentation mask may change if a new object obstructs the current object of interest. Therefore, system 102 can determine whether the segmentation mask changes by checking the segmentation mask in each scene for each object of interest. If the segmentation mask changes, system 102 can determine that a given object of interest is obstructed by a new object. In various examples, the segmentation mask described above defines a binary image so that system 102 can determine whether the segmentation mask changes by performing a pixel-by-pixel comparison between the "before" and "after" versions of the segmentation mask. If the number of common pixels between the "before" and "after" images is less than a predetermined threshold (e.g., 95%), system 102 can classify the object of interest as occluded or unexposed so that no positive grasping label is associated with the object. It will be understood that the predetermined threshold can vary as desired.

[0019] In various examples, using a trained neural network, system 102 can determine a singularity score for each object 106 in the depth image 401. The singularity score can represent or indicate the proportion of common pixels related to the preceding and succeeding images of the segmented mask provided by the rendering tool, or a predetermined threshold as described above. Thus, for example, the singularity score can indicate the degree to which each object is exposed to the end effector 116 so that each object can be grasped by the end effector 116 when the object is exposed. For example, an object placed directly beneath another object is typically difficult to grasp and therefore may have a lower singularity score compared to an object that is separated from and above other objects. In some cases, for example, system 102 can identify an object that defines a surface exposed to camera 118 in the transverse direction 120, where there are no objects between this surface and camera 118 in the transverse direction 120. In addition to or alternatively, the system 102 can identify the least crowded object compared to other objects, and the identified object determines the space and clearance for gripping by the end effector 116. For example, in 206, based on the depth image 401, the system 102 can rank the objects 106 based on their respective exposures. In one example, the exposure of a given object can be determined by calculating the extent to which other objects overlap the object in the transverse direction 120. Such overlap determination can take into account the size of a given object. In addition to or alternatively, the exposure of a given object can be determined, for example, based on the proportion of the surface area of ​​the exposed object in the transverse direction 120, such that no objects exist between the end effector 116 and the exposed surface area in the transverse direction 120.Therefore, in 206, the system 102 can identify the exposed object 106.

[0020] In 208, the system can determine candidate regions for grasping an exposed object. The candidate regions can define the graspable area or location of the exposed object. In particular, in 208, the system 102 can compare pixels of the depth image 401 with a graspability criterion to determine the candidate regions. In one example, the candidate regions relate to pixels that satisfy or exceed a grasp threshold determined by the graspability criterion. The graspability criterion may depend on various parameters of the environment 100, such as the type or size of the end effector 116, the distance of the camera 118 from the object 106, and the size or shape of the object 106. In some cases, by comparing the candidate regions with the graspability criterion, the system 102 can determine whether the object is large enough to place a vacuum gripper on it, and in particular whether each object has a sufficient exposed surface area (candidate region). In various examples, the system can define the dimensions of an end effector 116, such as a suction cup, in pixels, and identify candidate regions, and therefore objects that are candidates for grasping, by comparing the number of pixels defined by the suction cup with the number of pixels defined by each candidate region. The system 102 can map the physical size or dimensions of the end effector 116 to pixels in the image space defined by the depth image 401, based on the height of the camera 118 or the distance of the camera 118 from the lower end 111 in the transverse direction 120. In particular, for example, by comparing the pixels defined by the end effector 116 with the pixels defined by the exposed object, the system 102 can exclude exposed objects as grasp candidates that define boundaries that the end effector 116 is likely to straddle when attempting to grasp them.

[0021] In an example where the end effector 116 defines a vacuum gripper, system 102 can evaluate or identify planar areas on a given exposed object so that the vacuum gripper can form an adhesive seal on the exposed object in 210. For example, system 102 can run a planar fit algorithm on each pixel (or each exposed pixel) of the exposed object to determine the residual value of each exposed pixel. The residual values ​​can be used to rank the planes. For example, a surface that is flatter than other surfaces may have a smaller residual value, and vice versa. A set of ranked pixels, e.g., the top 10 or 20%, may satisfy the planarity condition so that they are sufficiently planar. It will be understood that the percentage of exposed pixels that satisfy the planarity condition can vary.

[0022] Therefore, system 102 can evaluate candidate regions of the exposed object and determine whether each candidate region is sufficiently flat. The region can define the space around the candidate gripping pixels. The space around the gripping pixels can be defined by the dimensions of the end effector 116, such as the dimensions of a suction gripper. In some examples, system 102 scores each candidate region based on how planar or flat the region defined by the candidate region is. In particular, for example, system 102 can perform gradient analysis, distance field criterion techniques, depth histogram techniques, and / or plane fitting techniques to calculate a plane score for each candidate region. The plane score can indicate the curvature defined by each candidate region. In one example, the normal vector to each plane defined by each candidate region determines the gripping approach angle of the robot 104, particularly the end effector 116, for each candidate region. Therefore, system 102 can compare the gripping approach angle for each candidate region with a predetermined tolerance range determined by robot 104, and exclude candidate regions with gripping approach angles outside the tolerance range of robot 104. In particular, for example, based on the planar score for each candidate region, system 102 can exclude candidate regions with gripping approach angles outside the predetermined tolerance range, such as those with gripping approach angles that are too steep, in order to determine or select the best candidate region. Thus, the planar score for the candidate region, and therefore the score for the candidate gripping pixel, can be used to select the best intra-object candidate, the best inter-object candidate, or a combination thereof. Intra-object candidates can refer to candidate gripping pixels on the same object, and inter-object candidates can refer to candidate gripping pixels on different objects. In one example, the selected candidate gripping regions are represented by labels 402 on the gripping label map 403, so that the gripping label map 403 can define possible gripping aggregates.

[0023] In some cases, system 102 can compare a candidate gripping region on a given object with candidate gripping regions on different objects in the same scene, such as environment 100. In particular, it can compare their planar scores and use the candidate gripping region with the higher score. For example, continuing to refer to Figure 2, in 212, based on the identified planar region indicating the best candidate region, system 102 can filter the best candidate regions to select the final gripping position or the best gripping position represented by a gripping label, such as an annotated gripping label 402. In particular, system 102 can filter or reject grips that are thought to result in physical instability of the gripped object, for example, due to the weight distribution of the object relative to the end effector 116. In some cases, system 102 determines the center of gravity of each object 106. Based on the center of gravity, system 102 can determine a boundary within which each object can be gripped stably. Grip positions that fall outside the boundary with respect to the center of gravity can be rejected. In addition to or instead of this, system 102 can determine boundaries or edges defined by the object. System 102 can determine thresholds for each edge and reject grips outside the thresholds that define grips that are too close to the edge. In various examples, the thresholds for the edges of an object and the boundaries for the centroid of an object can be determined based on the size of the end effector 116 relative to the size of each object. In 214, after filtering the candidate gripping regions, the system can generate an annotated synthetic data sample, such as a gripping label or probability map 403, which defines the best gripping position that can be identified, for example, by a gripping label 402. Furthermore, system 102 can classify hidden or crowded objects as negative grips so that the map 403 can include annotations for negative grips.The system can generate a probability map 403 from the synthetic depth image 401 such that the probability map 403 includes the grasping label 402 and defines a heuristically generated annotation. In this way, the depth image 401 and the map 403 can define an annotated synthetic dataset that can be used to train a neural network to identify grasps on objects in bins arranged in various configurations, such as stacked on top of each other. Referring to FIG. 4, the annotated depth image 400 includes a grasping label 402 overlaid on the depth image 401 for visualization.

[0024] In 216, for example, a synthetic dataset or image such as the synthetic data image 401 and the probability map 403 can be input into a neural network to train the neural network in a supervised manner to identify grasping positions on various objects in various arrangements by various end effectors or grippers. Thus, the synthetic depth image 401 can define the input image of the neural network, and the map 403 can define the input label for that image 401. In various examples, each positive pixel from the map 403 defines a grasping candidate on the synthetic depth image 401.

[0025] In particular, for example, referring to FIG. 3, a computing system, such as system 102, can define one or more systems or networks 300 that can be trained with a plurality of synthetic input images or label data 304. The input data 304 can include the synthetic depth image 401 and the probability map 403. It will be understood that the input data is not limited to the examples described herein. That is, the annotated synthetic data can be diverse. For example, the data can include various objects (e.g., different shapes and sizes) arranged in various configurations, and all such annotated synthetic data is considered to be within the scope of the present disclosure.

[0026] Continuing to refer to FIG. 3, network 300 can define an adversarial variational autoencoder (AVAE) system, such as a convolutional AVAE. An exemplary neural network 300 includes a plurality of layers, such as an input layer 302a configured to receive data, and an output layer 303b configured to generate a class score or an output score for the data or a portion of the data. For example, output layer 303b can be configured to determine a grasping score, an outlier score, or a planar score. Neural network 300 further includes a plurality of intermediate layers connected between input layer 302a and output layer 303b. In particular, in some cases, the intermediate layer and input layer 302a can define a plurality of convolutional layers 302. The intermediate layer can further include one or more fully connected layers 303. Convolutional layer 302 can include an input layer 302a configured to receive training data and test data, such as an annotated depth image. In some cases, the training data received by input layer 302a includes synthetic data of any object. Synthetic data can refer to training data generated, for example, by rendering objects at various positions within a container. Convolutional layer 302 can further include a final convolutional layer or last feature layer 302c, and one or more intermediate convolutional layers or second convolutional layers 302b disposed between input layer 302a and final convolutional layer 302c. It will be understood that the illustrated model 300 is simplified for purposes of illustration. In particular, for example, the model can include any number of layers, particularly any number of intermediate layers, as desired, and all such models are considered to be within the scope of the present disclosure.

[0027] A fully connected layer 303 may include a first layer 303a and a second layer or output layer 303b, and may include connections between fully connected layers. For example, neurons in the first layer 303a may transmit their outputs to all neurons in the second layer 303b, such that each neuron in the second layer 303b receives input from all neurons in the first layer 303a. Again, it will be understood that the model is simplified for illustrative purposes, and model 300 is not limited to the number of fully connected layers 303 shown. In contrast to fully connected layers, a convolutional layer 302 may be locally connected, for example, so that neurons in the hidden layer 302b may be connected to a limited number of neurons in the final convolutional layer 302c. Furthermore, the convolutional layer 302 may be configured to share connection strengths related to the strength of each neuron.

[0028] Referring further to Figure 3, the input layer 302a can be configured to receive an input 304, for example, a depth image 401 or an annotated gripping label map 403, and the output layer 303b can be configured to return an output 306. The output 306 may include one or more classifications or scores relating to the input 304. For example, the output 306 may include an output vector showing multiple scores 308 relating to different parts of the corresponding input 304, for example, pixels. Thus, the output layer 303b can be configured to generate gripping scores, singularity scores, or plane scores relating to the positions of objects drawn in the image 304 by generating scores 308 relating to the image 304, and in particular to the pixels of the image 304.

[0029] Therefore, although not bound by theory, the embodiments described herein can create custom composite scenes that define objects within bins and understand inter-object dependencies within bins in observation space (e.g., rendered depth images). Using the heuristic criteria for annotation described herein, annotated datasets specific to the geometry of a particular gripper, such as non-circular gripper cups or cups of different radii, can be generated. Simultaneously, according to various embodiments, a particular gripper can be characterized based on a general physical understanding of the reach of a robotic arm and its interaction with a given bin. Furthermore, in some cases, the pipeline is considered easily extensible to work with arrays consisting of multiple suction cups. Moreover, by creating annotations directly in observation space as described herein, the resulting annotated dataset can retain domain information, in contrast to mapping annotations from a higher-dimensional space to a lower-dimensional observation space. Furthermore, it is considered that annotations in such observation space can enable the system to operate faster and more efficiently compared to methods that map annotations from a higher-dimensional space to a lower-dimensional space.

[0030] Figure 5 shows an example of a computing environment in which embodiments of the present disclosure may be implemented. The computing environment 600 includes a computer system 610, which may include communication mechanisms such as a system bus 621 or other communication mechanisms for communicating information within the computer system 610. The computer system 610 further includes one or more processors 620 coupled to the system bus 621 for processing information. System 102 may include one or more processors 620, or may be coupled to one or more processors 620.

[0031] The processor 620 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processors known in the art. More generally, the processor described herein is a device for executing machine-readable instructions stored in a computer-readable medium in order to perform a task, and may comprise either hardware or firmware, or a combination thereof. Furthermore, the processor may include memory for storing executable machine-readable instructions for performing a task. The processor acts on information by manipulating, analyzing, modifying, transforming, or transmitting the information for use by an executable procedure or information device, and further / or routing the information to an output device. The processor may, for example, use or have the capabilities of a computer, controller, or microprocessor and be tuned with executable instructions to perform specialized functions that are not performed by a general-purpose computer. The processor may include, but is not limited to, any type of suitable processing unit, including, a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a composite instruction set computer (CISC) microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a digital signal processor (DSP), and so on. Furthermore, the processor 620 may have any suitable microarchitectural design that includes any number of components, such as registers, multiplexers, arithmetic logic units, a cache controller for controlling read / write operations on cache memory, branch predictors, etc. The microarchitectural design of the processor may be capable of supporting any of the various instruction sets. The processor may be coupled (electrically and / or including executable components) with any other processor and may be able to interact and / or communicate with each other.A user interface processor or generator is a known element comprising electronic circuitry, software, or a combination thereof for generating a display image or a part thereof. The user interface comprises one or more display images that enable user interaction with the processor or other devices.

[0032] The system bus 621 may include at least one of the following: a system bus, a memory bus, an address bus, or a message bus, enabling the exchange of information (e.g., data (including computer executable code), signaling, etc.) between various components of the computer system 610. The system bus 621 may also include, but is not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, etc. The system bus 621 may relate to any suitable bus architecture, but is not limited to, the Industry Standard Architecture (ISA), Microchannel Architecture (MCA), Extended ISA (EISA), Video Electronics Standards Association (VESA) architecture, Accelerated Graphics Port (AGP) architecture, Peripheral Component Interconnect (PCI) architecture, PCI Express architecture, Personal Computer Memory Card International Association (PCMCIA) architecture, Universal Serial Bus (USB) architecture, etc.

[0033] Continuing to refer to Figure 5, the computer system 610 may further include system memory 630 coupled to the system bus 621 for storing information and instructions executed by the processor 620. The system memory 630 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 631 and / or random access memory (RAM) 632. RAM 632 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, synchronous DRAM). ROM 631 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 630 may be used to store temporary variables or other intermediate information when instructions are executed by the processor 620. A basic input / output system 633 (BIOS) containing basic routines to assist in the transfer of information between elements within the computer system 610, such as at startup, may be stored in ROM 631. RAM 632 may contain data and / or program modules that are immediately accessible to the processor 620 and / or currently running on the processor 620. System memory 630 may further include, for example, an operating system 634, an application program 635, and other program modules 636. The application program 635 may further include a user portal for developing the application program that allows input of input parameters and modification as needed.

[0034] The operating system 634 may be loaded into memory 630 and can provide an interface between other application software running on the computer system 610 and the hardware resources of the computer system 610. More specifically, the operating system 634 may include a set of computer executable instructions for managing the hardware resources of the computer system 610 and providing common services to other application programs (e.g., managing memory allocation between various application programs). In certain exemplary embodiments, the operating system 634 may control the execution of one or more program modules, indicated as being stored in data storage 640. The operating system 634 may include, but is not limited to, any server operating system, any mainframe operating system, or any other dedicated or undedicated operating system, which are currently known or may be developed in the future.

[0035] Furthermore, the computer system 610 may include a disk / media controller 643 coupled to the system bus 621 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 641 and / or a removable media drive 642 (e.g., a floppy disk drive, compact disk drive, tape drive, flash drive, and / or solid-state drive). Storage devices 640 can be added to the computer system 610 using an appropriate device interface (e.g., Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire). Storage devices 641, 642 may be located outside the computer system 610.

[0036] Furthermore, the computer system 610 may include a field device interface 665 coupled to the system bus 621 for controlling field devices 666, such as equipment used in a manufacturing line. The computer system 610 may also include a user input interface or GUI 661, which may have one or more input devices such as a keyboard, touchscreen, tablet, and / or pointing device for interacting with a computer user and providing information to the processor 620.

[0037] The computer system 610 can execute some or all of the processing steps of embodiments of the present invention in response to the execution by the processor 620 of one or more sequences of one or more instructions contained in memory, such as system memory 630. Such instructions may be read into system memory 630 from another computer-readable medium in storage device 640, such as a magnetic hard disk 641 or a removable media drive 642. The magnetic hard disk 641 and / or removable media drive 642 may include one or more data stores and data files used by embodiments of the present disclosure. The data store 640 may include, but is not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores where data is stored on multiple nodes of a computer network, peer-to-peer network data stores, etc. The data store may store various types of data, such as skill data, sensor data, or any other data generated according to embodiments of the present disclosure. The contents of the data store and data files may be encrypted for enhanced security. The processor 620 may be used in a multiprocessing configuration to execute one or more sequences of instructions contained in system memory 630. In alternative embodiments, hardwired circuits may be used instead of, or in combination with, software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuits and software.

[0038] As described above, the computer system 610 may include at least one computer-readable medium or memory for holding instructions programmed according to embodiments of the present invention and for containing data structures, tables, records, or other data described herein. As used herein, the term “computer-readable medium” refers to any medium involved in providing instructions to the processor 620 for execution. Computer-readable mediums can take many forms, but are not limited to non-temporary, non-volatile, volatile, and transmission mediums. Non-limited examples of non-volatile mediums include optical disks, solid-state drives, magnetic disks, and magneto-optical disks such as magnetic hard disks 641 or removable media drives 642. Non-limited examples of volatile mediums include dynamic memory such as system memory 630. Non-limited examples of transmission mediums include coaxial cables, copper wires, and optical fibers, including wires that constitute the system bus 621. Transmission mediums may take the form of acoustic waves or light waves, such as those generated in radio and infrared data communications.

[0039] Computer-readable media instructions for performing the operations of the Disclosure may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages. Computer-readable program instructions may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to carry out an aspect of the present disclosure.

[0040] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable media instructions. The computing environment 400 may further include a computer system 610 operating in a network environment using logical connections to one or more remote computers, such as remote computing devices 680. A network interface 670 can enable communication with other remote devices 680 or systems and / or storage devices 641, 642, for example, via a network 671. The remote computing devices 680 may be personal computers (laptops or desktops), mobile devices, servers, routers, network PCs, peer devices, or other common network nodes, and typically include many or all of the elements described above with respect to the computer system 610. When used in a network environment, the computer system 610 may include a modem 672 for establishing communication on a network 671, such as the Internet. The modem 672 may be connected to the system bus 621 via the user network interface 670 or via another suitable mechanism.

[0041] Network 671 may be any network or system commonly known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or a series of connections, a mobile phone network, or any other network or medium that can facilitate communication between computer system 610 and other computers (e.g., remote computing device 680). Network 671 may be wired, wireless, or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection commonly known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection method commonly known in the art. Furthermore, several networks may operate individually or communicating with each other to facilitate communication in network 671.

[0042] The program modules, applications, computer executable instructions, code, etc., shown in Figure 5 as being stored in system memory 630 are merely illustrative and not exhaustive. It should be understood that processes described as being supported by any particular module may be distributed across multiple modules or executed by different modules. Furthermore, various program modules, scripts, plug-ins, application programming interfaces (APIs), or any other suitable computer executable code, hosted locally on computer system 610 and / or remote device 680, and / or on other computing devices accessible via one or more networks 671, may be provided to support the functions provided by the program modules, applications, or computer executable code shown in Figure 5, as well as / or additional or alternative functions. Moreover, the modularization of functions may vary; for example, processes described as being collaboratively supported by a collection of program modules shown in Figure 3 may be executed by fewer or more modules, or functions described as being supported by any particular module may be supported at least partially by other modules. In addition, program modules supporting the functions described herein can form part of one or more applications that can run across any number of systems or devices according to any suitable computing model, such as a client-server model or a peer-to-peer model. Furthermore, any of the functions described as being supported by any of the program modules shown in Figure 5 may be implemented at least partially in hardware and / or firmware across any number of devices.

[0043] It should be further understood that the computer system 610 may include alternative and / or additional hardware, software, or firmware components beyond those described or illustrated without departing from the scope of this disclosure. More specifically, it should be understood that the software, firmware, or hardware components shown as forming part of the computer system 610 are merely illustrative, and in various embodiments, some components may be omitted or additional components may be provided. Various exemplary program modules are illustrated and described as software modules stored in system memory 630, but it should be understood that the functions described as being supported by the program modules may be enabled by any combination of hardware, software, and / or firmware. It should be further understood that each of the modules described above may, in various embodiments, represent a logical division of the supported functions. This logical division is shown to facilitate the description of the functions and does not necessarily represent the structure of the software, hardware, and / or firmware for implementing the functions. Thus, it should be understood that functions described as being provided by a particular module may, in various embodiments, be provided at least partially by one or more other modules. Furthermore, one or more illustrated modules may not be present in a particular embodiment, while in other embodiments, additional modules not shown may exist that support at least some of the described functions and / or additional functions. Additionally, while certain modules may be illustrated and described as submodules of other modules, in certain embodiments, such modules may be provided as independent modules or as submodules of other modules.

[0044] While specific embodiments of this disclosure have been described, those skilled in the art will recognize that many other modifications and alternative embodiments are within the scope of this disclosure. For example, any of the functions and / or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Furthermore, while various exemplary implementations and architectures have been described in accordance with embodiments of this disclosure, those skilled in the art will understand that numerous other modifications to the exemplary implementations and architectures described herein are also within the scope of this disclosure. Moreover, it should be understood that any operation, element, component, data, etc. described herein as being based on another operation, element, component, data, etc. may be further based on one or more other operations, elements, components, data, etc. Accordingly, the expression "based on..." or any variation thereof should be interpreted as "at least partially based on...".

[0045] While embodiments have been described in language specific to structural features and / or methodological actions, it should be understood that this disclosure is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for carrying out the embodiments. Conditional language such as “can,” “could,” “might,” or “may,” in particular, is generally intended to convey that a particular embodiment includes certain features, elements, and / or steps, but other embodiments do not, unless otherwise explicitly stated or understood in the context of use. Thus, such conditional language does not generally mean that features, elements, and / or steps are required in any way in one or more embodiments, nor does it necessarily mean that one or more embodiments include logic for determining whether these features, elements, and / or steps are included in or performed in any particular embodiment, with or without user input or prompting.

[0046] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction set containing one or more executable instructions for performing a specified logical function. Note that in some alternative implementations, the functions mentioned in a block may be performed in an order other than that shown in the figure. For example, two consecutively shown blocks may actually be executed substantially simultaneously, depending on the function of the relationship, or possibly in reverse order. Also note that each block in a block diagram and / or flowchart, as well as combinations of blocks in a block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or operation, or executes a combination of dedicated hardware and computer instructions.

Claims

1. Obtaining depth images that define multiple objects placed at multiple different locations, In order to identify the exposed object, the set of objects from the plurality of objects, each of which defines the surface exposed to the end effector of the robot configured to grasp the plurality of objects, In each of the exposed objects, a gripping position is determined that defines the area of ​​the exposed object in which the end effector contacts the exposed object in order to grip the exposed object. The process involves generating a probability map containing a corresponding gripping annotation at the gripping position in each of the exposed objects, wherein the depth image and the probability map define and generate an annotated composite dataset. To determine the gripping position on objects arranged in multiple configurations, a neural network is trained on the annotated synthetic dataset, To determine candidate regions of the exposed object that meet or exceed the graspability criteria, the physical size or dimensions of the end effector are mapped to pixels in the image space defined by the depth image, and the exposed object is compared to the graspability criteria based on the end effector. including, Computer implementation methods.

2. The end effector defines a vacuum gripping device, and the method is The candidate regions are evaluated in such a way that a planar score is determined for each candidate region that represents the curvature defined by each candidate region. The method according to claim 1, further comprising:

3. This involves comparing each plane score with a predetermined threshold, Based on the above comparison, the grip annotation for each exposed object is determined. The method according to claim 2, further comprising:

4. The method according to any one of claims 1 to 3, wherein the plurality of configurations include objects arranged in a container so as to be stacked at least partially on top of each other, the objects having different shapes and sizes from each other.

5. A system comprising a robot having an end effector configured to grasp an object, Processor and Memory for storing instructions and Furthermore, When the above instruction is executed by the processor, Obtain a depth image that defines multiple objects placed at multiple different locations. In order to identify exposed objects, a set of objects is identified from among the plurality of objects, each of which defines a surface exposed to the end effector of the robot. For each of the exposed objects, a gripping position is determined that defines the area of ​​the exposed object in which the end effector contacts the exposed object in order to grip the exposed object. A probability map is generated that includes a corresponding gripping annotation at the gripping position in each of the exposed objects, and the depth image and the probability map define an annotated composite dataset. To determine the gripping position on an object arranged in multiple configurations, a neural network is trained on the annotated synthetic dataset. To determine candidate regions of the exposed object that meet or exceed the graspability criteria, the physical size or dimensions of the end effector are mapped to pixels in the image space defined by the depth image, and the exposed object is compared to the graspability criteria based on the end effector. The system is configured as follows: system.

6. The end effector defines a vacuum gripping device, and the memory, when executed by the processor, The candidate regions are evaluated in such a way that a planar score is determined for each candidate region that represents the curvature defined by each candidate region. The system according to claim 5, further storing instructions for further configuring the system as described above.

7. The memory, when executed by the processor, Each plane score is compared to a predetermined threshold, Based on the above comparison, the grip annotation for each exposed object is determined. The system according to claim 6, further storing instructions for further configuring the system as described above.

8. The system according to any one of claims 5 to 7, wherein the plurality of components include objects arranged in a container so as to be stacked at least partially on top of each other, the objects having different shapes and sizes from each other.

Images