Heuristic-based robot grabbing

Abstract

In some cases, the image and depth map can define a bin with randomly configured objects. The present invention recognizes that existing methods of training deep neural networks for grip calculations lack ability and efficiency, so that the resulting grip calculations and grip ways may be inaccurate or tedious, among other disadvantages. The composite depth image may be tagged with fetched annotations generated based on heuristic analysis, thereby defining an annotated composite dataset. The annotated synthetic dataset can be used to train a neural network to determine optimal capture locations for different objects arranged with each other in various locations.

Description

Background Art

[0001] Autonomous operations in unknown or dynamic environments, such as robotic grasping and manipulation, face various technical challenges. Autonomous operations in dynamic environments can be applied to mass customization (e.g., high-mix, low-volume manufacturing), on-demand flexible manufacturing processes in smart factories, warehouse automation in smart stores, and automated delivery from distribution centers in smart logistics. In order to perform autonomous operations (such as grasping and manipulation), robots can learn skills using machine learning, especially deep neural networks or reinforcement learning, in some cases.

[0002] Conventional robotic grasping or picking typically relies on a teach-based approach, where an operator can predetermine the poses that the robot uses for pick and drop locations. Advances in computer vision and deep learning have made flexible robotic bin picking possible, making it unnecessary to pre-teach pick and drop locations in some cases. For example, a camera system such as a red, green, blue, depth (RGB-D) camera can collect color pictures or images and a depth map or point cloud, and a neural network can perform grasping calculations based on the image or depth map. In some cases, such images and depth maps can define bins with randomly configured objects. However, various technical challenges remain associated with performing grasping calculations and grasping methods for such objects. For example, the present invention recognizes that existing methods of training deep neural networks to perform grasping calculations lack power and efficiency, such that, among other drawbacks, the resulting grasping calculations and grasping methods may be imprecise or cumbersome. Summary of the invention

[0003] Embodiments of the present invention address and overcome one or more of the disadvantages described herein by providing methods, systems, and apparatus for generating synthetic datasets for training grasping neural networks. For example, in various embodiments, synthetic depth images are labeled with grasp annotations generated based on heuristic analysis, thereby defining an annotated synthetic dataset. The annotated synthetic dataset can be used to train a neural network to determine optimal grasp positions for different objects arranged in various positions relative to each other.

[0004] In an example aspect, a system includes a robot defining an end effector configured to grasp an object. The system can also include a processor and a memory storing instructions that, when executed by the processor, configure the system to perform various operations. For an example, the system can obtain a depth image that defines a plurality of objects arranged in a plurality of corresponding positions. The system can identify a group of objects among the plurality of objects, each of which defines a corresponding surface exposed to an end effector of a robot configured to grasp the plurality of objects, thereby identifying the exposed objects. The system can also determine a grasping position on each of the exposed objects. The grasping position can define a corresponding exposed object range, in which the end effector contacts the exposed object to grasp the exposed object. In various examples, the system generates a probability map or a grasping label map, which includes corresponding grasping annotations at the grasping position on each of the exposed objects. The depth image and the probability map or the grasping label map can define an annotated synthetic data set including a plurality of grasping annotations. A neural network can be trained using the annotated synthetic data set to determine grasping positions on objects arranged in a plurality of configurations.

[0005] In some examples, the system compares the exposed object to the graspability criteria based on the end effector, thereby determining candidate areas of the exposed object that meet or exceed the graspability criteria. In an example where the end effector defines a vacuum gripper, the system can evaluate the candidate areas to determine a plane score associated with each candidate area. The plane score can indicate the curvature defined by the corresponding candidate area. Further, the system can compare each plane score to a predetermined threshold. In particular, based on the comparison, the system can determine a grasping annotation associated with each exposed object. Therefore, according to various embodiments, a neural network can be trained on the generated annotated synthetic data set. As an example, after training, the neural network can determine grasping locations on various objects located in a container so as to be at least partially stacked on top of each other. The various objects can include objects that define different shapes and sizes with respect to each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The above and other aspects of the present invention can be best understood by reading the following detailed description in conjunction with the accompanying drawings. For the purpose of illustrating the present invention, presently preferred embodiments are shown in the accompanying drawings, however, it should be understood that the present invention is not limited to the specific tools disclosed. The accompanying drawings include the following illustrations:

[0007] Figure 1 An example autonomous system is shown in an example physical environment including a box capable of containing various objects in accordance with an example embodiment.

[0008] Figure 2 is a computing system (e.g. Figure 1A flowchart of the operations that can be performed by the system.

[0009] Figure 3 A method that can be included in a computing system (e.g., Figure 1 The neural network model in the system shown in Figure 1.

[0010] Figure 4 An example synthetic dataset that can include annotated grasp labels on objects is shown, wherein the synthetic dataset can be generated by a computing system (e.g., Figure 1 The system shown in Figure 1 generates neural networks for training various neural networks (e.g., Figure 3 neural network model).

[0011] Figure 5 A computing environment is shown in which embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0012] First, the present invention recognizes that various technical challenges of training neural networks more specifically relate to preparing and annotating appropriate datasets that generalize well to various use cases. It is further recognized herein that the process of collecting datasets of real-world objects and robot grasping styles can be expensive due to various factors, such as different object types, different object shapes, camera sensor characteristics, experimental costs, etc. Further, these real-world datasets often do not generalize well, for example due to assumptions related to specific cameras, gripper physical models, object shapes, etc. In some cases, in order to better and / or more efficiently train neural networks, synthetic images (or datasets) of RGB-D images can be input into deep learning algorithms to learn grasping strategies for unseen environments. The present invention recognizes that such deep learning algorithms and neural networks can rely on the depth channel of camera frames, so that synthetic datasets require annotations for training. The embodiments described herein mainly address the challenges of annotating positive / negative grasping styles on such synthetic datasets, which can be transferred to real-world cases.

[0013] In particular, in the examples, the generated grasping dataset can summarize various object shapes, camera distances, gripper sizes, and box environments. In various examples, the observation space is defined by a depth image, so that annotations are made in the same space (depth image), rather than annotating on a 3D object space and mapping those annotations to a 2D space. The present invention recognizes that current vacuum adsorption-based grasping methods typically use analytical models to calculate the grasping direction of the 3D object space. For example, in some cases, a compliant adsorption contact model can quantify the formation of a seal using a quasi-static spring system. However, the present invention also recognizes that such an approach may require sampling each 3D point of a single object separately in world space, thereby preventing the encoding of static interactions within the object. Therefore, as an example, if the objects are in contact with each other, the current method may prevent the robot from successfully picking up a given object that defines a surface range that can form a good seal with the suction cup end effector, such as when the object is buried under another object. Furthermore, the present invention recognizes that these current approaches may result in datasets comprising cropped depth images centered around specific grasp candidates, which may unnecessarily increase the size of the dataset (e.g., due to oversampling) while failing to account for static relationships between objects (e.g., overlap).

[0014] Embodiments described herein can address such shortcomings by annotating in 2D space (e.g., depth images) rather than 3D space. Further, according to various embodiments, the generated grasp probability map encodes the spatial information of objects in a given scene, so that a sampling-based grasp strategy is not required. Such a map, such as a grasp probability map or a grasp label map 403 (see Figure 4 ), which can define a grab label for each pixel of the image. Additionally, this class of graphs can more efficiently train a fully convolutional neural network than using annotated 3D images and can be deployed with faster runtimes. For example, because a grab label can be generated for each pixel of a given image, the image and the label can define the same dimension to each other, enabling the training of a fully convolutional network. The present invention recognizes that a network with a single output channel can be trained using a sampling-based grab strategy instead of training a fully convolutional network. Thus, in an example, the system can determine the state of a given box and learn static interactions between objects in a scene, thereby identifying objects that are too close to the boundary, identifying objects that are cluttered and difficult to grab, identifying objects that are easier to grab than other objects, and the like. Specifically, a grab probability map (e.g., FIG. 403 ) can be generated for a depth image (e.g., depth image 401 ), and such probability maps and depth images can train various grab neural networks (e.g., fully convolutional networks) for efficient and accurate grab calculations.

[0015] See now Figure 1 , Figure 1An example of an industrial or physical environment or workspace 100 is shown. As used in the present invention, a physical environment or workspace can refer to any unknown or dynamic industrial environment. Unless otherwise specified, physical environment and workspace can be used interchangeably in the present invention without limitation. The reconstruction or model can define a virtual representation of a physical environment or workspace 100 or one or more objects 106 in the physical environment 100. For the purpose of example, the object 106 can be set in a box or container (e.g., box 107) to locate the grasping. Unless otherwise specified in the present invention, boxes, containers, pallets, boxes, etc. can be used interchangeably without limitation. As an example, the object 106 can be picked up from the box 107 by one or more robots and transported or placed in another location, such as the outside of the box 107. The example object 106 is shown as a rectangular object, such as a box, but it should be understood that the object 106 can be other shapes or define other structures as needed, and all such objects are included in the scope of the present disclosure.

[0016] The physical environment 100 can include a computerized autonomous system 102 configured to perform one or more manufacturing operations, such as assembly, transportation, etc. The autonomous system 102 can include one or more robotic devices or autonomous machines, such as autonomous machines or robotic devices 104, which are configured to perform one or more industrial tasks, such as box picking, grasping, etc. The system 102 can include one or more computing processors configured to process information and control the operation of the system 102, especially the autonomous machine 104. The autonomous machine 104 can include one or more processors, such as processor 108, which are configured to process information and / or control various operations associated with the autonomous machine 104. The autonomous system for operating the autonomous machine within the physical environment can also include a memory for storing modules. The processor can also be configured to execute modules to process information and generate models based on the information. It should be understood that the environment 100 and system 102 shown are simplified for the purpose of example. The environment 100 and system 102 can be changed as needed, and all such systems and environments are included in the scope of the present disclosure.

[0017] Still see Figure 1, the autonomous machine 104 can also include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 can include wheels 114, or can be otherwise configured to move within the physical environment 100. The autonomous machine 104 can also include an end effector 116 attached to the robotic manipulator 110. The end effector 116 can include one or more tools configured to grasp and / or move the object 106. Example end effectors 116 include finger grippers or vacuum grippers. The robotic manipulator 110 can be configured to move in order to change the position of the end effector 116, for example, in order to place or move the object 106 within the physical environment 100. The system 102 can also include one or more cameras or sensors, such as a depth camera or a three-dimensional (3D) point cloud camera 118, which is configured to detect or record the object 106 within the physical environment 100. The camera 118 can be mounted to the robotic manipulator 110 or otherwise configured to generate a 3D point cloud of a given scene (e.g., the physical environment 100). Alternatively or additionally, the one or more cameras of the system 102 can include one or more standard two-dimensional (2D) cameras that can record or capture images (e.g., RGB images or depth images) from different viewpoints. These images can be used to construct a 3D image. For example, a 2D camera can be mounted to the robotic manipulator 110 to capture images from a viewpoint along a given trajectory defined by the manipulator 110.

[0018] Still see Figure 1 , the camera 118 can be configured to capture images of the box 107, and therefore the object 106, along a first or lateral direction 120. In some cases, a deep neural network is trained on a set of objects. Based on its training, the deep neural network can calculate a grasp score for a corresponding area of ​​a given object (e.g., an object within the box 107). For example, the robotic device 104 and / or the system 102 can define one or more neural networks that are configured to learn various objects in order to recognize the poses, grasping points (or locations), and / or functional affordances of various objects that can be found in various industrial environments. According to various example embodiments, the example system or neural network model can be configured to learn objects and grasping locations based on, for example, images. For example, after the neural network completes training, an image of the object can be sent by the robotic device 104 to the neural network for classification, particularly classification of grasping locations or functional affordances.

[0019] See again Figure 1, the camera 118 can define a depth camera configured to capture a depth image of the workspace 100 from a perspective along a transverse direction 120. For example, the box 107 can define a top end 109 and a bottom end 111 opposite the top end 109 along the transverse direction 120. The box 107 can also define a first side 113 and a second side 115 opposite the first side 113 along a second or lateral direction 122 that is substantially perpendicular to the transverse direction 120. The box 107 can also define a front end 117 and a rear end 119 opposite the front end 117 along a third or longitudinal direction 124 that is substantially perpendicular to the transverse direction 120 and the lateral direction 122, respectively. Although the box 107 is shown to define a rectangular shape, it should be understood that the box or container can be other shapes or sizes, and all such boxes or containers are encompassed within the scope of the present disclosure.

[0020] See also Figure 2 , Figure 2 2 illustrates example operations 200 that can be performed by system 102, which can include one or more neural networks, such as example neural network 300 (see Figure 3 At 202, system 102 (particularly camera 118) may capture a depth image, such as depth image 401 (see Figure 4 ). Alternatively, the depth image 401 can define a synthetic depth image, such that at 202, the system 102 can obtain the synthetic depth image. The synthetic depth image can define an image that includes objects or information that are artificially generated rather than generated by real-world events (e.g., captured by a camera). In an example, the synthetic depth image 401 can include multiple objects 106. At 204, the system 102 can generate labels for the depth image. Synthetic images, such as the synthetic depth image 401, can be obtained using various open source or physics engines (e.g., pybullet, gazebo, mujoco) and open source rendering tools (e.g., pyrender). For example, in such engines, random objects can be loaded using the engine's mesh model. Objects can be placed or dropped in an artificial box to define a simulated scene. The rendering tool can then take a snapshot of the simulated scene to generate a synthetic data set, such as the synthetic depth image 401. In particular, for example, the system 102 can capture a snapshot (e.g., an RGB-D image) of a simulated scene that includes the object 106 in the box 107. As further described herein, system 102 can use a synthetic RGB-D image (such as depth image 401 ) to create a grasp label map (eg, map 403 ) as an annotation.

[0021] As an example, see also Figure 4, the system 102 can generate annotated grasping labels 402 associated with a given depth image 401, thereby defining an annotated depth image or an annotated synthetic data image 400. The grasping labels 402 can indicate locations on the object 106 where the corresponding object 106 can be grasped by the end effector 116. The grasping labels 402 can be generated based on the type of end effector 116 (e.g., finger gripper, vacuum gripper, etc.) or the geometry (e.g., size, shape) of the end effector 116. As an example, the grasping labels 402 can be different, such as indicating different locations, for a vacuum (e.g., suction cup) gripper defining a first radius compared to a vacuum gripper defining a second radius greater than the first radius. As another example, the grasping labels 402 can be different to indicate different grasping locations of the vacuum end effector 116 compared to the finger gripper end effector 116. The grasping labels 402 can also be based on the interaction of the object 106, which can be referred to as object interaction. Object interactions can indicate the position of an object relative to other objects. For example, a given label 402 may indicate an object interaction, which refers to the validity of the given label 402 on a specific object in the scene, which is touching another object. In the example, sufficiently separated or isolated objects 106 are marked as determined labels (capable of being grasped) so that other objects 106 that are hidden or obscured are not associated with the determined grasping labels because they cannot be grasped. In the example, each grasping label 402 can be defined as one or more pixels with a specific color on the map 403, thereby distinguishing the grasping location on the object from other locations on the object, but it should be understood that the grasping label can be an alternative shape or configured to identify the grasping location on the object, and all such alternatively configured annotated grasping labels are covered within the scope of the present disclosure.

[0022] Continue to see Figure 2, in order to generate annotated grab labels 402, at 206, the system 102 can identify exposed objects 106. In an example, the system 102 defines a rendering tool that is configured to generate a segmentation mask for each object of interest in the plurality of objects 106. The segmentation mask may change if a new object occludes the current object of interest. Therefore, the system 102 can determine whether the segmentation mask has changed by checking the segmentation mask of each object of interest in each scene. If the segmentation mask changes, the system 102 can determine that the given object of interest is occluded by the new object. In various examples, the above-mentioned segmentation mask defines a binary image, so that the system 102 can determine whether the segmentation mask has changed by comparing the "before" version and the "after" version of the segmentation mask pixel by pixel. If the common pixels of the "before" and "after" images are less than a predetermined threshold (e.g., 95%), the system 102 can classify the object of interest as occluded or not exposed, so that the object will not be associated with the determined grab label. It should be understood that the predetermined threshold can vary as needed.

[0023] In various examples, the system 102 can determine a singularity score associated with each of the objects 106 in the depth image 401 using a trained neural network. The singularity score can represent or indicate a percentage of common pixels associated with the before and after images of the segmentation mask provided by the rendering tool or the predetermined threshold described above. Thus, for example, the singularity score can indicate a degree of exposure of the corresponding object relative to the end effector 116, such that when the object is exposed, the corresponding object can be grasped by the end effector 116. As an example, an object disposed below other objects is generally more difficult to grasp than an object that is independent of and located above other objects, and therefore may have a lower singularity score. In some cases, for example, the system 102 can identify an object that defines a surface exposed to the camera 118 along the lateral direction 120, such that no object is located between the surface and the camera 118 along the lateral direction 120. Additionally or alternatively, the system 102 can identify an object that is least cluttered compared to other objects, such that the identified object defines a space and gap for grasping by the end effector 116. For example, at 206, based on the depth image 401, the system 102 can rank the objects 106 based on their respective exposure levels. In an example, the exposure level of a given object can be determined by calculating the degree to which other objects overlap the object along the lateral direction 120. Such determination of overlap can take into account the size of the given object. Additionally or alternatively, the exposure level of a given object can be determined based on a percentage of the surface area of ​​the object that is exposed, such as exposed along the lateral direction 120, such that no object is located between the end effector 116 and the surface area exposed along the lateral direction 120. Therefore, at 206, the system 102 can identify the exposed objects 106.

[0024] At 208, the system can determine candidate areas for grasping exposed objects. The candidate areas can define the range or location of the exposed objects that can be grasped. In particular, at 208, the system 102 can compare the pixels of the depth image 401 with the graspability criteria to determine the candidate areas. In an example, the candidate areas are associated with pixels that meet or exceed the grasping threshold determined by the graspability criteria. The graspability criteria can depend on various parameters of the environment 100, such as the type or size of the end effector 116, the distance between the camera 118 and the object 106, the size or shape of the object 106, etc. In some cases, by comparing the candidate areas with the graspability criteria, the system 102 can determine whether the objects are large enough, and in particular, whether each object defines a sufficient exposed surface range (candidate area) for placing the vacuum gripper on the corresponding object. In various examples, the system defines the dimensions of the end effector 116 (e.g., suction cup) in pixels, so that the number of pixels defined by the suction cup can be compared with the number of pixels defined by the corresponding candidate area, thereby identifying the candidate area, and therefore identifying the object as a grasping candidate. The system 102 can map the physical size or dimensions of the end effector 116 to pixels in the image space defined by the depth image 401 based on the height of the camera 118 or the distance of the camera 118 from the base 111 along the lateral direction 120. In particular, the system 102 can exclude the exposed object as a grasping candidate that defines a boundary that the end effector 116 would extend when attempting a grasp, for example, by comparing the pixels defined by the end effector 116 to the pixels defined by the exposed object.

[0025] In an example where the end effector 116 defines a vacuum gripper, at 210, the system 102 can evaluate or identify a planar area on an exposed object so that the vacuum gripper can form an adsorption seal on a given exposed object. For example, the system 102 can run a plane fitting algorithm on each pixel of the exposed object (or each exposed pixel) to determine a residual value for each exposed pixel. The residual values ​​can be used to sort the planar surfaces. For example, a surface that is more planar than other surfaces can have a lower residual value, and vice versa. A group of sorted pixels, such as the top 10% or 20%, may meet the planarity condition so as to be sufficiently planar. It should be understood that the percentage of exposed pixels that meet the planarity condition can vary.

[0026] Therefore, the system 102 can evaluate the candidate areas of the exposed object to determine whether each candidate area is sufficiently plane. The area can define the space around the candidate grasping pixel. The space around the grasping pixel can be defined by the dimensions of the end effector 116, such as the dimensions of the suction gripper. In some examples, the system 102 scores each candidate area based on the degree of plane or flatness of the area defined by the candidate area. In particular, for example, the system 102 can perform gradient analysis, distance field standard technology, depth histogram technology and / or plane fitting technology to calculate the plane score associated with each candidate area. The plane score can indicate the curvature defined by the corresponding candidate area. In the example, the normal vector of the corresponding plane defined by each candidate area defines the robot 104 (particularly the end effector 116) for each candidate area. The grasping approach angle. Therefore, the system 102 can compare the grasping approach angle associated with each candidate area with the predetermined tolerance defined by the robot 104, thereby filtering out the candidate area associated with the grasping approach angle that exceeds the tolerance range of the robot 104. In particular, for example, based on the plane scores associated with each candidate region, the system 102 can exclude candidate regions associated with grasping approach angles that exceed a predetermined tolerance range, such as candidate regions associated with grasping approach angles that are too steep, thereby defining preferred or optimal candidate regions. Therefore, the plane scores associated with the candidate regions and therefore the scores associated with the candidate grasped pixels can be used to select the best intra-object candidate, the best inter-object candidate, or a combination thereof. Intra-object candidates can refer to candidate grasped pixels located on the same object, and inter-object candidates can refer to candidate grasped pixels located on different objects. In the example, the selected candidate grasping regions are represented by labels 402 on the graph 403, so that the grasping label graph 403 can define an aggregation of possible grasping methods.

[0027] In some cases, system 102 can compare grasp candidate regions on one object with grasp candidate regions on a different object in the same scene (e.g., environment 100). In particular, corresponding plane scores can be compared so that grasp candidate regions associated with higher scores can be used. For example, see Figure 2At 212, based on the identified flat areas indicating the best candidate areas, the system 102 can filter the best candidate areas to select a final or best grasp position represented by a grasp label (e.g., annotated grasp label 402). In particular, for example, the system 102 can filter or reject grasping methods that will cause the grasped object to be physically unstable (e.g., due to the weight distribution of the object relative to the end effector 116). In some cases, the system 102 determines the center of mass of each object 106. Based on the center of mass, the system 102 can determine a boundary within which the corresponding object can be grasped in a stable manner. Grasping positions that fall outside the boundary associated with the center of mass can be rejected. Additionally or alternatively, the system 102 can determine a boundary or edge defined by the object. The system 102 can determine a corresponding threshold value with respect to the edge so that grasping methods outside the threshold value can be rejected, thereby defining a grasping method that is too close to the edge. In various examples, the threshold value with respect to the edge of the object and the boundary involving the center of mass of the object can be determined based on the size of the end effector 116 relative to the corresponding object. At 214, after filtering the candidate grasp regions, the system can generate annotated synthetic data samples, such as a grasp label map or probability map 403, which defines optimal grasp locations, such as locations that can be identified by grasp labels 402. The system 102 can also classify hidden or cluttered objects as negative grasps, such that the map 403 can include annotations associated with the negative grasps. The system can generate a probability map 403 from the synthetic depth image 401, such that the probability map 403 includes the grasp labels 402 and defines the heuristically generated annotations. Thus, the depth image 401 and the map 403 can define an annotated synthetic data set that can be used to train a neural network to recognize grasp patterns for objects in boxes arranged in multiple configurations (e.g., stacked on top of each other). See Figure 4 , the annotated depth image 400 includes a grasp label 402 overlaid on the depth image 401 for visualization.

[0028] At 216, a synthetic data set or image (e.g., synthetic data image 401 and probability map 403) can be input into a neural network to train the neural network in a supervised manner to recognize grasp locations on various objects in various arrangements and using various end effectors or grippers. Thus, synthetic depth image 401 can define an input image to the neural network, and map 403 can define an input label for image 401. In various examples, each determined pixel of map 403 defines a grasp candidate on synthetic depth image 401.

[0029] In particular, see, for example, Figure 3, a computing system (e.g., system 102) can define one or more systems or networks 300 that can be trained on multiple synthetic input images or labeled data 304. The input data 304 can include a synthetic depth image 401 and a probability map 403. It should be understood that the input data is not limited to the examples described in the present invention. That is, the annotated synthetic data can vary, for example, the data can include various objects (e.g., different shapes and sizes) located in various configurations, and all such annotated synthetic data are included within the scope of the present disclosure.

[0030] Continue to see Figure 3 , the network 300 can define an adversarial variational autoencoder (AVAE) system, such as a convolutional AVAE. The example neural network 300 includes multiple layers, such as an input layer 302a configured to receive data, an output layer 303b configured to generate a category or output score associated with the data or a portion of the data. For example, the output layer 303b can be configured to determine a grasping score, a singularity score, or a plane score. The neural network 300 also includes multiple intermediate layers connected between the input layer 302a and the output layer 303b. In particular, in some cases, the intermediate layers and the input layer 302a can define multiple convolutional layers 302. The intermediate layers can also include one or more fully connected layers 303. The convolutional layer 302 can include an input layer 302a configured to receive training and test data (such as annotated depth images). In some cases, the training data received by the input layer 302a includes synthetic data of any object. Synthetic data can refer to training data that has been generated to render objects in different positions, such as objects in a container. The convolutional layer 302 can also include a final convolutional layer or last feature layer 302c, and one or more intermediate or second convolutional layers 302b disposed between the input layer 302a and the final convolutional layer 302c. It should be understood that the model 300 shown is simplified for illustrative purposes. In particular, for example, the model can include any number of layers, in particular any number of intermediate layers, as desired, and all such models are encompassed within the scope of the present disclosure.

[0031] The fully connected layers 303 can include a first layer 303a and a second or output layer 303b, including connections between fully connected layers. For example, a neuron in the first layer 303a can transmit its output to each neuron in the second layer 303b, so that each neuron in the second layer 303b will receive input from each neuron in the first layer 303a. It should also be understood that the model is simplified for the purpose of explanation, and the model 300 is not limited to the number of fully connected layers 303 shown. In contrast to the fully connected layers, the convolutional layers 302 can be locally connected, such that, for example, a neuron in the intermediate layer 302b may be connected to a limited number of neurons in the final convolutional layer 302c. The convolutional layers 302 can also be configured to share the connection strength associated with the strength of each neuron.

[0032] Still see Figure 3 , the input layer 302a can be configured to receive an input 304, such as a depth image 401 or an annotated grasp label map 403, and the output layer 303b can be configured to return an output 306. The output 306 can include one or more classifications or scores associated with the input 304. For example, the output 306 can include an output vector indicating a plurality of scores 308 associated with respective portions (e.g., pixels) of the corresponding input 304. Thus, the output layer 303b can be configured to generate a score 308 associated with the image 304, and in particular, a score associated with the pixels of the image 304, thereby generating a grasp, singularity, or planarity score associated with the location of an object depicted in the image 304.

[0033] Thus, without being bound by theory, embodiments described herein enable the creation of custom synthetic scenes defining objects in a box and the understanding of dependencies between objects in the box in an observation space (e.g., a rendered depth image). Using the heuristic annotation criteria described herein, annotated datasets specific to certain gripper geometries, such as non-circular gripper cups or gripper cups of different radii, can be generated. At the same time, according to various embodiments, certain gripping methods can be characterized based on a general physical understanding of the reach of the robot arm and its mutual interference with a given box. In addition, in some cases, the present invention recognizes that the pipeline can be easily extended to support multi-suction cup arrays. Further, by annotating directly in the observation space, as described herein, the resulting annotated dataset can retain domain information, rather than mapping annotations from a higher dimensional space to a lower dimensional observation space. Further, the present invention recognizes that such annotations in the observation space can enable the system to run faster and more efficiently than methods that map annotations from a higher dimensional space to a lower dimensional space.

[0034] Figure 5An example of a computing environment in which embodiments of the present disclosure may be implemented is shown. Computing environment 600 includes a computer system 610, which may include a communication mechanism such as a system bus 621 or other communication mechanism for transmitting information within the computer system 610. The computer system 610 also includes one or more processors 620 coupled to the system bus 621 for processing information. The system 102 may include or be coupled to one or more processors 620.

[0035] The processor 620 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other processors known in the art. More broadly, a processor as described in the present invention is a device for executing machine-readable instructions stored on a computer-readable medium for performing tasks, and may include any one or a combination of hardware and firmware. The processor may also include a memory storing executable machine-readable instructions for performing tasks. The processor processes information by manipulating, analyzing, modifying, converting, or transmitting information for use by an executable process or information device, and / or by routing the information to an output device. The processor may use or include the capabilities of, for example, a computer, a controller, or a microprocessor, and may be adjusted using executable instructions to perform specific functions that a general-purpose computer does not perform. The processor may include any type of suitable processing unit, including but not limited to a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), a digital signal processor (DSP), and the like. In addition, the processor 620 can have any suitable micro-architecture design, which includes any number of constituent components, such as, for example, registers, multiplexers, arithmetic logic units, a cache controller for controlling read / write operations to cache memory, a branch predictor, etc. The micro-architecture design of the processor can be capable of supporting any of a variety of instruction sets. The processor can be coupled to any other processor (electrically coupled and / or as including executable components) to enable interaction and / or communication between them. The user interface processor or generator is a known element that includes electronic circuits or software or a combination of both for generating a display image or a display image of a portion thereof. The user interface includes one or more display images that enable a user to interact with the processor or other device.

[0036] The system bus 621 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between various components of the computer system 610. The system bus 621 may include, but is not limited to, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, etc. The system bus 621 may be associated with any suitable bus architecture, including, but not limited to, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnect (PCI) architecture, a PCI-Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, etc.

[0037] Continue to see Figure 5 , the computer system 610 may also include a system memory 630 coupled to the system bus 621 for storing information and instructions to be executed by the processor 620. The system memory 630 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 631 and / or random access memory (RAM) 632. RAM 632 may include other dynamic memory devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). ROM 631 may include other static memory devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 630 may be used to store temporary variables or other intermediate information during the execution of instructions by the processor 620. A basic input / output system 633 (BIOS) may be stored in ROM 631, which contains basic routines that help to transfer information between elements within the computer system 610, such as during startup. RAM 632 may contain data and / or program modules that are immediately accessible to the processor 620 and / or are currently being operated by the processor 620. System memory 630 may additionally include, for example, an operating system 634, application programs 635, and other program modules 636. Application programs 635 may also include a user portal for developing applications, allowing input parameters to be entered and modified as desired.

[0038] The operating system 634 may be loaded into the memory 630 and may provide an interface between other application software executing on the computer system 610 and the hardware resources of the computer system 610. More particularly, the operating system 634 may include a set of computer executable instructions for managing the hardware resources of the computer system 610 and providing common services to other application programs (e.g., managing memory allocations between various application programs). In certain example embodiments, the operating system 634 may control the execution of one or more program modules stored in the data storage 640. The operating system 634 may include any operating system now known or that may be developed in the future, including but not limited to any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

[0039] The computer system 610 may also include a disk / media controller 643 coupled to the system bus 621 to control one or more memory devices for storing information and instructions, such as a magnetic hard disk 641 and / or a removable media drive 642 (e.g., a floppy disk drive, an optical drive, a tape drive, a flash drive, and / or a solid-state drive). The memory device 640 may be added to the computer system 610 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), a universal serial bus (USB), or FireWire). The memory devices 641, 642 may be external to the computer system 610.

[0040] The computer system 610 may also include a field device interface 665 coupled to the system bus 621 to control field devices 666, such as devices used in a production line. The computer system 610 may include a user input interface or GUI 661, which may include one or more input devices, such as a keyboard, touch screen, tablet computer, and / or pointer device, for interacting with a computer user and providing information to the processor 620.

[0041] The computer system 610 may perform some or all of the processing steps of the embodiments of the present invention in response to the processor 620 executing one or more sequences of one or more instructions contained in a memory (such as a system memory 630). Such instructions may be read into the system memory 630 from another computer-readable medium (such as a magnetic hard disk 641 or a removable media drive 642) of the memory 640. The magnetic hard disk 641 and / or the removable media drive 642 may contain one or more data storage devices and data files for use in embodiments of the present disclosure. The data storage device 640 may include, but is not limited to, a database (e.g., relational, object-oriented, etc.), a file system, a flat file, a distributed data storage device (wherein data is stored on multiple nodes of a computer network), a peer-to-peer network data storage device, etc. The data storage device may store various types of data, such as skill data, sensor data, or any other data generated according to embodiments of the present disclosure. The data storage content and the data file may be encrypted to improve security. The processor 620 may also be implemented in a multi-processing device to execute one or more instruction sequences contained in the system memory 630. In an alternative embodiment, hard-wired circuits may be used instead of software instructions or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0042] As described above, the computer system 610 may include at least one computer-readable medium or memory for storing instructions programmed according to embodiments of the present invention and for containing data structures, tables, records, or other data described in the present invention. As used in the present invention, the term "computer-readable medium" refers to any medium that participates in providing instructions to the processor 620 for execution. Computer-readable media can take many forms, including but not limited to non-temporary, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid-state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disks 641 or removable media drives 642. Non-limiting examples of volatile media include dynamic memory, such as system memory 630. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, as well as wires that constitute the system bus 621. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0043] The computer-readable medium instructions for implementing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and traditional procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuits, thereby performing various aspects of the present disclosure.

[0044] The present invention describes various aspects of the present disclosure with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable medium instructions. The computing environment 400 may also include a computer system 610 operating in a networked environment, which uses a logical connection with one or more remote computers (such as a remote computing device 680). The network interface 670 can, for example, communicate with other remote devices 680 or systems and / or memory devices 641, 642 via a network 671. The remote computing device 680 can be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other public network node, and generally includes many or all of the above-mentioned elements about the computer system 610. When used in a networked environment, the computer system 610 may include a modem 672 for establishing communication through a network 671 (such as the Internet). The modem 672 may be connected to the system bus 621 via the user network interface 670 or via another appropriate mechanism.

[0045] The network 671 can be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or a series of connections, a cellular telephone network, or any other network or medium that can facilitate communication between the computer system 610 and other computers (e.g., a remote computing device 680). The network 671 can be wired, wireless, or a combination thereof. The wired connection can be implemented using Ethernet, a universal serial bus (USB), RJ-6, or any other wired connection generally known in the art. The wireless connection can be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellites, or any other wireless connection methods generally known in the art. In addition, several networks can work alone or communicate with each other to facilitate communication in the network 671.

[0046] It should be understood that Figure 5 The program modules, applications, computer executable instructions, codes, etc. depicted as stored in system memory 630 are merely illustrative and not exhaustive, and the processing described as being supported by any particular module may also be distributed among multiple modules or performed by different modules. In addition, various program modules, scripts, plug-ins, application programming interfaces (APIs), or other suitable computer executable codes hosted locally on computer system 610, remote devices 680, and / or hosted on other computing devices accessible via one or more networks 671 may be provided to support the processing by Figure 5 The functionality and / or additional or alternative functionality provided by the program modules, applications or computer executable codes depicted in the Figure 3 The processing jointly supported by the set of program modules depicted in the present invention may be performed by a fewer or greater number of modules, or the functionality described as supported by any particular module may be supported at least in part by another module. In addition, the program modules supporting the functions described in the present invention may form part of one or more applications that can be executed on any number of systems or devices according to any suitable computing model (such as, for example, a client-server model, a peer-to-peer model, etc.). In addition, the described by Figure 5 Any functionality supported by any program module depicted in the description may be implemented, at least in part, by hardware and / or firmware on any number of devices.

[0047] It should also be understood that the computer system 610 may include alternative and / or additional hardware, software or firmware components other than those described or depicted without departing from the scope of the present disclosure. More particularly, it should be understood that the software, firmware or hardware components depicted as forming part of the computer system 610 are merely illustrative, and that certain components may not be present or additional components may be provided in various embodiments. Although various illustrative program modules have been depicted and described as software modules stored in the system memory 630, it should be understood that the functions described as supported by the program modules may be enabled by any combination of hardware, software and / or firmware. It should also be understood that in various embodiments, each of the above modules may represent a logical partition of the supported functionality. The depiction of the logical partition is for the convenience of explaining the functionality and may not represent the structure of the software, hardware and / or firmware used to implement the functionality. Therefore, it should be understood that in various embodiments, the functionality described as provided by a particular module may be provided at least in part by one or more other modules. In addition, one or more of the depicted modules may not be present in some embodiments, while in other embodiments, additional modules that are not depicted may be present and may support at least part of the described functionality and / or additional functionality. Furthermore, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments such modules may be provided as stand-alone modules or sub-modules of other modules.

[0048] Although the specific embodiments of the present disclosure have been described, it will be appreciated by those of ordinary skill in the art that many other modifications and alternative embodiments are within the scope of the present disclosure. For example, any function and / or processing capability described with respect to a particular device or component can be performed by any other device or component. In addition, although various illustrative implementations and architectures have been described according to the embodiments of the present disclosure, it will be appreciated by those of ordinary skill in the art that many other modifications to the illustrative implementations and architectures described in the present invention are also within the scope of the present disclosure. In addition, it should be understood that any operation, element, component, data, etc. described in the present invention as being based on another operation, element, component, data, etc., can be additionally based on one or more other operations, elements, components, data, etc. Therefore, the phrase "based on" or its variants should be interpreted as "based at least in part on".

[0049] Although the embodiments have been described in language specific to structural features and / or methodological behaviors, it should be understood that the present disclosure is not necessarily limited to the specific features or behaviors described. Instead, specific features and behaviors are disclosed as illustrative forms of implementing the embodiments. Unless otherwise specifically stated or otherwise understood in the context of use, conditional language such as "may," "can," "can," "can," etc. is generally intended to convey that certain embodiments may include certain features, elements, and / or steps, while other embodiments do not include these features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments must include logic for determining whether these features, elements, and / or steps are included in any particular embodiment or whether they are to be performed in any particular embodiment with or without user input or prompting.

[0050] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality and operation of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, segment or partial instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions mentioned in the box may not occur in the order mentioned in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each box of the block diagram and / or flowchart illustration and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a dedicated hardware-based system that performs a specified function or behavior or implements a combination of dedicated hardware and computer instructions.

Claims

1. A computer-implemented method, the method comprising: obtaining a depth image, the depth image defining a plurality of objects arranged at a plurality of corresponding locations; identifying a group of objects among the plurality of objects, the group of objects each defining a respective surface exposed to an end effector of a robot configured to grasp the plurality of objects, thereby identifying the exposed objects; determining a grasping position on each of the exposed objects, the grasping position defining a range of the corresponding exposed object, within which the end effector contacts the exposed object to grasp the exposed object; generating a probability map including corresponding grasp annotations at the grasp locations on each of the exposed objects; The depth image and the probability map define an annotated synthetic dataset; and A neural network is trained using the annotated synthetic dataset to determine the grasp locations on the objects arranged in a plurality of configurations.

2. The method according to claim 1, further comprising: The exposed object is compared to a graspability criterion based on the end effector to determine a candidate region of the exposed object that exceeds the graspability criterion.

3. The method according to claim 2, wherein: The end effector defines a vacuum gripper, the method further comprising: The candidate regions are evaluated to determine a planarity score associated with each of the candidate regions, the planarity score being indicative of a curvature defined by the respective candidate region.

4. The method according to claim 3, further comprising: comparing each of said plane scores to a predetermined threshold; as well as Based on the comparison, the captured annotation associated with each of the exposed objects is determined.

5. A method according to any one of the preceding claims, wherein: The plurality of configurations include objects positioned in a container so as to be at least partially stacked on top of one another, the objects defining different shapes and sizes relative to one another.

6. A system comprising a robot, the robot defining an end effector configured to grasp an object, the system further comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the system to: obtaining a depth image, the depth image defining a plurality of objects arranged at a plurality of corresponding locations; identifying a group of objects among the plurality of objects, the group of objects each defining a respective surface exposed to the end effector of the robot, thereby identifying exposed objects; determining a grasping position on each of the exposed objects, the grasping position defining a range of the corresponding exposed object, within which the end effector contacts the exposed object to grasp the exposed object; generating a probability map including corresponding grasp annotations at the grasp locations on each of the exposed objects; The depth image and the probability map define an annotated synthetic dataset; and A neural network is trained using the annotated synthetic dataset to determine the grasp locations on objects arranged in a plurality of configurations.

7. The system of claim 6, wherein the memory further stores instructions that, when executed by the processor, further configure the system to: The exposed object is compared to a graspability criterion based on the end effector to determine candidate regions of the exposed object that meet or exceed the graspability criterion.

8. The system according to claim 7, wherein: The end effector defines a vacuum gripper, and the memory further stores instructions that, when executed by the processor, further configure the system to: The candidate regions are evaluated to determine a planarity score associated with each of the candidate regions, the planarity score being indicative of a curvature defined by the respective candidate region.

9. The system of claim 8, wherein the memory further stores instructions that, when executed by the processor, further configure the system to: comparing each of said plane scores to a predetermined threshold; and Based on the comparison, the captured annotation associated with each of the exposed objects is determined.

10. The system according to any one of claims 6 to 9, wherein: The plurality of configurations include objects positioned in a container so as to be at least partially stacked on top of one another, the objects defining different shapes and sizes with respect to one another.

Images