Deep machine learning method and device for robotic grasping

Abstract

The present invention relates to deep machine learning methods and apparatus for robotic grasping. Some embodiments involve training a semantic grasping model to predict a measure indicating whether motion data of an end effector of a robot will result in a successful grasp of an object; and predicting an additional measure indicating whether the object has a desired semantic feature. Some embodiments involve using the trained semantic grasping model to serve a grasping end effector of a robot to achieve a successful grasp of an object with a desired semantic feature.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201780018883.0, filed on March 2, 2017. Technical Field

[0003] The present disclosure relates to deep machine learning methods and apparatus for robotic grasping. Background Art

[0004] Many robots are programmed to grasp one or more objects using one or more end effectors. For example, a robot may utilize a grasping end effector such as an "impact" gripper or an "ingressive" gripper (e.g., physically penetrating an object using a pin, needle, etc.) to pick up an object from a first location, move the object to a second location, and drop the object at the second location. Some additional examples of robot end effectors that can grasp objects include an "astrictive" end effector (e.g., using suction or vacuum to pick up an object) and one or more "contigutive" end effectors (e.g., using surface tension, freezing, or adhesives to pick up an object), to name a few. Summary of the invention

[0005] The present specification generally relates to deep machine learning methods and apparatus related to manipulation of objects by an end effector of a robot. Some embodiments relate to training a grasping deep neural network, such as a convolutional neural network (also referred to herein as a "CNN"), to predict the probability that candidate motion data for an end effector of a robot will result in a successful grasp of one or more objects by the end effector and / or to predict spatial transformation parameters of a spatial transformer network (also referred to herein as a "STN"). For example, some embodiments enable the application of the following as input to the trained grasping deep neural network: (1) candidate motion vectors defining candidate motions (if any) of the robot's grasping end effector and (2) an image capturing at least a portion of the robot's workspace; and based on the application, generating: (1) a measure that directly or indirectly indicates the probability that the candidate motion vector will result in a successful grasp; and / or (2) spatial transformation parameters that indicate a location in the image (or additional images) to which the grasping end effector will move. The metric indicative of probability and / or spatial transformation parameters may then be used in servoing execution of a grasp attempt by a robot having a grasping end effector, thereby improving the robot's ability to successfully grasp objects in its environment.

[0006] Some of those embodiments also involve training a spatial transformer network to generate a spatial transformation of an image based on received spatial transformation parameters and / or involve training a semantic deep neural network (e.g., a semantic CNN) to predict the category (and / or other semantic features) of an object in the spatial transformation based on the spatial transformation. In some versions of those embodiments, the semantic deep neural network can be trained based on both: training samples generated based on grasping attempts by one or more robots; and training samples from other computer vision datasets including training samples not generated based on grasping attempts by one or more robots (e.g., training samples from the IMAGENET image database). Using training samples from other computer vision datasets can increase the accuracy and / or breadth of the semantic understanding of the semantic deep neural network.

[0007] Some embodiments relate to utilizing a trained grasping deep neural network, a trained spatial transformer network, and / or a trained semantic neural network to serve a grasping end effector of a robot to achieve a successful grasp of an object having desired semantic object features by the grasping end effector. For example, each trained network can be utilized in iterative updates of motion control commands of one or more actuators of the robot that control the posture of the grasping end effector of the robot, and each trained network can be utilized to determine when to generate a grasping control command to achieve an attempted grasp by the grasping end effector. For example, a grasping control command can be generated and provided to a corresponding actuator only when an output generated on the grasping deep neural network indicates that the likelihood of a successful grasp satisfies a threshold and when an output generated on the semantic deep neural network indicates that the grasp will most likely be for an object having desired semantic object features.

[0008] Using a trained network to achieve successful grasping of objects with desired semantic object features can enable a robot to perform various actions for objects with specific object features, where those specific object features are specified by human-generated user interface inputs and / or by another component (e.g., a higher-level task planner). For example, a user can provide user interface input (e.g., verbal, typed) indicating a desire to grasp an object with one or more specific object features, and the robot can utilize the trained network to attempt a grasp only when the grasp is likely to succeed and is likely to be for an object with the specific object features. For example, a user can provide a user interface input of "pick up a spoon" and the robot can attempt a grasp based on output generated on a grasping deep neural network indicating that the grasp is likely to succeed and based on output generated on a semantic deep neural network indicating that the grasp is likely to be for an object classified as "spoon".

[0009] In some embodiments, a method is provided, the method comprising: generating candidate end-effector motion vectors, the candidate end-effector motion vectors defining a motion to move a grasping end-effector of a robot from a current pose to an additional pose; and identifying a current image captured by a vision sensor associated with the robot. The current image captures the grasping end-effector and at least one object in the robot's environment. The method also comprises: applying the current image and the candidate end-effector motion vectors as inputs to a trained grasping convolutional neural network; generating a measure of successful grasping of an object with the motion applied on the trained grasping convolutional neural network. The measure is generated based on applying the image and the end-effector motion vectors to the trained grasping convolutional neural network. The method also comprises: identifying desired semantic features of the object; applying a spatial transformation of the current image or an additional image captured by the vision sensor as input to a semantic convolutional neural network; and generating an additional measure based on the spatial transformation on the semantic convolutional neural network, the additional measure indicating whether the desired semantic features of the object are present in the spatial transformation. The method also includes generating an end effector command based on the measure of successful grasping and an additional measure indicating whether a desired semantic feature of the object is present; and providing the end effector command to one or more actuators of the robot.

[0010] These and other implementations may include one or more of the following features.

[0011] In some embodiments, the method further comprises: generating spatial transformation parameters based on applying the image and the end effector motion vector to the trained grasping convolutional neural network on the trained grasping convolutional neural network; and generating a spatial transformation based on the spatial transformation parameters on the spatial transformation network.

[0012] In some implementations, desired semantic features of an object define a class of objects.

[0013] In some embodiments, the method further comprises receiving a user interface input from a user interface input device and identifying the desired object semantic feature based on the user interface input. In some of those embodiments, the user interface input device is a microphone, such as a microphone of a robot.

[0014] In some implementations, the spatial transformation is for the current image, such as a spatial transformation that crops out a portion of the current image.

[0015] In some embodiments, the method further comprises: determining a current measure of a successful grasp of the object without applying the motion. In some of those embodiments, generating the end effector command based on the measure comprises: generating the end effector command based on a comparison of the measure to a current measure. In some versions of those embodiments, the end effector command is a grasp command and generating the grasp command is in response to: determining that the additional measure indicates that the desired object feature is present in the spatial transformation; and determining that the comparison of the measure to the current measure satisfies one or more criteria.

[0016] In some implementations, the end effector command is an end effector motion command and generating the end effector motion command includes generating the end effector motion command to conform to the candidate end effector motion vector.

[0017] In some embodiments, the end effector command is an end effector motion command and conforms to a candidate end effector motion vector. In some of those embodiments, the end effector motion command is provided to one or more actuators to move the end effector to a new pose, and the method further comprises: generating additional candidate end effector motion vectors, the additional candidate end effector motion vectors defining a new motion to move the grasping end effector from the new pose to another additional pose; identifying a new image captured by a vision sensor associated with the robot, the new image capturing the end effector in the new pose and capturing an object in the environment; applying the new image and the additional candidate end effector motion vectors as input to a trained grasping convolutional neural network; generating a new motion vector on the trained grasping convolutional neural network after applying the new motion vector; The invention relates to a method for grasping an object in a moving state, the new metric being generated based on applying the new image and the additional end-effector motion vector to a trained grasping convolutional neural network; applying an additional spatial transformation of the new image or a new additional image captured by a vision sensor as an input to a semantic convolutional neural network; generating a new additional metric based on the additional spatial transformation on the semantic convolutional neural network, the new additional metric indicating whether a desired object feature is present in the spatial transformation; generating a new end-effector command based on the new metric of successful grasping and the new additional metric indicating whether a desired object feature is present; and providing the new end-effector command to one or more actuators of the robot.

[0018] In some implementations, applying the image and the candidate end-effector motion vector as input to a trained grasping convolutional neural network includes: applying the image as input to an initial layer of the trained grasping convolutional neural network; and applying the candidate end-effector motion vector to an additional layer of the trained grasping convolutional neural network. The additional layer may be downstream of the initial layer.

[0019] In some implementations, generating a candidate end effector motion vector includes: generating a plurality of candidate end effector motion vectors; and performing one or more iterations of cross entropy optimization on the plurality of candidate end effector motion vectors to select a candidate end effector motion vector from the plurality of candidate end effector motion vectors.

[0020] In some embodiments, a method is provided, the method comprising: identifying a current image captured by a visual sensor associated with a robot; generating, on a grasping convolutional neural network based on applying the current image to the grasping convolutional neural network: a measure of successful grasping of an object captured in the current image by a grasping end effector of the robot; and spatial transformation parameters. The method also includes: generating a spatial transformation based on the spatial transformation parameters on a spatial transformer network. The spatial transformation is for the current image or an additional image captured by the visual sensor. The method also includes: applying the spatial transformation as input to a semantic convolutional neural network; generating an additional measure based on the spatial transformation on the semantic convolutional neural network, the additional measure indicating whether desired semantic features of the object are present in the spatial transformation; generating an end effector command based on the measure and the additional measure; and providing the end effector command to one or more actuators of the robot.

[0021] In some embodiments, a method is provided, the method comprising: identifying a current image captured by a vision sensor of a robot; and generating, on a portion of a semantic grasping model based on applying the current image to the portion: a measure of successful grasping of an object captured in the current image by a grasping end effector of the robot; and spatial transformation parameters. The method also includes: generating a spatial transformation of the current image based on the spatial transformation parameters; applying the spatial transformation as input to an additional portion of the semantic grasping model; generating an additional measure based on the spatial transformation on the additional portion, the additional measure indicating whether a desired semantic feature of the object is present in the spatial transformation; generating an end effector command based on the measure and the additional measure; and providing the end effector command.

[0022] In some embodiments, a method is provided, the method comprising identifying a plurality of training samples generated based on sensor outputs from one or more robots during a plurality of grasping attempts by the robots. Each of the training samples comprises a training sample input, the training sample input comprising: an image of a corresponding time instance of a corresponding grasping attempt for the grasping attempt, the image capturing the robot end effector and one or more environmental objects at the corresponding time instance; and an end effector motion vector, the end effector motion vector defining the motion of the end effector from the time instance pose of the end effector at the corresponding time instance to the final pose of the end effector for the corresponding grasping attempt. Each of the training samples also comprises a training sample output, the training sample output comprising: at least one grasped object label, the at least one grasped object label indicating a semantic feature of an object grasped by the corresponding grasping attempt. The method also comprises training a semantic convolutional neural network based on the training samples by one or more of the processors.

[0023] These and other implementations may include one or more of the following features.

[0024] In some embodiments, training the semantic convolutional neural network based on the training samples includes: applying the training sample input of a given training sample in the training samples to the grasping convolutional neural network; generating spatial transformer network parameters based on the training sample input of the given training sample on the grasping convolutional neural network; using the spatial transformer network parameters to generate a spatial transformation of an image of the given training sample; generating an output based on the spatially transformed image on the semantic convolutional neural network; and performing back propagation on the semantic convolutional neural network based on the output and the training sample output of the given training sample. In some of those embodiments, the method also includes training the semantic convolutional neural network based on additional training samples that are not generated based on the grasping attempt. In some of those embodiments, applying the training sample input of the given training sample to the grasping convolutional neural network includes: applying the image of the given training sample as input to an initial layer of the grasping convolutional neural network; and applying the end effector motion vector of the given training sample to an additional layer of the grasping convolutional neural network, the additional layer being downstream of the initial layer.

[0025] In some embodiments, the method further comprises training the semantic convolutional neural network based on additional training samples that are not generated based on the crawling attempt. In some of those embodiments, the method further comprises training the crawling convolutional neural network based on the training samples. In some versions of those embodiments, training the crawling convolutional neural network based on the training samples comprises: generating a predicted crawling metric on the crawling convolutional neural network based on a training sample input for a given training sample; and performing back propagation on the crawling convolutional neural network based on the predicted crawling metric and the training sample output for the given training sample.

[0026] In some embodiments, the training samples include: a first group of training samples generated based on outputs of a plurality of first robot sensors from a first robot during a plurality of grasping attempts by the first robot; and a second group of training samples generated based on outputs of a plurality of second robot sensors from a second robot during a plurality of grasping attempts by the second robot. In some of those embodiments, the first robot sensor includes a first vision sensor that generates images of the first group of training samples, the second robot sensor includes a second vision sensor that generates images of the second group of training samples, and a first pose of the first vision sensor relative to a first base of the first robot is different from a second pose of the second vision sensor relative to a second base of the second robot.

[0027] In some implementations, the grasping attempts on which the plurality of training samples are based each include a plurality of random actuator commands that randomly move the end effector from a starting pose of the end effector to a final pose of the end effector and then grasp using the end effector at the final pose.

[0028] In some embodiments, a method is provided that includes identifying a semantic convolutional neural network trained based on training samples that are not generated based on a robot grasping attempt; identifying multiple training samples generated based on sensor outputs from one or more robots during multiple grasping attempts by the robot; and training the semantic convolutional neural network based on the training samples.

[0029] These and other implementations may include one or more of the following features.

[0030] In some embodiments, the method further comprises training a crawling convolutional neural network based on the training samples.

[0031] In some embodiments, the training samples include: a first group of training samples generated based on outputs of a plurality of first robot sensors from a first robot during a plurality of grasping attempts by the first robot; and a second group of training samples generated based on outputs of a plurality of second robot sensors from a second robot during a plurality of grasping attempts by the second robot. In some of those embodiments, the first robot sensor includes a first vision sensor that generates images of the first group of training samples, the second robot sensor includes a second vision sensor that generates images of the second group of training samples, and a first pose of the first vision sensor relative to a first base of the first robot is different from a second pose of the second vision sensor relative to a second base of the second robot.

[0032] In some implementations, the grasping attempts based on the plurality of training samples each include a plurality of random actuator commands that randomly move the end effector from a starting pose of the end effector to a final pose of the end effector and then grasp with the end effector at the final pose.

[0033] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor (e.g., a central processing unit (CPU) or a graphics processing unit (GPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Yet other embodiments may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0034] It should be understood that all combinations of the above-mentioned concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1A An example environment is illustrated in which crawl attempts can be performed by a robot, data associated with the crawl attempts can be utilized to generate training samples, and / or the training samples can be utilized to train one or more networks of a semantic crawling model.

[0036] Figure 1B Graphics Figure 1A A network of semantic crawling models and example data that can be applied to and generated on the network.

[0037] Figure 2 Graphics Figure 1A and Figure 1B An example of one of the robots and the movement of the robot's grasping end effector along a path.

[0038] Figure 3 is a flow chart illustrating an example method of performing a crawl attempt and storing data associated with the crawl attempt.

[0039] Figure 4 is a flow chart illustrating an example method for generating training samples based on data associated with a robot's grasping attempts.

[0040] Figure 5 is a flow chart illustrating an example method for training one or more networks of a semantic crawling model based on training samples.

[0041] Fig. 6A and Figure 6B Illustration of the architecture of an example crawling convolutional neural network, example inputs that may be applied to the crawling convolutional neural network, and example outputs that may be generated on the crawling convolutional neural network.

[0042] Figure 6C Illustration of a spatial transformer network and a semantic convolutional neural network, example inputs that may be applied to the networks, and example outputs that may be generated on the networks.

[0043] Figure 7 is a flow chart illustrating an example method for servo-grasping an end effector using a semantic grasping model.

[0044] Figure 8 An example architecture of a robot is schematically depicted.

[0045] Fig. 9 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0046] Some implementations of the techniques described herein involve training a semantic crawling model that includes a crawling deep neural network, a spatial transformer network, and a semantic deep neural network.

[0047] In some of those embodiments, a trained grasping deep neural network may be utilized to generate, based on a current image captured by a vision sensor associated with the robot: 1) a metric indicating a probability of successful grasping of one or more objects by an end effector; and 2) spatial transformation parameters. Additionally, a trained spatial transformer network may be utilized to generate, based on the spatial transformation parameters and the current image (and / or additional images): a spatial transformation of the current image (and / or additional images). Still additionally, a trained semantic deep neural network may be utilized to generate, based on the spatial transformation: additional metrics indicating one or more semantic features of the spatial transformation, such as one or more categories of objects present in the spatial transformation.

[0048] In some versions of those embodiments, the robot may attempt to grasp with an end effector based on a metric (indicating a probability of successful grasping) satisfying a threshold and based on an additional metric indicating that a desired object semantic feature is present in the spatial transformation. The desired object semantic feature may be based on, for example, a user interface input and / or input from one or more separate components. Thus, a trained semantic grasping model may enable a grasp attempt to be performed on an object based on determining that the likelihood of success of the grasp attempt satisfies a threshold and based on determining that the object is likely to have the desired object semantic feature.

[0049] As described in more detail herein, in various embodiments the output generated on the grasping deep neural network can be considered "dorsal" because it is primarily based on spatial reasoning and predicts whether the grasp will be successful and / or which part of the image to process without semantic understanding of the grasped object. In addition, in various embodiments the output generated on the spatial transformer network and the semantic deep neural network can be considered "ventral" because it is primarily based on semantic reasoning and predicts semantic features of the object being considered for grasping.

[0050] In some embodiments, a trained crawling deep neural network receives an image (I t and optionally I o ) and accept the end effector motion vector (v t ), such as task space motion vectors. The images are generated by the vision sensor and may include situations where the grasping end effector is not present (or where the end effector is in a position different from that in the current image (I t ) in different poses) to capture a “current” image of the scene (I t ) and additional images (I o ). t and optionally o ) and the end effector motion vector (v t ) applied to the trained grasping deep neural network can be used to generate on the grasping deep neural network: it executes commands to achieve the motion vector (v t ) defined motion and subsequent grasping will produce a predicted measure of successful grasping; and predicted spatial transformation parameters. The generated spatial transformation parameters can be used to generate a spatial transformation (e.g., a spatial transformation of the current image) on a trained spatial transformation network. The spatial transformation can be applied to a semantic deep neural network to generate measures on the network that indicate features of an object that are present in the spatial transformation. Whether to attempt a grasp by a grasp end effector can be based on the predicted measure of successful grasping and / or which features are indicated as being present in the spatial transformation. Additional descriptions of these and other embodiments of the technology are provided below.

[0051] refer to Figure 1A-Figure 6B , various implementations of the networks 125, 126 and / or 127 for training the semantic crawling model 124 are described. Figure 1A The diagram illustrates an example environment in which a crawl attempt can be performed by a robot (e.g., robot 180A, 180B, and / or other robots), data associated with the crawl attempt can be used to generate training samples, and / or the training samples can be used to train various networks 125, 126, and / or 127 of the semantic crawl model 124.

[0052] exist Figure 1A Example robots 180A and 180B are illustrated in FIG. Robots 180A and 180B are "robot arms" having multiple degrees of freedom to enable traversal of gripping end effectors 182A and 182B along any of multiple potential paths to position gripping end effectors 182A and 182B in a desired location. For example, referring to Figure 2 , illustrates an example of robot 180A traversing its end effector along path 201. Figure 2 Included are phantom and non-phantom images of the robot 180A showing two different poses among a set of poses assumed by the robot 180A and its end effector as it traverses along the path 201. Referring again to Figure 1A , robots 180A and 180B each also control two opposing "jaws" of their corresponding gripping end effectors 182A, 182B to actuate the jaws between at least an open position and a closed position (and / or optionally multiple "partially closed" positions).

[0053] exist Figure 1A Also illustrated in FIG. 1 are example vision sensors 184A and 184B. Figure 1A In the embodiment, the vision sensor 184A is mounted in a fixed attitude relative to the base or other stationary reference point of the robot 180A. The vision sensor 184B is also mounted in a fixed attitude relative to the base or other stationary reference point of the robot 180B. Figure 1A As illustrated in , the posture of visual sensor 184A relative to robot 180A is different from the posture of visual sensor 184B relative to robot 180B. As described herein, in some embodiments this can be beneficial to enable the generation of different training samples, which can be used to train a neural network that is robust to camera calibration and / or independent of camera calibration. Visual sensors 184A and 184B are sensors that can generate images related to the shape, color, depth and / or other features of objects in the line of sight of the sensor. Visual sensors 184A and 184B can be, for example, monocular vision (monographic) cameras, stereoscopic vision (stereographic) cameras and / or 3D laser scanners. 3D laser scanners include one or more lasers that emit light and one or more sensors that collect data related to the reflection of the emitted light. The 3D laser scanner can be, for example, a time-of-flight 3D laser scanner or a 3D laser scanner based on triangulation, and can include a position sensitive detector (PSD) or other optical position sensor.

[0054] Vision sensor 184A has a field of view of at least a portion of the workspace of robot 180A, such as a portion of the workspace that includes example object 191A. Figure 1AThe resting surface of the objects 191A is not shown, however those objects may be resting on a table, a tray, and / or other surface. The objects 191A include a shovel, a stapler, and a pencil. In other embodiments, more objects, fewer objects, additional objects, and / or alternative objects may be provided during all or part of the grasping attempts of the robot 180A as described herein. For example, in some embodiments, objects having the same one or more features (e.g., all objects having a first classification) may be provided during some grasping attempts of the robot 180A, objects having the same one or more different features (e.g., all objects having a second classification) may be provided during some other grasping attempts of the robot 180A, and the like.

[0055] Vision sensor 184B has a field of view of at least a portion of the workspace of robot 180B, such as a portion of the workspace that includes example object 191B. Figure 1A The resting surface of objects 191B is not shown, however they may be resting on a table, tray, and / or other surface. Objects 191B include pencils, staplers, and glasses. In other embodiments, more objects, fewer objects, additional objects, and / or alternative objects may be provided during all or part of a grasping attempt of the robot 180B as described herein. For example, in some embodiments, objects having the same one or more features (e.g., all objects of a first classification) may be provided during some grasping attempts of the robot 180B, objects having the same one or more different features (e.g., all objects of a second classification) may be provided during some other grasping attempts of the robot 180B, and the like.

[0056] Despite Figure 1A Specific robots 180A and 180B are illustrated in the drawings, however, additional and / or alternative robots may be utilized, including additional robotic arms similar to robots 180A and 180B, robots having other robotic arm forms, robots having humanoid forms, robots having animal forms, robots that move via one or more wheels (e.g., self-balancing robots), submersible vehicle robots, unmanned aerial vehicles (“UAVs”), etc. Figure 1A Specific gripping end effectors are illustrated in , however additional and / or alternative end effectors may be utilized, such as alternative impact gripping end effectors (e.g., those with gripping "plates," those with more or fewer "fingers" / "claws"), "intrusive" gripping end effectors, "retracting" gripping end effectors, or "contacting" gripping end effectors, or non-grasping end effectors. Additionally, although in Figure 1ASpecific mounting arrangements for vision sensors 184A and 184B are illustrated in FIG, however additional and / or alternative mounting arrangements may be used. For example, in some embodiments, the vision sensors may be mounted directly to the robot, such as on a non-actuable component of the robot or on an actuable component of the robot (e.g., on an end effector or on a component near an end effector). Additionally, for example, in some embodiments, the vision sensors may be mounted on a non-stationary structure separate from the robot with which they are associated and / or may be mounted in a non-stationary manner on a structure separate from the robot with which they are associated.

[0057] Robots 180A, 180B, and / or other robots may be utilized to perform a large number of grasping attempts and data associated with the grasping attempts may be utilized by the training sample generation system 110 to generate training samples for the training sample database 117. In some implementations, all or multiple aspects of the training sample generation system 110 may be implemented on the robot 180A and / or the robot 180B (e.g., via one or more processors of the robots 180A and 180B). For example, the robots 180A and 180B may each include an instance of the training sample generation system 110. In some implementations, all or multiple aspects of the training sample generation system 110 may be implemented on one or more computer systems that are separate from the robots 180A and 180B but in network communication with the robots 180A and 180B.

[0058] Each grasping attempt performed by robot 180A, 180B and / or other robots consists of T separate time steps or instances. At each time step, the current image captured by the vision sensor of the robot performing the grasping attempt is stored Also stores the current pose of the end effector The robot selects a path (translation and / or rotation) along which to move the gripper next. At the final time step T, the robot actuates (e.g., closes) the gripper and stores additional data and / or performs one or more additional actions to enable evaluation of the success of the grasp. The grasp success engine 116 of the training sample generation system 110 evaluates the success of the grasp, thereby generating a grasp success label (sl i ) and / or the grasped object label (ol i ), where the grasped object labels indicate the characteristics of the grasped objects (if any objects were actually grasped).

[0059] Each crawl attempt generates T training samples, through or That is, each training sample includes at least The observed image, the end effector motion vector from the pose at that time step to the pose finally reached (the final pose of the grasp attempt) and the crawl success tag of the crawl attempt (l i ) and / or the grasped object label (ol i ). Each end effector motion vector may be determined by an end effector motion vector engine 114 of the training sample generation system 110. For example, the end effector motion vector engine 114 may determine the transformation between the current pose and the final pose of the grasping attempt and use the transformation as the end effector motion vector. Training samples for multiple grasping attempts of multiple robots are stored in a training sample database 117 by the training sample generation system 110.

[0060] Data generated by sensors associated with the robot and / or data derived from the generated data may be stored in one or more non-transitory computer-readable media local to the robot and / or remote from the robot. In some embodiments, the current image may include multiple channels, such as a red channel, a blue channel, a green channel, and / or a depth channel. Each channel of the image defines a value for each of a plurality of pixels of the image, such as a value from 0 to 255 for each of the pixels of the image. In some embodiments, each of the training samples may include a current image and an additional image of the corresponding grasping attempt, wherein the additional image does not include a grasping end effector or includes an end effector in a different pose (e.g., a pose that does not overlap with the pose of the current image). For example, the additional image may be captured after any prior grasping attempt but before the end effector movement for the grasping attempt begins and when the grasping end effector is moved out of the field of view of the visual sensor. The current pose and the end effector motion vector from the current pose of the grasping attempt to the final pose may be represented in task space, in joint space, or in another space. For example, an end effector motion vector may be represented by five values ​​in task space: three values ​​defining a three-dimensional (3D) translation vector, and two values ​​representing a sine-cosine encoding of the end effector's orientation change about the end effector's axis.

[0061] In some embodiments, the grasp success label is a binary label, such as a "0 / successful" or "1 / unsuccessful" label. In some embodiments, the grasp success label can be selected from more than two options, such as 0, 1, and one or more values ​​between 0 and 1. For example, "0" can indicate a confirmed "unsuccessful grasp", "1" can indicate a confirmed successful grasp, "0.25" can indicate "most likely unsuccessful grasp" and "0.75" can indicate "most likely successful grasp".

[0062] In some embodiments, the grasped object tags each indicate whether a corresponding feature exists in the corresponding grasped object. For example, a grasped object tag for a grasped object may indicate which of one or more object categories the object belongs to. Object categories of various granularities may be utilized. The grasped object tags may each be binary (e.g., a feature exists or does not exist) or selected from more than two options (e.g., a feature is likely not present, most likely not present, likely present, most likely present). As described herein, the grasp success engine 116 may utilize one or more techniques to assign grasped object tags to grasp attempts. For example, the grasp success engine 116 may assign the tag to a grasp attempt based on the grasp attempt coming from a set of objects that all have a grasped object tag (e.g., assigning a "toothbrush" tag when the object is grasped from multiple objects that are all "toothbrushes"). In addition, for example, the grasping success engine 116 can capture an image of the object grasped in the grasping attempt (e.g., by moving the grasping end effector in front of it while the visual sensor is still grasping the object), and use the captured image to determine the grasped object label. For example, the grasping success engine 116 can provide the captured image to the computing device of one or more human reviewers and use the input from one or more human reviewers (via their corresponding computing devices) to determine the grasped object label. In addition, for example, the grasping success engine 116 can apply the captured image (or its spatial transformation) to the trained semantic CNN (optionally separated from the semantic CNN 127) and use the output generated on the trained semantic CNN to determine the grasped object label. In some of those instances, the trained semantic CNN can be optionally trained based on captured images labeled by human reviewers (e.g., training samples with training sample inputs based on captured images and training sample outputs based on human reviewer labels) and / or additional training samples (e.g., training samples from IMAGENET).

[0063] The training engine 120 trains one or more of the networks 125, 126, and 127 of the semantic crawling model 124 based on the training samples of the training sample database 117. The training engine 120 may use some training samples of the training sample database 117 to train only the crawling CNN 125. For example, some training samples of the training sample database 117 may include the corresponding time step The observed image, the end effector motion vector from the pose at that time step to the pose finally reached (the final pose of the grasp attempt) And the successful capture tag (l i )—but does not include the grasped object label of the grasp attempt (ol i ) (e.g., because the crawl was unsuccessful and / or because the tags were not applied to a successful crawl).

[0064] The training engine 120 may also utilize some other training samples of the training sample database 117 to additionally or alternatively train the grasping CNN and STN 126 and / or the semantic CNN 127. For example, some training samples of the training sample database 117 may include the following: The observed image, the end effector motion vector from the pose at that time step to the pose finally reached (the final pose of the grasp attempt) Crawling success tag (l i ), and also includes the grasped object label (ol i ).

[0065] In some embodiments, the training engine 120 also trains the semantic CNN 127 using additional training samples from the semantic training sample database 119. The training samples of the database 119 may include training samples from other computer vision datasets, including training samples that are not generated by one or more robots based on grasping attempts (e.g., training samples from the IMAGENET image database). In some of those embodiments, the additional training samples of the database 119 may be used to train the semantic CNN 127 before any training based on training samples generated based on grasping attempts. In addition, in some embodiments, such additional training samples may be used to pre-train the semantic CNN 127. For example, the semantic CNN 127 may be a pre-trained object recognition network, such as an initial network trained on IMAGENET data. Using additional training samples from other computer vision datasets may increase the accuracy and / or breadth of the semantic understanding of the semantic CNN 127. The training engine 120 may optionally utilize various domain adaptation techniques when training the semantic CNN 127 based on additional training samples and / or based on training samples generated based on grasping attempts.

[0066] Reference below Figure 3-5 Additional descriptions of embodiments of performing crawling attempts, generating training data based on the crawling attempts, and training the semantic crawling network 124 are provided.

[0067] Figure 1B Graphics Figure 1A The networks 125, 126, and 127 of the semantic grasping model 124 and the example data that can be applied to and generated on the trained versions of the networks 125, 126, and 127. For example, one or more processors of the robot can apply and / or generate based on the networks 125, 126, and 127. Figure 1B For example, one or more processors may apply and / or generate during a given iteration (eg, a control loop) Figure 1B data, said given iteration being preceded and / or followed by other iterations that similarly generate and / or apply corresponding data.

[0068] One or more images 161 are applied as input to the grasping CNN 125, such as the current image and optionally additional images (e.g., images that capture the same scene as the current image but omit grasping the end effector). End effector motion vectors 162 are also applied to the grasping CNN 125. In some embodiments, the images 161 are applied to the initial layers of the grasping CNN 125 and the end effector motion vectors 162 are applied to downstream layers of the grasping CNN 125.

[0069] Grasp metrics 177 and STN parameters 178 are generated on the grasp CNN 125 based on the applied image 161 and end effector motion vectors 162. In some embodiments, the STN parameters may be transformation parameters of the STN 126, such as translation coefficients of an affine transformation, optionally with a fixed scale for the cropped region (e.g., a fixed scale of 128.0 / 472.0).

[0070] At least one of the STN parameters 178 and the image 161 (e.g., the current image) is applied as input to the STN 126. The STN 126 is utilized to generate a spatial transform 179 that is a spatial transform of the applied image 161 based on the STN parameters 178. In some embodiments, the spatially transformed image 179 is a crop of a portion of the applied image 161, where the portion is selected based on the approved STN parameters 178. In some embodiments, additional and / or alternative transforms may be generated on the STN 126.

[0071] The spatial transformation 179 is applied as input to the semantic CNN 127 and semantic features 180 are generated on the semantic CNN 127 based on the applied spatially transformed image 179. For example, the semantic features 180 may indicate to which of one or more categories an object in the spatially transformed image 179 belongs, such as the categories of “eraser”, “scissors”, “comb”, “shovel”, “torch”, “toy”, “hairbrush”, and / or other categories of greater or lesser granularity.

[0072] One or more processors of the robot can determine whether to execute the end effector motion vector 162, attempt a grasp, or perform a correction trajectory based on the grasp metrics 177 and the semantic features 180. For example, one or more desired object semantic features can be provided (e.g., via a user interface input) and a grasp can be attempted based on the semantic features 180 indicating those semantic features and based on the grasp metrics 177 and / or grasp metrics of a previous iteration satisfying one or more criteria. Figure 7 Additional description of implementations utilizing a semantic crawling model is provided.

[0073] Figure 3 is a flow chart illustrating an example method 300 of performing a grasp attempt and storing data associated with the grasp attempt. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as a processor and / or a robot control system of the robot 180A, 180B, 825 and / or other robots. Furthermore, while the operations of the method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0074] At box 352, the system begins a grasp attempt. At box 354, the system stores an image of the environment without the end effector being present in the image. For example, the system may move the grasp end effector out of the field of view of the vision sensor (i.e., not obstructing the view of the environment) and capture an image at the instance when the grasp end effector leaves the field of view. The image may then be stored and associated with the grasp attempt.

[0075] At block 356, the system determines and implements end effector movement. For example, the system may generate one or more motion commands to cause one or more of the actuators that control the pose of the end effector to actuate, thereby changing the pose of the end effector.

[0076] In some embodiments and / or iterations of block 356, the motion commands may be random within a given space, such as a workspace accessible by the end effector, a confined space within which the end effector is confined for a grasping attempt, and / or a space defined by the position and / or torque limits of an actuator that controls the pose of the end effector. For example, before initial training of a neural network is completed, the motion commands generated by the system at block 356 to effect movement of the end effector may be random within a given space. Random as used herein may include true random or pseudo-random.

[0077] In some embodiments, the motion commands generated by the system at box 356 to effectuate movement of the end effector may be based at least in part on the current version of the trained grasping CNN and / or on other criteria. In some embodiments, in the first iteration of box 356 for each grasping attempt, the end effector may be "out of position" based on it being moved out of the field of view at box 354. In some of those embodiments, the end effector may be moved "back into position" randomly or otherwise prior to the first iteration of box 356. For example, the end effector may be moved back to a set "starting position" and / or to a randomly selected position within a given space.

[0078] At box 358, the system stores: (1) an image capturing the end effector and environment at the current instance of the grasp attempt and (2) the pose of the end effector at the current instance. For example, the system can store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., using a timestamp). In addition, for example, the system can determine the current pose of the end effector based on data from one or more joint position sensors of the robot's joints (whose positions affect the robot's pose), and the system can store the pose. The system can determine and store the pose of the end effector in task space, joint space, or another space.

[0079] At block 360, the system determines whether the current instance is the final instance for the crawl attempt. In some embodiments, the system may increment an instance counter at blocks 352, 354, 356, or 358 and / or increment a time counter as time passes—and determine whether the current instance is the final instance based on comparing the value of the counter to a threshold. For example, the counter may be a time counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and / or other values. In some embodiments, the threshold may vary between one or more iterations of method 300.

[0080] If the system determines at block 360 that the current instance is not the final instance of the grasp attempt, the system returns to block 356 where it determines and implements another end effector movement, and then proceeds to block 358 where it stores the image and pose at the current instance. Through multiple iterations of blocks 356, 358, and 360 for a given grasp attempt, the pose of the end effector will be modified through multiple iterations of block 356, and the image and pose will be stored at each of those instances. In many embodiments, blocks 356, 358, 360, and / or other blocks may be performed at a relatively high frequency, thereby storing a relatively large amount of data for each grasp attempt.

[0081] If the system determines at block 360 that the current instance is the final instance of the grasping attempt, the system proceeds to block 362 where it actuates the gripper of the end effector. For example, for an impact-type end effector gripper, the system may cause one or more plates, fingers, and / or other components to close. For example, the system may cause the components to close until they are in a fully closed position or a torque reading measured by a torque sensor associated with the component satisfies a threshold.

[0082] At box 364, the system stores additional data and optionally performs one or more additional actions to enable determination of the success of the grasp of box 360 and / or to enable determination of grasped object features of the object grasped at box 360 (if any). In some embodiments, the additional data stored to enable determination of the success of the grasp is a position reading, a torque reading, and / or other readings from the grasping end effector. For example, a position reading greater than a certain threshold (e.g., 1 cm) immediately following box 362 can indicate a successful grasp (e.g., can indicate that the grasping end effector is grasping an object because it can close beyond the threshold). In some embodiments, the additional data stored to enable determination of grasped object features of the object grasped at box 360 is an indication of one or more object features shared by all objects available for grasping during the grasp attempt. For example, only paintbrushes may be available in the robot's workspace during the grasp attempt and object features indicating a classification of "paintbrushes" can be stored. Object features shared by all objects in a grasp attempt may be determined based on operator provided indications and / or sensed indications (eg, machine readable codes indicative of object features provided in the workspace and captured by a vision sensor).

[0083] In some embodiments, at box 364 the system additionally and / or alternatively: (1) maintains the end effector in an actuated (e.g., closed) position and moves the end effector and any object that may be grasped by the end effector to a position proximate to the vision sensor; and (2) captures an image using the vision sensor while the end effector (and any grasped object) is in a position proximate to the vision sensor. In some of those embodiments, the system may only move the end effector and / or capture such an image if it is first determined based on other data that the grasp is likely to be successful (e.g., if the position reading indicates a successful grasp). As described herein, such captured images may then be utilized to determine whether the grasp attempt was successful and / or to determine one or more grasped object features of an object grasped by the end effector. For example, the captured images may be provided to computing devices of one or more human reviewers, who may utilize their computing devices to indicate whether the grasp was successful based on the captured images and / or to indicate one or more grasped object features of an object grasped in the captured images. Additionally, for example, the captured image (or a spatial transformation thereof) may be applied to a trained semantic CNN (optionally separate from the semantic CNN 127) and outputs generated on the trained semantic CNN to determine one or more grasped object features of the grasped object.

[0084] At block 366 , the system resets the counter (eg, instance counter and / or time counter) and proceeds back to block 352 to begin another grab attempt.

[0085] In some embodiments, Figure 3 The method 300 is implemented on each of a plurality of robots - the plurality of robots optionally operating in parallel during one or more (e.g., all) of their respective iterations of the method 300. This can enable more grasp attempts to be achieved in a given time period compared to if only one robot were operating the method 300. Furthermore, in embodiments where one or more of the plurality of robots includes an associated visual sensor having a pose with respect to the robot that is unique to the poses of one or more visual sensors associated with other robots, training samples generated based on grasp attempts from the plurality of robots can provide robustness to the visual sensor poses in a neural network trained based on those training samples. In addition, in embodiments where the grasping end effectors and / or other hardware components of the multiple robots vary and / or wear differently, and / or where different robots (e.g., the same brand and / or model and / or different brands and / or models) interact with different objects (e.g., objects of different sizes, different weights, different shapes, different transparencies, different materials) and / or in different environments (e.g., different surfaces, different lighting, different environmental obstacles), training samples generated based on grasping attempts from multiple robots can provide robustness to individual robot and / or environmental configurations.

[0086] In some embodiments, the objects that can be reached by a given robot and for which a grasp attempt can be made can be different during different iterations of method 300. For example, a human operator and / or another robot can add and / or remove objects to the robot's workspace between one or more grasp attempts by the robot. In addition, for example, the robot itself can discard one or more objects from its workspace following a successful grasp of those objects. This can increase the diversity of the training data. In some embodiments, environmental factors such as lighting, surfaces, obstacles, etc. can additionally and / or alternatively be different during different iterations of method 300, which can also increase the diversity of the training data.

[0087] Figure 44 is a flowchart illustrating an example method 400 for generating training samples based on data associated with a grasping attempt of a robot. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include one or more components of a robot and / or another computer system, such as a processor of a robot 180A, 180B, 825 and / or a robot control system, and / or a processor of a training sample generation system 110 and / or other systems that may optionally be implemented separately from the robot. In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0088] At block 452, the system begins training sample generation. At block 454, the system selects a crawl attempt. For example, the system can access a database including data associated with a plurality of stored crawl attempts and select one of the stored crawl attempts. The selected crawl attempt can be, for example, based on Figure 3 The crawl attempt generated by method 300.

[0089] At block 456, the system determines a grasp success label and optionally a grasped object label for the selected grasp attempt based on the stored data for the selected grasp attempt. For example, as described with respect to block 364 of method 300, additional data may be stored for the grasp attempt to enable determination of a grasp success label and / or a grasped object label for the grasp attempt. The stored data may include data from one or more sensors, where the data is generated during and / or after the grasp attempt.

[0090] As an example, the data may be position readings, torque readings, and / or other readings from a grasping end effector. In such an example, the system may determine a grasp success label based on the readings. For example, where the readings are position readings, the system may determine a "successful grasp" label if the readings are greater than a certain threshold (e.g., 1 cm)—and an "unsuccessful grasp" label if the readings are less than a certain threshold (e.g., 1 cm).

[0091] As another example, the data may be an image of an end effector grasping an object grasped in a grasp attempt. For example, the image may be captured by a vision sensor after the end effector and any object that may be grasped by the end effector are moved to a position close to the vision sensor. In such an example, the system may utilize the captured image to determine whether the grasp attempt was successful and / or determine one or more grasped object features of the object grasped by the end effector. For example, the system may provide the captured image to a computing device of one or more human reviewers, who may utilize their computing devices to indicate whether the grasp was successful and / or indicate one or more grasped object features of the object grasped in the captured image based on the captured image. As another example, the system may apply the captured image to a trained semantic CNN, generate an output on the trained semantic CNN, and utilize the output to determine one or more grasped object features of the grasped object.

[0092] As an example of a grasped object label that may be determined, the grasped object label may indicate whether the grasped object has each of one or more classifications in a set of classifications. For example, the grasped object label may indicate that the grasped object has the "pen" classification in the set, but does not have any other classifications in the set.

[0093] In some iterations of box 456, the system only determines the grasping success label of the grasping attempt. In some other iterations of box 456, the system additionally or alternatively determines the grasped object label of the grasping attempt. Whether the system determines the grasping success label and / or the grasped object label of the grasping attempt in a given iteration can be based on various factors. For example, in some iterations of box 456, if the grasping of the grasping attempt is unsuccessful, only the grasping success label is determined. In addition, for example, in some iterations of box 456, it may be desirable to generate training samples for training only the grasping CNN and only determine the grasping success label. In addition, for example, in some iterations of box 456, it may be desirable to generate training samples for training both the grasping CNN and the semantic CNN and determine both the grasping success label and the grasped object label. In addition, for example, in some cases, computational and / or other costs can be considered when determining whether it is desired to generate grasped object labels for training samples in a given iteration.

[0094] At block 458, the system selects an instance for the scrape attempt. For example, the system can select data associated with the instance based on a timestamp and / or other boundaries associated with the data that distinguish the instance from other instances of the scrape attempt.

[0095] At box 460, the system generates an end effector motion vector for the instance based on the pose of the end effector at the instance and the pose of the end effector at the final instance of the grasp attempt. For example, the system can determine a transformation between the current pose and the final pose of the grasp attempt and use the transformation as the end effector motion vector. The current pose and the end effector motion vector from the current pose of the grasp attempt to the final pose can be represented in task space, in joint space, or in another space. For example, the end effector motion vector can be represented by five values ​​in task space: three values ​​define a three-dimensional (3D) translation vector, and two values ​​represent a sine-cosine encoding of the change in orientation of the end effector around an axis of the end effector.

[0096] At box 462, the system generates a training sample for the instance, the training sample including: (1) the stored image for the instance, (2) the end effector motion vector generated for the instance at box 460, and (3) the grasp success label and / or grasped object label determined at box 456. In some embodiments, the system generates a training sample that also includes additional images stored for the grasp attempt (such as additional images that at least partially omit the end effector and are captured before the grasp attempt). In some of those embodiments, the system concatenates the stored image for the instance and the additional images stored for the grasp attempt to generate a cascaded image of the training sample. The cascaded image includes both the stored image for the instance and the additional images stored. For example, where both images include X times Y pixels and three channels (e.g., red, blue, green), the cascaded image may include X times Y pixels and six channels (three for each image). As described herein, the current image, additional images, and a vector from the current pose to the final pose of a grasp attempt of a training sample may be used as training sample inputs; and a grasp success label and / or a grasped object label may be used as training sample outputs.

[0097] In some embodiments, the system may optionally process the image at block 462. For example, the system may optionally resize the image to fit a defined size of an input layer of one or more networks, remove one or more channels from the image, and / or normalize the values ​​of a depth channel (in embodiments where the image includes a depth channel).

[0098] At block 464, the system determines whether the selected instance is the final instance of the crawl attempt. If the system determines that the selected instance is not the final instance of the crawl attempt, the system returns to block 458 and selects another instance.

[0099] If the system determines that the selected instance is the final instance of the grasp attempt, the system proceeds to block 466 and determines whether there are additional grasp attempts to process. If the system determines that there are additional grasp attempts to process, the system returns to block 454 and selects another grasp attempt. In some embodiments, determining whether there are additional grasp attempts to process may include determining whether there are any remaining unprocessed grasp attempts. In some embodiments, determining whether there are additional grasp attempts to process may additionally and / or alternatively include determining whether a threshold number of training samples have been generated and / or whether other criteria have been met.

[0100] If the system determines that there are no additional grab attempts to process, the system proceeds to block 466 and the method 400 ends. Another iteration of the method 400 may be performed again. For example, the method 400 may be performed again in response to at least a threshold number of additional grab attempts being performed.

[0101] Although method 300 and method 400 are illustrated in separate figures for clarity herein, it should be understood that one or more blocks of method 400 may be performed by the same component that performs one or more blocks of method 300. For example, one or more (e.g., all) of the blocks of method 300 and method 400 may be performed by a processor of a robot. Additionally, it should be understood that one or more blocks of method 400 may be performed in conjunction with one or more blocks of method 300 or before or after one or more blocks of method 300.

[0102] Figure 5 1 is a flowchart illustrating an example method 500 for training one or more networks of a semantic crawling model based on training samples. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include one or more components of a computer system, such as a training engine 120 operating on networks 125, 126, and 127 of a semantic crawling model 124 and / or a processor (e.g., a GPU) of another computer system. In addition, although the operations of method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0103] At block 552, the system begins training. At block 554, the system selects training samples. For example, the system may select a training sample based on Figure 4 The training samples generated by method 400.

[0104] At box 556, the system applies the image of the instance of the training sample and the additional image of the selected training sample to the initial layer of the grasping CNN. For example, the system can apply the image to the initial convolutional layer of the grasping CNN. As described herein, the additional image can at least partially omit the end effector. In some embodiments, the system cascades the image and the additional image and applies the cascaded image to the initial layer. In some other embodiments, the image and the additional image have been cascaded in the training sample.

[0105] At box 558, the system applies the end effector motion vector of the selected training sample to an additional layer of the grasped CNN. For example, at box 556 the system may apply the end effector motion vector to an additional layer of the grasped CNN downstream of the initial layer to which the image is applied. In some embodiments, to apply the end effector motion vector to the additional layer, the system passes the end effector motion vector through a fully connected layer to generate an end effector motion vector output, and concatenates the end effector motion vector output with the output from the immediately upstream layer of the grasped CNN. The immediately upstream layer is immediately upstream of the additional layer to which the end effector motion vector is applied and may optionally be one or more layers downstream of the initial layer to which the image is applied at box 556. In some embodiments, the initial layer is a convolutional layer, the immediately upstream layer is a pooling layer, and the additional layer is a convolutional layer.

[0106] At block 560, the system performs back propagation on one or more layers of the grasping CNN based on the grasping success labels of the training samples. For example, the system can generate a predicted measure of successful grasping on the grasping CNN based on the application of block 558, determine an error based on a comparison of the predicted measure of successful grasping with the grasping success labels of the training samples, and back propagate the error through one or more layers of the grasping CNN.

[0107] Blocks 562, 564, and 566 are now described. Figure 5 564, 566). For example, some of the training samples selected at block 554 may not include the grasped object labels, and as a result, training of the grasped CNN may be performed based on those training samples without performing blocks 562, 564, and 566 based on those training samples.

[0108] At block 562, the system applies the image and STN parameters of the training sample to the STN to generate a spatial transformation of the image. The STN parameters are generated by the system on the grasping CNN based on the application of block 558. For example, based on the application of block 558, both a prediction measure of successful grasping and the STN parameters can be generated.

[0109] At box 564, the system applies the spatial transformation generated at box 562 to the semantic CNN.

[0110] At box 566, the system performs back propagation of the semantic CNN and / or STN based on the captured object labels of the training samples. For example, the system can generate predicted semantic object features of objects present in the spatially transformed image on the semantic CNN based on the application of box 564, determine errors based on comparison of the predicted semantic object features and the captured object labels of the training samples, and back propagate the errors through one or more layers of the semantic CNN. In some embodiments, the errors can be further back propagated through the STN parameter layer of the captured CNN (but optionally not through any other layer of the captured CNN). The back propagation of the STN and / or STN parameter layer can enable the STN parameters and / or STN to be adapted to enable the spatially transformed image to be generated for the location / region to be captured over multiple iterations during training.

[0111] At box 568, the system determines whether additional training samples exist. If the system determines that additional training samples exist, the system returns to box 554 and selects another training sample. In some embodiments, determining whether additional training samples exist may include determining whether there are any remaining training samples that have not been utilized to train the one or more networks of the semantic crawling model. In some embodiments, determining whether additional training samples exist may additionally and / or alternatively include determining whether a threshold number of training samples have been utilized and / or whether other criteria have been met.

[0112] If the system determines that there are no additional training samples and / or some other criteria has been met, the system proceeds to block 570 or block 572 .

[0113] At block 572, the training of the semantic grasping model may be completed. The semantic grasping model may then be provided for use by one or more robots in servo-grasping end effectors to achieve successful grasping of an object by a grasping end effector. For example, a robot may perform Figure 7 The method 700 utilizes a semantic crawling model.

[0114] At block 572, the system may additionally and / or alternatively provide the trained grasping CNN to generate additional training samples based on the trained grasping CNN. For example, one or more robots may utilize the trained grasping CNN when performing grasping attempts and data from those grasping attempts may be utilized to generate additional training samples. For example, one or more robots may utilize the trained grasping CNN when performing grasping attempts and data from those grasping attempts may be utilized to generate additional training samples. Figure 7 Some blocks of method 700 utilize a trained grasp CNN when performing grasp attempts and data from those grasp attempts is utilized to perform grasp attempts based on Figure 4 The method 400 generates additional training samples. The robot whose data is utilized to generate the additional training samples may be a robot in a laboratory / training setting and / or a robot actually used by one or more consumers.

[0115] At block 574, the system may perform further training of one or more networks of the semantic crawling model based on additional training samples generated in response to providing the trained crawling CNN at block 572. For example, the system may perform additional iterations of blocks 554, 556, 558, and 560 (and optionally blocks 562, 564, and 566) based on the additional training samples.

[0116] As indicated by the arrow extending from block 574 to block 572, the updated grasping CNN may be provided again at block 572 to generate additional training samples and those training samples utilized at block 574 to further train one or more networks of the semantic grasping model. In some implementations, grasping attempts performed in association with future iterations of block 572 may be grasping attempts that are longer in time than those grasping attempts performed in earlier iterations and / or grasping attempts performed without utilizing the trained grasping CNN. For example, grasping attempts performed without utilizing the trained grasping CNN Figure 3 Implementations of method 300 may have grasp attempts that are shortest in time, those performed using the initially trained grasp CNN may have grasp attempts that are longer in time, those performed using the next iteration of the trained grasp CNN may have grasp attempts that are further longer in time, etc. This may optionally be implemented via an optional instance counter and / or time counter of method 300.

[0117] Fig. 6A and Figure 6B An example architecture of a crawling CNN 600 is illustrated in accordance with various embodiments. Fig. 6A and Figure 6B The capture of CNN600 can be based on Figure 5 The method uses 500 trained crawling CNN examples. Fig. 6A and Figure 6B The crawling CNN 600 can still be based on Figure 7 An example of a grasping CNN utilized when servo grasping an end effector using method 700. Generally, a convolutional neural network is a multi-layer learning framework that includes an input layer, one or more convolutional layers, optional weights and / or other layers, and an output layer. During training, the convolutional neural network is trained to learn a hierarchy of feature representations. The convolutional layers of the network are convolved with filters and optionally downsampled by pooling layers. Generally, the pooling layers aggregate values ​​in smaller areas by one or more downsampling functions such as maximum, minimum, and / or normalized sampling.

[0118] The crawling CNN 600 includes an initial input layer 663 that is a convolutional layer. In some embodiments, the initial input layer 663 is a 6×6 convolutional layer with a stride of 2 and 64 filters. Fig. 6A Also illustrated in FIG. 6 is an image 661A with an end effector and an image 661B without an end effector. Images 661A and 661B are further illustrated as being cascaded (indicated by a merged line extending from each image) and the cascaded image is fed to an initial input layer 663. In some embodiments, images 661A and 661B may each be 472 pixels by 472 pixels by 3 channels (e.g., 3 channels may be selected from a depth channel, a first color channel, a second color channel, and a third color channel). Thus, the cascaded image may be 472 pixels by 472 pixels by 6 channels. Other sizes may be used, such as different pixel sizes or more or fewer channels. Images 661A and 661B are convolved to the initial input layer 663. The weights of the features of the initial input layer and other layers of the grasping CNN 600 are learned during training the grasping CNN 600 based on a plurality of training samples.

[0119] The initial input layer 663 is followed by a max pooling layer 664. In some embodiments, the max pooling layer 664 is a 3×3 max pooling layer with 64 filters. The max pooling layer 664 is followed by six convolutional layers, two of which are in Fig. 6A denoted by 665 and 666. In some embodiments, each of the six convolutional layers is a 5×5 convolutional layer with 64 filters. Convolutional layer 666 is followed by a max pooling layer 667. In some embodiments, max pooling layer 667 is a 3×3 max pooling layer with 64 filters.

[0120] exist Fig. 6A The end effector motion vector 662 is also shown in FIG. The end effector motion vector 662 is concatenated with the output of the maximum pooling layer 667 (e.g., by Fig. 6A ) and the concatenated output is applied to the convolutional layer 670 ( Figure 6B). In some embodiments, concatenating the end effector motion vector 662 with the output of the max pooling layer 667 includes processing the end effector motion vector 662 by a fully connected layer 668, the output of which is then added point-wise to each point in the response map of the max pooling layer 667 by tiling the output in the spatial dimensions via the tiling vector 669. In other words, the end effector motion vector 662 is passed through the fully connected layer 668 and replicated in the spatial dimensions of the response map of the max pooling layer 667 via the tiling vector 669.

[0121] Now turn to Figure 6B , the concatenation of the end effector motion vector 662 and the output of the max pooling layer 667 is provided to a convolutional layer 670, which is followed by five more convolutional layers (the last of which is shown in FIG. Figure 6B In some embodiments, convolutional layers 670 and 671 and the four middle convolutional layers are each 3×3 convolutional layers with 64 filters.

[0122] The convolutional layer 671 is followed by a max pooling layer 672. In some embodiments, the max pooling layer 672 is a 2×2 max pooling layer with 64 filters. The max pooling layer 672 is followed by three convolutional layers, two of which are in Fig. 6A It is represented by 673 and 674.

[0123] The final convolutional layer 674 of the grasping CNN 600 is fully connected to a first fully connected layer 675, which in turn is fully connected to a second fully connected layer 676. The fully connected layers 675 and 676 can be vectors, such as a vector of size 64. The output of the second fully connected layer 676 is utilized to generate a measure of successful grasping 677. For example, a sigmoid can be utilized to generate and output the measure 677.

[0124] The output of the second fully connected layer 676 is also utilized to generate STN parameters 678. For example, sigmoid may be utilized to generate and output STN parameters 678. In some embodiments, separate fully connected layers may be fully connected to the final convolutional layer 674 and those fully connected layers may be STN layers utilized to generate STN parameters 678 (while separate fully connected layers 675 and 676 are separately utilized to generate a measure of successful grasping 677).

[0125] Figure 6CThe spatial transformer network 126 and the semantic convolutional neural network 127 are illustrated, example inputs that can be applied to these networks, and example outputs that can be generated on these networks. Although not explicitly illustrated, it should be understood that in many embodiments the STN 126 and the semantic CNN 127 will each have multiple layers, such as convolutional layers, pooling layers, and / or other layers.

[0126] exist Figure 6C In, based on Fig. 6A and Figure 6B The STN parameters 678 generated by the grasping CNN 600 are applied as input to the STN 126. In addition, the image 661A with the end effector is also applied as input to the STN 126. The spatial transformation 679 is generated on the STN 126 based on the applied STN parameters 678 and the image 661A.

[0127] The spatial transformation 679 is applied as an input to the semantic CNN 127. The semantic features 680 are generated based on the output generated on the semantic CNN 127 based on the application of the transformed image 679. For example, the output of the semantic CNN 127 may be a vector having N values, where each value indicates whether an object of the transformed image 679 has a classification corresponding to the value.

[0128] In some embodiments of training the crawling CNN 600, STN 126, and / or semantic CNN 127, various values ​​of epochs, learning rates, weight decay, dropout probabilities, and / or other parameters may be utilized. In some embodiments, one or more GPUs may be used to train and / or utilize the crawling CNN 600, STN 126, and / or semantic CNN 127. Fig. 6A , Figure 6B and Figure 6C The specific architecture and arrangement of the crawling CNN 600, STN 126, and semantic CNN 127 are illustrated in FIG, however variations are possible. For example, more or fewer convolutional layers may be provided in the crawling CNN 600, one or more layers of the crawling CNN 600 may have a different size than those provided as examples, etc.

[0129] Once a semantic grasping model is trained according to the techniques described in this paper, it can be used to servo grasp an end-effector. Figure 7, a flowchart illustrating an example method 700 of using a semantic grasping model to servo a grasping end effector is shown. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include one or more components of a robot, such as processors (e.g., CPUs and / or GPUs) and / or a robot control system of robots 180A, 180B, 825, and / or other robots. When implementing one or more blocks of method 700, the system may operate on a trained semantic grasping model, which may be stored locally at the robot, for example, and / or may be stored remotely from the robot. Additionally, although the operations of method 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0130] At block 752, the system generates a candidate end effector motion vector. Depending on the input parameters of the trained CNN to be utilized in additional blocks, the candidate end effector motion vector may be defined in task space, joint space, or other spaces.

[0131] In some embodiments, the system generates a candidate end effector motion vector that is random within a given space, such as a workspace reachable by the end effector, a restricted space within which the end effector is confined for a grasping attempt, and / or a space defined by the position and / or torque limits of the actuators that control the pose of the end effector.

[0132] In some embodiments, the system may utilize one or more techniques to sample a set of candidate end effector motion vectors and select a subgroup from the sampled set. For example, the system may utilize an optimization technique such as the cross-entropy method (CEM). CEM is a derivative-free optimization algorithm that samples a batch of N values at each iteration, fits a Gaussian distribution to M < N of these samples, and then samples a new batch of N values according to this Gaussian. For example, the system may utilize CEM with M = 64 and N = 6, and perform three iterations of CEM to determine the best available (according to CEM) candidate end effector motion vector.

[0133] In some implementations, one or more constraints may be imposed on the candidate end effector motion vectors that may be generated at block 752. For example, the candidate end effector motions evaluated by CEM or other techniques may be constrained based on constraints. One example of a constraint is a computer-generated and / or human-input constraint (e.g., via a user interface input device of a computer system) that imposes constraints on an area that may be attempted to grasp, constraints on specific object features that may be attempted to grasp, and the like. Yet other examples of constraints include, for example, constraints based on a robot's workspace, joint limits of a robot, torque limits of a robot, constraints provided by a collision avoidance system and that limit the robot's motion to prevent collisions with one or more objects, and the like.

[0134] At block 754, the system identifies a current image that captures the end effector and one or more environmental objects. In some embodiments, the system also identifies additional images that at least partially omit the end effector, such as additional images of environmental objects captured by the vision sensor when the end effector is at least partially out of view of the vision sensor. In some embodiments, the system cascades the image and the additional images to generate a cascaded image. In some embodiments, the system optionally performs processing of the image and / or the cascaded image (e.g., to resize to an input to a grabbing CNN).

[0135] At box 756, the system applies the current image and the candidate end effector motion vector to the trained grasping CNN. For example, the system can apply a cascaded image including the current image and the additional image to the initial layer of the trained grasping CNN. The system can also apply the candidate end effector motion vector to the additional layer of the trained grasping CNN downstream of the initial layer. In some embodiments, when applying the candidate end effector motion vector to the additional layer, the system passes the end effector motion vector through a fully connected layer of the grasping CNN to generate an end effector motion vector output and cascades the end effector motion vector output with the upstream output of the grasping CNN. The upstream output is from the immediately upstream layer of the grasping CNN, which is immediately upstream of the additional layer and downstream of the initial layer and downstream of one or more intermediate layers of the grasping CNN.

[0136] At box 758, the system generates metrics and STN parameters for successful grasps on the trained grasping CNN. The metrics and STN parameters are generated based on applying the current image (and optionally additional images) and the candidate end effector motion vectors to the trained grasping CNN at box 756 and determining the metrics and STN parameters based on the learned weights of the trained grasping CNN.

[0137] At block 760 , the system generates a spatial transform of the current image of block 754 based on the STN parameters of block 758 on the STN.

[0138] At box 762, the system generates semantic features based on the spatial transformation of box 760 on the semantic CNN.

[0139] At box 764, the system generates an end effector command based on the measure of successful grasping of box 758 and the semantic features of box 762. Generally, at box 764, the system generates an end effector command that attempts to achieve (through one or more iterations of method 700) a successful grasp of an object having desired object semantic features. The desired object semantic features can be based on, for example, user interface inputs provided to the system and / or inputs from one or more separate components. As an example, if the measure of successful grasping of box 758 meets a threshold indicating a successful grasp and the semantic features of box 762 indicate desired semantic object features, the end effector command can generate an end effector command corresponding to the end effector motion vector of box 752 (and then optionally perform a grasp).

[0140] In some embodiments, multiple candidate end effector motion vectors are considered in a given iteration of method 700 and a measure and semantic features of a successful grasp are determined for each of the candidate end effector motion vectors (all based on the same current image). In some of those embodiments, at block 764 the system determines the end effector motion vector having the determined corresponding measure of successful grasp that indicates success to the highest degree and having the determined corresponding semantic features that conform to the desired semantic object features, and generates an end effector command corresponding to the end effector motion vector.

[0141] As an example of those embodiments, the system may generate one or more additional candidate end effector motion vectors at box 752, and generate: a measure of successful grasping for those additional candidate end effector motion vectors at an additional iteration of box 758 (based on applying the current image and the additional candidate end effector motion vectors to the grasping CNN); and a semantic feature for those additional candidate end effector motion vectors at an additional iteration of box 762. The additional iterations of boxes 758 and 762 may optionally be performed in parallel by the system. In some of those embodiments, the system may generate an end effector command at box 764 based on an analysis of all generated measures of successful grasping and corresponding semantic features. For example, the system may generate an end effector command to fully or substantially conform to a candidate end effector motion vector having: a measure of successful grasping that indicates a successful grasp to the highest degree; and a corresponding semantic feature that matches a desired semantic feature of the object. For example, a control system of a robot of the system may generate a motion command to actuate one or more actuators of the robot to move the end effector based on the end effector motion vector.

[0142] In some iterations of block 764, if a new motion command is not generated using a candidate end effector motion vector, the system additionally and / or alternatively generates an end effector command based on the current measure of successful grasps and the current semantic features. For example, if one or more comparisons of the current measure of successful grasps to the measure of successful grasps determined at block 758 fail to meet a threshold, and the current semantic features indicate the desired semantic features of the object, the end effector motion command may be a "grasp command" that causes the end effector to attempt a grasp (e.g., close the fingers of an impact grasp end effector). For example, if the result of the current measure of successful grasps divided by the measure of successful grasps determined at block 758 for the candidate end effector motion vector that most highly indicates a successful grasp is greater than or equal to a first threshold (e.g., 0.9), the end effector command may be a grasp command (under the rationale of stopping grasping early if closing the gripper is as likely to produce a successful grasp as moving it). Additionally, for example, if the result is less than or equal to a second threshold (e.g., 0.5), the end effector command may be a motion command to effectuate a trajectory correction (e.g., lift the grasping end effector "up" at least X meters) (under the rationale that the grasping end effector is most likely not positioned in a good configuration and a relatively large motion is required). Additionally, for example, if the result is between the first threshold and the second threshold, a motion command may be generated that substantially or completely conforms to the candidate end effector motion vector having a measure of successful grasping determined at block 758 that is indicative of successful grasping to the greatest extent and also includes corresponding semantic features corresponding to desired semantic features of the object. The end effector command generated by the system may be a single group of one or more commands or a sequence of one or more groups of commands.

[0143] In the absence of utilizing a candidate end effector motion vector to generate a new motion command, a metric of a successful grasp can be based on a metric of a candidate end effector motion vector utilized in a previous iteration of method 700 and / or based on applying a “null” motion vector and the current image (and optionally additional images) to a trained grasping CNN at an additional iteration of box 758 and generating the metric based on an additional iteration of box 758.

[0144] At block 762, the system determines whether the end effector command is a grasp command. If the system determines at block 762 that the end effector command is a grasp command, the system proceeds to block 764 and implements the grasp command. In some implementations, the system may optionally determine whether the grasp command resulted in a successful grasp (e.g., using the techniques described herein), and if unsuccessful, the system may optionally adjust the pose of the end effector and return to block 752. Even in the event that the grasp is successful, the system may return to block 752 at a later time to grasp another object.

[0145] If the system determines at block 762 that the end effector command is not a grasp command (e.g., it is a motion command), the system proceeds to block 766 and implements the end effector command, and then returns to block 752, where it generates another candidate end effector motion vector. For example, at block 766 the system may implement an end effector motion command that substantially or completely conforms to the end effector motion vector having the grasp metric that indicates a successful grasp to the highest degree and also includes semantic features that conform to desired semantic features of the object.

[0146] In many embodiments, blocks of method 700 may be executed at a relatively high frequency, thereby enabling iterative updating of end effector commands and enabling servoing of the end effector along a trajectory informed by a trained CNN to result in a relatively high probability of successful grasping of an object having desired object features.

[0147] Figure 8 An example architecture of a robot 825 is schematically depicted. The robot 825 includes a robot control system 860, one or more operating components 825a-825n, and one or more sensors 842a-842m. The sensors 842a-842m may include, for example, vision sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, etc. Although the sensors 842a-m are depicted as being integral to the robot 825, this is not intended to be limiting. In some embodiments, the sensors 842a-m may be located external to the robot 825, for example, as a stand-alone unit.

[0148] The operating components 825a-825n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to achieve movement of one or more components of the robot. For example, the robot 825 may have multiple degrees of freedom and each of these actuators may control the actuation of the robot 825 in one or more degrees of freedom in response to a control command. As used herein, the term actuator includes a mechanical or electrical device (e.g., a motor) that produces motion, in addition to any driver that may be associated with the actuator and converts a received control command into one or more signals for driving the actuator. Therefore, providing a control command to an actuator may include providing the control command to a driver that converts the control command into an appropriate signal for driving an electrical or mechanical device to produce the desired motion.

[0149] The robotic control system 860 may be implemented with one or more processors, such as a CPU, GPU, and / or other controller of the robot 825. In some embodiments, the robot 825 may include a "brain box" that may contain all or multiple aspects of the control system 860. For example, the brain box may provide real-time bursts of data to the operating components 825a-n, where each of the real-time bursts includes a set of one or more control commands that, among other things, specify motion parameters (if any) for each of the one or more operating components 825a-n. In some embodiments, the robotic control system 860 may perform one or more aspects of the methods 300, 400, 500, and / or 700 described herein.

[0150] As described herein, in some embodiments all or multiple aspects of the control commands generated by the control system 860 when positioning the end effector to grasp an object can be based on end effector commands generated based on the use of a semantic grasping model. For example, the visual sensor of the sensors 842a-m can capture the current image and the additional image, and the robot control system 860 can generate candidate motion vectors. The robot control system 860 can provide the current image, the additional image, and the candidate motion vectors to the trained semantic grasping model and utilize the measure of successful grasping and the semantic features generated based on the application to generate one or more end effector control commands for controlling the movement and / or grasping of the robot's end effector. Although the control system 860 is in Figure 8 825, however, in some embodiments, all or multiple aspects of the control system 860 may be implemented in a component that is separate from the robot 825 but in communication with the robot 825. For example, all or multiple aspects of the control system 860 may be implemented on one or more computing devices (such as computing device 910) that communicate with the robot 825 in wired and / or wireless communication.

[0151] Fig. 9is a block diagram of an example computing device 910 that can be optionally utilized to perform one or more aspects of the techniques described herein. For example, in some embodiments, the computing device 910 can be utilized to provide desired semantic features of objects for grasping by the robot 825 and / or other robots. The computing device 910 typically includes at least one processor 914 that communicates with multiple peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 - including, for example, a memory subsystem 925 and a file storage subsystem 926, a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow a user to interact with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0152] The user interface input devices 922 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 910 or onto a communication network.

[0153] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also, for example, provide a non-visual display via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and information is output from computing device 910 to a user or to the mode of another machine or computing device.

[0154] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include an executable Figure 3 , Figure 4 , Figure 5 and / or Figure 7 The logic of the selected aspect of the method.

[0155] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used in the storage subsystem 924 may include multiple memories, including a main random access memory (RAM) 930 for storing instructions and data during program execution and a read-only memory (ROM) 932 storing fixed instructions. The file storage subsystem 926 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules that implement the functionality of certain embodiments can be stored in the storage subsystem 924 by the file storage subsystem 926, or stored in other machines that can be accessed by the processor 914.

[0156] The bus subsystem 912 provides a mechanism for the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0157] Computing device 910 may be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Fig. 9 The description of computing device 910 depicted in FIG. 1 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 910 may have more Fig. 9 The computing devices depicted may have greater or fewer components.

[0158] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which the teachings are used. Those skilled in the art will recognize or be able to use routine experimental scopes to ascertain many equivalents of the specific embodiments described herein. It should therefore be understood that the above embodiments are presented only as examples and within the scope of the appended claims and their equivalents, embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, product, material, kit, and / or method described herein. In addition, if such features, systems, products, materials, kits, and / or methods do not contradict each other, any combination of two or more such features, systems, products, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method for robot grasping implemented by one or more processors, comprising: Identify desired object semantic features of grasp attempts; generating a candidate end-effector motion vector that defines a motion to move a grasping end-effector of the robot from a current pose to an additional pose; identifying a current image captured by a vision sensor associated with the robot, the current image capturing the grasping end effector and an object in the robot's environment; applying the current image and the candidate end-effector motion vector as input to a trained semantic grasping model; Based on processing the current image and the candidate end-effector motion vector, the trained semantic grasping model is used to generate: a measure of successful grasping of said object with application of said motion; an additional measure indicating whether the object has the desired object semantic feature; generating a grasp command based on determining that the measure of successful grasping satisfies one or more criteria and the additional measure indicates that the object has the desired object semantic characteristic; as well as The grasp command is provided to one or more actuators of the robot to cause the grasp end effector to attempt grasping of the object.

2. The method according to claim 1, wherein: The desired object semantic features define an object class.

3. The method according to claim 1, further comprising: receiving user interface input from a user interface input device; Wherein identifying the desired object semantic feature is based on the user interface input.

4. A robot comprising: a vision sensor that views the robot's environment; End effector; an actuator, the actuator controlling the posture of the end effector; user interface input devices; one or more deep neural networks stored in one or more non-transitory computer-readable media; at least one processor configured to: receiving user interface input from a user via the user interface input device; identifying desired object semantic features of a grasp attempt based on the user interface input; generating a candidate end-effector motion vector that defines a motion to move a grasping end-effector of the robot from a current pose to an additional pose; identifying a current image captured by the vision sensor, the current image capturing an object in the robot's environment; generating an output based on processing the candidate end effector and the current image using the one or more deep neural networks; Based on the output, determine: The object has the desired object semantic feature indicated by the user interface input; as well as a measure of successful grasping of the object while applying the motion satisfies one or more criteria; In response to determining that the object has the desired object semantic feature and the measure of successful grasping satisfies the one or more criteria: An end effector command based on the candidate end effector motion vector is provided to cause the one or more actuators to adjust a pose of the end effector.

5. The robot according to claim 4, wherein: The user interface input device comprises a microphone, and wherein the user interface input is a verbal input.

6. The robot according to claim 4, wherein: The desired object semantic features define an object class.

7. The robot according to claim 4, wherein: The end effector command is a grasp command.

8. The robot according to claim 4, wherein: In the process, the image is applied to a first portion of the one or more deep neural networks and the candidate end effector motion vectors are applied to a separate portion of the one or more deep neural networks.

9. The robot according to claim 4, wherein: The end effector is a retractable end effector.

10. The robot according to claim 4, wherein: The end effector is an impact type end effector.

Images