Pixel-wise prediction for grasp generation
A neural network-based system generates optimal grasp poses from depth image data, addressing the reliance on complete 3D models and enhancing robot grasping stability and efficiency.
Patent Information
- Application Number
- JP2023533714
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-12-02
- Filing Date
- 2021-12-01
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-12-01
AI Technical Summary
Existing robot grasping technologies rely heavily on complete 3D models of objects, which are not always available, limiting their effectiveness in real-world scenarios, and often lack robust methods for determining stable and collision-free grasps.
A data processing device using a neural network to generate pixel-wise predictions from depth image data, aggregating these predictions to determine optimal grasp poses, incorporating additional heuristic data and reinforcement learning for improved grasp stability and efficiency.
Enables robust and efficient grasp generation in real-world scenarios without complete 3D models, enhancing robot grasping performance by improving stability and reducing collisions.
Smart Images

Figure 0007581520000001 
Figure 0007581520000002 
Figure 0007581520000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to robotic control systems, and more particularly, to systems and methods for use in generating grasps. [Background technology]
[0002] Robotic automation is a field that can improve productivity, safety, and economy when performing tasks. For robots that manipulate objects, generating a grasp pose is often a critical component of the task. For example, a robot may observe an object and determine where to position the gripper (according to its position and orientation, e.g., "pose") so that the robot can pick up the object. The stability of an individual grasp may depend on the object and gripper geometry, object mass distribution, as well as surface friction, among other factors. The geometry around the object can impose additional constraints by limiting the grasp points that can be reached without causing the robot manipulator to collide with other objects in the scene. In some cases, the problem is approached by geometrically inspired heuristics to select promising grasp points around the object, sometimes followed by a more detailed geometric analysis of the stability and reachability of the sampled grasps. However, many of these approaches rely on the availability of a complete 3D model of the object to be grasped, which can be a severe limitation in realistic picking scenarios. Thus, improved methods of grasp determination are needed. Summary of the Invention
[0003] A data processing device for grasp generation is provided and is configured to obtain image data comprising depth data representative of an image of an object captured by a camera, provide the image data to a grasp model comprising a neural network trained to predict a plurality of outcomes associated with a grasp operation independently based on the image of the graspable object, obtain a plurality of pixel-wise predictions corresponding to the plurality of outcomes, each pixel-wise prediction representing a pixel-wise probability value corresponding to a predetermined outcome among the plurality of outcomes associated with the grasp operation, aggregate the plurality of pixel-wise predictions to obtain an aggregated pixel-wise prediction, and output the aggregated pixel-wise prediction for selection therefrom of one or more pixels that are the basis for generating one or more grasp poses for grasping the object.
[0004] A computer-implemented method is further provided, comprising: acquiring image data representing an image of an object captured by a camera, the image data including depth data; providing the image data to a grasping model including a neural network trained to predict a plurality of outcomes associated with a grasping action independently based on the image of the graspable object; acquiring a plurality of pixel-wise predictions corresponding to the plurality of outcomes, each pixel-wise prediction representing a pixel-wise probability value corresponding to a predetermined outcome among the plurality of outcomes associated with the grasping action; aggregating the plurality of pixel-wise predictions to obtain an aggregated pixel-wise prediction; and outputting the aggregated pixel-wise prediction for selection therefrom of one or more pixels that are the basis for generating one or more grasp poses for grasping the object.
[0005] Also provided is a computer system comprising one or more processors and a computer readable memory storing executable instructions which, when executed by the one or more processors, cause the computer system to perform the above-described method. Similarly, a machine readable medium is provided having stored thereon a set of instructions which, when executed by one or more processors, cause the one or more processors to perform the above-described method.
[0006] In general terms, the present description introduces a system and method that uses a grasp model that includes a neural network to obtain pixel-by-pixel predictions (e.g., score maps or heat maps) for corresponding grasp results in a set of grasp results. The pixel-by-pixel predictions are output as a single aggregated pixel-by-pixel prediction for use in generating a grasp, e.g., by sampling the pixel-by-pixel predictions to obtain two-dimensional grasp positions at which grasp poses (e.g., with six degrees of freedom) in the robot's task space can be generated. By providing a set of grasp predictions as a single aggregated pixel-by-pixel prediction, other pixel-by-pixel data, such as heuristic data and / or segmentation data related to the scene, can be incorporated into the scoring of pixels as candidate grasp positions. Additionally, reinforcement learning techniques can be applied to adjust weights at least one of two levels, e.g., when aggregating the pixel-by-pixel predictions into a single aggregated pixel-by-pixel prediction and / or when aggregating the single aggregated pixel-by-pixel prediction with additional pixel-by-pixel data. Thus, the grasping model described herein can be implemented in a grasping pipeline for a robot and tuned to improve the robot's performance metrics at a macro level, such as the robot's speed or throughput.
[0007] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numbers indicate the same or corresponding parts, and in which: [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a schematic diagram of a robotic picking system according to an example. [Diagram 2] FIG. 2 is a schematic diagram of input and output data of a grasping model according to an example. [Figure 3A] FIG. 3A is a schematic diagram of input and output data of a grasping model according to an example. [Figure 3B]FIG. 3B is a schematic diagram of aggregation of multiple pixel-by-pixel predictions according to an example. [Figure 4] FIG. 4 is a schematic diagram of a neural network used as part of the example grasping model. [Diagram 5] FIG. 5 is a process flow chart. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] In the drawings, like features are designated by like reference numerals where appropriate.
[0010] In the following description, a number of specific details are included to provide a thorough understanding of various disclosed embodiments. However, one of ordinary skill in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In some cases, well-known structures associated with the gripper assembly and / or robotic manipulator, such as processors, sensors, storage devices, network interfaces, workpieces, tension members, fasteners, electrical connectors, mixers, etc., are not shown or described in detail to avoid unnecessarily obscuring the description of the disclosed embodiments.
[0011] Unless the context requires, throughout this specification and the appended claims, the word "comprise" and variations thereof, such as "comprises" and "comprising," are to be interpreted in their open and inclusive sense, including but not limited to.
[0012] References throughout this specification to "one embodiment," "embodiment," "one example," or "example" mean that the particular referenced feature, structure, or characteristic described in connection with an embodiment, example, or implementation is included in at least one embodiment or implementation. Thus, the appearances of the phrase "in one embodiment" or the like in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, examples, or implementations.
[0013] It should be noted that, as used herein and in the appended claims, user forms such as "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. Thus, for example, a reference to a gripper assembly including a "finger element" includes one finger element or two or more finger elements. It should be noted that the term "or" is generally used in its sense including "and / or" unless the content clearly dictates otherwise.
[0014] 1, there is illustrated an example of a robotic picking system 100 such as may be adapted for use with the present assemblies, devices, and methods. The robotic picking system 100 may form part of an online retail operation, such as an online grocery retail operation, but may also be applied to any other operation requiring picking and / or sorting of items.
[0015] In this example, the robotic picking system 100 includes a manipulator device 102 that includes a robotic manipulator 121. The robotic manipulator 121 is configured, for example, to pick an item from a first location and place the item at a second location.
[0016] The manipulator device 102 is communicatively coupled via a communication interface 104 to other components of the robotic picking system 100, such as one or more optional operator interfaces 106, from which an observer can observe or monitor the operation of the system 100 and the manipulator device 102. The operator interface 106 may include a WIMP interface and an output display of a descriptive text or dynamic representation of the manipulator device 102 in a context or scenario. For example, the dynamic representation of the manipulator device 102 may include video and audio feeds, such as computer-generated animation. Examples of suitable communication interfaces 104 include a wire-based network or communication interface, an optical-based network or communication interface, a wireless network or communication interface, or a combination of wired, optical, and / or wireless network or communication interfaces.
[0017] The robotic picking system 100 further comprises a control system 108 including at least one controller 110 communicatively coupled to the manipulator device 102 and any other components of the robotic picking system 100 via a communication interface 104. The controller 110 comprises a control unit or computing device having one or more electronic processors. Embedded within the one or more processors is computer software including a set of control instructions provided as processor-executable data that, when executed, causes the controller 110 to issue actuation commands or control signals to the manipulator system 102. For example, the actuation commands or control signals cause the manipulator 121 to perform various methods and operations, such as identifying and manipulating items.
[0018] The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphic processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), program logic units (PLUSs), etc. In some implementations, the controller 110 is a smaller processor-based device, such as a mobile phone, single board computer, embedded computer, etc., which may be interchangeably called or referred to as a computer, server, or analyzer. A set of control instructions may be provided as processor executable data associated with the operation of the system 100 and the manipulator device 102 contained in a non-transitory computer readable storage device 112 that forms part of the robotic picking system 100 and is accessible to the controller 110 via the communication interface 104.
[0019] In some implementations, the storage device 112 includes two or more separate devices. The storage device 112 can include, for example, one or more volatile storage devices, such as random access memory (RAM), and one or more non-volatile storage devices, such as read-only memory (ROM), flash memory, magnetic hard disk (HDD), optical disk, solid state disk (SSD), etc. Those skilled in the art will appreciate that the storage devices may be implemented in a variety of ways, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other form of computer and processor readable memory or storage medium, and / or combinations thereof. The storage devices may be read-only or read-write, as appropriate.
[0020] The robotic picking system 100 includes a sensor subsystem 114 that comprises one or more sensors that detect, sense, or measure a status or condition of the manipulator apparatus 121 and / or a condition in the environment or workspace in which the manipulator 102 operates, and generate or provide corresponding sensor data or information. The sensor information includes environmental sensor information representative of environmental conditions in the workspace of the manipulator 121, as well as information representative of a status or condition of the manipulator apparatus 102, including its various subsystems and components, as well as characteristics of the item being manipulated. The acquired data can be transmitted via the communication interface 104 to the controller 110 to instruct the manipulator 121 accordingly. Such information can include, for example, diagnostic sensor information useful for diagnosing a status or condition of the manipulator apparatus 102 or the environment in which the manipulator 121 operates.
[0021] Such sensors include, for example, one or more cameras or imagers 116 (e.g., responsive in the visible and / or non-visible ranges of the electromagnetic spectrum, including infrared and ultraviolet). Other sensors in the sensor subsystem 114 may include one or more of contact sensors, force sensors, strain gauges, vibration sensors, position sensors, attitude sensors, accelerometers, radar, sonar, lidar, touch sensors, pressure sensors, load cells, microphones 118, weather sensors, chemical sensors, etc. In some implementations, the sensors include diagnostic sensors that monitor the status and / or health of an on-board power source (e.g., a battery array, an ultracapacitor array, a fuel cell array) in the manipulator device 102.
[0022] In some implementations, the one or more sensors comprise a receiver for receiving position and / or orientation information regarding the manipulator 121, for example a Global Positioning System (GPS) receiver for receiving two or more time signals for the controller 110 to generate a position measurement based on data in the signals, such as GPS data, time of flight, signal strength, or other data to achieve a position measurement. The manipulator 121 may also be provided with one or more accelerometers, which may form part of the manipulator arrangement 102, for example, to obtain inertial or orientation data in one, two, or three axes regarding its movement.
[0023] The robotic manipulator 121 of the system 100 may be manipulated by a human operator at the operator interface 106. In a human operator controlled or maneuvering mode, the human operator observes a representation of sensor data, e.g., video, audio, or haptic data received from one or more sensors of the sensor subsystem 114. The human operator then acts conditioned by his perception of the representation of the data and creates information or executable control instructions to direct the manipulator 121 accordingly. In the maneuvering mode, the manipulator device 102 can execute control instructions received from the operator interface 106 in real time (e.g., without added delay) without considering other control instructions based on the sensed information.
[0024] In some implementations, the manipulator device 102 operates autonomously, i.e., without a human operator creating control instructions at the operator interface 106 to direct the manipulator 121. The manipulator device 102 can operate in an autonomous control mode by executing autonomous control instructions. For example, the controller 110 can use sensor data from one or more sensors of the sensor subsystem 114, which is correlated with operator-generated control instructions from one or more times that the manipulator device 102 was in pilot mode, to generate autonomous control instructions for subsequent use. In the autonomous mode, for example by using deep learning techniques to extract features from the sensor data, the manipulator device 102 autonomously recognizes features or conditions in its environment and the items being manipulated, and accordingly executes a defined action, set of actions, task, or pipeline or sequence of tasks. In some implementations, the controller 110 autonomously perceives features and / or conditions in the environment surrounding the manipulator 121, as represented by sensor data from the sensor subsystem 114 and one or more virtual items synthesized into the environment, and in response to a representation being presented, issues control signals to the manipulator device 102 to perform one or more actions or tasks.
[0025] In some cases, manipulator device 102 may be autonomously controlled at some times and piloted, operated, or controlled by a human operator at other times, i.e., operate under an autonomous control mode and change to operate under a piloted (i.e., non-autonomous) mode. In another mode of operation, manipulator device 102 may replay or execute control instructions previously executed in a human operator controlled (or piloted) mode. That is, manipulator device 102 may operate without sensor data based on replayed piloting data.
[0026] Manipulator device 102 further includes a communication interface subsystem 124 (e.g., a network interface device) communicatively coupled to bus 126 and providing bidirectional communication with other components of system 100 (e.g., controller 110) via communication interface 104. Communication interface subsystem 124 may be any circuitry that affects bidirectional communication of processor readable data and processor executable instructions, such as a radio (e.g., radio or microwave frequency transmitter, receiver, transceiver), communication port, and / or associated controller. Suitable communication protocols include FTP, HTTP, Web Services, SOAP with XML, WI-FI® compliant, BLUETOOTH® compliant, cellular (e.g., GSM®, CDMA), and the like.
[0027] The manipulator 121 is an electromechanical machine that includes one or more appendages, such as a robotic arm 120, and a gripper assembly or end effector 122 attached to the end of the robotic arm 120. The end effector 122 is a device of complex design that is configured to interact with an environment to perform a number of tasks, including, for example, grasping, holding, releasably engaging, or otherwise interacting with an item. Examples of end effectors 122 include jaw grippers, finger grippers, magnetic or electromagnetic grippers, Bernoulli grippers, vacuum suction cups, electrostatic grippers, van der Waals grippers, capillary grippers, cryogenic grippers, ultrasonic grippers, and laser grippers.
[0028] Manipulator device 102 further includes a motion subsystem 130 that is communicatively coupled to robotic arm 120 and end effector 122. Motion subsystem 130 comprises one or more motors, solenoids, other actuators, linkages, drive belts, etc. operable to move robotic arm 120 and / or end effector 122 through a range of motion in accordance with actuation commands or control signals issued by controller 110. Memory subsystem 130 is communicatively coupled to controller 110 via bus 126.
[0029] The manipulator device 102 also includes an output subsystem 128 that includes one or more output devices, such as a speaker, light, or display, that enable the manipulator device 102 to transmit signals into the workspace, for example, to communicate with an operator and / or another manipulator device 102.
[0030] Those skilled in the art will appreciate that components within manipulator device 102 may be modified, combined, divided, omitted, etc. In some examples, one or more of communication interface subsystem 124, output subsystem 128, and / or motion subsystem 130 may be combined. In other examples, one or more of the subsystems (e.g., motion subsystem 130) are divided into additional subsystems.
[0031] 2 illustrates a schematic input and output pipeline of the grasp model 200. The grasp model 200 may be implemented as part of a grasp operation performed by the robotic picking system 100 of the previous example. For example, the control system 108 includes a grasp control system, such as the controller 110 described with reference to FIG. 1, configured to issue actuation commands or control signals to the manipulator system 102. The grasp control system may include, for example, one or more processors communicatively coupled to the grasp model 200.
[0032] The grasping model 200 includes a neural network trained to predict multiple outcomes associated with a grasping action based on an image of a graspable object. Image data, including depth data, such as RGB-D data, is input to the grasping model 200, which processes the image data to generate independent predictions for multiple outcomes. In other words, the same grasping model 200 (including the trained neural network) is configured to generate multiple independent predictions for a corresponding set of outcomes associated with a grasping action. In an example, the grasping model 200 includes a trained fully convolutional network (FCN). In such a case, the grasping model 200 may be referred to as a fully convolutional grasping (FCG) model.
[0033] 2 illustrates an exemplary RGB-D image 210 including three color (red, green, blue) channels and a depth channel as input to the grasping model 200. The image 210 illustrates multiple objects in a workspace of a picking robot. The grasping model 200 is configured to process the (RGB-D) image data to generate multiple pixel-wise predictions corresponding to multiple outcomes. The pixel-wise predictions can be represented as pixel-wise prediction maps (or "heat maps") 220. Each of the heat maps 220 is a visualization of the generated pixel-wise probability values corresponding to a respective outcome associated with a grasping operation.
[0034] The set of outcomes associated with the gripping operation may include two or more of: successful gripping of the object, successful scanning of the object, successful placement of the object, successful subsequent gripping of another object, avoidance of the object from gripping another object, and avoidance of stalling the gripping operation. For example, successful gripping of the object corresponds to the robotic manipulator completing a gripping operation or gripping duty cycle for the object (e.g., including picking the object at a first location, moving the object to a second location, and placing the object at the second location) without dropping the item. Detecting that the object has fallen from the manipulator, e.g., an end effector of the manipulator, prior to placing the object constitutes, e.g., dropping the item. The manipulator or its end effector may include at least one contact sensor configured to detect contact with the picked item. Thus, detecting that the item has fallen may be determined based on feedback from the at least one contact sensor. Successful placement of an object corresponds, for example, to placement of the object at a location within a predetermined threshold distance, or a measure of how centered the placement of the object is within a target placement area. Successful scanning of an object corresponds, for example, to a registered or completed scan of an identification marker (e.g., a barcode) on the object during a gripping operation or duty cycle. Avoiding simultaneous gripping of another object, e.g., a "double pick," corresponds, for example, to a determination that no double pick detection occurred during a gripping operation or duty cycle. Avoiding stopping a gripping operation may correspond to a determination that no stop of the gripping operation or duty cycle occurred during a gripping operation or duty cycle. For example, a protective stop or "p-stop" may be triggered by the robot controller during a gripping operation to protect the robot, for example, from damage. In other cases, a gripping operation may be stopped due to the robot failing to complete the gripping operation, for example, within a predetermined time threshold.
[0035] 3A shows another example of the inputs and outputs of the grasp model 200 for generation of a grasp as part of a grasping operation. Image data 310, including color data 311 and depth data 312, is input into the grasp model 200, which includes a trained neural network, to generate multiple independent pixel-wise predictions 320 for the grasp operation. In the example of FIG. 3A showing five output pixel-wise heat maps 320 corresponding to five different outcomes associated with the grasp operation, five pixel-wise predictions are generated.
[0036] 3B shows an example of aggregating multiple pixel-wise heatmaps into an aggregated pixel-wise heatmap, where the five heatmaps 320 generated in the example of FIG. 3A are aggregated into a single pixel-wise prediction 330. In the example, a weighted sum is used for aggregation, but other pixel-wise aggregation methods are possible, such as a weighted product. When weighted pixel-wise aggregation is used, the weights corresponding to each pixel-wise prediction, e.g., heatmap, can be predetermined or adjusted via reinforcement learning methods (described below).
[0037] The aggregated pixel-wise heatmap 330 may be used to select grasp positions therefrom on which to generate a grasp pose for grasping the object. For example, the aggregated pixel-wise heatmap 330 can be sampled deterministically (e.g., via a greedy algorithm) or probabilistically (e.g., per a probability distribution, such as a probability distribution derived from the aggregated pixel-wise heatmap 330) to obtain pixels corresponding to candidate grasp positions.
[0038] [Neural Networks] FIG. 4 illustrates an example of a neural network architecture for implementation as part of the grasping model 200. The example neural network 400 is a fully convolutional neural network (FCN). An example FCN is the U-Net architecture developed by the Computer Science Department at the University of Freiburg, although other FCNs and convolutional neural networks (CNNs) can be used, such as the VGG-16 network architecture. The input 401 to the FCN 400 includes image data, which in this example includes depth data. The input image data 401 is a predetermined number of pixels wide and a predetermined number of pixels high, and includes one or more color channels (e.g., a red channel, a green channel, and a blue channel) and a depth channel. The convolutional layers 402 of the FCN 400 typically extract certain features from the input data 401 to create feature maps, which may operate on small portions of the image.
[0039] In the example of FIG. 4, the output of at least one of the convolutional layers 402 undergoes pooling, e.g., in a pooling layer, before being input to the next convolutional layer in the sequence of convolutional layers 402. Pooling allows, e.g., values for a region of an image or feature map to be aggregated or combined, e.g., by taking the highest value in the region. For example, in 2×2 max pooling, the highest value of the output of a given convolutional layer within a 2×2 pixel patch of the feature map output from the given convolutional layer is used as the input to the subsequent convolutional layer in the set of layers 402, rather than forwarding the entire output. Pooling can thus reduce the amount of computation for subsequent layers of the neural network 400. The effect of pooling is shown diagrammatically in FIG. 4 as a reduction in the size of the frames in the relevant layer. It should be understood that the schematic diagram of the neural network 400 in FIG. 4 is greatly simplified for ease of explanation, and that typical neural networks can be significantly more complex. For example, neural network 400 includes one or more upsampling (e.g., upconvolutional) layers and / or one or more skip (or "shortcut") connections. In general, neural network architectures used in grasping model 200 do not utilize fully connected layers. For example, the U-Net architecture uses only convolutional, pooling, and upsampling layers, along with skip connections. When a pre-trained architecture (e.g., VGG-16) is used, only the convolutional "backbone" (e.g., a set of feature extraction layers) is used with appropriate modifications (e.g., additional layers) to ensure that the output prediction map (e.g., heat map) has the same dimensions as the input image.
[0040] In general, a neural network, such as the FCN 400 of FIG. 4, may undergo what is called a "training phase" during which the neural network is trained for a particular purpose. A neural network typically includes layers of interconnected artificial neurons that form a directed weighted graph, where each vertex (corresponding to a neuron) or edge (corresponding to a connection) of the graph is associated with a weight. The weights can be adjusted throughout training to change the output of individual neurons and therefore the output of the entire neural network. In contrast to using a CNN for object classification, the FCN in the grasping model is configured to make pixel-by-pixel predictions of a particular outcome.
[0041] In the present context, the neural network 400 is trained to predict grasping outcomes by processing image data, e.g., to determine pixel-wise probability values corresponding to each grasping outcome. Training the neural network 400 in this manner, for example, generates weight data representing weights to be applied to the input image data (e.g., different weights are associated with different respective layers of a multi-layer neural network architecture). Each of these weights is multiplied by the corresponding pixel value of the image patch, e.g., to convolve a weight kernel with the image patch.
[0042] Specific to the context of grasp prediction, the neural network 400 is trained with a training dataset that includes a number of grasps performed with the robot. Each "grasp" in the dataset comprises a captured image, including depth data, of the robot's task space (e.g., bin) before the grasp, the grasp pose in world coordinates, a mapping from world coordinates to pixel coordinates of the image, and the resulting measurement (e.g., grasp success, barcode scan success, etc.). Given an input image, the FCG model generates an output prediction, e.g., a heat map, of the same dimensions as the input image with one channel for each predicted outcome. For example, a given pixel (corresponding to a grasp location) in the output heat map is associated with a vector that represents the grasp model's predicted probability of each outcome if a grasp is performed in the immediate vicinity of that pixel. Each component of the vector includes, for example, the predicted probability of the corresponding outcome. The training procedure increases, e.g., maximizes, the likelihood of an outcome at a pixel corresponding to a grasp location as seen in the training dataset. Once trained, the neural network 400 can be used to predict pixel-by-pixel probability values that correspond to any of the grasp outcomes for which the network 400 was trained.
[0043] In an example, one or more processors of a grasp control system communicatively coupled to the grasp model 200 are configured to obtain and provide image data to the grasp model 200 and receive the generated pixel-by-pixel predictions. The one or more processors aggregate the pixel-by-pixel predictions and output the aggregated pixel-by-pixel predictions for selection from among a grasp position to generate one or more grasp poses for grasping the object. In some implementations, the one or more processors of the control system 108 implement the grasp model 200 locally.
[0044] [heuristic] In some implementations, the grip control system is configured to obtain the aggregated pixel-wise prediction map 330 and combine it with one or more pixel-wise heuristic maps. The pixel-wise heuristic data, for example, represents pixel-wise heuristic values corresponding to a given heuristic. Such heuristic data may include height data, surface data (e.g., surface normal data), image segmentation data, or any other heuristic data separate from the grip data that the grip model 200 is trained to predict. Thus, the grip model 200 configured to generate pixel-wise grip prediction data allows other pixel-wise data, for example, pixel maps, to be combined therewith. The grip control system combines the aggregated pixel-wise prediction (e.g., aggregate heat map) with one or more pixel-wise heuristics (e.g., heuristic heat map) to obtain a combined pixel-wise map.
[0045] [Grasp Generation] In some implementations, the controller 110 of the robotic system 100 cooperates with a pose generator (or pose estimator) configured to generate (e.g., determine or estimate) gripping poses. The pose information includes processor-readable information representing a position, an orientation, or both. Each gripping pose comprises a six-dimensional (6D) pose of the robotic manipulator 121 for gripping an object, where the six dimensions include three translational dimensions and a rotational orientation of the object.
[0046] One or more grip positions can be input to a pose generator to generate one or more grip poses based on the one or more grip positions. The one or more grip positions correspond to one or more pixels selected from the aggregated pixel-wise prediction or the combined pixel-wise map (wherein one or more pixel-wise heuristics are combined with the aggregated pixel-wise prediction). For example, the one or more pixels are sampled from the associated pixel-wise map, e.g., deterministically or probabilistically, as described in the previous examples. Determining a grip pose, e.g., having six degrees of freedom, based on pixel positions in the image can be done heuristically (e.g., using a top-down approach, surface normals, etc.) or using a trained neural network configured for pose estimation. For example, determining a grip pose involves mapping two-dimensional pixel positions in an image to a six-dimensional pose.
[0047] The controller 110, for example, obtains one or more generated gripping poses from the pose generator and controls the robot manipulator based thereon. In an example, the one or more gripping poses are ranked according to their corresponding pixel values, e.g., pixel-wise probability values in an aggregated pixel-wise prediction, or a combined "score" in a combined pixel-wise map incorporating heuristic data. The controller 110 can then issue actuation commands or control signals to cause the manipulator 121 to grip the object according to one or more gripping poses in the ranked order.
[0048] 5 shows a flow chart depicting a corresponding computer-implemented process for generating a grasp. For example, the process may be encoded into instructions executable by one or more processors, such as a processor of a grasp control system communicatively coupled to the grasp model 200 in the previous example. The executable instructions are stored in a computer-readable memory, such as the storage device 112, or in a machine-readable medium, such as an electrical, magnetic, or optical storage medium.
[0049] The process 500 starts with obtaining 501 image data comprising depth data, representing an image of an object captured by a camera. One or more camera parameters (e.g., intrinsic parameters such as focal length, and / or extrinsic parameters such as camera pose in world coordinates) are also obtained for use in mapping pixel coordinates in the image (e.g., in 2D) to world coordinates (e.g., in 3D). The image data is provided 502 to a grasping model including a neural network trained to predict multiple outcomes associated with a grasping action independently based on the image of the graspable object. Then, multiple pixel-wise predictions corresponding to the multiple outcomes are obtained 503. Each pixel-wise prediction, which can be visualized, for example, as a heat map, represents a pixel-wise probability value corresponding to a given outcome among multiple outcomes associated with the grasping action. The exemplary outcomes associated with the grasping action described in the previous system implementations also apply here.
[0050] The multiple pixel-wise predictions are then aggregated 504 to obtain an aggregated pixel-wise prediction, e.g., a heatmap. The process 500 ends with outputting 505 the aggregated pixel-wise predictions for selecting one or more pixels therefrom on which the generation of one or more grasp poses for grasping the object is based. Thus, the process 500 can be applied to any robotic system in which an item is grasped and used for some other task (or tasks), e.g., as part of a grasp generation pipeline, and the results of these downstream tasks can be measured and quantified such that the resulting data can be used as training data for a grasp model. In other words, grasping an item is not an end in itself, and in fact objects are often picked in order to do something else with it (e.g., picking up a spoon to stir coffee).
[0051] In some implementations, the process 500 involves processing the image data using a grasp model that comprises a neural network to independently predict multiple outcomes. Multiple pixel-by-pixel predictions corresponding to multiple outcomes are generated by the grasp model, for example, as part of the process 500.
[0052] In an example, the process 500 involves selecting, e.g., sampling, one or more pixels from the aggregated pixel-by-pixel predictions based on one or more corresponding probability values in the pixel-by-pixel predictions. For example, one or more gripping positions corresponding to the one or more selected pixels are output for the robot to grasp the object based thereon. For example, one or more gripping poses are generated based on the one or more gripping positions. The one or more gripping poses may be output to the robot to implement the one or more gripping poses to grasp the object.
[0053] In another example, the process 500 involves obtaining one or more pixel-wise heuristic maps, each representing a pixel-wise heuristic value corresponding to a given heuristic, in addition to the aggregated pixel-wise prediction. The one or more pixel-wise heuristic maps are combined with the aggregated pixel-wise prediction, for example, to obtain a combined pixel-wise map. In such a case, generating or estimating the grip pose involves obtaining one or more grip locations by sampling one or more corresponding pixels from the combined pixel-wise map. One or more grip poses can then be generated or estimated based on the one or more grip locations and output to a robot for implementation.
[0054] The neural network of the grasping model is trained, for example, based on at least one of historical grasping data or simulated grasping data instead of, for example, hand-annotated grasping data. For example, the historical grasping data is collected during implementation of the grasping robot, for example, in a production setting, for example, in a supervised manner, using a predefined grasping policy. Additionally or alternatively, the robotic grasping may be simulated, for example, a controller of the robotic manipulator issues actual control signals or actuation commands for grasping, but these are implemented by a simulated manipulator operating in a simulated environment.
[0055] The grasping model, e.g., a neural network, may be updated based on additional grasping data obtained from grasping additional objects as part of process 500. For example, if the grasping model is implemented in a live environment, e.g., in an industrial or production environment, the instructions and results of the live grasping can be collected and added to a training set for updating the grasping model. For example, the neural network can be retrained, e.g., iteratively, using new grasping data collected during the robot implementation.
[0056] In some implementations, a reinforcement learning (RL) model is applied using multiple pixel-wise predictions corresponding to multiple outcomes as feature inputs to the RL model. For example, if weighted aggregation is used for pixel-wise aggregation of multiple pixel-wise predictions, one or more weights corresponding to each pixel-wise prediction (e.g., a heat map) in the set of pixel-wise predictions are adjusted as part of the RL process.
[0057] In this way, the grasp model can be interpreted as a "Q-function" for a Q-learning RL approach, or as a "critic" network in an actor-critic RL framework that associates each action (e.g., a grasp performed at a given pixel location + 6d pose) with a score or set of scores. Thus configured, RL techniques can be applied to both the training and data collection steps. For example, this can inform grasp selection strategies to appropriately balance exploration (e.g., picking where the model is unsure about the outcome in order to build a better understanding of the task space) and exploitation (e.g., picking where the model is more certain of success).
[0058] In some implementations, the process 500 handles one or more gripping parameters that are not directly inferable from pixel coordinates. Examples of such gripping parameters include grip height, end effector orientation, end effector selection, and grip strategy. These additional gripping parameters can be handled in different ways. For example, an additional input corresponding to each value of the parameter (e.g., each end effector selection) can be passed to the gripping model to generate a pixel-by-pixel prediction conditional on the value of the additional parameter. Alternatively, the gripping model can be configured to generate a separate pixel-by-pixel prediction (e.g., heat map) for each value of the parameter. As an example, for two gripping outcomes (e.g., grip success and placement success) and three additional parameter values (e.g., three possible vacuum cup types), the gripping model is configured to generate six output pixel-by-pixel predictions (heat maps): one for each combination of gripping outcome and additional parameter value (e.g., vacuum cup type). As a further alternative, the grasping model can be configured (e.g., the model includes an additional output head) to generate separate outputs to predict parameter values (e.g., end effector selection) at each pixel. The separate outputs are included, for example, as channels in the pixel-by-pixel prediction (e.g., a heat map). In the exemplary case of three additional parameter values (e.g., corresponding to three possible vacuum cup types), the output pixel-by-pixel prediction for each grasping outcome includes a three-channel image, with each channel corresponding to a single parameter value (e.g., vacuum cup type).
[0059] Returning to the context of implementing the described grasping model as part of a grasping pipeline, the robot control system 108 (e.g., controller 110) of the robot may initiate a process for generating grasp candidates. For example, the robot control system 108 sends a request for grasp candidates to the grasping control system. The grasping control system obtains image data, including depth data, corresponding to the current state of the workspace (e.g., a picking bin or container). The image data is processed using one or more predictive (or "scoring") models, including the grasping models described herein (e.g., FCG models). Other scoring models include, for example, the heuristic models and segmentation models (e.g., for recognizing where different objects are located in an image) described above. If additional scoring models are used in addition to the grasping models, the output score maps are combined into a single output score map. The output score map is sampled to obtain pixels corresponding to the two-dimensional grasp candidates. The 2D grasp candidates are processed with one or more grasp generation operators, e.g., pose generators, to convert the 2D image positions into 3D positions in the robot's task space. In some examples, the one or more grasp generation operators also determine an orientation of the manipulator, e.g., an end effector, based on surface normals in the task space. Thus, one or more 3D positions and orientations (e.g., grasp poses) in the task space are returned to the robot control system 108 along with corresponding scores. The corresponding scores may be adjusted by the one or more grasp generation operators, in examples, e.g., based on surface normals or other heuristics. The robot control system 108 then issues actuation commands to the robot based on one or more of the one or more grasp poses and their corresponding scores.
[0060] In examples using data processing, a processor may be used as part of the associated system, which may be a general-purpose processor such as a central processing unit (CPU), microprocessor, graphic processing unit (GPU), digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination of these designed to perform the data processing functions described herein.
[0061] In examples involving neural networks, a dedicated processor may be used as part of the associated system. The dedicated processor may be an NPU, a Neural Network Accelerator (NNA), or other version of a hardware accelerator dedicated for neural network functions. Additionally or alternatively, the neural network processing workload may be at least partially shared by one or more standard processors, e.g., a CPU or GPU.
[0062] The above examples should be understood as illustrative examples. Further examples are envisioned. For example, another outcome of interest to include in the multiple outcomes that the neural network is trained to predict may be scan speed. For example, instead of or in addition to making a binary prediction of whether an item is successfully scanned or not, the neural network may be trained to predict which gripping position corresponds to the shortest amount of time spent trying to obtain a successful barcode scan. In the context of grocery picking, for example, one outcome of the multiple outcomes may be bagging success, and the gripping model learns how to pick the item so that it can be successfully bagged. The model's prediction of a "bagging success" outcome may be conditional on, for example, the current state of the bag (e.g., as represented by an RGB-D image of the output bin or tote).
[0063] In general terms, the described systems and methods can be used as part of a grasp generation or grasp synthesis pipeline that takes input images including depth information and outputs ranked grasp poses, e.g., having six degrees of freedom. Input to the grasp generation pipeline includes, e.g., an input image captured by a camera along with the camera's orientation in a world coordinate frame. The grasp generation pipeline then includes passing the input image through the described FCN grasp model and optionally one or more other heuristic map computations, aggregating the resulting heat map, sampling pixel locations from the aggregated map, mapping the sampled pixel locations (e.g., in 2D) to corresponding grasp poses (e.g., in 6D), and optionally adjusting the ranking of the corresponding grasp poses based on some combination of the heuristic and / or grasp model predictions.
[0064] In the described examples, the training data for the grasping model is collected from historical grasping results, e.g., rather than human-annotated results. For example, the grasping model is intended to learn "what would happen if I grasped here?" rather than "what would happen if I grasped here?". However, in some implementations, one or more human pilots are used as part of the production system. Specifically, a human can suggest grasping positions for a small percentage of picks if one or more predefined conditions are met. In such cases, there may be some grasps in the historical data set where a grasping position (or grasping strategy) was suggested by a human. Nevertheless, the grasping results are still collected from the physical robotic system.
[0065] It is also understood that any feature described in connection with any one example may also be used alone or in combination with other features described, or in combination with one or more features of any other example, or in any combination of any other example. Moreover, equivalents and modifications not described above may also be employed without departing from the scope of the appended claims. The following is a summary of the claims as originally filed: [C1] A data processing device for generating a grasp, comprising: obtaining image data representing an image of an object captured by a camera, the image data including depth data; providing the image data to a grasp model, the grasp model comprising a neural network trained to predict a plurality of outcomes associated with a grasp operation independently based on an image of a graspable object; obtaining a plurality of pixel-by-pixel predictions corresponding to the plurality of outcomes, where each pixel-by-pixel prediction represents a pixel-by-pixel probability value corresponding to a given outcome of the plurality of outcomes associated with the grasping motion; aggregating the plurality of pixel unit predictions to obtain an aggregated pixel unit prediction; and configured to output the aggregated pixel-by-pixel predictions for selection of one or more pixels therefrom on which one or more grasp poses for grasping the object are based. [C2] A data processing device according to C1, configured to implement the grasping model. [C3] The plurality of outcomes associated with the grasping action include: a successful grasp of said object; a successful scan of said object; successful placement of said object; Successful subsequent grasping of another object, avoidance of said object from grasping another object; and avoiding the stopping of the gripping motion. [C4] A gripping control system for a robot comprising the data processing device according to any one of C1 to C3, The gripping control system includes: obtaining the aggregated pixel-wise predictions and one or more pixel-wise heuristic maps, each representing a pixel-wise heuristic value corresponding to a given heuristic; a grasping control system configured to combine the aggregated pixel-wise predictions and the one or more pixel-wise heuristic maps to obtain a combined pixel-wise map. [C5] a pose generator, the pose generator comprising: obtaining one or more grasp locations corresponding to one or more sampled pixels from the combined pixel-wise map; 5. The gripping control system of claim 4, configured to determine one or more gripping postures based on the one or more gripping positions. [C6] A gripping control system for a robot, comprising: a data processing device according to any one of C1 to C3; and a posture generator; The attitude generator includes: obtaining one or more grasp locations corresponding to one or more pixels selected from the aggregated pixel-by-pixel predictions based on one or more corresponding probability values in the pixel-by-pixel predictions; A gripping control system configured to determine one or more gripping postures based on the one or more gripping positions. [C7] 7. The gripping control system of claim 5 or 6, comprising a controller configured to obtain the one or more gripping poses from the pose generator and control a robotic manipulator to grip the object based on the one or more gripping poses. [C8] A robot comprising the gripping control system according to C7 and a robot manipulator. [C9] The robot of C8, wherein the robot manipulator has an end effector for gripping the object, the end effector comprising at least one of a jaw gripper, a finger gripper, a magnetic or electromagnetic gripper, a Bernoulli gripper, a vacuum suction cup, an electrostatic gripper, a van der Waals gripper, a capillary gripper, a cryogenic gripper, an ultrasonic gripper, or a laser gripper. [C10] 1. A computer-implemented method comprising: acquiring image data representing an image of an object captured by a camera, the image data including depth data; providing the image data to a grasp model, the grasp model comprising a neural network trained to predict a plurality of outcomes associated with a grasp operation independently based on an image of a graspable object; obtaining a plurality of pixel-by-pixel predictions corresponding to the plurality of outcomes, where each pixel-by-pixel prediction represents a pixel-by-pixel probability value corresponding to a given outcome of the plurality of outcomes associated with the grasping motion; aggregating the plurality of pixel unit predictions to obtain an aggregated pixel unit prediction; and and outputting the aggregated pixel-by-pixel predictions for selection of one or more pixels therefrom on which one or more grasp poses are based to generate for grasping the object. [C11] The plurality of outcomes associated with the grasping action include: a successful grasp of the object; and a successful scan of the object; successful placement of the object; and a successful subsequent grasp of another object; avoidance of said object from grasping another object; and and avoiding halting of the gripping motion. [C12] processing the image data using the grasp model including the neural network to independently predict the plurality of outcomes; and generating the plurality of pixel unit predictions corresponding to the plurality of results. [C13] The computer-implemented method of any one of C10 to C12, wherein the neural network is trained with at least one of historical grasp data or simulated grasp data. [C14] The computer-implemented method of any one of C10 to C13, comprising updating the grasp model based on further grasp data obtained from grasping other objects. [C15] The computer-implemented method of C14, further comprising retraining the neural network based on the further grasping data. [C16] selecting the one or more pixels from the aggregated pixel-by-pixel predictions based on one or more corresponding probability values in the pixel-by-pixel predictions; and outputting one or more grasp positions for a robot to grasp the object based on the selected one or more pixels. [C17] determining one or more gripping postures based on the one or more gripping positions; and outputting the one or more grasp postures to the robot to perform the one or more grasp postures to grasp the object. [C18] obtaining the aggregated pixel-wise predictions and one or more pixel-wise heuristic maps, each representing a pixel-wise heuristic value corresponding to a given heuristic; and combining the aggregated pixel-wise predictions with the one or more pixel-wise heuristic maps to obtain a combined pixel-wise map. [C19] obtaining one or more grasp locations by sampling one or more corresponding pixels from the combined pixel-wise map; and determining one or more gripping postures based on the one or more gripping positions. [C20] 20. The computer-implemented method of any one of C10 to 19, comprising applying a reinforcement learning model using the pixel-by-pixel predictions corresponding to the outcomes as feature inputs. [C21] The computer-implemented method of C20, wherein applying the reinforcement learning model includes adjusting one or more weights corresponding to each pixel-unit prediction of the plurality of pixel-unit predictions. [C22] The computer-implemented method of any one of C10 to 21, wherein the neural network comprises a fully convolutional neural network. [C23] one or more processors; and a computer-readable memory storing executable instructions that, when executed by the one or more processors, cause the computer system to perform a method according to any one of C10 to C22. [C24] A machine-readable medium storing a set of instructions that, when executed by one or more processors, cause the one or more processors to perform a method according to any one of C10 to C22.
Claims
1. A data processing device for generating a grasp, comprising: obtaining image data representing an image of an object captured by a camera, the image data including depth data; processing the image data with a grasp model comprising a neural network trained to predict a plurality of independent outcomes associated with a grasp operation based on an image of a graspable object; obtaining a plurality of pixel-by-pixel predictions corresponding to the plurality of independent outcomes, where each pixel-by-pixel prediction represents a pixel-by-pixel probability value corresponding to a given outcome of the plurality of independent outcomes associated with the grasping motion; aggregating the plurality of pixel unit predictions using weighted aggregation to obtain an aggregated pixel unit prediction; applying a reinforcement learning model to adjust one or more weights corresponding to each pixel-wise prediction of the weighted aggregate; and outputting the aggregated pixel-by-pixel predictions for selection of one or more pixels on which one or more grasp poses for grasping the object are based.
2. A data processing apparatus according to claim 1 , configured to implement the grasp model.
3. The plurality of independent outcomes associated with the grasping action include: a successful grasp of said object; a successful scan of said object; successful placement of said object; Successful subsequent grasping of another object, avoidance of said object from grasping another object; The data processing device according to claim 1 or 2, further comprising two or more of: a) preventing the gripping motion from stopping;
4. A gripping control system for a robot comprising the data processing device according to any one of claims 1 to 3, The gripping control system includes: obtaining the aggregated pixel-wise predictions and one or more pixel-wise heuristic maps, each representing a pixel-wise heuristic value corresponding to a given heuristic; A grasping control system configured to combine the aggregated pixel-wise predictions and the one or more pixel-wise heuristic maps to obtain a combined pixel-wise map.
5. a pose generator, the pose generator comprising: obtaining one or more grasp locations corresponding to one or more sampled pixels from the combined pixel-wise map; The gripping control system of claim 4 , configured to determine one or more gripping postures based on the one or more gripping positions.
6. A gripping control system for a robot, comprising: a data processing device according to any one of claims 1 to 3; and a posture generator, The attitude generator includes: obtaining one or more grasp locations corresponding to one or more pixels selected from the aggregated pixel-by-pixel predictions based on one or more corresponding probability values in the pixel-by-pixel predictions; A gripping control system configured to determine one or more gripping postures based on the one or more gripping positions.
7. 7. The gripping control system of claim 5, comprising a controller configured to obtain the one or more gripping poses from the pose generator and to control a robotic manipulator to grip the object based on the one or more gripping poses.
8. A robot comprising the gripping control system according to claim 7 and a robot manipulator.
9. 9. The robot of claim 8, wherein the robotic manipulator comprises an end effector for gripping the object, the end effector comprising at least one of a jaw gripper, a finger gripper, a magnetic or electromagnetic gripper, a Bernoulli gripper, a vacuum suction cup, an electrostatic gripper, a van der Waals gripper, a capillary gripper, a cryogenic gripper, an ultrasonic gripper, or a laser gripper.
10. 1. A computer-implemented method, comprising: acquiring image data representing an image of an object captured by a camera, the image data including depth data; processing the image data with a grasp model comprising a neural network trained to predict a plurality of independent outcomes associated with a grasp operation based on an image of a graspable object; obtaining a plurality of pixel-by-pixel predictions corresponding to the plurality of independent outcomes, where each pixel-by-pixel prediction represents a pixel-by-pixel probability value corresponding to a given outcome of the plurality of independent outcomes associated with the grasping motion; aggregating the plurality of pixel-unit predictions using weighted aggregation to obtain an aggregated pixel-unit prediction; applying a reinforcement learning model to adjust one or more weights corresponding to each pixel-wise prediction of the weighted aggregate; and outputting the aggregated pixel-by-pixel predictions for selection of one or more pixels on which one or more grasp poses are based to generate for grasping the object.
11. The plurality of independent outcomes associated with the grasping action include: a successful grasp of the object; and a successful scan of the object; successful placement of the object; and a successful subsequent grasp of another object; avoidance of said object from grasping another object; and and avoiding stalling of the grasping motion.
12. The computer-implemented method of claim 10 or 11, wherein the neural network is trained on at least one of historical grasp data or simulated grasp data.
13. 13. The computer-implemented method of claim 10, further comprising updating the grasp model based on further grasp data obtained from grasping other objects.
14. The computer-implemented method of claim 13 , further comprising retraining the neural network based on the additional grasp data.
15. selecting the one or more pixels from the aggregated pixel unit predictions based on one or more corresponding probability values in the pixel unit predictions; and outputting one or more grasp positions for a robot to grasp the object based on the selected one or more pixels.
16. determining one or more gripping postures based on the one or more gripping positions; and outputting the one or more grasp poses to the robot for performing the one or more grasp poses to grasp the object.
17. obtaining the aggregated pixel-wise predictions and one or more pixel-wise heuristic maps, each representing a pixel-wise heuristic value corresponding to a given heuristic; and combining the aggregated pixel-wise predictions with the one or more pixel-wise heuristic maps to obtain a combined pixel-wise map.
18. obtaining one or more grasp locations by sampling one or more corresponding pixels from the combined pixel-wise map; and determining one or more grasp poses based on the one or more grasp positions.
19. 19. The computer-implemented method of claim 10, wherein the neural network comprises a fully convolutional neural network.
20. one or more processors; and a computer readable memory storing executable instructions that, when executed by the one or more processors, cause the computer system to perform a method according to any one of claims 10 to 19.
21. A machine-readable storage medium having stored thereon a set of instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 10 to 19.
Citation Information
Patent Citations
Autonomous grabbing and stacking method and system based on mechanical arm
CN111331607A
Learning device, learning method, learning model, detection device, and holding system
JP2019164836A
Deep machine learning method and apparatus for robotic grasping
JP2019217632A
Computing system and machine learning method
JP2020179438A
Grasp generation using a variational autoencoder
US20200361083A1
