Methods and systems for picking objects

Abstract

Disclosed herein are systems and methods for picking objects, comprising receiving an input image and point cloud for one or more objects to be picked, annotating the point cloud by converting points thereof into one or more candidate grasping surfaces, generating one or more heatmaps that encode angles of 3D grasp poses relative to a common plane, or a type of those poses, running a Zero-Shot object detection to detect the one or more objects in the input image, and obtaining, for each detected object, a grasping location based on each of the candidate grasping surface.

Description

[0001] METHODS AND SYSTEMS FOR PICKING OBJECTS

[0002] Technical Field

[0003] The present disclosure relates, in general terms, to a method for picking objects (i.e. picking up objects using a robotic arm). In particular, the present disclosure relates to, but is not limited to, a method for finding grasp poses for picking randomly placed unknown objects without prior object registration.

[0004] Background

[0005] This background description is provided for the purpose of generally presenting the context of the disclosure. Contents of this background section are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0006] With the development of robot grasping technology, there is a present challenge of picking up objects automatically and accurately. Using robots to grasp objects has wide-ranging applications, including bin picking in manufacturing and order fulfilment in logistics.

[0007] While the methods of picking up objects in a structured environment, or picking up objects in a random environment, but with master registration are well developed, there is a need to develop a method for picking objects where no model of the objects are available beforehand.

[0008] Picking up objects in a random environment without a master registration has traditionally been done with a suction cup. While the use of a suction cup is rotationally invariant, i.e. the direction which the suction cup approaches the object does not matter, a suction cup requires a flat object surface to work.

[0009] Picking up objects in a random environment with a gripper, e.g. a parallel gripper, is a relatively more complex problem. A grasp can be parameterised with seven degrees-of-freedom, with three corresponding to the translation vector of the centre of the grasp, three corresponding to the rotational vector of the grasp at the centre, and one corresponding to an opening width of the gripper. It is hence an object of the present invention to provide a method for finding a grasp configuration of randomly placed unknown objects without prior object registration.

[0010] Summary

[0011] Disclosed herein is a computer-implemented method for picking objects, the method comprising : receiving an input image and a point cloud, each comprising one or more objects, the point cloud comprising points corresponding to a respective surface of one or more said objects to be picked; using a neural network model on the input image and the point cloud to produce a plurality of heatmaps; each heatmap encoding at least one of a grasp angle for picking a said object relative to a common picking plane, and a grasp type; generating a plurality of candidate 3D grasp poses by analysing the plurality of heatmaps, to identify high-scoring regions with a specific said grasp angle, and grasp type; performing Zero-Shot object detection to detect the one or more objects to be picked, in the input image; obtaining, from the plurality of candidate 3D grasp poses, a 3D grasp pose for each detected object to be picked; picking one or more of the said objects to be picked by moving an end effector to the obtained 3D grasp poses.

[0012] In some embodiments, the method may further involve picking one of the one or more objects, based on a criterion for ranking.

[0013] In some embodiments, the method may involve defining a sequence of objects to be picked - the sequence being the order in which the objects are to be picked.

[0014] In some embodiments, each candidate grasping surface is ranked according to the grasp angle relative to the common picking plane, corresponding to each object to be picked. In some embodiments, each candidate 3D grasp pose in the system is further ranked according to the distance between a location of the candidate grasping surface and the end effector.

[0015] In some embodiments, a collision check is done during the picking process to ensure collision-free grasping. This involves, for each of the plurality of objects with the corresponding grasp location, increasing a grasp opening width and decreasing an approaching distance until the grasp contacts another object of the plurality of objects.

[0016] In some embodiments, picking objects is carried out with one of: a two-finger grasp, a single suction cup, a single multi-fingered gripper or a combination of suction cups and gripper.

[0017] Disclosed herein is also a method for training a model for picking objects, the method comprising : receiving a plurality of annotated inputs, each input comprising an input image, a point cloud each comprising one or more objects to be picked, and ground-truth 3D grasp poses corresponding to the one or more objects, the respective point cloud further comprising points corresponding to a respective surface of one or more said objects to be picked; encoding each ground-truth 3D grasp pose into a heatmap, each heatmap representing a grasp type and grasp angle, relative to a common picking plane, of respective ground-truth 3D grasp poses; training a neural network model, based on the inputs and respective heatmaps, to predict heatmaps for a new input; and evaluating the trained neural network model on validation and test subsets of the annotated data set to identify an optimal model.

[0018] In some embodiments, the method comprises training the neural network model to detect one or more predetermined objects. In this context, a "predetermined object" is a known object as opposed to an object that is unknown. Consider, for example, a production line of a specific type of object. That object is predetermined since it can be expected to be detected in an input image. Therefore, a model can be trained to accurately detect that predetermined object. In some embodiments, the method comprises training the neural network model to detect one or more uninteresting objects. An uninteresting object is an object that is not intended to be picked - for example, a background object or container containing the objects to be picked.

[0019] In some embodiments, generating one or more heatmaps is done by random rotation augmentation, random translation augmentation, or random image intensity augmentation.

[0020] In some embodiments, encoding the grasp angle and the grasp type comprises performing Gaussian smoothing for reducing discretisation of the corresponding heatmap.

[0021] In some embodiments, during the encoding step, for a score of "1" on an heatmap, the surrounding pixels within a predetermined radius R may also be set at "1". This similarly serves to provide smoother heatmaps.

[0022] In some embodiments, obtaining a grasp location based on each of the candidate grasping surfaces comprises obtaining two selected points on opposite sides of the detected object to be picked.

[0023] In some embodiments, in training a model for picking objects, prior to receiving the annotated inputs, each pixel of the input image or point of the point cloud is annotated by specifying one or both of a 3D grasp angle and grasp type.

[0024] Disclosed herein is a system for picking objects, comprising : an input device for receiving a plurality of input pairs, each pair comprising an input image and a point cloud, the input image and the point cloud each comprising one or more objects to be picked, the respective point cloud comprising points corresponding to a respective surface of one or more said objects to be picked; an annotating module, for producing annotations on the point cloud by adding 3D grasp poses that are suitable for picking the objects to be picked, the annotation specifying a type of grasp for each 3D grasp pose, such as suction cup, two-fingered grasping, or multiple-fingered grasping; an encoding module, for encoding the 3D grasp poses into one or more of the heatmaps, the heatmaps encoded by grasp type and grasp orientation projected onto a common image plane centred about the grasp position; a neural network module comprising a neural network model for generating a plurality of heatmaps from the input pairs; a training module for training the neural network model based on training inputs, each training input comprising an input image and corresponding point cloud, wherein at least one of the input image and point cloud is annotated with 3D grasp poses; a grasp generation module, for generating 3D grasp poses by analysing the heatmaps to identify high-scoring regions with specific grasp angle and grasp type; a detection module for performing Zero-shot object detection to detect the one or more objects to be picked; and an end effector for picking objects to be picked.

[0025] In some embodiments, each candidate 3D grasp pose is ranked according to the grasp angle relative to the common picking plane, and a distance between a grasp location for each of the one or more objects to be picked and the end effector.

[0026] Brief description of the drawings

[0027] Some embodiments of the methods and systems for picking objects, in accordance with the present disclosure, will now be described, by way of non-limiting example only, with reference to the accompanying drawings in which:

[0028] Figure 1 illustrates a computer-implemented method 100 for picking objects.

[0029] Figure 2 shows two-finger grasps annotated on a coloured point cloud.

[0030] Figure 3 illustrates a series of heatmaps, presently being angle heatmaps, corresponding to an input image with three objects to be picked.

[0031] Figure 4 shows the scene coordinate frames X-Y-Z, the 3D grasp pose coordinate frame x-y-z and the projected axis x'. Detailed description

[0032] Embodiments of the present disclosure relate to a computer-implemented method for picking objects. The shape, structure and dimensions of the objects to be picked may be unknown. The environment surrounding the objects may be random and unstructured. There may not be a master, i.e. all objects may be distinctly shaped and different from objects used in the training method also disclosed.

[0033] Figure 1 illustrates such a method (100), computer-implemented, for picking objects. The method comprises (102) receiving an input pair comprising an image and a point cloud. The image and point cloud are captured by a camera (or derived from the image(s) captured by an image capture device - e.g., camera), and will typically be paired - i.e., the point cloud is a point cloud corresponding to the objects, background and other features shown in the image - and the image and the corresponding point cloud are thus referred to as an image pair. In some embodiments, the point cloud is instead generated from the input images. Particularly, the point cloud may also be generated via the stereoscopic 3D reconstruction method, wherein two images as captured by a left camera and a right camera are used to generate a 3-dimensional model of a scene, to provide for depth awareness, or via the monocular depth estimation method, in which the point cloud may be generated from a single image using a neural network model (104). The point cloud may also be generated by time of flight (ToF) cameras or a position angle sensor. The point cloud and image (i.e. the image pair) each comprise one or more objects to be picked.

[0034] A neural network model generates a plurality of heatmaps from the input image pair. Each heatmap encodes the probability of one or more 3D grasp poses of a specific type, having a specific relative angle on a common picking plane. The method (100) generates a plurality of 3D grasp poses by analysing the plurality of heatmaps (106) - for each heatmap, identifying high scoring regions, combining the specific angle of the heatmap with the surface normal of a common picking plane (e.g., a plane perpendicular to the camera plane, or a plane tangential to a region on the point cloud) and the specific grasp type of the heatmap. The heatmaps may therefore be binary maps - e.g., each location in each heatmap is either hot (i.e., "1") or cold (i.e., "0") - or they may have varying temperatures. For heat maps with varying temperatures, the 3D grasp poses may be separated into bins depending on angle - e.g., angles between 0° and 18° may be in a first bin, 19° to 36° in a second bin and so on - with each heatmap corresponding to a bin and thus a temperature based on the specific angle, where the temperature is normalised over the range of angles corresponding to the bin. Notably, each location is each heatmap corresponds to, or is registered to, a pixel in the corresponding input image. There may be a 1-to-l relationship between all pixels in the input image and the heatmap (and, incidentally, the point cloud) such that the heatmap, input image, and point cloud have the same dimensions.

[0035] The number of heat maps generated may also depend on the desired grasp type. For suction cup end effectors, the angle of the grasp pose on the picking plane may be of less importance, and thus a single heatmap may be sufficient to generate the 3D grasp poses. For gripper end effectors, the angle or orientation of the grasp pose is important, since the gripper must be appropriately angled relative to the object to pick the object up, thus multiple heatmaps, each corresponding to a specific angle relative to a common picking plane, are required to generate the 3D grasp poses. For a combination of end-effectors, combinations of groups of heatmaps are required to generate the 3D grasp poses. Thereafter, a Zero-Shot object detection is performed to identify objects to be picked in the image (108). From this, a 3D grasp pose is obtained for each detected object (110).

[0036] When particular ones of the objects are picked, the heatmaps may be executed hierarchically. In this case, 3D grasp poses from the "easiest" heat map (for either a suction cup or the heatmap corresponding to the first bin) are selected first, and then 3D grasp poses from the second easiest heat map (either a particular type of end effector or grasp type, or the second bin) are selected, and so on. Alternatively, or in addition, an object may be picked from a heatmap and then the heatmaps are regenerated to account for movement in the objects between successive picking actions.

[0037] Step 102 involves receiving an input image and a point cloud, each comprising one or more objects. The point cloud comprises points corresponding to a respective surface of one or more said objects to be picked. The image and point cloud are captured by a camera (or derived from the image(s) captured by an image capture device - e.g., camera), and will typically be paired - i.e., the point cloud is a point cloud corresponding to the objects, background and other features shown in the image - and thus referred to as an image pair. The image capture device may be a digital camera, a mobile phone camera, a 3D camera based on the Time-of-flight technology or stereoscopic 3D reconstruction technology. The image and point cloud may be captured by a single device or multiple devices. The image and point cloud may be registered, i.e., a point in the point cloud corresponds to a known point in the image. If the image and point cloud are not registered (i.e. not every point in the point cloud has a correspondingly known point in the image), an additional processing step may be required to perform the registration before the disclosed method can be applied. Each point in the point cloud corresponds to a surface of one or more objects. The one or more objects may be an object to be picked, or it may be an uninteresting object. An uninteresting object is an object that is not intended to be picked - for example, a background object or a container containing the objects to be picked. Each point in the point cloud has a set of coordinates. The coordinates may reference a common coordinate system, i.e., the camera image frame, in which an end effector is used to pick the objects, also has known or determinable coordinates.

[0038] Step 106 comprises generating a plurality of candidate 3D grasp poses by analysing the heatmaps. This consists of, for each heatmap, identifying high scoring regions and combining the specific angle of the heatmap with a common picking plane (e.g., a plane perpendicular to the camera axis, or a plane tangential to the corresponding region on the point cloud) and / or the specific grasp type of the heatmap.

[0039] In a first component of the generation of candidate 3D grasp poses, regions with high scores in the plurality of heatmaps are identified. For each region, a candidate 3D grasp pose may be identified. The heatmap that the region belongs to may be used to associate a specific grasp type and a specific angle relative to a common picking plane. Figure 4 illustrates the coordinate transformation - the common picking plane is represented by XYZ (130), the grasp pose coordinate frame is represented by xyz (131), and the transformed axis, x', is marked by (132), indicating a projection of x onto the picking plane XY. The relative angle on a common picking plane may be defined as the angle between the x-axis of the grasp projected onto the plane via a coordinate transformation. The transformed axis may be termed x'. The transformation for x' may be given from the following transformation x' = Z x y , where Z is the scene z-axis that may be parallel to the camera's optical axis. The transformation can be derived by the perpendicularity of x' and Z and y.

[0040] In a second component of the generation of candidate 3D grasp poses, a series of surface normals is estimated for the identified high-scoring regions. The surface normals may be estimated as the normal of the nearest K points from the centre of a region. K may take on a value of 30, for a heatmap with size 96 by 160 pixels. An algorithm might be used to perform this estimation, for example, the Principal Component Analysis algorithm. A learning-based method might be used to perform this estimation to account for the imperfections in the point cloud quality. The surface normal may be used as the z-axis of the candidate 3D grasp poses (131). A processing logic may be applied to convert the surface normal to the z-axis of the candidate grasp pose. The z-axis of the grasp is a normal vector at the grasping point. A surface normal for a particular object to be picked would correspond to the normal at the centre of the end effector during picking of the object.

[0041] By considering the projected x'-axis and the z-axis as generated by the above two steps, the coordinate frame, i.e., the 3-axis x-y-z (131), of a candidate 3D grasp pose may be obtained. This may be done by the following transformation: y = z x x'; and x = y x z.

[0042] At this point, the three axes x-y-z of a candidate 3D grasp pose are known.

[0043] Advantageously, by predicting the projection of the 3D grasp pose on a common picking plane instead of predicting the full 3D grasp pose, the degrees-of-freedom of the output space of the neural network may be reduced from 5, as is in comparable methods, to 3. This dimensionality reduction allows high data efficiency in the disclosed training method and faster inference time in the disclosed computer- implemented picking method. Conventionally, the 5 degrees-of-freedom of the output space comprise 3 degrees-of-freedom arising from rotation in 3D space, and 2 degrees-of-freedom arising from positions in 3D space, wherein the depth (z) is excluded due to it being derived from the point cloud. The disclosed method reprojects the grasp into the image plane to obtain an angle of rotation, the rotational degrees-of-freedom are further constrained to a single degree-of-freedom about the normal of the plane. Consequently, dimensions of the output heatmap (i.e. the sum of all heatmaps generated during step 108) can be increased, for denser grasp prediction, while keeping the same speed of predicting a grasp.

[0044] Step 108 comprises running a Zero-Shot object detection to detect the one or more objects to be picked, in the input image. The Zero-Shot object detection may be performed using a Tag2Text, Grounding DINO and MobileSAM models. The detection may also be performed with any zero-shot instance segmentation algorithm.

[0045] Step 110 comprises obtaining, from the plurality of candidate 3D grasp poses and Zero-shot object detection result, for each detected object a 3D grasp pose and a grasp type, e.g., suction cup or two fingers gripper.

[0046] The method 100 may further involve picking one of the one or more objects, based on a criterion for ranking. For example, the object with the grasping surface ranked highest (i.e. easiest to pick) may be picked first. Where there are multiple such objects, the object which is closest to the end effector may be selected first. Any other desired process may instead be used for selecting which object to pick, or the order in which the objects should be picked.

[0047] The method 100 may also involve defining a sequence of objects to be picked - the sequence being the order in which the objects are to be picked. However, in other embodiments the method 100 is run after each pick. This enables the system to account for when objects have moved as a result of other objects having been picked. This also enables the system to identify higher ranked grasping surfaces that have become accessible because of objects that have been picked.

[0048] In some embodiments, the method comprises ranking each candidate grasping surface according to the surface normals corresponding to each detected object.

[0049] In some embodiments, the method comprises ranking each candidate grasping surface according to the distance between the grasping location for each of the one or more objects and the end effector. In some embodiments, a collision check is done during the picking process to ensure collision-free grasping. For each of the plurality of objects with a predicted grasp pose, the opening width of the grasp is increased gradually and concurrently with an approaching distance, until the grasps contact another one of the plurality of objects. This provides an indication of space which a gripper occupies while grasping the object, without collision with a nearby object which is not the selected object to be picked.

[0050] Also disclosed is a method for training a neural network model for picking objects. The method for training the neural network model comprises receiving a plurality of input pairs, each pair comprising an image and a point cloud, each comprising one or more objects to be picked, annotating the input pairs with ground-truth 3D grasp poses (annotations may be performed manually, for the training phase, or using a pre-trained neural network model); and encoding the ground-truth 3D grasp poses into one or more angle heatmaps. By providing a data set of input images and point clouds, and the set of heatmaps representing the ground-truth 3D grasp poses, a neural network model is trained to predict the grasping heatmaps. The training process may also be compatible with any neural network model architectures used for image segmentation. This process may involve training a neural network to learn relationships between each input pair and the corresponding heatmap (or heatmaps) with which it is paired. This model may be further used on a new image and point cloud, to obtain new angle heatmaps and grasp candidates which might be used to further train the neural network model.

[0051] Prior to neural network model training, the input pair is annotated with ground-truth 3D grasp poses that are suitable for one or more said objects to be picked. The annotation may specify the type of grasp. The type of grasp may be one of: suction cup, two-fingered grasping, or multiple-fingered grasping. The annotation process may comprise placing a two-finger gripper 3D model (or a gripper 3D model corresponding to the type of grasp being annotated) on the input pair, wherein the annotated 3D gripper model simulates possible positions taken by a gripper to grasp said objects to be picked. Annotating the point cloud may also comprise placing more than one of a plurality of different grasp types on the point cloud, wherein the different grasp types may be one or more of the following : a suction cup (a one-point contact), an array of suction cups (multiple points of contacts), or four finger grasps. The model may be used to generate grasps for only a specific grasp type or end effector, or it may be used to generate grasps for multiple grasp types (thus requiring grasp type to be part of the annotation, or manually provided as a user input), with the system operating based on the end effector carried by the robotic arm of the system. The ground-truth 3D grasp poses may be arbitrarily placed, or they may be placed based on the locations of points in groups of points in the point cloud - e.g. by identifying a group of points that appear to lie in a common plane or define a common surface and defining the grasp based on that surface, such as being in a plane of a surface normal corresponding to that surface. Annotating the point cloud may be performed using a 3D editor program, e.g., Blender. One such annotation has been presented in Figure 2, where two-finger grasps have been annotated on the point cloud.

[0052] The annotated ground-truth 3D grasp poses are encoded into a plurality of heatmaps. The coordinate frames of the 3D grasp poses are projected onto a common picking plane centred on the grasp positions to calculate relative angles, which are subsequently encoded in one or more heatmaps (Figure 4).

[0053] For each heatmap, a grasp type and a relative angle are encoded within, wherein a score of "1" is assigned to a pixel if a ground-truth 3D grasp pose of the specific type projected to a common image plane (e.g., the picking plane being the logical selection, though another plane may be chosen arbitrarily) centred about the grasp orientation has the corresponding pixel position and the corresponding relative angle. A score of "0" is assigned otherwise.

[0054] A relative angle is defined from the projection of the coordinate frame of a grasp pose onto a common picking plane. This angle, therefore, represents the projection of a three-dimensional rotation onto a one-dimensional rotation angle about the normal of the common picking plane. For a range of angles spanning from 0° to 180°, the range may be segmented into a plurality of N bins. Each bin may correspond to a heatmap.

[0055] In one example, N takes a value of 18. Here the range of 0° to 180° is divided into 18 bins, each corresponding to an angle taking the central values of {0°, 10°, 20°, ..., 160°, 170°} and the width of 10°. Grasping angles falling in a particular range (e.g., >175° to 5°, >5° to 15°, >15° to 25° and so on) are placed in the corresponding bin - e.g., a grasping angle of 4° will be placed in the 0° bin, and a grasping angle of 39° will be placed in the 40° bin. One or more bins may correspond to an angle heatmap of a predefined size, the size corresponding to the input image and the point cloud. Figure 3 illustrates a series of angle heatmaps corresponding to an input image with three objects to be picked, and wherein the picking angles range of 0° to 180° has been divided into N = 18 bins. The three objects to be picked may generally be described as (i) a cylindrical object standing upright with its circular face facing upward, (ii) a second cuboid object, and (iii) an irregularly shaped bottle with a varying cross-section, and its longitudinal axis lying in the plane of the surface on which it resides.

[0056] Grasp angles and their corresponding locations in the angle heatmap may be calculated by re-projecting the ground-truth 3D grasp poses onto the image plane, then encoded as "1" in the corresponding angle heatmap. To illustrate, a cylindrical object standing upright with its circular face facing upward may be picked from any grasping angle. Consequently, a coordinate on the angle heatmaps corresponding to the cylindrical object would take a temperature value of "1", indicating the presence and position of the cylindrical object in the image. Given that an upright object may be picked from any grasping angle, the point value of "1" would appear on all 18 heatmaps corresponding to picking angles of {0°, 10°, 20°, ..., 160°, 170°} at the location of the centre of the object.

[0057] A second, cuboid object present in the image plane would not be picked from merely any grasping angle. Due to the shape of the object, an optimal picking angle may be about 65° to about 75°. Consequently, a coordinate on the angle heatmap corresponding to the cuboid object would take a temperature value of "1", for the angle heatmap corresponding to a picking angle of 70°, and a temperature value of "0" otherwise.

[0058] A third, irregular object present in the image plane would similarly not be picked from merely any grasping angle. Due to the shape of the object, an optimal picking angle may be about 105° to about 115°. Consequently, a coordinate on the angle heatmap corresponding to the object would take a temperature value of "1", for the angle heatmap corresponding to a picking angle of 110°, and a temperature value of "0" otherwise. By summing all angle heatmaps, a verification that all objects have been annotated may be obtained. A sum of all angle heatmaps (as indicated by the plot titled "Sum heatmap") further verifies that the density of the annotations for each object is appropriate.

[0059] In some embodiments, the method comprises training the neural network model to detect one or more predetermined objects. In this context, a "predetermined object" is a known object as opposed to an unknown object to be detected. Detecting predetermined objects in addition to grasp prediction may be used to improve the accuracy of the method (i.e., predetermined objects may be more readily verified), or to satisfy specific system requirements.

[0060] In some embodiments, the method comprises training the neural network model to detect one or more uninteresting objects. Detecting uninteresting objects may be used to remove inaccurate grasp detection results, thereby improving the accuracy of the method, or to satisfy specific system requirements.

[0061] In some embodiments, one or more data points of input image pairs and heatmaps may be generated using random rotation augmentation, random translation augmentation or random image intensity augmentation of input pairs and heatmaps generated in the previous procedure. Random rotation augmentation is a technique used to change the angles by which the objects appear during training, to account for rotational variance in the object in practice.

[0062] In some embodiments, encoding the grasp further comprises performing a smoothing operation on the heatmaps. This serves to provide smoother heatmaps, for the purpose of encoding valid object picking grasp poses in regions which may be broader than individual pixels in the input image and point cloud pair.

[0063] In some embodiments, for a defined score of "1" on an angle heatmap corresponding to a particular picking angle, the surrounding pixels within a predetermined radius R may also be set at "1". This similarly serves to provide smoother heatmaps, for the purpose of encoding valid object picking grasp poses in regions which may be broader than individual pixels in the input image and point cloud pair. The neural network model may subsequently be trained to predict the heatmap from an input image and a point cloud. The deep learning model may be evaluated on, and subsets of the annotated data may be tested to identify an optimal model.

[0064] Disclosed herein is also a system for picking objects. The system comprises an input device for receiving a plurality of input pairs, each pair comprising an input image and a corresponding point cloud. Each input pair comprises one or more objects to be picked. The point clouds further comprise points corresponding to a respective surface of the one or more said objects to be picked. The input device may receive the input image from an image capture device, or it may be the image capture device used to capture the images. The input device may also receive the point cloud image as generated by 3D scanners or reconstruction methods, or it may receive a 3D image generated via a neural network model. The system also comprises an annotating module, for producing annotations on the point cloud by adding ground-truth 3D grasp poses that are suitable for picking the objects to be picked. The annotation may specify the type of grasp for each 3D grasp pose, e.g., suction cup, two-fingered grasping or multiple-fingered grasping. The system also comprises an encoding module, for encoding the 3D grasp poses into one or more of the heatmaps, wherein the heatmaps encode grasp type and grasp orientation projected onto a common image plane centred about the grasp position. The system may also comprise a training module for training neural networks models to predict relationships between input pairs and heatmaps. The system may also comprise a neural network model for generating heatmaps from the input pairs of images and point clouds. The system further comprises a grasp generation module for generating 3D grasp poses by analysing the heatmaps to identify high-scoring regions with a specific grasp angle and grasp type and by calculating surface normals at these respective regions on the point cloud. The system further comprises a detection module for performing Zeroshot object detection to detect the one or more objects in the input image and an end effector for picking detected objects.

[0065] It will be appreciated that many further modifications and permutations of various aspects of the described embodiments are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Throughout this disclosure, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps, but not the exclusion of any other integer or step or group of integers or steps.

[0066] The reference to any prior art in this disclosure is not, and should not be taken as, an acknowledgement or any form of suggestion that the prior art forms part of the common general knowledge.

Claims

CLAIMS1. A computer-implemented method for picking objects, comprising : receiving an input image and a point cloud, each comprising one or more objects, the point cloud comprising points corresponding to a respective surface of one or more said objects to be picked; using a neural network model on the input image and the point cloud to produce a plurality of heatmaps, each heatmap encoding at least one of a grasp angle for picking a said object relative to a common picking plane, and a grasp type; generating a plurality of candidate 3D grasp poses by analysing the plurality of heatmaps to identify high-scoring regions with a specific said grasp angle, and grasp type; performing Zero-Shot object detection to detect the one or more objects to be picked, in the input image; obtaining, from the plurality of candidate 3D grasp poses, a 3D grasp pose for each detected object to be picked; and picking one or more of the said objects to be picked by moving an end effector to the obtained 3D grasp poses.

2. A method for training a model for picking objects, comprising : receiving a plurality of annotated inputs, each input comprising an input image, a point cloud each comprising one or more objects to be picked, and ground-truth 3D grasp poses corresponding to the one or more objects, the respective point cloud further comprising points corresponding to a respective surface of one or more said objects to be picked; encoding each ground-truth 3D grasp pose into a heatmap, each heatmap representing a grasp type and grasp angle, relative to a common picking plane, of respective ground-truth 3D grasp poses; training a neural network model, based on the inputs and respective heatmaps, to predict heatmaps for a new input; and evaluating the trained neural network model on validation and test subsets of the annotated data set to identify an optimal model.

3. The method according to claim 2, further comprising training the neural network model to detect one or more objects to be picked.

4. The method according to any one of claim 1 to claim 3, wherein generating one or more heatmaps is done by random rotation augmentation, random translation augmentation or random image intensity augmentation.

5. The method according to any one of claims 2 to 4, wherein encoding a grasp angle and a grasp type comprises performing Gaussian smoothing for reducing discretisation of each heatmap.

6. The method according to any one of claims 1, 4 or 5, wherein the method further comprises ranking each candidate grasping surface, according to the grasp angle relative to the common picking plane, corresponding to each object to be picked.

7. The method according to any one of claims 1 or 4 to 6, wherein the method further comprises ranking each candidate 3D grasp pose, according to the distance between a grasp location for each of the one or more objects to be picked and the end effector.

8. The method according to any one of claims 1 to 7, wherein the method further comprises, for each of the plurality of objects with the corresponding grasp location, increasing a grasp opening width and decreasing an approaching distance until the grasp contacts another object of the plurality of objects.

9. The method according to any one of claims 1 to 8, wherein picking objects is carried out with a two-finger grasp.

10. The method according to any one of claims 1 to 9, wherein picking objects is carried out with a single suction cup, a single multi-fingered gripper or a combination of suction cups and gripper.

11. The method according to any one of claims 1 to 10, wherein obtaining a grasp location based on each of the candidate grasping surfaces comprises obtaining two selected points on opposite sides of the detected object to be picked.

12. The method of claim 2, further comprising, prior to receiving the annotated inputs, annotating each pixel of the input image or point of the point cloud by specifying one or both of a 3D grasp angle and grasp type.

13. A system for picking objects, comprising: an input device for receiving a plurality of input pairs, each pair comprising an input image and a point cloud, the input image and the point cloud each comprising one or more objects to be picked, therespective point cloud comprising points corresponding to a respective surface of one or more said objects to be picked; an annotating module, for producing annotations on the point cloud by adding 3D grasp poses that are physically suitable for picking the objects to be picked, the annotation specifying a grasp type for each 3D grasp pose; an encoding module, for encoding the 3D grasp poses into one or more of the heatmaps, each 3D grasp pose comprising a grasp orientation projected onto a common image plane centred about a grasp position to calculate a grasp angle, for the 3D grasp pose, that is encoded into the one or more heatmaps; a neural network module comprising a neural network model for generating a plurality of heatmaps from the input pairs; a training module for training the neural network model based on training inputs, each training input comprising an input image and corresponding point cloud, wherein at least one of the input image and point cloud is annotated with 3D grasp poses; a grasp generation module, for calculating a surface normal corresponding to each 3D grasp pose and selecting one or more of the 3D grasp poses based on the respective one or more heatmaps, grasp angle and grasp type; a detection module for performing Zero-shot object detection to detect the one or more objects to be picked; and an end effector for picking objects to be picked.

14. The system according to claim 13, wherein each candidate 3D grasp pose is ranked according to the grasp angle relative to the common picking plane, and a distance between a grasp location for each of the one or more objects to be picked and the end effector.

15. The system according to claim 13 or claim 14, wherein the end effector is a two-finger grasper.

16. The system according to any one of claims 13 to 15, wherein the encoding module further performs Gaussian smoothing for reducing discretisation of each heatmap.

Images