Method for non-prehensile manipulation of an object

Abstract

A computer-implemented method for controlling a robotic picking system 200 for the non-prehensile manipulation an object 252 within a workspace or container space 254 is disclosed in which the robotic

Description

The present disclosure generally relates to the non-prehensile manipulation of an object using a robotic manipulator. Some aspects of the disclosure relate to a method of controlling a robotic manipulator for the non-prehensile manipulation of an object and a robotic picking system comprising the robotic manipulator. BACKGROUND Robotic manipulators having a parallel jaw gripper can be used to pick a vast range of stock keeping units (SKUs) from bins, containers or shelves. However, the success of a pick depends on the arrangement of the SKUs in the container. Benchmark data has shown that most grasp failures occur when the SKU arrangement requires extra manipulations within the container. In some cases, a grasp may not even be attempted due to a lack of space between SKUs. It is against this background that the invention has been devised. SUMMARY The invention accordingly provides, in a first aspect, a computer-implemented method for controlling a robotic picking system for the non-prehensile manipulation of an object within a workspace (e.g., a container space). The robotic picking system comprises an imaging device and a robotic manipulator. In one implementation, the robotic manipulator comprises an end effector assembly having two finger elements (e.g., a parallel jaw gripper) and a force-sensing arrangement for sensing forces applied to respective finger elements. In other implementations, the robotic manipulator comprises an end effector assembly having a single element with which to manipulate the object. In this case, the force-sensing arrangement will be able to sensing force applied to different sections of the single element. The method comprises the steps of obtaining image data representative of the object within the container space; determining principal components the object in dependence on the image data; determining a direction in which to push the object in dependence on the principal components; determining a start position for the end effector in dependence on the direction; and, controlling the robotic manipulator to: move the end effector from the start position to bring the finger element into engagement with the object; and, adjust the pose of the end effector prior to initiating a push manoeuvre in response to a force detected by the force-sensing arrangement. Optionally, the method further comprises determining a contour of the object in dependence on the image data; and, determining the centroid of the object in dependence on the contour of the object. Optionally, the method further comprises determining a boundary of the space; determining axes along which to move the object in dependence on the principal components; and, determining notional collision points of the object along the axes in dependence on the boundary of the container space. Optionally, the method further comprises determining contour lines of other objects within the container space in dependence on the image data; and, determining notional collision points of the object along the axes in dependence on the contour line of the other objects. Optionally, the method further comprises selecting from the axes, an axis with the fewest notional collision points. Optionally, the method further comprises determining relative distances between the notional collision points on the axes; and, selecting from the axes, the axis with the largest distance between two consecutive notional collision points. Optionally, the method further comprises determining the position of the centroid along the selected axis; and, determining the direction in which to push the object in dependence on the position of the centroid. Optionally, the method further comprises determining a boundary of the container space; determining axes along which to move the object in dependence on the principal components; and, determining empty areas within the container space on each side of the object along the axes in dependence on the boundary of the container space. Optionally, the method further comprises determining contour lines of other objects within the container space in dependence on the image data; and, determining empty areas within the container space on each side of the object along the axes in dependence on the boundary of the container space and the contour lines of the other objects. Optionally, the method further comprises selecting from the axes, the axis with empty areas on opposing sides of the object. Optionally, the method further comprises selecting from the axes, the axis having a comparatively greater portion of its length extending through empty areas. Optionally, the method further comprises determining the direction in which to push the object in dependence on the size of the empty areas associated with the selected axis. Optionally, the method further comprises controlling the robotic manipulator to rotate the end effector in response to the force-sensing arrangement detecting a force on only one finger element. Optionally, the method further comprises controlling the robotic manipulator to rotate the end effector in a direction so as to reduce the magnitude of a force component extending perpendicular to a gripping surface of the one finger element. Optionally, the method further comprises controlling the robotic manipulator to stop rotating the end effector in response to the force-sensing arrangement detecting forces on both finger elements. Optionally, the method further comprises controlling the robotic manipulator to move the end effector laterally in response to the force-sensing arrangement detecting a higher force on one of the finger elements when compared to the force applied to the other of the finger elements. Optionally, the method further comprises controlling the robotic manipulator to move the end effector laterally in the direction of the higher force. Optionally, the method further comprises determining the centre of motion / centre of reaction forces of the object in dependence on the forces detected by the force-sensing arrangement; and, controlling the robotic manipulator to move the end effector laterally relative to the centre of motion to effect rotation of the object during the push manoeuvre. In another aspect, there is provided a controller for a robotic manipulator, wherein the controller is configured to perform the computer-implemented method of the first aspect. In another aspect, there is provided, a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the computer-implemented method of the first aspect. In another aspect, there is provided, a computer-readable data carrier having stored thereon the computer program of the previous aspect. In yet another aspect, there is a robotic picking system comprising the controller of an earlier aspect and a robotic manipulator for picking an object. BRIEF DESCRIPTION OF THE DRAWINGS These and other aspects of the invention will now be described, by way of example only, with reference to the accompanying drawing, in which: FIG. 1 is a schematic diagram of a robotic sorting / picking system according to an embodiment; FIG. 2 is a schematic diagram of another robotic picking system according to an embodiment; FIGs. 3a-d are schematic diagrams of a view captured by an overhead imaging device of the robotic picking system of FIG. 2; FIGs. 4a-d illustrate examples of how the robotic picking system of FIG. 2 might be used in the non-prehensile manipulation of an object; FIG. 5 shows a flowchart depicting a computer-implemented method of controlling the robotic picking system of FIG. 2 for the non-prehensile manipulation of an object; FIG. 6 shows a flowchart depicting a sub-method that might be used in the method of FIG. 5; FIG. 7 shows a flowchart depicting a sub-method that might be used in the method of FIG. 5; FIG. 8 shows a flowchart depicting a sub-method that might be used in the method of FIG. 5; FIG. 9 shows a flowchart depicting a sub-method that might be used in the method of FIG. 5; and, FIG. 10 shows a flowchart depicting a sub-method that might be used in the method of FIG. 5; In the drawings, like features are denoted by like reference signs where appropriate. DETAILED DESCRIPTION In the following description, some specific details are included in the following description to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognise that embodiments may be practised without one or more of these specific details or with other methods, components, materials, etc. In some instances, well-known structures associated with gripper assemblies and / or robotic manipulators (such as processors, sensors, storage devices, network interfaces, workpieces, tensile members, fasteners, electrical connectors, and the like) are not shown or described in detail to avoid unnecessarily obscuring descriptions of the disclosed embodiments. Unless the context requires otherwise, the word “comprise” and its variants like “comprises” and “comprising” are to be construed in this description and appended claims in an open, inclusive sense, i.e., as “including, but not limited to”. Reference throughout this specification to “one”, “an”, or “another” applied to “embodiment” or “example”, means that a particular referent feature, structure, or characteristic described in connection with the embodiment, example, or implementation is included in at least one embodiment, example, or implementation. Thus, the appearances of the phrase “in one embodiment” or the like in various places throughout this specification do not necessarily refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, examples, or implementations. It should be noted that, as used in this specification and the appended claims, the used forms “a”, “an”, and “the” include plural referents unless the content clearly dictates otherwise. It should also be noted that the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise. Regarding FIG. 1, there is illustrated an example of a robotic picking system 100 that may be adapted for use with the present assemblies, devices, and methods. The robotic picking system 100 may form part of an online retail operation, such as an online grocery retail operation. Still, it may also be applied to any other operation requiring the picking and packing of items. For example, the robotic picking system 100 may also be adapted for picking and / or sorting apparel and might sometimes be referred to as a “pick and place robot”. The robotic picking system 100 includes a manipulator apparatus 102 comprising a robotic manipulator 121. The manipulator 121 is an electro-mechanical machine comprising one or more appendages, such as a robotic arm 120, and an end effector 122 mounted on an end of the robotic arm 120. The end effector 122 is a device configured to interact with the environment in order to perform tasks, including, for example, gripping, grasping, releasably engaging or otherwise interacting with an item. In the present example, the end effector 122 is an assembly comprising at least two finger elements, such as a parallel jaw gripper, and a force-sensing arrangement for sensing forces applied to respective finger elements. The robotic manipulator 121 can manipulate an object (i.e., rearrange a storage space / workplace in a specific manner, e.g., create an opportunity for picking or prepare a part for an assembly step, etc.) and, in some cases, subsequently grasp the object. In the case of a pick and place application, the robotic manipulator 121 is configured to pick an item or object from a first location and place the item in a second location, for example. The manipulator apparatus 102 is communicatively coupled via a communication interface 104 to other components of the robotic picking system 100, e.g., one or more optional operator interfaces 106 from which an observer may observe or monitor the system 100, including the manipulator apparatus 102. The operator interfaces 106 may include a WIMP interface and an output display of explanatory text or a dynamic representation of the manipulator apparatus 102 in a context or scenario. For example, the dynamic representation of the manipulator apparatus 102 may include a video feed, for instance, a computer-generated animation. Examples of suitable communication interface 104 include a wire-based network or communication interface, an optical-based network or communication interface, a wireless network or communication interface, or a combination of wired, optical, and / or wireless networks or communication interfaces. The example robotic picking system 100 also includes a control system 108, including at least one controller 110 communicatively coupled to the manipulator apparatus 102 and any other components of the robotic picking system 100 via the communication interface 104. The control system 108 may further comprise a machine learning system communicatively coupled to the controller 110. The controller 110 comprises a data processing apparatus or computational device having one or more electronic processors. Embedded within the one or more processors is computer software comprising a set of control instructions provided as processor-executable data that, when executed, cause the controller 110 to issue actuation commands or control signals to the manipulator system 102. For example, the actuation commands or control signals cause the manipulator 121 to carry out various methods and actions, such as identifying and manipulating items. The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), programmed logic units (PLUs), or the like. In some implementations, the controller 110 is a smaller processor-based device like a mobile phone, single-board computer, embedded computer, or the like, which may be termed or referred to interchangeably as a computer, server, or analyser. The set of control instructions may also be provided as processor-executable data associated with the operation of the system 100 and manipulator apparatus 102 included in a non-transitory computer-readable storage device 112, which forms part of the robotic picking system 100 and is accessible to the controller 110 via the communication interface 104. In some implementations, the storage device 112 includes two or more distinct devices. The storage device 112 can, for example, include one or more volatile storage devices, e.g., random access memory (RAM), and one or more non-volatile storage devices, e.g., readonly memory (ROM), flash memory, magnetic hard disk (HDD), optical disk, solid-state disk (SSD), or the like. A person of skill in the art will appreciate storage may be implemented in a variety of ways such as a read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other forms of computer- and processor-readable memory or storage medium, and / or a combination thereof. Storage can be read-only or read-write as needed. The robotic picking system 100 includes a sensor subsystem 114 comprising one or more sensors that detect, sense or measure conditions or states of the manipulator apparatus 102 and / or conditions in the environment or workspace in which the manipulator 121 operates and produce or provide corresponding sensor data or information. Sensor information includes environmental sensor information, representative of environmental conditions within the workspace of the manipulator 121, as well as information representative of condition or state of the manipulator apparatus 102, including the various subsystems and components thereof, and characteristics of the item or object to be manipulated. The acquired data may be transmitted, via the communication interface 104, to the controller 110 for directing the manipulator 121 accordingly. Such information can, for example, include diagnostic sensor information that is useful in diagnosing a condition or state of the manipulator apparatus 102 or the environment in which the manipulator 121 operates. Such sensors include, for example, one or more cameras or imagers 116 (e.g., responsive within visible and / or non-visible ranges of the electromagnetic spectrum including, for instance, infrared and ultraviolet). The one or more cameras 116 may include an RGB-D camera, e.g., a stereo camera, to capture depth data alongside colour channel data in an imaged scene, or a software-based (or Al-based) monocular camera producing depth sensing or an Al-based stereo camera. Other sensors of the sensor subsystem 114 may include one or more of: contact sensors, force sensors, strain gages, vibration sensors, position sensors, attitude sensors, accelerometers, radars, sonars, lidars, touch sensors, pressure sensors, load cells, microphones 118, meteorological sensors, chemical sensors, or the like. In some implementations, the sensors include diagnostic sensors to monitor a condition and / or health of an on-board power source within the manipulator apparatus 102 (e.g., a battery array, ultra-capacitor array, or fuel cell array). In some implementations, the one or more sensors comprise receivers to receive position and / or orientation information concerning the manipulator 121. For example, a global position system (GPS) receiver to receive GPS data, two or more time signals for the controller 110 to create a position measurement based on data in the signals, such as time-of-flight, signal strength, or other data to affect a position measurement. Also, for example, one or more accelerometers, which may also form part of the manipulator apparatus 102, could be provided on the manipulator 121 to acquire inertial or directional data, in up to six axes, e.g., translational and rotational, regarding the movement thereof. The robotic manipulator 121 may be piloted by a human operator at the operator interface 106. Ina human operator-controlled (or “piloted”) mode, the human operator observes representations of sensor data, e.g., video, audio, or haptic data received from the one or more sensors of the sensor subsystem 114. The human operatorthen acts, conditioned by a perception of the representation of the data, and creates information or executable control instructions to direct the manipulator 121 accordingly. In the piloted mode, the manipulator apparatus 102 may execute control instructions in real-time (e.g., without added delay) as received from the operator interface 106 without taking into account other control instructions based on the sensed information. In some implementations, the manipulator apparatus 102 operates autonomously, i.e., without a human operator creating control instructions at the operator interface 106 for directing the manipulator 121. The manipulator apparatus 102 may operate in an autonomous control mode by executing autonomous control instructions. For example, the controller 110 can use sensor data from one or more sensors of the sensor subsystem 114. The sensor data is associated with operator-generated control instructions from one or more times during which the manipulator apparatus 102 was in the piloted mode to generate autonomous control instructions for subsequent use. For example, deep learning techniques can be used to extract features from the sensor data. Thus, in the autonomous mode, the manipulator apparatus 102 can autonomously recognise features or conditions of its environment and the item to be manipulated. In response, the manipulator apparatus 102 performs one or more defined acts or tasks. For example, the manipulator apparatus 102 performs a pipeline or sequence of acts or tasks. In some implementations, the controller 110 autonomously recognises features or conditions of the environment surrounding the manipulator 121 and one or more virtual items composited into the environment. The environment is represented by sensor data from the sensor subsystem 114. In response to being presented with the representation, the controller 110 issues control signals to the manipulator apparatus 102 to perform one or more actions or tasks. In some instances, the manipulator apparatus 102 may be controlled autonomously at a given time while being piloted, operated, or controlled by a human operator at another time. That is, the manipulator apparatus 102 may operate under the autonomous control mode and change to operate under the piloted (i.e., non-autonomous) mode. In another mode of operation, the manipulator apparatus 102 can replay or execute control instructions previously carried out in the piloted mode. That is, the manipulator apparatus 102 can operate based on replayed pilot data without sensor data. The manipulator apparatus 102 further includes a communication interface subsystem 124 (e.g., a network interface device) communicatively coupled to a bus 126 and which provides bi-directional communication with other components of the system 100 (e.g., the controller 110) via the communication interface 104. The communication interface subsystem 124 may be any circuitry affecting bidirectional communication of processor-readable data and processor-executable instructions, such as radios (e.g., radio or microwave frequency transmitters, receivers, transceivers) ports, and / or associated controllers. Suitable communication protocols include FTP, HTTP, Web Services, SOAP with XML, cellular (e.g., GSM, CDMA), Wi-Fi® compliant, Bluetooth® compliant, and the like. The manipulator apparatus 102 further includes a motion subsystem 130, communicatively coupled to the robotic arm 120 and end effector 122, and the controller 110 via the bus 126. The motion subsystem 130 comprises one or more motors, solenoids, other actuators, linkages, drive-belts, or the like operable to cause the robotic arm 120 and / or end effector 122 to move within a range of motions in accordance with the actuation commands or control signals issued by the controller 110. The actuation commands are processorexecutable motion instructions or data that cause the robotic arm 120 and end effector 122 to move within its environment using the components in the motion subsystem 130. For example, processor-executable motion instructions may aid the robotic arm 120 in performing: motion plan creation, inverse kinematics, or other motion related tasks. Processor-executable motion instructions or data may include forward kinematic instructions, which, when executed, controls at least one of the robotic arm 120 or end effector 122 through specification of joint positions (e.g., angles, displacements). Processor-executable motion instructions or data may include inverse kinematic instructions, which, when executed, controls at least one of the robotic arm 120 or end effector 122 by calculations of joint positions given a specification of a position of the robotic arm 120 or end effector 122. Processor-executable motion instructions or data may implement, in part, various methods described herein, including those in and in relation to FIGs. 5 to 10. The manipulator apparatus 102 also includes an output subsystem 128 comprising one or more output devices, such as speakers, lights, or displays that enable the manipulator apparatus 102 to send signals into the workspace to communicate with, for example, an operator and / or another manipulator apparatus 102. - see 0068 A person of ordinary skill in the art will appreciate the components in manipulator apparatus 102 may be varied, combined, split, omitted, or the like. In some examples, one or more of the communication interface subsystem 124, the output subsystem 128, and the motion subsystem 130 are combined. In other instances, one or more subsystems (e.g., the motion subsystem 130) are split into further subsystems. FIG. 2 shows an example of a robotic picking system 200 including an implementation (designated by 221) of the robotic manipulator 121 described in previous examples. In accordance with such examples, the robotic manipulator 221 includes a robotic arm 220, an end effector 222, and a motion subsystem 230. The motion subsystem 230 is communicatively coupled to the robotic arm 220 and end effector 222 and configured to cause the robotic arm 220 and / or end effector 222 to move in accordance with actuation commands or control signals issued by a control system 232. The control system 232 may form part of a robotic manipulator apparatus, together with the robotic manipulator 221, or part of the wider robotic picking system 200, e.g., like control system 108 described in previous examples. The robotic picking system 200 may be implemented in an automated storage and retrieval system (ASRS), e.g., in a picking station thereof. An ASRS typically includes multiple containers arranged to store items and one or more load-handling device or automated guided vehicle (AGV) to retrieve one or more containers 244 during fulfilment of a customer order. At a picking station, items are picked from and / or placed into the one or more retrieved containers 244. The one or more containers in the picking station may be considered as being either storage or delivery containers. A storage container is a container which remains within the ASRS and holds eaches of products which can be transferred from the storage container to a delivery container. A delivery container is a container that is introduced into the ASRS when empty and that has a number of different products loaded into it as part of fulfilling a customer delivery. The robotic picking system 200 can therefore be used to pick an item from one container, e.g., a storage container, and place the item into another container, e.g., a delivery container, at a picking station. The picking station may thus have two sections: one section for the storage container and one for the delivery container. The arrangement of the picking station, e.g., the sections thereof, can be varied and selected by the skilled person. For example, the two sections may be arranged on two sides of an area or with one section above or below the other. In some cases, the picking station is located away from the storage locations of the containers in the ASRS, e.g., away from the storage grid in a gridbased ASRS. The load handling devices may therefore deliver and collect the containers to / from one or more ports of the ASRS which are linked to the picking station, e.g., by chutes. In other instances, the picking station is located to interact directly with a subset of storage locations in the ASRS, e.g., to pick and place items between containers located at the subset of storage locations. For example, in the case of a grid-based ASRS, the picking station may be located on the grid of the ASRS. Like with the previous robotic manipulator 121, the robotic manipulator 221 of this example comprises an end effector 222 comprising at least two finger elements 222a, 222b, such as a parallel jaw gripper, and a force-sensing arrangement (not shown) for sensing forces applied to respective finger elements 222a, 222b during the end effector’s 222 interaction with its workspace and any product or object being manipulated. The robotic picking system 200 of FIG. 2 also includes an imaging device or camera 216 positioned above the workspace of the robotic manipulator 221 and arranged to capture images of objects within the container 244. In this example, the overhead imaging device 216 is supported by a frame structure 240, which is shown in a simplified form, but could take any suitable structural form as will be appreciated by the skilled person. Another camera might be mounted to the robotic manipulator 221 to capture images of objects within the container 244 as an alternative or in addition to the overhead imaging device 216. Accordingly, the imaging device 216, as shown, may correspond to the one or more cameras or imagers 116 in the sensor subsystem 114 of the robotic picking system 100 described with reference to FIG. 1. The control system 232 comprises at least one controller 234 communicatively coupled, via a communication interface, to the robotic manipulator 221, imaging device 216, and any other components of the robotic picking system 200. The control system 232 further comprises a machine learning system (not shown) that is communicatively coupled to the controller 234. The machine learning system may include a number of machine learning services including a model for instance segmentation, such as a Mask Region-based Convolutional Neural Network (Mask R-CNN), for detecting objects in an image and generating a segmentation mask for each instance (i.e., object). The controller 234 comprises a data processing apparatus or computational device having one or more electronic processors. Embedded within the one or more processors is computer software comprising a set of control instructions provided as processor-executable data that, when executed, cause the controller 234 to issue actuation commands or control signals to the system 200. For example, the actuation commands or control signals may cause the manipulator 221 to carry out various actions, such as the manipulation of an object while it is grasped by the end effector 222. The end effector 222 might also be used for a non-prehensile manipulation of an object within the container 244 in order to move the object into a position where it can be more easily grasped. For example, an object might be too close to other objects within the container 244, preventing access for the end effector 222 to grasp the object and risking damage to neighbouring objects. In this situation, the end effector 222 might be used to push the object, moving it clear of the other objects, to create a comparatively better grasping opportunity. To that end, the controller 234 is configured to obtain image data representative of an image 250 captured by the overhead imaging device 216 of objects 252, 253, 256 within the container space 254, an example of which is shown in FIG. 3a. In this example, the image 250 shows three elongated objects 252, 253, 256 within the container space 254, but the skilled person will understand that the number and shape of objects is illustrative and that the invention can be practised with fewer or more than three objects. With reference to FIG. 3b, if, for example, the end effector 222 is to be used for the non-prehensile manipulation of object 252, a start position for the end effector 222 must first be determined. To that end, the centroid 252c of the object 252 may be determined, by the controller 234, in dependence on its boundary or contour. In this case, the controller 234 processes the image data using the model for instance segmentation to detect each object 252, 253, 256 within the image 250 and generates segmentation masks to establish the boundary or contour of each object 252, 253, 256. The contour of the object 252 is then used, by the controller 234, to determine its centroid 252c. For example, in the present example, in which the object 252 appears in the image 250 as a rectangle, although it is of course cubical, the controller 234 may first determine the length and width of the object 252 based on its respective boundary or contour, and then ascertain that the centroid 252c of the object 252 is located at a position halfway along its respective length and width. In another example, the controller 234 uses the Mask R-CNN algorithm to provide a mask and contour for each segmented object 252 in the workspace or container space 254. The centroid 252c of the segmented object 252 is computed by averaging all the points in the corresponding mask. These are just a couple of examples of a method by which the controller 234 may determine the centroid of an object. Other ways of calculating centroids of different shapes will be known to the skilled reader but will not be shown or described in detail to avoid unnecessarily obscuring the disclosure. Having determined the centroid 252c, the controller 234 then determines the principal components of the object 252. The principal components can be defined by the major and minor axes 252a, 252b of the object 252. For example, the major axis 252a might be defined by a longitudinal axis of the object 252, i.e., an axis along the lengthwise direction of the object 252, which passes through the centroid 252c. The major axis 252a can also be defined by endpoints of the longest line that can be drawn through the object 252 represented in the image 250. The major axis endpoints, e.g., pixel coordinates (x1, y1) and (x2, y2) in the image 250, are found by the controller 234 computing the pixel distance between every combination of border pixels in the boundary or contour of the object 252 and finding the pair with the maximum length, for example. Similarly, the minor axis 252b could be defined by an axis that also passes through a centroid 252c of the object 252, but extending perpendicular with respect to the longitudinal axis or by the endpoints of the shortest line that can be drawn through the object 252. The points in the mask provided by the Mask R-CNN algorithm can also be used to compute the principal components of the object 252 using Principal Components Analysis (PCA). Once the principal components have been computed, the points in the mask with minimum and maximum coordinates along the principal axes determine the dimensions of the object 252 along its principal axes. A direction in which to push the object 252 is then determined, by the controller 234, in dependence on the principal components 252a, 252b. In determining the direction, the controller 234 may first determine a boundary 258 of the container space 254. The boundary 258 could be a predetermined boundary that is accessible to the controller 234, necessitating the assumption that the container 244 is always in the same position within the image 250. Alternatively, the controller 234 processes the image data using the model for instance segmentation to detect the container 244 within the image 250 and generate a segmentation mask to establish the boundary 258. The controller 234 then determines candidate axes along which to move the object 252 in dependence on the principal components 252a, 252b. With reference to FIG. 3c, the controller 234 may form the candidate axes 252d, 252e by lengthening the principal components 252a, 252b to the boundary 258 of the container space 254. Notional collision points of the object 252 along the axes 252d, 252e are then determined by the controller 234. Firstly, notional collision points 260, 262, 264, 266 are identified by the controller 234 at the points at which the axes 252d, 252e and the boundary 258 of the container space 254 intersect. For example, the controller 234 may process the image data to identity 3D points which can be mapped to colour or depth image pixels, or points in a point cloud, to determine pixels of the candidate axes 252d, 252e that are adjacent pixels forming the boundary 258 of the container space 254, and label those pixels of the axes 252d, 252e as notional collision points 260, 262, 264, 266. The controller 234 is further configured to determine further notional collision points in dependence on the objects 253, 256 within the container space 254, i.e., at the points at which the axes 252d, 252e traverse the contours or boundaries of the other objects 253, 256. In this instance, the controller 234 might process the image data to identify pixels of the candidate axes 252d, 252e that are adjacent to the border pixels of the boundary or contour of the objects 253, 256, and label those pixels of the axes 252d, 252e as notional collision points. In this example, a further four collision points 270, 272, 274, 276 would be identified, all located along axis 252e. The controller 234 is further arranged to select, from the candidate axes 252d, 252e, the most appropriate axis along which to move the object 252. To that end, the controller 234 may select the candidate axis that is associated with the fewest notional collision points, i.e., candidate axis 252d, which is associated with two notional collision points 260, 264. Alternatively, or in the event that the candidates axes 252d, 252e are each associated with the same number of notional collision points, the controller 234 may be further configured to process the image data to determine the relative distances between the notional collision points along each axes (that is, the relative distance, based on the number of pixels, between notional collision points 260, 264 along candidate axis 252d and the distances between points 262, 266, 270, 272, 274, 276 on axis 252e) and select the candidate axis with the largest distance between two consecutive notional collision points. Under this criteria, the controller 234 would select candidate axis 252d as its associated notional collision points 260, 264 are separated by a distance greater than the distances separating the notional collision points 262, 266, 270, 272, 274, 276 along the other candidate axis 252e. Having selected the appropriate candidate axis 252d, the controller 234 is further arranged to determine a direction in which to push the object 252. The controller 234 achieves this by first determining the position of the centroid 252c along selected candidate axis 252d. For example, the controller 234 may process the image data to determine a midpoint 280 of the axis 252d, based on the number of pixels forming the axis 252d, and establish which side of the midpoint 280 the centroid 252c is located. The controller 234 is configured to then determine a direction in which to move the object 252 along the selected axis 252d that, at least initially, moves the centroid 252c towards the midpoint 280 as indicated by arrow 282. Alternatively, instead of determining the midpoint 280 of the axis 252d, the controller 234 may process the image data to determine the distances along the selected axis 252d either side of the centroid 252c between the centroid 252c and the boundary 258 of the container space 254. The controller 234 is configured then to determine a direction 282 in which to move the object 252 along the selected axis 252d that would decrease the larger of the two distances either side of the centroid 252c. The length of the push is a factor of the length of the object along the chosen principal component. For example, if the factor is 0.5, the length of the push is the half of the length of the object along the chosen principal component. Having established the direction 282 in which to move the object 252, the controller 234 is further configured to determine the start position for the end effector 222 in dependence on the direction 282. This may be carried out by the controller 234 by identifying a section 284 of the boundary of the object 252 that is substantially perpendicular to the direction 282 in which the object 252 is to be pushed and designating an area 286 of the container space 254 adjacent to the section 284 as the start position. The controller 234 may alternatively assess space around the object 252 when determining the direction in which to move it instead of basing the decision on notional collision points. To that end, and with reference to FIG. 3d, the controller 234 is configured to determine the boundary 258 of the container space 254 and the candidate axes 252d, 252e along which to move the object 252 as described above. The controller 234 is then further configured to determine empty areas within the container space 254 adjacent to each side of the object 252 along the axes 252d, 252e in dependence on boundaries within or defining the container space 254. In order to do this, the controller 234 is configured to process the image data to identify pixels within the container space 254 along the axes 252d, 252e that are adjacent to the border pixels forming the boundary of the object 252 and any other pixels between those adjacent pixels and pixels forming another boundary, i.e., the boundary 258 of the container space 254 or the boundaries of other objects 253, 256 within the container space 254. The identified pixels are then labelled by the controller 234 as forming an empty area or space with the container space 254 adjacent to the object 252. In this example, three empty areas 261, 263, 265 of the container space 254 were identified by the controller 234: two areas 261, 263 either side of the object 252 along candidate axis 252d and, because of the presence of the neighbouring object 253 and its associated boundary, only one area 265 along candidate axis 252e. The controller 234 is configured then to select an axis from the candidate axes 252d, 252e with empty areas on opposing sides of the object 252. In the present example, the controller 234 would select candidate axis 252d as it traverses two empty areas 261, 263 located at opposing sides of the object 252. In the event that both axes 252d, 252e traverse empty spaces at opposing sides of the object 252, the controller 234 is configured to select from the candidate axes 252d, 252e, the axis 252d having a comparatively greater portion of its length extending through the empty spaces 261,263. One could consider ranking all the suitable pushes according to how empty their corresponding squares are. This would indicate a lower chance of collisions. The controller 234 is configured then to determine a direction 282 in which to push the object 252 along the selected axis 252d in dependence on the size of the empty areas 261, 263 associated with the axis 252d. For example, the controller 234 might determine the direction 282 as one that, when executed, would move the centroid 252c towards the larger of the two areas 261, 263. Alternatively, the controller 234 might determine the direction 282 as one that, when executed, would decrease the size of the larger area 261. Alternatively, as mentioned above, the principal components can be computed using PCA to determine the dimensions of the object 252 along its principal axes. These dimensions can then be used to compute the position for a square before the object 252 and for a square after the object. In order to simplify the search space for suitable pushes, it is assumed that the line of the push is parallel to one of the principal components and intercepts the centroid 252c. Therefore, there are four possibilities for each object 252. The squares before and after the object 252 are used to determine if there is enough space to push the object 252. If too many points from the point cloud higher than a certain threshold are inside one of these squares, the corresponding push is considered to be unsuitable. All of the suitable pushes given a certain scene are determined. Then, using some heuristics, one of the suitable pushes is chosen to be executed by the robotic manipulator 221. The heuristics can be for example such that the algorithm would choose to push the object whose centroid is the closest to the centre of the workspace. Other heuristics could try to minimise the distance between the current pose of the end effector 222 and the target push start pose. The length of the push is a factor of the length of the object along the chosen principal component. For example, if the factor is 0.5, the length of the push is the half of the length of the object along the chosen principal component. Having established the direction 282 in which to move the object 252, the controller 234 is configured further to determine the start position for the end effector 222 in dependence on the direction 282. This may be carried out by the controller 234 by identifying a section 284 of the boundary of the object 252 that is substantially perpendicular to the direction 282 in which the object 252 is to be pushed and designating an area 286 of the container space 254 adjacent to the section 284 as the start position. The controller 234 is configured further to issue one or more actuation commands or control signals to the system 200 causing the end effector 222 to move to the start position 286 and, from there, to move the end effector 222 from the start position 286 to bring at least one finger element 222a, 222b into engagement with the object 252. There are four general poses that the end effector 222 could be in during this latter manoeuvre. In an ideal scenario, as shown in FIG. 4a, the end effector 222 would be positioned such that the finger elements 222a, 222b, shown in cross-section, are aligned with the direction 282 of the end effector 222 to come into engagement with the object 252 substantially at the same time, and are equidistant from the select axis 252d. In this scenario, the forces, designated by 290a, 290b, applied to the finger element 222a, 222b, as detected by the force-sensing arrangement, are equal in magnitude and direction. It should be noted that the forces 290a, 290b are shown for illustrative purposes only and would, of course, only be applied to the finger elements 222a, 222b when the end effector 222 has actually engaged the object 252. In this example, the controller 234 maintains the current pose of the end effector 222 when initiating the push manoeuvre, prior to the object 252 being grasped by the end effector 222. With reference to FIG. 4b, in another situation, the end effector 222 might be rotationally displaced about the selected axis 252d when initially brought into contact with the object 252. In such a situation, only one of the finger elements 222b engages the object 252, and consequently experiences a resultant force 290b comprising force components extending perpendicular and parallel to the direction 282 of the end effector 222 and selected axis 252d. In response to the force-sensing arrangement detecting such a force 290b, the controller 234 is configured to issue actuation commands causing the end effector 222 to rotate in a direction so as to reduce the magnitude of the force component extending perpendicular to the direction 282 of the end effector 222 (i.e., in a clockwise direction in this example, as indicated by arrow 292). The controller 234 causes the end effector 222 to simultaneously rotate 292 and move in the direction 282 along the selected axis 252d to maintain its engagement with the object 252, and is configured to stop rotating the end effector 222 in response to the force-sensing arrangement detecting forces applied to both finger elements 222a, 222b. From here, the controller 234 is configured then to initiate the push manoeuvre and move the object 252 along the selected axis 252d. On other occasions, the end effector 222 may be displaced laterally (i.e., where the finger elements 222a, 222b are not equidistant from the selected axis 252d, as shown in FIG. 4c). When the end effector 222 is moved by the controller 234 into engagement with the object 252 in this pose, the force-sensing arrangement might detect a higher force 290b on one finger element 222b when compared to the force 290a applied to the other finger element 222a. A similar force distribution might also be apparent when the end effector 222 is displaced laterally with respect to the centre of motion / centre of reaction forces of the object 252. In response, the controller 234 is configured further to issue one or more actuation commands to the robotic manipulator 221 to cause the end effector 222 to move sideways or laterally, as indicated by arrow 292, in the direction of the higher force 290b until equilibrium is achieved across the two finger elements 222a, 222b, as detected by the force-sensing arrangement. From here, the controller 234 is configured then to initiate the push manoeuvre and move the object 252 along the selected axis 252d. As indicated above, the force distribution across the finger element 222a, 222b, as detected by the force-sensing arrangement, might also be used to determine the lateral position of the end effector 222 with respect to the centre of motion of the object 252. For example, the finger element 222a, 222b to which the comparatively higher force is applied will be laterally closer to the centre of motion of the object 252. During the push manoeuvre, the controller 234 is arranged further to issue control signals configured to control the robotic manipulator 221 to move the end effector 222 laterally relative to the centre of motion of the object 252 to achieve an imbalance of forces applied to finger elements 222a, 222b causing the object 252 to rotate. In other scenarios, the end effector 222 might be both rotationally and laterally displaced with respect to the selected axis 252d as shown in FIG. 4d. In that case, the controller 234 is configured to first correct for the rotational displacement as described with respect to FIG. 4b and then correct for the lateral displacement as described with respect to FIG. 4c. FIG. 5 is a flowchart illustrating a computer-implemented method 300 that includes the steps described above for the non-prehensile manipulation of the object 252 by the end effector 222 according to an embodiment of the invention. The method 300 starts at step 302 and proceeds to step 304 where the controller 234 obtains image data representative of an image 250 captured by the overhead imaging device 216 of the objects 252, 253, 256 within the container space 254. At step 306, the controller 234 determines the principal components 252a, 252b of the object 252 in dependence on the image data. With reference to FIG. 6, steps 304 and 306 may include one or more steps of sub-method 400. Sub-method 400 might be initiated by the controller 234 after step 304 and then proceeds to step 402, where the controller 234 processes the image data using the model for instance segmentation to detect each object 252, 253, 256 within the image 250 and generates segmentation masks to establish the boundary or contour of each object 252, 253, 256. The sub-method 400 then proceeds to step 404 where the boundary of the object 252 is then used, by the controller 234, to ascertain its centroid 252c. This ascertainment can include, for example, the controller 234 determining the length and width of the object 252 based on its respective boundary or contour, and then deducing that the centroid 252c of the object 252 is located at a position halfway along its respective length and width. In another example, the controller 234 uses the Mask R-CNN algorithm to provide a mask and contour for each segmented object 252 in the workspace or container space 254. The centroid 252c of the segmented object 252 is computed by averaging all the points in the corresponding mask. These are just a couple of examples of a method by which the controller 234 may determine the centroid of an object. Referring back to FIG. 5, from step 404 the sub-method 400 reverts back to step 306 where the method 300 then proceeds to step 308. At step 308, the controller 234 determines the direction 282 in which to push the object 252 in dependence on the principal components 252a, 252b. Step 308 may include one or more steps of sub-methods 500, 600, 700. Referring to FIG. 6, following step 306, sub-method 500 might be initiated by the controller 234 and proceed to step 502, where the controller 234 determines the boundary 258 of the container space 254. As previously mentioned, the controller 234 might process the image data using the model for instance segmentation to detect the container 244 within the image 250 and generate a segmentation mask to establish the boundary 258 or the boundary 258 could be a predetermined boundary that is accessible to the controller 234. The sub-method 500 then proceeds to step 504 where the controller 234 determines candidate axes 252d, 252e along which the object 252 might be moved in dependence on the principal components 252a, 252b of the object 252 and boundary 258 of the container space 254. This can include the controller 234 forming the candidate axes 252d, 252e by lengthening the principal components 252a, 252b of the object 252 to the boundary 258 of the container space 254. The sub-method 500 then proceeds to step 506 where the controller 234 determines notional collision points 260, 262, 264, 266 of the object 252 along the axes 252d, 252e in dependence on the boundary 258 of the container space 254. The notional collision points 260, 262, 264, 266 may be identified by the controller 234 at the points at which the axes 252d, 252e and the boundary 258 of the container space 254 intersect. For example, the controller 234 may process the image data to identify pixels of the candidate axes 252d, 252e that are adjacent pixels forming the boundary 258 of the container space 254, and label those pixels of the axes 252d, 252e as notional collision points 260, 262, 264, 266. At step 508, the controller 234 then determines if other objects 253, 256 are present in the container space 254. If it is determined that other objects 253, 256 are present, the submethod 500 proceeds to step 510 where the controller 234 determines, using the model for instance segmentation, the boundaries or contour lines of the other objects 253, 256 within the container space 254 in dependence on the image data. This determination might have been made earlier on in the sub-method 400. Either way, sub-method 500 then proceeds to step 512 where the controller 234 determines notional collision points 270, 272, 274, 276 of the object 252 along the axes 252d, 252e in dependence on the contour lines of the other objects 253, 256. To this end, the controller 234 might process the image data to identify pixels of the candidate axes 252d, 252e that are adjacent to the border pixels of the boundary or contour of the objects 253, 256, and label those pixels of the axes 252d, 252e as notional collision points 270, 272, 274, 276. From here, the sub-method 500 proceeds to step 514 where the controller 234 selects from the candidate axes 252d, 252e, the axis 252d with the fewest notional collision points. Referring back to step 508, if it is determined that no other objects are present in the container space 254, the sub-method 500 proceeds directly to step 514 where an axis 252d is selected from candidate axes 252d, 252e. At step 516, controller 234 determines the position of the centroid 252c along selected axis 252d. For example, the controller 234 may process the image data to determine a midpoint 280 of the axis 252d, based on the number of pixels forming the axis 252d, and establish which side of the midpoint 280 the centroid 252c is located. At step 518, the controller 234 then determines a direction 282 in which to move the object 252 along the selected axis 252d that would move the centroid 252c towards the midpoint 280. Alternatively, instead of determining the midpoint 280 of the axis 252d, the controller 234, at step 516, may process the image data to determine the distances along the selected axis 252d either side of the centroid 252c, i.e., between the centroid 252c and the boundary 258 of the container space 254. At step 518, the controller 234 then determines a direction 282 in which to move the object 252 along the selected axis 252d that would decrease the larger of the two distances either side of the centroid 252c. Referring back to FIG. 5, from step 518, the sub-method 500 reverts back to the method 300 then proceeds to step 310 where a start position for the end effector 222 is determined in dependence on the direction 282. FIG. 7 shows another example of a sub-method 600 that might be included in steps 304 and 306 of method 300. Steps 602 to 612 of sub-method 600 are substantially the same as the equivalent steps in sub-method 500. However, at step 614, instead of selecting the axis 252d with the fewest notional collision points as is the case with sub-method 500, the controller 234 processes the image data to determine the relative distances between the notional collision points along each axes and selects, at step 616, the candidate axis 252d with the largest distance between two consecutive notional collision points. These steps, 614, 616, could also be used in sub-method 500 in the event that the candidate axes 252d, 252e are each associated with the same number of notional collision points. From there, the sub-method 600 proceeds to step 617 where controller 234 determines the position of the centroid 252c along the selected axis 252d. As mentioned before, for example, the controller 234 may process the image data to determine a midpoint 280 of the axis 252d, based on the number of pixels forming the axis 252d, and establish which side of the midpoint 280 the centroid 252c is located. At step 618, the controller 234 then determines a direction 282 in which to move the object 252 along the selected axis 252d that would move the centroid 252c of the object 252 towards the midpoint 280. Alternatively, instead of determining the midpoint 280 of the axis 252d, the controller 234, at step 617, may process the image data to determine the distances along the selected axis 252d either side of the centroid 252c, i.e., between the centroid 252c and the boundary 258 of the container space 254. At step 618, the controller 234 then determines a direction 282 in which to move the object 252 along the selected axis 252d that would decrease the larger of the two distances either side of the centroid 252c. Referring back to FIG. 5, from step 618, the sub-method 600 reverts back to the method 300 then proceeds to step 310 where a start position for the end effector 222 is determined in dependence on the direction 282. FIG. 8 shows yet another example of a sub-method 700 that might be included in steps 304 and 306 of method 300. Steps 702 and 704 of sub-method 700 are substantially the same as the equivalent steps in sub-method 500. At step 706, the controller 234 determines empty areas within the container space 254 adjacent to each side of the object 252 along the axes 252d, 252e in dependence on the boundary 258 defining the container space 254. At step 708, the controller 234 then determines if other objects 253, 256 are present in the container space 254. If it is determined that other objects 253, 256 are present, the submethod 700 proceeds to step 710 where the controller 234 determines, using the model for instance segmentation, the boundaries or contour lines of the other objects 253, 256 within the container space 254 in dependence on the image data. This determination might have been made earlier on in the sub-method 400. Either way, sub-method 700 then proceeds to step 712 where the controller 234 determines empty areas within the container space 254 on each side of the object 252 along the axes 252d, 252e in dependence on the boundary 258 of the container space 254 and contour lines of the other objects 253, 256. The controller 234 then selects, at step 714, an axis from the candidate axes 252d, 252e with empty areas on opposing sides of the object 252. In the present example, the controller 234 would select candidate axis 252d as it traverses two empty areas 261, 263 located at opposing sides of the object 252. In the event that both axes 252d, 252e traverse empty spaces at opposing sides of the object 252, the controller 234 is configured to select from the candidate axes 252d, 252e, the axis 252d having a comparatively greater portion of its length extending through the empty areas 261, 263. Referring back to step 708, if it is determined that no other objects are present in the container space 254, the sub-method 700 proceeds directly to step 714 where an axis is selected from the candidate axes 252d, 252e. At step 716, the controller 234 then determines a direction 282 in which to push the object 252 along the selected axis 252d in dependence on the size of the empty areas 261, 263 associated with the axis 252d. For example, the controller 234 might determine the direction 282 as one that, when executed, would move the centroid 252c towards the larger of the two areas 261, 263. Alternatively, the controller 234 might determine the direction 282 as one that, when executed, would decrease the size of the larger area 261. Referring back to FIG. 5, from step 716, the sub-method 700 reverts back to the method 300 then proceeds to step 310 where a start position for the end effector 222 is determined in dependence on the direction 282. Having established the direction 282 in which to move the object 252, the controller 234 then determines the start position for the end effector 222 in dependence on the direction 282 in step 310. This may be carried out by the controller 234 by identifying a section 284 of the boundary of the object 252 that is substantially perpendicular to the direction 282 in which the object 252 is to be pushed and designating an area 286 of the container space 254 adjacent to the section 284 as the start position. At step 312, the controller 234 then causes the end effector 222 to move to the start position 286 and, from there, to move the end effector 222 from the start position 286 to bring at least one finger element 222a, 222b into engagement with the object 252. The controller 234 then adjusts the pose of the end effector 222, in step 314, prior to initiating a push manoeuvre in response to a force detected by the force-sensing arrangement. With reference to FIG. 10, step 314 may include one or more steps of submethod 800. Sub-method 800 might be initiated by the controller 234 after step 312 and then proceed to step 802 where it is determined, using the force sensing arrangement, how many finger elements 222a, 222b have engaged the object 252. If only one of the finger elements 222a, 222b has engaged the object 252, the controller 234 then rotates the end effector 222 until the force-sensing arrangement detects forces on both finger elements 222a, 222b in step 804. On the other hand, if both finger elements 222a, 222b engage the object 252, the sub-method 800 proceeds to step 806 where it is determined if a higher force is detected, by the force sensing arrangement, on one of the finger elements 222a, 222b when compared to the other of the finger elements 222a, 222b. If a comparatively higher force is detected, the sub-method 800 moves to step 808 where the controller 234 causes the end effector to move laterally to better equalise the forces applied to the two finger elements 222a, 222b. Referring back to FIG. 5, from step 808, the sub-method 800 reverts back to the method 300 and then proceeds to step 316 where the method 300 is ended. Similarly, if it is determined at step 806 that the forces applied to the two finger elements 222a, 222b are substantially equal, the sub-method 800 reverts to method 300 and then proceeds to step 316 where the method 300 is ended. Many of the methods described herein can be performed with variations. For example, many of the methods may include additional acts, omit some acts, and / or perform acts in a different order than as illustrated or described. The various examples, implementations, and embodiments described above can be combined to provide further embodiments. Aspects of the embodiments can be modified, if necessary, to employ systems, circuits, devices, methods, and concepts in various patents, applications, and publications to provide yet further embodiments. These and other changes can be made to the examples, implementations, and embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled.

Claims

1. A computer-implemented method for controlling a robotic picking system for the non-prehensile manipulation an object within a workspace, the robotic picking system comprising an imaging device and a robotic manipulator, wherein the robotic manipulator comprises an end effector assembly having at least one finger element and a force-sensing arrangement for sensing forces applied to the finger element, the method comprising:obtaining image data representative of the object within the container space; determining principal components the object in dependence on the image data;determining a direction in which to push the object in dependence on the principal components;determining a start position for the end effector in dependence on the direction; and,controlling the robotic manipulator to:move the end effector from the start position to bring at least one of the finger elements into engagement with the object; and,adjust the pose of the end effector prior to initiating a push manoeuvre in response to a force detected by the force-sensing arrangement.

2. A computer-implemented method according to claim 1, further comprising: determining a contour of the object in dependence on the image data; and, determining the centroid of the object in dependence on the contour of the object.

3. A computer-implemented method according to claim 1 or 2, further comprising: determining a boundary of the container space;determining axes along which to move the object in dependence on the principal components; and,determining notional collision points of the object along the axes in dependence on the boundary of the container space.

4. A computer-implemented method according to any preceding claim, further comprising:determining contour lines of other objects within the container space in dependence on the image data; and,determining notional collision points of the object along the axes in dependence on the contour line of the other objects.

5. A computer-implemented method according to claim 3 or 4, further comprising: selecting from the axes, an axis with the fewest notional collision points.

6. A computer-implemented method according to claim 3 or 4, further comprising: determining relative distances between the notional collision points on the axes; and, selecting from the axes, the axis with the largest distance between two consecutive notional collision points.

7. A computer-implemented method according to claim 5 or 6, further comprising: determining the position of the centroid along the selected axis; determining the direction in which to push the object in dependence on the position of the centroid.

8. A computer-implemented method according to claim 1 or 2, further comprising: determining a boundary of the container space;determining axes along which to move the object in dependence on the principal components; and,determining empty areas within the container space on each side of the object along the axes in dependence on the boundary of the container space.

9. A computer-implemented method according to claim 8, further comprising: determining contour lines of other objects within the container space in dependence on the image data; and, determining empty areas within the container space on each side of the object along the axes in dependence on the boundary of the container space and the contour lines of the other objects.

10. A computer-implemented method according to claim 8 or 9, further comprising: selecting from the axes, the axis with empty areas on opposing sides of the object.

11. A computer-implemented method according to claim 10, further comprising: selecting from the axes, the axis having a comparatively greater portion of its length extending through empty areas.

12. A computer-implemented method according to claim 10 or 11, further comprising:determining the direction in which to push the object in dependence on the size of the empty areas associated with the selected axis.

13. A computer-implemented method according to any preceding claim, further comprising controlling the robotic manipulator to:rotate the end effector in response to the force-sensing arrangement detecting a force on only one finger element.

14. A computer-implemented method according to claim 13, further comprising controlling the robotic manipulator to:rotate the end effector in a direction so as to reduce the magnitude of a force component extending perpendicular to a gripping surface of the one finger element.

15. A computer-implemented method according to claim 13 or 14, further comprising controlling the robotic manipulator to:stop rotating the end effector in response to the force-sensing arrangement detecting forces on both finger elements.

16. A computer-implemented method according to any preceding claim, further comprising controlling the robotic manipulator to:move the end effector laterally in response to the force-sensing arrangement detecting a higher force on one of the finger elements when compared to the force applied to the other of the finger elements.

17. A computer-implemented method according to claim 16, further comprising controlling the robotic manipulator to:move the end effector laterally in the direction of the higher force.

18. A computer-implemented method according to any preceding claim, further comprising:determining the centre of motion of the object in dependence on the forces detected by the force-sensing arrangement; and,controlling the robotic manipulator to:move the end effector laterally relative to the centre of motion to effect rotation of the object during the push manoeuvre.

19. A controller for a robotic manipulator, wherein the controller is configured to perform the computer-implemented method of any preceding claim.5 20. A computer program comprising instructions which, when the program is executed bya computer, cause the computer to carry out the computer-implemented method of any one of claims 1 to 18.

21. A computer-readable data carrier having stored thereon the computer program of10 claim 20.

22. A robotic picking system comprising the controller of claim 19 and a robotic manipulator for picking an object.

Images