Control of a robot manipulator for packing objects.

Abstract

A computer-implemented method of controlling a robot manipulator for packing an object is provided, the method including acquiring an image of an object grasped by an end effector of the robot manipulator and determining a principal axis of the object in the image. A first object pose is determined, where the principal axis of the object is aligned with an axis of the storage space. The robot manipulator is controlled to manipulate the object into the first object pose above the storage space and move the object from the first object pose downward into the storage space. In response to detecting, by a force sensor, a contact force at the end effector that exceeds a predetermined force threshold, the robot manipulator is controlled to manipulate the object to a second object pose above the storage space to initiate a further attempt to place the object within the storage space.

Description

[Technical field]

[0001] The present disclosure relates to robotic control systems and, more particularly, to systems and methods for use in packing objects into receptacles. [Background technology]

[0002] Bin packing is a core problem in computer vision and robotics. The goal is to have a system with sensors and a robot to grasp items using a suction gripper, parallel gripper, or other type of robot end effector and pack the items into bins, e.g., receptacles. The packing system may be combined with a bin picking system that uses the same or a different robot to initially pick up objects with random poses (positions / orientations) from different bins using the same or a different type of end effector.

[0003] However, current systems have problems including focusing on planning, avoiding all contacts during packing, and assuming that only rigid objects are packed. This means that the system is not practical in real-world scenarios. For example, general-purpose packing solutions typically do not consider the specificity of the grocery packing problem. For example, a packing algorithm should be able to handle unexpected errors and be robust when performing packing attempts in real-world scenarios. Summary of the Invention

[0004] A computer-implemented method for controlling a robotic manipulator for packing objects is provided, the method comprising: acquiring an image of an object grasped by an end effector of a robotic manipulator; Determining a principal axis of an object in an image; determining a first object pose, where a principal axis of the object is aligned with an axis of the storage space; manipulating the object into a first object orientation above the containment volume; moving the object downwardly into the containment volume from a first object orientation; and in response to detecting by the force sensor a contact force at the end effector that exceeds a predetermined force threshold, controlling the robot manipulator to manipulate the object to a second object pose above the storage space to initiate a further attempt to place the object within the storage space.

[0005] Also provided is a data processing apparatus comprising a processor configured to perform the method.A computer program is provided comprising instructions which, when executed by a computer, cause the computer to perform the method.Similarly, a computer readable medium is provided comprising instructions which, when executed by a computer, cause the computer to perform the method.

[0006] Further provided is a robotic packing system comprising a robotic manipulator for packing objects and a controller for the robotic manipulator, acquiring an image of an object grasped by an end effector of the robot manipulator; determining the principal axes of objects in the image; determining a first object pose, where a principal axis of the object is aligned with an axis of the containment volume; manipulating the object into a first object orientation above the containment volume; moving the object downwardly into the containment volume from a first object orientation; In response to detecting a contact force by the force sensor at the end effector that exceeds a predetermined force threshold, the control unit is configured to control the robot manipulator to manipulate the object to a second object pose above the storage space to initiate a further attempt to place the object within the storage space.

[0007] Generally, the present description introduces a system and method for packing an object into a receptacle, e.g., a container, using a robotic manipulator by aligning the object with the receptacle (based on imaging analysis) and iterating packing attempts (based on force feedback). A contact force threshold, which may include a torque threshold, means that the system is sensitive to forces that may damage the object being packed or the contents of the receptacle. For example, if a contact force / torque exceeding a set threshold is detected, the packing attempt is aborted and the robotic manipulator is resumed for another packing attempt starting from a shifted initial pose.

[0008] Aligning the object with the receptacle reduces the likelihood of contact forces occurring between the object and the receptacle during one or more packing attempts. However, because unexpected contact forces may still occur, e.g., between the object and the receptacle contents, iterating the packing attempts along the path of the shifted initial pose may increase the chances of achieving a successful packing attempt, e.g., where a set force / torque threshold is not exceeded and the object is released into the receptacle. Thus, the iterative nature of the presented packing solution adds robustness to the algorithm compared to known systems and methods by being able to react to unexpected contacts and attempt a new pack run with a modified initial position.

[0009] Overall, the present systems and methods combine a proactive component, the aligner, that uses image analysis to suggest initial poses, and a reactive component, the iterative packer, that drives a robotic manipulator to perform packing attempts. Together, the combined system improves the efficiency of achieving successful packing attempts over implementing either or the other component independently. [Brief description of the drawings]

[0010] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numbers indicate the same or corresponding parts, and in which: [Figure 1] FIG. 1 is a schematic diagram of a robotic packing system according to one embodiment. [Diagram 2] FIG. 2 is a schematic front view of a robotic packing system according to one embodiment. [Figure 3A] FIG. 3A is a schematic perspective view from an overhead camera of a robotic packing system according to one embodiment. [Figure 3B] FIG. 3B is a representation of a point cloud image corresponding to the overhead camera perspective view of FIG. 3A. [Figure 4A] FIG. 4A is a representation of a point cloud image of an object separated from the point cloud image of FIG. 3B. [Figure 4B] FIG. 4B is a diagram illustrating the projection of the point cloud image of FIG. 4A onto a two-dimensional plane. [Diagram 5] FIG. 5 shows a flowchart depicting a computer-implemented method for controlling a robotic manipulator for packing objects, according to one embodiment.

[0011] In the drawings, like features are designated by like reference numerals where appropriate. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the description that follows, some specific details are included to provide a thorough understanding of various disclosed embodiments. However, one of ordinary skill in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In some examples, well-known structures associated with the gripper assembly and / or robotic manipulator (processors, sensors, storage devices, network interfaces, workpieces, tension members, fasteners, electrical connectors, mixers, etc.) are not shown or described in detail to avoid unnecessarily obscuring the description of the disclosed embodiments.

[0013] Unless the context requires otherwise, the word "comprise" and variations thereof, such as "comprises" and "comprising," are to be interpreted in this specification and the appended claims in an open and inclusive sense, i.e., "including but not limited to."

[0014] References throughout this document to "one," "an," or "another" applied to an "embodiment," "example," or "implementation" mean that a particular feature, structure, or characteristic described in connection with that embodiment, example, or implementation is included in at least one embodiment, example, or implementation. Thus, the appearances of phrases such as "in one embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, examples, or implementations.

[0015] As used in this specification and the appended claims, it should be noted that the forms "a," "an," and "the" are inclusive of the plural unless the content clearly dictates otherwise. It should be noted that the term "or" is generally used in its sense including "and / or" unless expressly dictated otherwise.

[0016] With reference to FIG. 1, an example of a robotic packing system 100 is shown that may be adapted for use with the present assemblies, apparatus, and methods. The robotic packing system 100 may form part of an online retail operation, such as an online grocery retail operation. Furthermore, it may be applied to any other operation that requires packing of items. For example, the robotic packing system 100 may be adapted to pick or sort articles, for example as a robotic picking / packing system, sometimes referred to as a "pick and place robot."

[0017] The robotic packing system 100 includes a manipulator apparatus 102 that includes a robotic manipulator 121. The manipulator 121 is an electromechanical machine that includes one or more appendages, such as a robotic arm 120, and an end effector 122 attached to the end of the robotic arm 120. The end effector 122 is a device configured to interact with an environment to perform tasks including, for example, grasping, gripping, releasably engaging, or otherwise interacting with an item. Examples of end effectors 122 include jaw grippers, finger grippers, magnetic or electromagnetic grippers, Bernoulli grippers, vacuum suction cups, electrostatic grippers, van der Waals grippers, capillary grippers, cryogenic grippers, ultrasonic grippers, and laser grippers.

[0018] The robotic manipulator 121 can grasp and manipulate objects. In a pick-and-place case, the robotic manipulator 121 is configured, for example, to pick an item from a first location and place the item at a second location.

[0019] The manipulator device 102 is communicatively coupled via a communication interface 104 to other components of the robotic packing system 100, such as one or more optional operator interfaces 106 that allow an observer to observe or monitor the system 100 and the manipulator device 102. The operator interface 106 may include a WIMP interface and an output display of a descriptive text or dynamic representation of the manipulator device 102 in a context or scenario. For example, the dynamic representation of the manipulator device 102 may include a video feed, such as a computer-generated animation. Examples of suitable communication interfaces 104 include a wire-based network or communication interface, an optical-based network or communication interface, a wireless network or communication interface, or a combination of wired, optical, and / or wireless network or communication interfaces.

[0020] The exemplary robotic packing system 100 also includes a control system 108 including at least one controller 110 communicatively coupled to the manipulator apparatus 102 and any other components of the robotic packing system 100 via a communication interface 104. The controller 110 comprises a control unit or computing device having one or more electronic processors. Embedded within the one or more processors is computer software including a set of control instructions provided as processor executable data that, when executed, causes the controller 110 to issue actuation commands or control signals to the manipulator apparatus 102. For example, the actuation commands or control signals cause the manipulator 121 to perform various methods and actions, such as identifying and manipulating items.

[0021] The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), programmed logic units (PLUs), etc. In some implementations, the controller 110 is a smaller processor-based device, such as a mobile phone, single board computer, embedded computer, etc., which may be interchangeably referred to or as a computer, server, or analyzer. A set of control instructions may be provided as processor executable data associated with the operation of the system 100 and the manipulator device 102 is contained in a non-transitory computer readable storage device 112, which forms part of the robotic packing system 100 and is accessible to the controller 110 via the communication interface 104.

[0022] In some implementations, the storage device 112 includes two or more separate devices. The storage device 112 can include, for example, one or more volatile storage devices, such as random access memory (RAM), and one or more non-volatile storage devices, such as read-only memory (ROM), flash memory, magnetic hard disk (HDD), optical disk, solid state disk (SSD), etc. Those skilled in the art will recognize that storage can be implemented in a variety of ways, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other form of computer and processor readable memory or storage medium, and / or combinations thereof. Storage can be read-only or read-write, as appropriate.

[0023] The robotic packing system 100 includes a sensor subsystem 114 that comprises one or more sensors that detect, sense, or measure the state of the manipulator apparatus 102 and / or conditions within the environment or workspace in which the manipulator 121 operates, and generate or provide corresponding sensor data or information. The sensor information includes environmental sensor information representative of environmental conditions within the workspace of the manipulator 121, as well as information representative of the condition or state of the manipulator apparatus 102, including its various subsystems and components, and characteristics of the item to be manipulated. The acquired data can be transmitted via the communication interface 104 to the controller 110 to instruct the manipulator 121 accordingly. Such information can include, for example, diagnostic sensor information useful for diagnosing the state or status of the manipulator apparatus 102, or the environment in which the manipulator 121 operates.

[0024] Such sensors include, for example, one or more cameras or imagers 116 (responsive in the visible and / or non-visible ranges of the electromagnetic spectrum, including, for example, infrared and ultraviolet). The one or more cameras 116 may include a depth camera, such as, for example, a stereo camera, to capture depth data along with color channel data within the imaged scene. Other sensors in the sensor subsystem 114 may include one or more of contact sensors, force sensors, strain gauges, vibration sensors, position sensors, attitude sensors, accelerometers, radar, sonar, lidar, touch sensors, pressure sensors, load cells, microphones 118, weather sensors, chemical sensors, and the like. In some implementations, the sensors include diagnostic sensors for monitoring the status and / or health of an on-board power source (e.g., a battery array, an ultracapacitor array, or a fuel cell array) within the manipulator device 102.

[0025] In some implementations, the one or more sensors comprise a receiver for receiving position and / or orientation information regarding the manipulator 121. For example, a Global Positioning System (GPS) receiver for receiving GPS data, two or more time signals for the controller 110 to create a position measurement based on data in the signal, such as time of flight, signal strength, or other data to achieve a position measurement. The manipulator 121 may also be provided with one or more accelerometers, which may also form part of the manipulator device 102, for example, to obtain inertial or orientation data in one, two, or three axes regarding its movement.

[0026] The robotic manipulator 121 of the system 100 may be piloted by a human operator at the operator interface 106. In a human operator control (or "pilot") mode, the human operator observes a representation of sensor data, e.g., video, audio, or haptic data, received from one or more sensors of the sensor subsystem 114. The human operator then acts conditioned by his perception of the representation of the data and creates information or executable control instructions to guide the manipulator 121 accordingly. In the pilot mode, the manipulator device 102 may execute control instructions in real time (e.g., without additional delay) as received from the operator interface 106 based on the sensed information and without consideration of other control instructions.

[0027] In some implementations, the manipulator device 102 operates autonomously, i.e., without a human operator creating control instructions in the operator interface 106 to direct the manipulator 121. The manipulator device 102 may operate in an autonomous control mode by executing autonomous control instructions. For example, the controller 110 may use sensor data from one or more sensors of the sensor subsystem 114. The sensor data may be associated with operator-generated control instructions from one or more times that the manipulator device 102 was in pilot mode to generate autonomous control instructions for subsequent use. For example, deep learning techniques may be used to extract features from the sensor data. Thus, in the autonomous mode, the manipulator device 102 may autonomously recognize features or conditions of its environment and the items to be manipulated. In response, the manipulator device 102 performs one or more defined actions or tasks. For example, the manipulator device 102 executes a pipeline or sequence of actions or tasks.

[0028] In some implementations, the controller 110 autonomously perceives characteristics or conditions of the environment surrounding the manipulator 121 and one or more virtual items synthesized into the environment. The environment is represented by sensor data from a sensor subsystem 114. In response to the representation being presented, the controller 110 issues control signals to the manipulator device 102 to perform one or more actions or tasks.

[0029] In some examples, the manipulator device 102 may be autonomously controlled at a given time while being piloted, operated, or controlled by a human operator at another time. That is, the manipulator device 102 may change to operate under an autonomous control mode and operate under a piloted (i.e., non-autonomous) mode. In another mode of operation, the manipulator device 102 may replay or execute control instructions previously executed in a pilot mode. That is, the manipulator device 102 may operate based on replayed pilot data without sensor data.

[0030] Manipulator apparatus 102 further includes a communication interface subsystem 124 (e.g., a network interface device) communicatively coupled to bus 126 and providing bidirectional communication with other components of system 100 (e.g., controller 110) via communication interface 104. Communication interface subsystem 124 may be a wireless (e.g., radio or microwave frequency transmitter, receiver, transceiver) port and / or any circuitry that provides bidirectional communication of processor-readable data and processor-executable instructions with an associated controller, etc. Suitable communication protocols include FTP, HTTP, web services, SOAP with XML, cellular (e.g., GSM, CDMA), Wi-Fi compliant, Bluetooth compliant, etc.

[0031] Manipulator device 102 further includes a motion subsystem 130 that is communicatively coupled to robotic arm 120 and end effector 122. Motion subsystem 130 comprises one or more motors, solenoids, other actuators, linkages, drive belts, etc. operable to move robotic arm 120 and / or end effector 122 within a range of motion in accordance with actuation commands or control signals issued by controller 110. Motion subsystem 130 is communicatively coupled to controller 110 via bus 126.

[0032] The manipulator device 102 also includes an output subsystem 128 that includes one or more output devices, such as a speaker, light, or display, that enable the manipulator device 102 to transmit signals into the workspace, for example, to communicate with an operator and / or another manipulator device 102.

[0033] Those skilled in the art will understand that components of manipulator device 102 may be altered, combined, divided, omitted, etc. In some examples, one or more of communication interface subsystem 124, output subsystem 128, and motion subsystem 130 are combined. In other examples, one or more subsystems (e.g., motion subsystem 130) are divided into additional subsystems.

[0034] 2 illustrates an example of a robotic packing system 200 that includes a robotic manipulator 121, such as an implementation of the robotic manipulator 221 described in the previous example. According to such an example, the robotic manipulator 221 includes a robotic arm 220, an end effector 222, and a motion subsystem 230. The motion subsystem 230 is communicatively coupled to the robotic arm 220 and the end effector 222 and configured to move the robotic arm 220 and / or the end effector 222 according to actuation commands or control signals issued by a controller (not shown). The controller, such as the controller 110 described in the previous example, is part of a manipulator apparatus that includes the robotic manipulator 221.

[0035] The robotic manipulator 221 is arranged to manipulate an object grasped by, for example, the end effector 222 within the workspace to pack the object into a receiving space, for example, a container (or "bin" or "tote") 244. For example, the robotic packing system 200 may be implemented in an automated storage and retrieval system (ASRS), for example, at a picking station thereof. The ASRS typically includes a number of containers arranged to store items and one or more load handling devices or automated guided vehicles (AGVs) for retrieving one or more containers 244 during the fulfillment of a customer order. At the picking station, items are picked from and / or placed into one or more retrieved containers 244. The one or more containers in the picking station may be considered to be storage containers or shipping containers. A storage container is a container that holds each of the products that remain in the ASRS and may be transferred from the storage container to a shipping container. A shipping container is a container that is introduced into the ASRS when empty and has a number of different products loaded into it. A shipping container can contain one or more bags or cartons that the product can be loaded into. A shipping container can be substantially the same size as a storage container. Alternatively, a shipping container can be slightly smaller than a storage container such that the shipping container can be nested within the storage container.

[0036] Thus, the robotic packing system 200 may be used to pick an item from one container, e.g., a storage container, and place the item in another container, e.g., a shipping container, at a picking station. Thus, the picking station may have two sections, i.e., one section for storage containers and one section for shipping containers. The arrangement of the picking station, e.g., its sections, may be varied and selected by one skilled in the art. For example, the two sections may be arranged on two sides of an area, or one section above or below the other. In some cases, the picking station is located away from the storage locations of the containers in the ASRS, e.g., away from the storage grid in a grid-based ASRS. Thus, the load handling device may deliver and collect the containers from one or more ports of the ASRS that are connected to the picking station, e.g., by a chute. In other examples, the picking station is located to directly interact with a subset of storage locations in the ASRS, e.g., to pick and place items between containers located in the subset of storage locations. For example, in the case of a grid-based ASRS, the picking stations may be located on the grid of the ASRS.

[0037] The robot manipulator 221 may include one or more end effectors 222. For example, the robot manipulator 221 may include two or more different types of end effectors. In some examples, the robot manipulator 221 may be configured to exchange a first end effector for a second effector. In some cases, the controller may send instructions to the robot manipulator 221 regarding which end effector 222 to use for each different object or product (or stock keeping unit, "SKU") being packed. Alternatively, the robot manipulator 221 may determine which end effector to use based on the weight, size, shape, etc. of the product. Previous success and / or failure to grasp and move an item may be used to update the selection of an end effector for a particular SKU. This information may be fed back to the controller so that success / failure information is stored and shared among different picking / packing stations. The robot manipulator 221 may be capable of changing end effectors. For example, the picking / packing station may include a storage area capable of receiving one or more end effectors. The robotic manipulator 221 may be configured such that an end effector in use may be removed from the robotic arm 220 and placed in an end effector storage area. Additional end effectors may be removably attached to the robotic arm 220 such that they may be used in subsequent picking / packing operations. The end effectors may be selected according to the planned picking / packing task.

[0038] The robotic packing system 200 of Figure 2 includes a camera 216 positioned above the workspace of the robotic manipulator 221. The overhead camera 216 is supported by a frame structure 240, which is shown in simplified form in the drawings, but may take any suitable structural form as will be understood by one of ordinary skill in the art. For example, the frame structure 240 may comprise a scaffolding to which the overhead camera 216 is mounted.

[0039] The overhead camera 216 is positioned to capture an image of an object grasped by the end effector 222 of the robot manipulator 221. For example, the overhead camera 216 is positioned to have a view of the workspace of the robot manipulator 221 including the object when grasped by the end effector 222. FIG. 3A shows a schematic diagram of the workspace of the robot manipulator 221 as seen by the overhead camera 216. The workspace includes a containment space, such as a container 344, in which the robot manipulator 221 is positioned to pack the grasped object. In this example, the container 344 is positioned within a rig 348 that is positioned to support the container 344 at a specific location within the workspace of the robot manipulator 221. As previously mentioned, the container 344 may be positioned in a picking station of a grid-based ASRS or in a storage location within a grid structure in other examples.

[0040] The camera 216 may correspond to one or more cameras or imagers 116 in the sensor subsystem 114 of the robotic packing system 100 described with reference to Figure 1. In an example, the camera 216 of the robotic packing system 200 comprises a depth camera configured to capture a depth image. For example, the depth (or "depth map") image includes depth information of a scene viewed by the camera 216.

[0041] The point cloud generator may be associated with an overhead camera or imaging device 216, e.g., a depth camera or LIDAR sensor, positioned to view the workspace below, e.g., the container 244 and its contents. In a typical setup, the underside of the container 244 is positioned horizontally (e.g., in the xy plane of FIG. 2) with the sensor of the overhead camera or imaging device 216 pointing vertically down (e.g., in the -z direction of FIG. 2) to view the contents of the container 244. Examples of structured light devices for use in point cloud generation include a Kinect device by Microsoft®, a time-of-flight device, an ultrasonic device, a pair of stereo cameras, and a laser stripper. These devices typically generate depth map images.

[0042] It is usual in the art to calibrate depth map images for aberrations in the camera's lens and sensor. Once calibrated, the depth map may be converted into a set of metric 3D points, known as a point cloud. Preferably, the point cloud is an organized point cloud, meaning that each three-dimensional point is on the line of sight of a separate pixel resulting in a one-to-one correspondence between the 3D point and the pixel. The organization is desirable as it allows for more efficient point cloud processing. In a further part of the calibration process, the pose of the camera, i.e. its position and orientation, is determined relative to the reference frame of the robot packing system 200 or the robot manipulator 221. The reference frame may be the base of the robot manipulator 221, but any known reference frame will work, for example a reference frame located at the wrist joint of the robot arm 220. Thus, a point cloud may be generated based on the depth map and information about the lens and sensor used to generate the depth map. Optionally, the generated depth map may be converted into the reference frame of the robot packing system 200 or the robot manipulator 221. For simplicity, the camera or imager 116, 216 is shown as a single unit in Figures 1 and 2. However, it will be appreciated that each of the functions of depth map generation and depth map calibration may be performed by separate units, for example, the depth map calibration means may be integrated within the controller of the robotic packing system 100, 200.

[0043] 3B shows an example depth image 300 of the workspace of the robotic manipulator 221, corresponding to the view of FIG. 3A, captured by the overhead camera 216. The depth image 300 includes a point cloud in this example. The image 300 includes a rig 348, a container 344, a bag (e.g., a grocery bag or tote bag) 346 within the container 344, and an object 350 grasped by the end effector 222 of the robotic manipulator 221.

[0044] A controller for the robotic manipulator 221, such as the controller 100 communicatively coupled to the manipulator apparatus of the previous example, is configured to acquire an image 300 captured by the overhead camera 216. As described herein, the image 300 includes an object 350 grasped by the end effector 222 of the robotic manipulator 221.

[0045] The controller processes the image 300 to determine a principal axis of the object 350 in the image 300, e.g., the longitudinal axis of the object 350. For example, the longitudinal axis of the object 350 is an axis along the length of the body of the object, which may pass through the center of gravity or mass. The principal axis may also be defined by the endpoints of the longest line that can be drawn through the object 350 represented in the image 300. The principal axis endpoints, e.g., pixel coordinates (x1,y1) and (x2,y2) in the image 300, are found, for example, by calculating the pixel distance between all combinations of boundary pixels within the object boundary and finding the pair with the maximum length.

[0046] The object 350 may have other axes, e.g., minor or transverse axes, that are not substantially equal in length to the longitudinal axis. The minor axis is, for example, defined by the endpoint of the longest line that may be drawn through the object 350 represented in the image but that remains perpendicular to the major axis. The minor axis endpoint is, for example, determined by calculating the pixel distance between the two boundary pixel endpoints. The transverse axis is perpendicular to the longitudinal axis. In the third dimension, a sagittal axis may be defined that is perpendicular to both the longitudinal and transverse axes of the object 350. An object that is symmetric in two or three dimensions, for example, has two or three respective axes that are substantially equal in length. For example, a spherical object has symmetry in all three dimensions such that all axes passing through the center of the sphere are equal in length. Thus, any such axis may be determined as the major axis of the spherical object. Similarly, a cylindrical object 350, such as the example of FIG. 3B, is axisymmetric and has cylindrical symmetry about its longitudinal axis.

[0047] In an example, the controller processes the depth image 300 to remove any features therefrom that have associated depth values ​​outside a range that corresponds to a volume between the end effector 222 and the top surface 262 of the containing space. For example, if the image 300 is a point cloud, the controller removes points from the point cloud that are between the end effector 222 and the top plane 362 of the containing space. The top surface 262 of the containing space coincides with the top of the containing space, e.g., the container 244 in the example of FIG. 2. In an example where the image 300 includes depth information, e.g., a layer of depth values ​​that correspond pixel-by-pixel to color (e.g., RGB) or intensity channel data, the controller removes pixels or pixel values ​​from the image 300 that have associated depth values ​​outside a defined range. Thus, the controller can use the depth information to separate the object 350 from the image 300.

[0048] FIG. 4A shows an example image 470 of an object 350 separated from an image 300 captured by an overhead camera 216. In an example, the controller processes the image 470 to obtain a set of two-dimensional (2D) points, as shown in the example image 475 of FIG. 4B, projecting a depth image of the object 300 onto a plane 260 parallel to the bottom surface of the containment space (e.g., container 244, 344). The controller then determines a principal axis 480 of the object 350 based on the set of 2D points. For example, the controller performs a principal component analysis (PCA) using the set of 2D points. Performing a PCA, for example, determines as the principal axis 480 the axis that maximizes the variance of the 2D points projected onto this axis, which is more robust than using a line between the most distant points, for example, because it is not affected by outliers (such as some distant points at the bottom of FIG. 4B).

[0049] Once the principal axis 480 of the object 350 is determined, the controller is configured to determine a first object pose in which the principal axis 480 is aligned with the axis of the storage space. In the example of FIG. 3B, the axis of the storage space is the principal axis of the selected bag 346 in the container 344. In an example, the principal axis 480 of the object and the axis of the storage space are aligned when they are substantially parallel to each other, for example, within an angular separation of 1 or 2 degrees. For example, the two axes do not have to be superimposed, aligned, or overlap in a common plane to be aligned with each other. In certain cases, in addition to aligning the axes, a first object pose is determined that at least partially overlaps the two axes when projected onto a common plane. However, subsequent initial object poses corresponding to further packing attempts (described in a later example) may keep the axes aligned but may shift the principal axis 480 of the object such that the two axes no longer overlap each other.

[0050] A body pose represents the position and orientation of an associated body in space. For example, a six-dimensional (6D) pose of an body includes values ​​in three translational dimensions (e.g., corresponding to position) and three rotational dimensions (e.g., corresponding to orientation) of the body.

[0051] In some implementations, the controller interfaces with a pose generator (or pose estimator) configured to generate (e.g., determine or estimate) an object pose. For example, determining a given object pose involves mapping two-dimensional pixel locations in an image of the object to a six-dimensional pose.

[0052] The controller controls the robot manipulator 221 to manipulate the object into a first object pose above a receiving space, e.g., a container 244. For example, the controller issues actuation commands or control signals to the motion subsystem 230 of the robot manipulator 221 to cause the robot arm 220 and / or the end effector 222 to manipulate, e.g., move and / or rotate, the object into the first object pose. In a particular example, the controller determines a planar rotation of the end effector 222, e.g., to align a principal axis of the object with an axis of the receiving space, e.g., in the first object pose. The controller may then control the robot manipulator 221 to perform the planar rotation of the end effector 222.

[0053] In an example, the first object pose includes a predetermined value of the object's height in a z-direction perpendicular to a bottom surface of the containment space (e.g., container 244). For example, the first object pose includes a translational position that is in a plane 264 above the container 244, the plane 264 being perpendicular to the z-direction, i.e., parallel to the bottom plane 260 or top plane 262 of the container 244. The xy position of the first object pose may be selected to be inside the xy boundary of the container 244, for example. In some cases, the first object pose for placement attempt may be determined by adjusting the xy position of the initial object pose (e.g., determined based on the axis alignment) relative to the containment space, e.g., shifting the xy position of the initial object pose toward the center of the container 244. For example, the initial object pose may have an object overhanging beyond the xy boundary of the container 244, and in response, the initial object pose is adjusted such that less of the object overhangs the container 244 (or more of the object area overlays the container area). With respect to orientation (again relative to the global reference frame shown in FIG. 2), a first body pose can be determined by a controller that keeps the body's roll and pitch angles stable while varying the yaw angle to achieve alignment of the respective axes.

[0054] The controller controls the robot manipulator 221 to move the object from the first object pose downward into the containment space. For example, with the object's orientation defined by the first object pose, the controller controls the robot manipulator 221 to move the object in the (negative) z-direction towards and into the container 244, for example by adjusting the z-value of the first object pose while maintaining other values ​​of the first object pose. For example, the xy position and orientation of the object are kept the same by the robot manipulator 221 while the z-value of the object's pose is changed as the robot manipulator 221 moves the object into the container 244.

[0055] The robot manipulator 221 includes at least one force sensor (not shown). For example, the robot manipulator 221 includes at least one of a torque sensor for detecting torque forces and a linear force sensor for detecting linear forces acting on the robot manipulator 221, for example at the robot arm 220 or the end effector 222. The force / torque sensor is installed, for example, between the robot arm 220 and the end effector 222.

[0056] There are several types of force / torque sensors that can be selected by one skilled in the art, including strain gauges, capacitive sensors, and optical sensors. The principle of operation of any force sensor is to generate a measurable response to an applied force. Some force sensors are created using force-sensing resistors, for example, using electrodes and a sensing polymer film. Force-sensing resistors are based on contact resistance, where a conductive polymer film changes its electrical resistance in a predictable manner under a force applied to its surface. In a particular implementation, the robot manipulator 221 has a built-in force / torque sensor, for example, an e-series industrial robot (e.g., UR3e, UR5e, UR10e) manufactured by Universal Robots A / S (Odense, Denmark) may be used with its built-in force / torque sensor. Additionally or alternatively, the robot manipulator 221 may be retrofitted with at least one force / torque sensor, for example, a HEX force / torque sensor manufactured by OnRobot A / S of Odense, Denmark.

[0057] During an attempt to pack an object by forcing it into the storage space, unintended contact may occur with the storage space, its components, or objects already packed in the storage space. For example, referring to Figures 3A-3B where the storage space corresponds to a bag in a tote 344, unintended contact may occur with the edge of the tote, the bag, or objects already inside the tote 344 or bag 346.

[0058] A predetermined force threshold is set for the contact force detected by the force / torque sensor. If the contact force at the end effector 222 is detected to exceed the predetermined force threshold, the controller controls the robot manipulator 221 to manipulate the object to a different second object pose above the receiving space to initiate further attempts to pack the object. For example, if the detected contact force is too high (relative to the pre-set force / torque threshold), the controller causes the robot manipulator 221 to re-initialize the object to attempt to pack the object again, but starting from a different initial object pose. In an example, the normal of the detected force or torque signal is compared to the pre-set force or torque threshold. In some cases, there are separate force and torque thresholds such that the normals of the detected force and torque signals are compared to the respective force and torque thresholds.

[0059] The force / torque thresholds may be predetermined manually, for example, by testing whether the system is sensitive enough or oversensitive at a given force / torque threshold for a collision to occur. In some cases, only the difference between the measured force / torque values ​​and the corresponding "pre-packing" values ​​set before the start of packing are considered, to account for sensor biases that drift over time.

[0060] In some cases, different thresholds are used for different SKU categories, for example a more sensitive threshold may be used for bags of crisps compared to other SKUs to reduce the chance of damage to their contents during packing.

[0061] In an example, the first and second object poses are in a common plane 264 parallel to a top surface 262 or a bottom surface 260 of the receiving space, e.g., the container 244. For example, the controller reinitializes the robot manipulator 221 to position the object at a predetermined height above the receiving space, but shifts the position (and therefore the pose) of the object in the plane 264 at each reinitialization. Such shifting of the object's pose in this initialization plane 264 may be performed according to a predetermined path or route in the plane, e.g., a linear path such as a helical, figure-of-eight, or zigzag tool path. The predetermined path defines, for example, a sequence of pose displacements between subsequent poses in the path. The robot manipulator 221 may thus iterate along the predetermined path after each packing attempt until the object is successfully packed into the receiving space. For example, in the case of a spiral path, after each failed packing attempt, different dimensions of the initial object position within plane 264 are modified such that the sequence of end effector positions spirals within plane 264, and the object orientation is also modified in alternating directions during the packing attempt.

[0062] For example, a packing attempt is successful when the controller determines that the object 350 and the end effector 222 are positioned within the receiving space 244, 344 without detecting a contact force exceeding a set force threshold. For example, the pose of the end effector can be determined, for example, by the controller from forward kinematics techniques known to those skilled in the art. In essence, the pose of the end effector 222 at the end of the robot arm 220 can be determined using knowledge of all the joint positions of the robot manipulator 221 (provided by the robot sensors) and knowing all the relationships between the links of the robot arm 220. The controller can then determine whether the position (x, y, z) coordinates of the end effector 222 are located inside a volumetric region defined by the boundaries of the receiving space 244, 344, e.g., a tote or the like.

[0063] Once the controller determines that the packing attempt is successful, the controller causes the end effector 222 to release the object 350 in the receiving space 244, 344, for example, by issuing an actuation command or control signal to the motion subsystem 230 of the robotic manipulator 221. After releasing the object 350, the robotic manipulator 221 may return to a reset position above the receiving space, for example, ready to perform another picking and / or packing task.

[0064] In some examples, during a packing attempt, the force / torque sensor detects a contact force, and the controller responsively determines a direction in which the contact force will move away from the detected contact point. For example, when the controller detects a contact force that exceeds a set threshold, it determines a normal force vector. The controller further controls the robot manipulator 221 to move the end effector 222 and the object in the determined direction away from the contact point, e.g., along the determined normal vector. Thus, the controller can use force and / or torque feedback to reduce damage to the object being packed, and any other objects already in the containment space, when aborting a packing attempt. The selection of the second object pose can also be based on the determined direction away from the contact point. For example, if a contact point is detected on the right side of the end effector, the next starting pose for a subsequent packing attempt may be shifted to the left to move the end effector 222 away from the area where the unexpected contact occurred. In examples where the subsequent initial object poses are based on a predetermined path, the next starting pose may be selected from the predetermined path of poses while in a direction away from the contact point (e.g., projected onto the initialization plane 264). Additionally, information regarding any contact points detected in a previous packing attempt may be used in the subsequent packing attempt to, for example, alter the path of the end effector 222 through the receiving space 244 to avoid the same contact occurring during the packing attempt.

[0065] 5 illustrates a computer-implemented method 500 for controlling a robotic manipulator for packing an object. The robotic manipulator may be one of the example robotic manipulators 121, 221 described with reference to Figures 1 and 2. The method 500 may be performed by one or more components of the packing system 100 previously described, such as the control system 108 or the controller 110.

[0066] At 501, an image of an object grasped by an end effector of a robotic manipulator is acquired. In some examples, the image comprises a depth image. In certain cases, the method 500 includes removing from the depth image any features (e.g., points in the point cloud, or pixels in the image having depth values) having associated depth values ​​outside the range between the respective depth values ​​of the end effector and the top plane of the containment space. Such removal of features isolates the object in the image because only objects grasped by the end effector are positioned between the end effector and the top of the containment space, e.g., a container or tote. Depending on the type of depth image, the features removed from the image may comprise pixels or points in the point cloud.

[0067] At 502, the principal axes of the object in the image are determined. In some cases, the depth image of the object is projected onto a plane parallel to the bottom surface of the containment space to flatten the image into 2D points. The principal axes of the object can then be determined from the flattened image using, for example, PCA with the set of 2D points.

[0068] At 503, a first object pose is determined, where the object's principal axis is aligned with the receiving space's axis. For example, a rotation of the end effector is determined that aligns the object's principal axis with the receiving space's axis. The receiving space's axis may be the principal axis or may be another axis, such as a minor axis perpendicular to the principal axis. The rotation may be determined based on a calculated angle between the object's principal axis and the receiving space's axis. For example, the rotation corresponds to an angular displacement that reduces the angle between the two axes to zero, such that the two axes are aligned.

[0069] In an example, the spatial dimensions of the object are determined based on an acquired image of the object. For example, the area of ​​the object in the projection of the 2D points is determined. The spatial dimensions of the storage space are also acquired, for example, as input parameters, or are determined based on an image including the storage space as well. A first object pose for attempting to place the object in the storage space may be determined based on a comparison of the spatial dimensions of the object and the spatial dimensions of the storage space. For example, an initial pose of the object may be determined from the axis alignment step and then adjusted based on the spatial dimension comparison. If, in the initial pose, the object overhangs (e.g., protrudes) beyond the boundary of the storage space (e.g., a container and / or a bag within the container), the initial pose may be adjusted to reduce the overhang. For example, in the 2D projection image, the determined overhang area of ​​the object beyond the edge of the storage space can be reduced by shifting the initial pose toward the center of the storage space. In some cases, an adjustment vector for shifting the initial object pose is calculated for each edge of the storage space where the object overhangs. A resulting adjustment vector can then be calculated by summing the individual component adjustment vectors. If an object overhangs opposite edges of the containing space, the resulting adjustment vector applied to the initial object pose can be calculated to evenly distribute the overhang on both sides of the containing space, for example, to center the object with respect to the containing space along the axis along which the double overhang occurs.

[0070] In some examples, the determined first object pose is adjusted based on the determined end effector pose. For example, the end effector pose can be calculated based on forward kinematics, as described above. If the end effector pose is outside the boundary of the storage space in the determined first object pose, the first object pose can be adjusted, e.g., shifted, to move the corresponding end effector pose to or into the edge of the storage space. Thus, in a first attempt to place an object in the storage space, the end effector is positioned at least above the edge of the storage space, if not in the storage space. This can improve the likelihood that the object will be placed in the storage space, e.g., instead of falling outside the storage space after release by the end effector.

[0071] At 504, the robotic manipulator is controlled to manipulate the object into a first object pose above the storage space. For example, a control signal is sent to a motion subsystem of the robotic manipulator to effect a determined rotation of the end effector to align the object's principal axis with the associated axis of the storage space. Manipulation of the object into the first object pose may additionally or alternatively include a translational motion. In some cases, the determined rotation to align the associated axis is a planar rotation (e.g., determined in a plane parallel to a bottom surface of the storage space), and manipulation to place the object into the first object pose may further include a rotation in a different plane.

[0072] At 505, the robot manipulator is controlled to move the object downward into the storage space from the first object pose. For example, the height of the object above the storage space, e.g., in the z direction shown in Figure 2, is reduced to move the object into the storage space, e.g., across the top surface 262 of the container 244. The z component of the object pose may be altered while other components (e.g., five, for a 6D object pose) are held constant.

[0073] At 506, the robotic manipulator is controlled to manipulate the object in response to detecting, by the force sensor, a contact force at the end effector that exceeds a predetermined force threshold. In such case, the object is manipulated to a second object pose above the storage space to initiate a further attempt to place the object within the storage space.

[0074] In some examples, the first and second object poses are in a common plane parallel to a top or bottom surface of the containment space. The displacement of the second object pose relative to the first object pose in the common plane may be determined according to a predetermined path of pose displacement. For example, the robot manipulator is controlled to repeat the manipulation of the object in the common plane along the predetermined path in response to each failed placement attempt, i.e., when a contact force exceeding a predetermined threshold is detected. After a failed packing attempt, the robot manipulator is controlled to return the object to the common plane, e.g., an initialization plane, but to shift the position of the object in the plane according to a predetermined path, e.g., a series of initialization positions.

[0075] Upon determining that the object and end effector are positioned within the containment space without detecting a contact force exceeding a predetermined force / torque threshold, the end effector is controlled to release the object within the containment space. For example, a control signal is sent to the motion subsystem of the robot manipulator to cause the end effector to release its grip on the object. The robot manipulator can then return to a reset position, for example, in an initialization plane above the containment space or another predetermined position, ready to perform another task.

[0076] The method 500 of controlling a robotic manipulator for packing an object can be implemented by a control system or controller for the robotic manipulator, such as the control system or controller of the aforementioned packing system 100. For example, the control system or controller includes one or more processors for performing the method 500 according to instructions, such as computer program code, stored on a computer readable data carrier or storage medium.

[0077] The above examples should be understood as illustrative examples. Further examples are envisioned. For example, the robot manipulator 221 may further include one or more cameras attached to the robot arm 220. The cameras may be mounted on or near the end effector, for example, on or near the wrist of the robot arm. Additionally or alternatively, the cameras may be mounted on or near the elbow of the robot arm. The use of one or more cameras attached to the robot arm may be in addition to, or may be a replacement for, the overhead camera 216 of the robot packing system. Each camera may include a lighting element to illuminate the interior of the container as the items are being packed. The one or more cameras may be located elsewhere as part of the robot packing system. For example, the cameras may be used as barcode scanners.

[0078] Further, in the described example, the object grasped by the end effector is separated from the rest of the image using depth data, i.e., when the image includes a depth image. In other examples, the object may be determined in the image using object detection methods, e.g., with an artificial neural network (ANN), to separate the object and determine its major axis. In certain cases, a neural network model may be trained to determine the major axis of an object directly from an image captured by an overhead camera. Additionally or alternatively, an ANN may be trained and implemented to reconstruct a portion of an object that is missing from its image due to obfuscation by the robot manipulator, e.g., the end effector, or an external structure. FIG. 4A, for example, shows a missing portion of an object with a part of the robot manipulator positioned between the object and the camera during imaging. These ANN (or "deep learning") methods can be explicit (e.g., when an ANN model provides a reconstructed point cloud) or implicit (e.g., when the principal axes of an object can be determined even though some parts of the object are not visible, due to the generalization of the ANN being able to identify that the visible parts of the object belong to a wider range of items than are shown in the image).

[0079] It will also be understood that any feature described in connection with any one example may be used alone or in combination with other features described, or in combination with one or more features of any other example, or any combination of any other example. Additionally, equivalents and modifications not described above may be used without departing from the scope of the appended claims.

Claims

1. 1. A computer-implemented method for controlling a robotic manipulator for packing an object, comprising: acquiring an image of an object grasped by an end effector of the robotic manipulator; determining a principal axis of the object in the image; determining a first object pose, where the principal axis of the object is aligned with an axis of a storage space; maneuvering the object into the first object orientation above the storage space; moving the object downwardly into the containment volume from the first object orientation; in response to detecting a contact force by a force sensor at the end effector that exceeds a predetermined force threshold, manipulating the object to a second object pose above the storage space to initiate a further attempt to place the object within the storage space. and controlling the robotic manipulator.

2. 2. The computer-implemented method of claim 1, comprising: in response to determining that the object and the end effector are positioned within the containment space without detecting any further contact force exceeding the predetermined force threshold, controlling the robotic manipulator to release the object from the end effector.

3. The computer-implemented method of claim 1 or 2, wherein the first and second object poses lie in a common plane parallel to a top plane of the containment volume.

4. The computer-implemented method of claim 3 , wherein a displacement of the second object pose relative to the first object pose in the common plane is determined according to a predetermined path of pose displacement.

5. responsive to detecting the contact force by the force sensor, determining a direction away from a contact point associated with the detected contact force; and controlling the robotic manipulator to move the end effector and object in the determined direction away from the contact point.

6. The computer-implemented method of claim 1 , wherein the image comprises a depth image.

7. 7. The computer-implemented method of claim 6, further comprising removing from the depth image any features having associated depth values ​​outside a range corresponding to a volume between the end effector and a top plane of the containment space.

8. 8. A computer-implemented method according to claim 6 or 7, comprising projecting the depth image onto a plane parallel to a bottom surface of the storage space to obtain a set of two-dimensional points.

9. The computer-implemented method of claim 8 , wherein determining the principal axis of the object in the image is based on the set of two-dimensional points.

10. The computer-implemented method of claim 9 , wherein determining the principal axes of the object in the image is done by performing a principal component analysis using the set of two-dimensional points.

11. Controlling the robotic manipulator to manipulate the object into the first object pose comprises: determining a planar rotation of the end effector to align the primary axis of the object with the axis of the receiving space; and controlling the robot manipulator to perform the planar rotation of the end effector.

12. determining a spatial dimension of the object based on the image; Obtaining a spatial dimension of the accommodation space; and determining the first object pose based on a comparison of the spatial dimensions of the object and the spatial dimensions of the containing space.

13. 13. The computer-implemented method of claim 12, wherein determining the first object pose comprises adjusting an initial object pose to reduce a determined overhang of the spatial dimension of the object over the spatial dimension of the containment space.

14. determining an end effector pose of the end effector; and adjusting the first object pose based on the end effector pose.

15. A computer program comprising instructions which, when executed by a computer, cause the computer to perform a computer-implemented method according to any one of claims 1 to 14.

16. 16. A computer readable data carrier storing the computer program of claim 15.

17. A controller for a robotic manipulator, the controller being configured to execute a computer-implemented method according to any one of claims 1 to 14.

18. 20. A robotic packing system comprising: a controller according to claim 17; and the robotic manipulator for packing objects.

Images