Method and control system for controlling a robotic manipulator

Abstract

A control system and method for controlling a robotic manipulator, comprising: acquiring depth data based on an image of the container captured by a camera following placement or removal of a first object into or from the container by the robotic manipulator, and based on the depth data, a determination is made as to whether the first object or a given object comprising a different second object within the container exceeds a height threshold associated with the container. In response to determining that the given object exceeds the height threshold, a signal is generated indicating that the container is in an over-height condition, and a control signal is output and configured to control the robotic manipulator to manipulate the given object within the container based on the generated signal.

Description

[Technical Field]

[0001] The present disclosure relates to robotic control systems, and more particularly to systems and methods for use in packing objects into containers. [Background technology]

[0002] Bin packing is a central problem in computer vision and robotics. The goal is to have a system involving sensors and a robot that uses a suction gripper, parallel gripper, or other type of robotic end effector to grasp items and pack them into bins, e.g., containers. The packing system can be combined with a bin-picking system that uses the same or a different robot to initially pick objects in random poses (positions / orientations) from different bins using the same or a different type of end effector.

[0003] However, current systems have problems, including a focus on planning and avoiding all contacts during packing and an assumption that only rigid objects are being packed, which means the system is not practical in real-world scenarios. For example, general-purpose packing solutions typically do not consider the specificity of the grocery packing problem: for example, a packing algorithm should be able to handle unexpected errors and be robust when performing packing attempts in real-world scenarios. Summary of the Invention

[0004] A method for controlling a robotic manipulator is provided, the method comprising: acquiring depth data based on an image of the container captured by a camera following placement of a first object in the container or removal of a first object from the container by the robotic manipulator; determining, based on the depth data, whether a given object, comprising the first object or a different second object in the container, exceeds a height threshold associated with the container; generating a signal indicating that the container is in an over-height state in response to determining that the given object exceeds the height threshold; and outputting, based on the generated signal, a control signal configured to control the robotic manipulator to manipulate the given object in the container.

[0005] Optionally, the control signals are generated by teleoperation of the robotic manipulator. Optionally, the teleoperation is based on image data from another camera attached to the robotic manipulator.

[0006] Optionally, the control signals are generated by a manipulation algorithm. Optionally, the manipulation algorithm is configured to generate the control signals for the robot manipulator based on image data from another camera attached to the robot manipulator.

[0007] Optionally, the method comprises, following manipulation of the given object by the robotic manipulator, acquiring further depth data based on a further image of the container captured by the camera, determining whether the given object exceeds a height threshold based on the further depth image, and in response to determining that the given object exceeds the height threshold, generating a further signal indicative of the container being in an over-height condition, and outputting the further signal in a request for teleoperation of the robotic manipulator to further manipulate the given object within the container. Optionally, the method comprises outputting a further control signal configured to control the robotic manipulator to manipulate the given object within the container based on the further generated signal.

[0008] Optionally, the height threshold is a first threshold, the excessive height condition is a first excessive height condition, the signal is a first signal, and the control signal is a first control signal, and the method comprises: determining, based on the depth data, whether a given object in the container exceeds a second height threshold that is less than the first height threshold; generating a second signal indicating that the container is in a second excessive height condition in response to determining that the given object exceeds the second height threshold and does not exceed the first height threshold; and outputting a second control signal generated by a manipulation algorithm based on the generated second signal and configured to control a robotic manipulator to manipulate the given object in the container. Optionally, the method comprises outputting, based on the generated first signal, a request to teleoperate a robotic manipulator to manipulate the given object in the container. Optionally, the method comprises outputting the first control signal generated by teleoperation of the robotic manipulator.

[0009] Optionally, the control signals are configured to control the robotic manipulator to re-grasp the object, move the object and release the object into the container.

[0010] Optionally, the control signal is configured to control the robotic manipulator to manipulate the object in the container by a non-grasping operation.

[0011] Optionally, the camera is attached to a robotic manipulator.

[0012] Optionally, the camera is configured for use in an automated pick and place process in which a robotic manipulator is controlled to pick and place an object between selected containers from a plurality of containers including the container based on images captured by the camera.

[0013] Optionally, the height threshold corresponds to the top of the container.

[0014] Optionally, the height threshold is between 1 mm and 50 mm above the top of the container.

[0015] Optionally, determining whether the object exceeds a height threshold comprises searching for points in the depth data located within a predetermined search area defined based on the height threshold, Optionally, the search area is defined based on a dimension of the container.

[0016] Optionally, the signal indicating that the container is in an over-height condition comprises position data representing the position of a given object. Optionally, the position data represents the position of the given object relative to a camera or robotic manipulator.

[0017] In a related aspect, there is provided a computer program comprising instructions which, when executed by a computer, cause the computer to perform the provided method. In a further related aspect, there is provided a computer readable data carrier having stored thereon a computer program.

[0018] In a related aspect, a control system for a robotic manipulator is provided, wherein the controller is configured to perform the provided method.

[0019] In a related aspect, there is provided a robotic packing system comprising the control system described above and a robotic manipulator for packing objects. [Brief explanation of the drawings]

[0020] Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which like reference numerals indicate the same or corresponding parts, and in which: [Figure 1] FIG. 1 is a schematic diagram of a robotic packing system according to an embodiment. [Figure 2A] FIG. 2A is a schematic side view of a robotic packing system according to an embodiment. [Figure 2B] FIG. 2B is a schematic side view of a robotic packing system according to an embodiment. [Figure 3] FIG. 3 is a perspective view of a robotic packing system including a representation of an over-height region, according to an embodiment. [Figure 4] FIG. 4 is a perspective view of a robotic packing system positioned on a grid structure, according to an embodiment. [Figure 5] FIG. 5 shows a flowchart illustrating a method for controlling a robotic manipulator according to an embodiment.

[0021] In the drawings, like features are designated by like reference numerals where appropriate, incremented by, for example, multiples of 10 or 100 according to the figure number. DETAILED DESCRIPTION OF THE INVENTION

[0022] In the description that follows, several specific details are included in the following description to provide a thorough understanding of various disclosed embodiments. However, one skilled in the art will recognize that the embodiments may be practiced without one or more of these specific details, or with other methods, components, materials, etc. In some instances, well-known structures associated with gripper assemblies and / or robotic manipulators (processors, sensors, storage devices, network interfaces, workpieces, tension members, fasteners, electrical connectors, mixers, etc.) have not been shown or described in detail to avoid unnecessarily obscuring the description of the disclosed embodiments.

[0023] Unless the context requires otherwise, the word "comprise" and variations thereof, such as "comprises" and "comprising," are to be interpreted in this specification and the appended claims in their open and inclusive sense, i.e., "including but not limited to."

[0024] References throughout this specification to "one," "a," or "another" as applied to an "embodiment" or "example" mean that a particular reference feature, structure, or characteristic described in connection with an embodiment, example, or implementation is included in at least one embodiment, example, or implementation. Thus, the appearances of phrases such as "in one embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments, examples, or implementations.

[0025] It should be noted that, as used in this specification and the appended claims, the forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. It should also be noted that the term "or" is generally used in its sense including "and / or" unless the content clearly dictates otherwise.

[0026] The phrase "movement in the n-direction," where n is one of x, y, and z, (and related phrases) is intended to mean movement substantially along or parallel to the n-axis in either direction (i.e., toward the positive end of the n-axis or toward the negative end of the n-axis). In this document, the term "connect" and its derivatives are intended to include the possibility of direct and indirect connections. For example, "x is connected to y" is intended to include the possibility that x is directly connected to y with no intervening components, and the possibility that x is indirectly connected to y with one or more intervening components. When a direct connection is intended, "directly connected," "directly connected," or similar terms are used. Similarly, the term "support" and its derivatives are intended to encompass the possibilities of direct contact and indirect contact. For example, "x supports y" is intended to encompass the possibilities of x directly supporting and directly contacting y without any intervening components, and the possibilities of x indirectly supporting y with one or more intervening components contacting x and / or y. The term "attach" and its derivatives are intended to encompass the possibilities of direct and indirect attachment. For example, "x is attached to y" is intended to encompass the possibilities of x being directly attached to y without any intervening components, and the possibilities of x being indirectly attached to y with one or more intervening components. As used herein, the term "comprises" and its derivatives are intended to have an inclusive, rather than exclusive, meaning. For example, "x comprises y" is intended to encompass the possibilities of x including one and only one y, multiple ys, or one or more ys and one or more other elements. When an exclusive meaning is intended, the phrase "x consists of y" is used to mean that x includes only y and nothing else. As used herein, "controller" is intended to include any hardware suitable for controlling (e.g., providing instructions to) one or more other components. For example, a processor with one or more memories and appropriate software processes data for one or more components and sends appropriate instructions to the components to enable the components to perform their / their intended function.

[0027] As used throughout this specification, the term "pose" refers to the position and orientation of a given object in space. For example, the six-dimensional (6D) pose of an object includes values ​​in three translational dimensions (e.g., corresponding to position) and three rotational dimensions (e.g., corresponding to orientation) of the object.

[0028] Generally, this description introduces a system and method for automatically checking whether a container available to receive items manipulated by a robotic manipulator is in an "over-height" condition, e.g., whether it has one or more items protruding from the top, e.g., upper edge, of the container. This is done using depth data acquired via a camera, e.g., a depth image of the container captured after interaction between the robotic manipulator and the container. Based on the depth image captured after a pick, place, or pick-and-place operation, it is determined whether an object within the container protrudes above the top of the container (e.g., a set height threshold at or above the top edge or plane of the container). A positive determination triggers an over-height condition for the container. By manipulating the protruding object within the container, the over-height condition is signaled for the robotic manipulator to resolve the over-height condition (automatically and / or via remote control).

[0029] Automated overheight checks are used (e.g., as a microservice) to reduce the likelihood that a container will be packed or picked by a robotic manipulator, for example, at a picking station, leaving one or more items protruding above the tote height, which could cause problems when storing or moving the container. For example, a container in an overheight condition may make it more difficult to store or move the container with equipment. In the context of an automated storage and retrieval system (ASRS or AS / RS), a container handling device (e.g., a retrieval robot) may struggle to handle a container in an overheight condition. For example, one or more items protruding above the tote height may obstruct the container handling device when handling the container.

[0030] Overall, the present system and method avoids the need to install additional sensors, such as laser scanners or infrared presence sensors, and their associated cabling, compared to known systems and methods. Thus, the spatial constraints of a picking station, particularly on or within a grid-like ASRS where the number of storage grid cells picked up by the picking station is limited, are more easily addressed for installing additional sensors. Similarly, the present system and method avoids the need to mount such sensors (e.g., laser or infrared scanners) on a robotic manipulator, which would add bulk and make the robotic manipulator impractical for performing pick-and-place operations.

[0031] Instead, the present system and method detect over-height containers by utilizing a camera that may already be available to the robotic manipulator for other tasks. For example, a point cloud captured by a depth camera is automatically scanned for any point in three-dimensional space measured above the container. Such points are measurements corresponding to one or more objects that cause the container to be over-height. For example, measurement data from the point cloud, including the location of one or more over-height portions of one or more objects, can be used to control the robotic manipulator to manipulate one or more objects within the container.

[0032] General System 1 illustrates an example of a robotic packing system 100 that may be adapted for use with the present assemblies, devices, and methods. The robotic packing system 100 may form part of an online retail operation, such as an online grocery retail operation. Furthermore, the present invention may be applied to any other operation requiring packing of items. For example, the robotic packing system 100 may be adapted to pick or sort articles, e.g., as a robotic picking / packing system, sometimes referred to as a "pick-and-place robot."

[0033] The robotic packing system 100 includes a manipulator apparatus 102 that includes a robotic manipulator 121. The manipulator 121 is an electromechanical machine that includes one or more appendages, such as a robotic arm 120, and an end effector 122 attached to the end of the robotic arm 120. The end effector 122 is a device configured to interact with an environment to perform tasks, including, for example, grasping, grasping, releasably engaging, or otherwise interacting with an item. Examples of end effectors 122 include jaw grippers, finger grippers, magnetic or electromagnetic grippers, Bernoulli grippers, vacuum suction cups, electrostatic grippers, van der Waals grippers, capillary grippers, cryogenic grippers, ultrasonic grippers, and laser grippers.

[0034] The robotic manipulator 121 is capable of grasping and manipulating objects. In the case of a pick-and-place application, the robotic manipulator 121 is configured, for example, to pick an item from a first location and place the item at a second location.

[0035] The manipulator device 102 is communicatively coupled via a communication interface 104 to other components of the robotic packing system 100, such as one or more optional operator interfaces 106 that allow an observer to observe or monitor the system 100 and the manipulator device 102. The operator interface 106 may include a WIMP interface and an output display of descriptive text or a dynamic representation of the manipulator device 102 in a context or scenario. For example, the dynamic representation of the manipulator device 102 may include a video feed, e.g., a computer-generated animation. Examples of suitable communication interfaces 104 include a wire-based network or communication interface, an optical-based network or communication interface, a wireless network or communication interface, or a combination of wired, optical, and / or wireless networks or communication interfaces.

[0036] The exemplary robotic packing system 100 also includes a control system 108 including at least one controller 110 communicatively coupled to the manipulator device 102 and any other components of the robotic packing system 100 via a communication interface 104. The controller 110 comprises a control unit or computing device having one or more electronic processors. Embedded within the one or more processors is computer software comprising a set of control instructions provided as processor-executable data that, when executed, causes the controller 110 to issue actuation commands or control signals to the manipulator device 102. For example, the actuation commands or control signals cause the manipulator 121 to perform various methods and actions, such as identifying and manipulating items.

[0037] The one or more electronic processors may include at least one logic processing unit, such as one or more microprocessors, central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), application specific integrated circuits (ASICs), programmable gate arrays (PGAs), program logic units (PLUs), etc. In some implementations, the controller 110 is a smaller processor-based device, such as a mobile phone, single-board computer, embedded computer, etc., which may be referred to, or interchangeably referred to, as a computer, server, or analyzer. A set of control instructions may also be provided as processor-executable data associated with the operation of the system 100 and manipulator device 102 contained on a non-transitory computer-readable storage device 112, which forms part of the robotic packing system 100 and is accessible to the controller 110 via the communications interface 104.

[0038] In some implementations, storage device 112 includes two or more separate devices. Storage device 112 may include, for example, one or more volatile storage devices, such as random access memory (RAM), and one or more non-volatile storage devices, such as read-only memory (ROM), flash memory, magnetic hard disk drive (HDD), optical disk, solid-state disk (SSD), etc. Those skilled in the art will appreciate that storage devices may be implemented in a variety of ways, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), network drive, flash memory, digital versatile disk (DVD), any other form of computer- and processor-readable memory or storage medium, and / or combinations thereof. Storage devices may be read-only or read-write, as desired.

[0039] The robotic packing system 100 includes a sensor subsystem 114 comprising one or more sensors that detect, sense, or measure the condition or state of the manipulator apparatus 121 and / or conditions within the environment or workspace in which the manipulator 102 operates, and generate or provide corresponding sensor data or information. The sensor information includes environmental sensor information representative of environmental conditions within the workspace of the manipulator 121, as well as information representative of the condition or state of the manipulator apparatus 102, including its various subsystems and components, and characteristics of the item being manipulated. The acquired data may be transmitted via the communication interface 104 to the controller 110 to instruct the manipulator 121 accordingly. Such information may include, for example, diagnostic sensor information useful for diagnosing the state or status of the manipulator apparatus 102 or the environment in which the manipulator 121 operates.

[0040] Such sensors include, for example, one or more cameras or imagers 116 (e.g., responsive in the visible and / or non-visible range of the electromagnetic spectrum, including infrared and ultraviolet). The one or more cameras 116 may include a depth camera, e.g., a stereo camera, for capturing depth data along with color channel data within the imaged scene. Other sensors in the sensor subsystem 114 may include one or more of contact sensors, force sensors, strain gauges, vibration sensors, position sensors, attitude sensors, accelerometers, radar, sonar, lidar, touch sensors, pressure sensors, load cells, microphones 118, weather sensors, chemical sensors, etc. In some implementations, the sensors include diagnostic sensors for monitoring the status and / or health of an on-board power source (e.g., a battery array, an ultracapacitor array, or a fuel cell array) within the manipulator device 102.

[0041] In some implementations, the one or more sensors comprise a receiver for receiving position and / or orientation information about the manipulator 121, for example, a Global Positioning System (GPS) receiver for receiving GPS data, two or more time signals from which the controller 110 generates a position measurement based on data in the signal, such as time of flight, signal strength, or other data to provide a position measurement. Also, for example, one or more accelerometers, which may also form part of the manipulator device 102, may be provided on the manipulator 121 to obtain inertial or orientation data about its movement in one, two, or three axes.

[0042] The robotic manipulator 121 of the system 100 may be piloted by a human operator at the operator interface 106. In a human operator control (or "pilot") mode, the human operator observes a representation of sensor data, e.g., video, audio, or tactile data, received from one or more sensors of the sensor subsystem 114. The human operator then acts conditioned by his or her perception of the representation of the data and creates information or executable control instructions to direct the manipulator 121 accordingly. In a pilot mode, the manipulator device 102 may execute control instructions in real time (e.g., without additional delay) as received from the operator interface 106 based on the sensed information and without consideration of other control instructions.

[0043] In some implementations, the manipulator device 102 operates autonomously, i.e., without a human operator creating control instructions in the operator interface 106 to direct the manipulator 121. The manipulator device 102 may operate in an autonomous control mode by executing autonomous control instructions. For example, the controller 110 can use sensor data from one or more sensors in the sensor subsystem 114. The sensor data can be associated with operator-generated control instructions from one or more times the manipulator device 102 was in pilot mode to generate autonomous control instructions for subsequent use. For example, deep learning techniques can be used to extract features from the sensor data. Thus, in autonomous mode, the manipulator device 102 can autonomously recognize the characteristics or state of its environment and the item being manipulated. In response, the manipulator device 102 performs one or more defined actions or tasks. For example, the manipulator device 102 executes a pipeline or sequence of actions or tasks.

[0044] In some implementations, the controller 110 autonomously perceives features or conditions of the environment surrounding the manipulator 121 and one or more virtual items composited into the environment. The environment is represented by sensor data from the sensor subsystem 114. In response to the representations being presented, the controller 110 issues control signals to the manipulator device 102 to perform one or more actions or tasks.

[0045] In some examples, the manipulator device 102 may be autonomously controlled at a given time while being piloted, manipulated, or controlled by a human operator at another time. That is, the manipulator device 102 may change to operate under an autonomous control mode and operate under a pilot (i.e., non-autonomous) mode. In another mode of operation, the manipulator device 102 may replay or execute control instructions previously executed in the pilot mode. That is, the manipulator device 102 may operate based on replayed pilot data without sensor data.

[0046] Manipulator device 102 further includes a communication interface subsystem 124 (e.g., a network interface device) communicatively coupled to bus 126 and providing bidirectional communication with other components of system 100 (e.g., controller 110) via communication interface 104. Communication interface subsystem 124 may be any circuitry that provides bidirectional communication of processor-readable data and processor-executable instructions, such as a wireless (e.g., radio or microwave frequency transmitter, receiver, transceiver) port and / or associated controller. Suitable communication protocols include FTP, HTTP, web services, SOAP with XML, cellular (e.g., GSM, CDMA), Wi-Fi compliant, Bluetooth compliant, etc.

[0047] Manipulator device 102 further includes a motion subsystem 130 communicatively coupled to robotic arm 120 and end effector 122. Motion subsystem 130 comprises one or more motors, solenoids, other actuators, linkages, drive belts, etc. operable to move robotic arm 120 and / or end effector 122 through a range of motion in accordance with actuation commands or control signals issued by controller 110. Motion subsystem 130 is communicatively coupled to controller 110 via bus 126.

[0048] The manipulator device 102 also includes an output subsystem 128 that includes one or more output devices, such as a speaker, a light, or a display, that enable the manipulator device 102 to transmit signals into the workspace, for example, to communicate with an operator and / or another manipulator device 102.

[0049] Those skilled in the art will understand that the components of manipulator device 102 may be varied, combined, divided, or omitted. In some examples, one or more of communication interface subsystem 124, output subsystem 128, and motion subsystem 130 are combined. In other examples, one or more subsystems (e.g., motion subsystem 130) are divided into additional subsystems.

[0050] Robot System 2 illustrates an example of a robotic packing system 200 that includes a robotic manipulator 221, such as an implementation of the robotic manipulator 121 described in the previous example. According to this example, the robotic manipulator 221 includes a robotic arm 220, an end effector 222, and a motion subsystem 230. The motion subsystem 230 is communicatively coupled to the robotic arm 220 and the end effector 222 and configured to move the robotic arm 220 and / or the end effector 222 according to actuation commands or control signals issued by a controller (not shown). The controller, such as the controller 110 described in the previous example, is part of a manipulator apparatus that includes the robotic manipulator 221.

[0051] The robotic manipulator 221 is positioned within the workspace to manipulate an object grasped by, for example, the end effector 222 to pack the object into a receiving space, such as, for example, a container (or “bin” or “tote”) 244. For example, the robotic packing system 200 may be implemented in an automated storage and retrieval system (ASRS), such as a picking station thereof. The ASRS typically includes multiple containers arranged to store items and one or more load handling devices or automated guided vehicles (AGVs) for retrieving one or more containers 244 during the fulfillment of customer orders. At the picking station, items are picked from and / or placed into one or more retrieved containers 244. One or more containers in the picking station may be considered storage containers or shipping containers. A storage container is a container that remains within the ASRS and holds each of the products that can be transferred from the storage container to a shipping container. A shipping container is a container that is introduced into the ASRS when empty and is loaded with multiple different products. The shipping container may comprise one or more bags or cartons that the product may be loaded into. The shipping container may be substantially the same size as the storage container. Alternatively, the shipping container may be slightly smaller than the storage container so that the shipping container can be nested within the storage container.

[0052] Thus, the robotic packing system 200 can be used at a picking station to pick an item from one container, e.g., a storage container, and place the item in another container, e.g., a shipping container. Thus, the picking station can have two sections, one for storage containers and one for shipping containers. The arrangement of the picking station, e.g., its sections, can be varied and selected as appropriate. For example, the two sections can be located on two sides of an area, or one section can be located above or below the other. In some cases, the picking station is located away from the storage locations of containers within the ASRS, e.g., away from the storage grid in a grid-based ASRS. Thus, a load handling device can deliver and collect containers to / from one or more ports of the ASRS linked to the picking station, e.g., by a chute. In other examples, the picking station is positioned to directly interact with a subset of storage locations within the ASRS, e.g., to pick and place items between containers located in the subset of storage locations. For example, in the case of a grid-based ASRS, the picking stations may be positioned on the grid of the ASRS.

[0053] 4 shows an example of a robotic packing system 400, which includes a robotic manipulator 421 as described, located on a section of a grid 405, which in the example forms part of an ASRS. For example, a load handling device (or "retrieval robot") may navigate along two orthogonal axes of the grid 405 to retrieve a container from a stack of containers below the grid 405. Meanwhile, a robotic manipulator 421 located at a picking station on the grid is configured to pick and pack items among containers arranged in an array of grid spaces that form part of the picking station, e.g., containers retrieved by the retrieval robot. The containers (not shown in FIG. 4) located at picking locations 440 may be storage containers or shipping containers.

[0054] In the schematic diagram of a robotic packing system 400 shown in FIG. 4 , which includes a robotic picking station on an ASRS, the robotic manipulator 421 is received on a pedestal connected to the framework of the storage system, e.g., grid structure 405, such that the robotic arm is mounted on the storage system. For example, the pedestal may be connected to one or more of the upright and / or horizontal members of the storage system. In an alternative example, a mount may be used to connect the robotic arm to the framework of the grid structure 405. For example, one or more mounting members may attach the robotic arm 421, e.g., a base of the robotic arm, to one or more members of the storage system.

[0055] The robotic manipulators 221, 321 of the present systems 200, 300 may include one or more end effectors 222, 322. For example, the robotic manipulators 221, 321 may include two or more different types of end effectors. In some examples, the robotic manipulators 221, 321 may be configured to exchange a first end effector for a second effector. In some cases, a controller may send instructions to the robotic manipulators 221, 321 regarding which end effector 222, 322 to use for each different object or product (or stock-keeping unit “SKU”) being packed. Alternatively, the robotic manipulators 221, 321 may determine which end effector to use based on the product's weight, size, shape, etc. Previous success and / or failure in grasping and moving items may be used to update the end effector selection for a particular SKU. This information may be fed back to the controller so that success / failure information can be stored and shared among different picking / packing stations. The robot manipulators 221, 321 may be capable of changing end effectors. For example, a picking / packing station may include a storage area that can receive one or more end effectors. The robot manipulators 221, 321 may be configured to allow an end effector in use to be removed from the robot arm 220 and placed in the end effector storage area. An additional end effector may then be removably attached to the robot arm 220 so that it can be used for a subsequent picking / packing operation. The end effector may be selected according to the planned picking / packing operation.

[0056] The robotic packing system 200 of FIG. 2 includes a depth camera 216 attached to the robotic manipulator 221. For example, the depth camera 216 may be mounted on or near an end effector, such as on or near the wrist of the robotic arm. Additionally or alternatively, the depth camera 216 may be mounted on or near the elbow of the robotic arm. In other examples, the depth camera 216 is supported by a frame structure 240, for example, comprising a scaffold to which the depth camera 216 is attached. Depth cameras, also known as RGB-D cameras or "range cameras," generate depth information by illuminating a scene with "structured light" or infrared speckle patterns using techniques such as time-of-flight, LIDAR, interferometry, and stereo triangulation.

[0057] The depth camera is configured to capture depth data, e.g., a depth image, of a scene including one or more container locations 340, as shown in FIG. 3 . Each container 344 can be positioned at a respective container location 340 such that the end effector 322 of the robotic manipulator 321 can interact with items stored therein. For example, the depth camera is positioned to have a view of the workspace of the robotic manipulator 221, including a given container 344, following placement of an object in or removal of an object from the container 344 by the robotic manipulator 321. As previously mentioned, the container locations 340 may be at picking stations and / or may correspond to storage locations within the grid structure of a grid-based ASRS.

[0058] In an example, the depth camera is configured for use in an automated pick-and-place process in which a robotic manipulator 321 is controlled to pick and place objects between selected containers 344 based on images captured by the depth camera.

[0059] The depth camera 216 may correspond to one or more cameras or imagers 116 in the sensor subsystem 114 of the robotic packing system 100 described with reference to Figure 1. As described, the depth camera 216 of the robotic packing system 200 is configured to capture a depth image. For example, the depth (or "depth map") image includes depth information of the scene viewed by the camera 216.

[0060] The point cloud generator may be associated with a depth camera or imager 216, such as a LIDAR sensor, positioned to view the workspace, e.g., a given container 344 and its contents. Examples of structured light devices for use in point cloud generation include a Kinect® device by Microsoft®, a time-of-flight device, an ultrasound device, a stereo camera pair, and a laser stripper. These devices typically generate depth map images that are processed by the point cloud generator to generate the point cloud.

[0061] It is common to calibrate the depth map image for aberrations in the camera's lens and sensor. Once calibrated, the depth map can be converted into a set of metric 3D points known as a point cloud. Preferably, the point cloud is an organized point cloud, meaning that each 3D point lies on the line of sight of a separate pixel, resulting in a one-to-one correspondence between the 3D point and the pixel. Organization is desirable because it allows for more efficient point cloud processing. A further part of the calibration process determines the camera's pose, i.e., its position and orientation, relative to the reference frame of the robot packing system 200 or the robot manipulator 221. The reference frame may be the base of the robot manipulator 221, although any known reference frame will work, for example, a reference frame located at the wrist joint of the robot arm 220. Thus, a point cloud can be generated based on the depth map and information about the lens and sensor used to generate the depth map. Optionally, the generated depth map can be converted into the reference frame of the robot packing system 200 or the robot manipulator 221. For simplicity, the depth camera or imager 116, 216 is shown as a single unit in Figures 1, 2A, and 2B. However, it will be appreciated that the functions of depth map generation and depth map calibration may each be performed by a separate unit, for example, the depth map calibration means may be integrated into the control system of the robotic packing system 100, 200.

[0062] Control System A control system of the robotic manipulator 221, e.g., the control system 108 communicatively coupled to the manipulator device of the previous example, is configured to acquire depth data based on images captured by the depth camera 216. As described herein, the images include the container 244, 344 after placement / removal of an object into / from the container 244, 344 by the robotic manipulator 221, 321.

[0063] The control system processes the depth data to determine whether a given object, which may be an object just placed in the container or a different object already located in the container, exceeds a height threshold associated with the container. An object is considered to exceed the height threshold, for example, when at least a portion of the object exceeds the height threshold.

[0064] In some examples, the height threshold corresponds to the top of the container and can be expressed as, for example, a plane coincident with the top of the container. Thus, an object within the container that extends beyond the top of the container can be considered to exceed the height threshold. FIG. 2A shows another example in which the height threshold 264 is offset from a plane 262 coincident with the top of the container 244. For example, the height threshold 264 is 1 mm to 50 mm above the top of the container. Thus, the height threshold 264 can be expressed as a plane parallel to the plane 262 coincident with the top of the container 244. The parallel planes 262, 264 can be coincident or offset by a predetermined distance, for example, 1 mm to 50 mm.

[0065] In an example, determining whether a given object in a container 244, 344 exceeds the height threshold 264 involves searching for points in the depth data that fall within a search region (or "excess height region") 370 (shown in FIG. 3) above the container 244, 344. The search region 370 is bounded (in the z direction), for example, by the height threshold as a lower limit. The upper limit of the search region 370 (in the z direction) can be set, for example, to a predetermined height or depth value, or a set depth difference from the lower limit, to define the height of the search region 370. The boundaries of the search region 370 in the other orthogonal (x and y) directions, in an example, are based on the dimensions of the container 344. For example, the length and width of the container 344 are set as the length and width of the search region 370. Thus, the search region 370 is located above the container in depth space, and its lower boundary is set as the height threshold that coincides with the top of the container or is a predetermined height above the top of the container.

[0066] Thus, the control system can process the depth data to find features having associated depth values ​​within the boundaries of search area 370. For example, the control system may extract features from the depth data by, e.g., removing features from the image that have depth values ​​outside search area 370. If the depth data comprises a point cloud, for example, the control system extracts points from the point cloud that are within search area 370 by, e.g., removing points from the point cloud that are outside search area 370. The control system can thus isolate features of the depth data that are within search area 370, e.g., "excess height" features or points, based on the depth information.

[0067] In some examples, outlier points detected within the search area 370 are removed from the determined excess height points. For example, if excess height points are clustered within the area of ​​the search area 370, they can be considered part of an excess height object. On the other hand, if isolated points are detected within the search area 370, e.g., far away from the detected cluster, these outliers are removed from the set of excess height points. For example, statistical methods are used to remove points that are further away from their neighbors compared to the average distance of the point cloud using a threshold (e.g., based on the standard deviation of the average distance across the point cloud). Other methods for filtering depth data, e.g., point clouds, can be used, such as fitting a smooth surface to the points and removing outliers that are a large distance from the fitted surface.

[0068] In some cases, search area 370 is modified to exclude portions of the perimeter, such as a picking station. Thus, search area 370 may first be defined based on the excess height threshold and the container dimensions, and then modified to exclude, e.g., subtract from search area 370, any overlapping exclusion regions defined based on the dimensions of features in the perimeter area.

[0069] In response to determining that the object exceeds the height threshold, the control system generates a signal indicating that the container is in an over-height condition, for example, an over-height condition is a defined state of the container that represents a container containing an object that extends beyond a set height threshold.

[0070] The control system outputs control signals configured to control the robotic manipulators 221, 321 to manipulate the over-height objects detected within the container based on the generated signals. For example, a signal indicating that the container is in an over-height state may be transmitted between different controllers of the control system, e.g., from a controller associated with the depth camera 216 to a controller associated with the motion subsystem 230 configured to move the robotic arm 220 and / or end effector 222 according to the control signals issued by the controller, e.g., in a vision system comprising the depth camera 216. In such a case, the control signals may be generated by a manipulation algorithm based on the over-height signal. The manipulation algorithm is configured to generate control signals for the robotic manipulators 221, 321 based on image data from another camera (not shown) attached to the robotic manipulators 221, 321, for example. For example, a camera attached to the wrist of the robotic manipulators 221, 321 may acquire an image of a scene including the end effector 222, 322 to control the end effector 222, 322 within that environment. In these examples, the robotic packing system 200, 300 can automatically, e.g., without human intervention, manipulate over-height objects within a container that are detected by the control system based on depth images from the depth camera 216.

[0071] In another example, the control signal is generated by remote control of the robotic manipulator. For example, a signal indicating that a container is in an overheight state may be sent externally from the control system, e.g., at the request of remote control of the robotic manipulator. The control system may receive the control signal generated by the remote control, e.g., at an interface, and output the control signal, e.g., from a controller associated with the motion subsystem 230 of the robotic manipulator 221, 321. As described for the manipulation algorithm, the remote control may be based on image data from a separate camera mounted on the robotic manipulator 221, 321, e.g., on its wrist. During teleoperation, a human operator remotely controls the movement of the robotic manipulator 221, 321, e.g., at a different location. A communication channel between the operator and the robotic manipulator 221, 321 allows signals to be transmitted between them. For example, sensory information may be transmitted from the control system of the robotic manipulator 221, 321, e.g., including image data captured by a camera mounted on the robotic manipulator 221, 321. In some cases, the detected excess height heat map is overlaid on a color image displayed to the teleoperator. The teleoperator can generate control signals using a human interface device, such as a joystick, gamepad, keyboard, pointing device, or other input device. The control signals are sent to the robotic manipulator and control the robotic manipulator via a control system.

[0072] In some cases, a hybrid of manipulation algorithms and teleoperation is used to generate control signals for the robot manipulator 221, 321. For example, an operator may use a human input device to define the area of ​​the over-height item to be grasped by the robot manipulator 221, 321—e.g., the flat surface of a box if the end effector 222, 322 is equipped with a suction end effector (as shown in the example of FIG. 3 ). The defined area of ​​the over-height item to be grasped may then be used as input to an automated picking attempt. In other words, in the hybrid case, grasp generation is performed by manual input rather than fully automatically by the manipulation algorithm. In some cases, teleoperation commands generated, for example, by a teleoperator, comprise a strategy, for example, a motion strategy and / or a grasping strategy. The teleoperator can click a single point in an image of a scene to move the robot manipulator, for example, in the direction of, e.g., the clicked (“target”) point in the scene. The robot manipulator may be moved, for example, from the edge of the over-height area to the target point, which is calculated automatically. If the automated picking attempt is still unsuccessful in grasping or otherwise manipulating the over-height object, the operator can fully operate the robotic manipulator 221, 321 to manipulate the object, as described above. It should be appreciated that some form of machine learning techniques may be utilized in the automated manipulation algorithms. In such cases, data generated during the remote operation of the robotic manipulator 221, 321 by the remote operator may be used to refine the manipulation algorithms used in the automated operation of the robotic manipulator 221, 321.

[0073] For example, the output control signals generated by the manipulation algorithm and / or teleoperation are configured to control the robotic manipulator 221, 321 to manipulate the over-height object detected in the container. For example, the control signals are configured to control the robotic manipulator 221, 231 to re-grasp the object, move the object, and release the object into the container. Additionally or alternatively, the control signals are configured to control the robotic manipulator 221, 321 to manipulate the object in the container via a non-grasp operation. A non-grasp operation involves the robotic manipulator 221, 321 manipulating the object without grasping it, for example, by gently pushing the object. The purpose of the control signal operation is to reposition the over-height object detected in the container so that it no longer exceeds the height threshold and the container is no longer in an over-height state.

[0074] In an example, the control system performs another over-height check after manipulating the over-height object in accordance with the output control signal. For example, the control system acquires an additional depth image of the container 244, 344 from the depth camera 216 following manipulation of the over-height object by the robotic manipulator 221, 321. The control system can determine whether the given object exceeds the height threshold based on the additional depth image. In response to determining that the given object exceeds the height threshold, the control system generates, for example, an additional signal indicating that the container is in an over-height state.

[0075] Further signals output by the control system may comprise a request for teleoperation of the robotic manipulator to further manipulate a given object within the container. Thus, in an example where an initial determination of an over-height object within a container results in automatic manipulation of the over-height object by the robotic manipulator 221, 321, a further determination that the container is in an over-height condition (due to the initial over-height object or a different object within the container) may result in a teleoperation request to resolve the over-height condition of the container.

[0076] A further control signal based on the further generated signal indicating that the container 244, 344 is in an over-height condition is output by the control system in the example. The further control signal is configured to control the robotic manipulator to manipulate the over-height object within the container 244, 344, for example, to attempt to resolve the over-height condition of the container 244, 344 determined in a check after the initial manipulation of the over-height object.

[0077] As described above for the control signals configured to control the robot manipulators 221, 321, further control signals may be generated by a teleoperation or automated operation algorithm and output via a controller associated with the motion subsystem 230 of the robot manipulators 221, 321.

[0078] In addition to or instead of the further over-height check, another check can be performed after manipulating the over-height object in accordance with the output control signal, i.e., the container pose can be redetermined to check whether the container has moved due to manipulation by the robotic manipulator 221, 321. For example, the over-height region (370) can be recalculated after interaction with the robotic manipulator, e.g., based on the new container pose.

[0079] In some examples, there are two or more height thresholds for determining an excessive height condition for a container. Figure 2B illustrates such a scenario. In this example, there is a first height threshold 264, as described in the previous example, and the control system is configured to generate a first signal indicating that the container is in a first excessive height condition in response to determining that a given object exceeds the first height threshold 264. The control system is further configured to output a first control signal configured to control the robotic manipulator 221 to manipulate the given object within the container 244 based on the generated first signal.

[0080] In the example of FIG. 2B , the control system is also configured to determine, based on the depth image captured by the depth camera 216, whether a given object within the container exceeds a second height threshold 266 that is less than the first height threshold 264. For example, the first height threshold 264 is a stricter height threshold than the second height threshold 266, i.e., it is at a higher height above the container 244. The first height threshold 264 and the second height threshold 266 can be considered as parallel planes that are offset from each other. For example, the two planes have a predetermined displacement between them in the vertical z-direction, or each have a predetermined displacement from the top of the container 244. In some examples, the second height threshold 266 corresponds to the top of the container and can be expressed as a plane that coincides with the top of the container, for example.

[0081] In response to determining that a given object exceeds the second height threshold but does not exceed the first height threshold, the control system is configured to generate a second signal indicating that the container is in, for example, a second over-height condition. Thus, the first and second over-height conditions allow for differentiation between levels of over-height, e.g., how far a given object extends beyond the upper boundary of the container 244, rather than a binary determination of whether the container is in an over-height condition or not.

[0082] In such instances involving different over-height conditions of the container, different actions can be taken depending on the over-height condition determined by the control system. For example, the control system is configured to output a second control signal generated by the manipulation algorithm based on the generated second signal and configured to control the robotic manipulator to manipulate a given object within the container. Thus, the control system triggers an automatic response to the second over-height condition involving the manipulation algorithm generating a second control signal to manipulate the over-height object in an attempt to resolve the second over-height condition of the container 244.

[0083] In some cases, the control system outputs a teleoperation request when a first signal is generated indicating that the container is in a first excessive height condition. Thus, in these examples, a first excessive height condition associated with a higher first height threshold exceeded by an object in the container has a different response than a second excessive height condition. A less severe second excessive height condition may trigger an automatic response involving a manipulation algorithm generating a control signal, while a more severe first excessive height condition triggers a more complex response including a teleoperation request. In the latter case, the control signal for controlling the robotic manipulator 221 to manipulate the object may be generated by teleoperation and obtained by the control system for implementation in the robotic manipulator 221, e.g., via the motion subsystem 230, as described in other examples.

[0084] Control Method 5 illustrates a computer-implemented method 500 for controlling a robotic manipulator, which may be one of the example robotic manipulators 121, 221, 321, 421 described with reference to FIGS. 1-4. Method 500 may be performed by one or more components of system 100 previously described, such as control system 108 or controller 110.

[0085] At 501, following placement of a first object into or removal of a first object from a container by a robotic manipulator, depth data based on an image of the container is acquired. The image is captured by a camera having a field of view of the container, for example, the camera is mounted on the robotic manipulator. In an example, the depth data comprises a point cloud.

[0086] At 502, method 500 involves determining, based on the depth data, whether a given object, comprising a first object or a different second object within the container, exceeds a height threshold associated with the container. An object is considered to exceed the height threshold, for example, when at least a portion of the object exceeds the height threshold. In an example, it is determined whether a given object (e.g., the first object or the second object) protrudes above a threshold plane having a set height in space based on the height of the container. For example, the threshold plane may coincide with the top edge of the container or be offset a set amount above the top edge of the container.

[0087] In response to determining that a given object exceeds the height threshold, a signal indicating that the container is in an over-height condition is generated at 503. For example, the generated signal may be used as an input to an operation algorithm for generating control signals for a robotic manipulator, or as an output to a teleoperation system for a remote operator for generating control signals for the robotic manipulator.

[0088] At 504, a control signal configured to control the robotic manipulator to manipulate a given object within the container is output based on the generated signal. As described herein, the control signal may be generated by the robotic manipulator's teleoperation and / or manipulation algorithms based on, for example, image data from another camera attached to the robotic manipulator. The other camera is configured to acquire sensory data, e.g., visual data, associated with the robotic manipulator's environment for use in controlling the robotic manipulator.

[0089] In some examples, method 500 involves obtaining further depth data based on further images of the container captured following manipulation of the given object by the robotic manipulator, e.g., in accordance with a control signal output as part of the provided method 500. It can then be determined based on the further depth data whether the given object still exceeds the height threshold, e.g., as a check that manipulation of the over-height object has resolved the over-height condition of the container. In response to determining that the given object still exceeds the height threshold, a further signal is generated, e.g., indicating that the container is in an over-height condition. The further signal may be included in a request for teleoperation of the robotic manipulator to further manipulate the given object within the container. For example, the further signal may comprise position data representing the position of the given object in the scene represented in the further image. Such position information may, for example, be relative to a coordinate system of the robotic manipulator. In some examples, a further control signal configured to control the robotic manipulator to manipulate the given object within the container is output based on the further generated signal. For example, further control signals may be generated by the required teleoperation and output to the robotic manipulator, e.g., its motion subsystem, for implementation in controlling the robotic manipulator.

[0090] As described in other examples, there may be multiple height thresholds. For example, the height threshold may be a first threshold, the excessive height condition may be a first excessive height condition, the signal may be a first signal, and the control signal may be a first control signal. The method may then involve determining, based on the depth data, whether a given object in the container exceeds a second height threshold that is less than the first height threshold. In response to determining that the given object exceeds the second height threshold but does not exceed the first height threshold, a second signal may be generated, indicating that the container is in a second excessive height condition, for example. Thus, the first and second signals may be used, for example, to distinguish between first and second excessive height conditions of the container. Thus, in such cases, different responses may be made to different excessive height conditions of the container. For example, the method may involve outputting a second control signal generated by a manipulation algorithm based on the generated second signal and configured to control a robotic manipulator to manipulate the given object in the container. If a given object in the container is determined to exceed a first height threshold (e.g., as well as a second height threshold), a request for teleoperation of the robotic manipulator may be made based on the generated first signal. The first control signal generated by teleoperation of the robotic manipulator may be output as part of the method, in examples.

[0091] The control signals output as part of method 500 are configured to control a robotic manipulator to manipulate an over-height object detected within the container. In some examples, the manipulation involves re-grasping the object, moving the object, and releasing the object within the container. In other examples, the manipulation is non-grasping, e.g., moving the object within the container without grasping the object.

[0092] In some examples, method 500 involves searching for points in the depth data that lie in a search region above the container. For example, the search region is defined as a region of space above the container, e.g., based on the (calibrated) image and the pose of the container. The pose of the container represents the location and orientation of the container in space. For example, a six-dimensional (6D) pose of the container includes respective values ​​in three translational dimensions (e.g., corresponding to position) and three rotational dimensions (e.g., corresponding to orientation) of the container.

[0093] The lower limit of the search area (in the z-direction) corresponds to a height threshold for determining whether the container is in an over-height condition. As described in other examples, the upper limit of the search area (in the z-direction) can be set, for example, to a predetermined height or depth in space or to a set depth difference from the lower limit. The boundaries of the search area in the x- and y-directions can be defined based on the dimensions of the container, for example, based on a (CAD) model of the container and / or direct height measurements of the container using camera-derived depth data (e.g., a depth map of the container captured by a depth camera). In such examples, if the search area is determined to be non-empty, the container is determined to be in an over-height condition.

[0094] The points present in the search area can be reprojected to the depth view to obtain pixel-wise detection. The pixel-wise detection can be represented as a pixel-wise detection map (or "heat map"). A heat map is a two-dimensional visualization of the detected points in the search area, for example, projected onto a single plane in the z direction. For example, the detected points in the search area are projected onto the bottom surface of the search area corresponding to a height threshold, and the heat map comprises a map of the points in the x and y directions and color or gradient information representing the respective heights of the points above the projection plane.

[0095] The pixel-by-pixel detection, e.g., a heat map, may be output as part of method 500 in examples. In some cases, the location of one or more of the over-height points or regions is output along with the heat map. For example, the pixel-by-pixel over-height detection is output along with a signal indicating that the container is in an over-height condition.

[0096] The method 500 described in the examples can be implemented by a control system for a robotic manipulator, such as one or more controllers, such as the control systems described above. For example, the control system may include one or more processors for executing the method 500 according to instructions, such as computer program code, stored on a computer-readable data carrier or storage medium.

[0097] Overall, the present system and method leverage the capabilities of depth cameras, which may already be mounted on, for example, robotic manipulators, to detect over-height containers between item picks. The compactness advantage makes the present system and method suitable for use in pick stations of an ASRS, for example, on top of a grid structure in a cube-based storage system. In such an implementation, one or more storage containers (or "totes") may be placed at the on-grid picking station by one or more retrieval robots (or "bots") for interaction with an on-grid robotic manipulator. The robotic manipulator is configured to pick items from a container (e.g., a storage tote) and place them in another container (e.g., a shipping tote). During picks by the robotic manipulator, the present system and method are used to check whether any totes are in an over-height condition, in which one or more items protrude beyond the top edge of the tote. In such a condition, the retrieval bots may not be able to grasp and lift the tote, for example, using their tote-gripper assemblies. Thus, the on-grid robotic manipulator may be configured to resolve an over-height condition by manipulating one or more over-height items detected within a given tote before the tote is removed from the picking station by the recovery bot.

[0098] In an example, the on-grid picking station may further include an optical sensor that may be positioned on the top surface of the pedestal that supports the robotic manipulator. The optical sensor may be used to identify the product during the picking process. The picking station may include multiple such optical sensors. In one example, the picking station may include four optical scanners, one optical scanner located at or near each corner of the pedestal. Each optical scanner may include a barcode reader. In an alternative configuration, one or more barcode scanners may be mounted on the robot arm, such that the barcode scanner moves with the arm. In a particular implementation, two barcode scanners may be mounted on the arm.

[0099] The above examples should be understood as illustrative. Further examples are contemplated. For example, the acquired depth data may be based on multiple images captured by multiple cameras at different respective positions or orientations, e.g., using two color cameras for stereoscopic vision. The multiple images are processed, for example, to obtain a depth map.

[0100] Additionally, the presented systems and methods involve acquiring and processing depth data. In many of the described examples, the depth data is captured by a depth camera, e.g., a type of camera equipped with one or more depth sensors and configured to analyze a scene to determine the distances of features and objects in the environment, from which the camera can create a 3D map of the scene. However, in other examples, the camera is not a specific depth camera, and the depth data is acquired by processing (two-dimensional) image data captured by the camera.

[0101] Thus, the depth data is based on an image captured by a camera, which may be a depth image or depth map captured by a depth camera, or an image that does not initially contain depth information and is later processed to obtain the depth data. For example, the depth data may be obtained directly from the depth camera, e.g., as a depth image or depth map output by the depth camera. Alternatively, there is intermediate processing of the camera output to obtain the depth data. For example, a depth image captured by a depth camera is processed to obtain a point cloud. Alternatively, an image captured by a camera that does not contain depth information is processed to obtain a depth image indicative of the depth data.

[0102] Processing of images to derive depth data that can be interpreted as an output image showing depth information can be performed using computer vision techniques, such as machine learning. For example, multiple images captured by a camera can be processed to generate a depth map, where multiple images are captured at different locations and processed as a stereo image. Alternatively, a single 2D image captured by a camera can be processed, for example, by a trained neural network to obtain depth information. Examples include using convolutional neural networks, deep neural networks, and supervised learning on segmented regions of an image using features such as texture variation, texture gradient, interposition, and shading to perform scene depth inference from a single image. This process involves assigning a depth value to each pixel in the image to convert, for example, an RGB image captured by an RGB camera into an RGB-D image of the type obtainable by a depth-sensing camera. For example, a depth estimator neural network is trained to determine distance for every pixel in a color image received from a camera (e.g., it can only return color information from a 2D scene). The neural network is trained, for example, in a fully supervised manner, using RGB images as input and estimated depth as output. For training the neural network, a synthetic dataset can be used, including, for example, RGB images, depth maps, and semantic segmentation from a stereo camera. As illustrated in the examples, the output depth images can be used to derive a 3D point cloud.

[0103] Further examples of container overheight detection using machine learning are envisioned. For example, a control method may include acquiring image data representing an image of a container captured by a camera following placement or removal of a first object into or from the container by a robotic manipulator. The camera may be a color, e.g., RGB, camera, or a depth camera in which separate color channels provide image data. The image data is processed with a neural network trained to detect containers containing objects that protrude beyond the top of the container, for example, within a set tolerance. Based on the processing, it is determined whether the container in the image contains such an overheight object and is therefore in an overheight condition. In response to a positive determination, a signal is generated indicating that the container is in an overheight condition, as described in the previous example. A control signal based on the generated signal is output to control the robotic manipulator to manipulate the overheight object within the container.

[0104] In some cases, a neural network is trained to determine a binary classification mapping from (color) image data to an over-height determination, i.e., whether the container contains an over-height object. In other cases, a neural network is trained, for example, by an object detection model, to determine a mapping from (color) image data input to an instance segmentation output, where over-height objects in an image are identified and assigned a unique label / identifier, or pixel-level masking, where a pixel-level mask is generated for the identified object. In the latter, the mask specifies the image pixels that belong to that particular object, which allows for precise location and boundary delineation of each instance of the over-height object in the image. Thus, a signal indicating that a container is in an over-height state can include the location of the over-height object in the scene for a robotic manipulator to manipulate based on the control signal.

[0105] The mapping learned by the neural network during training may, in some examples, include depth data as well as color image data. For example, the neural network may be trained to determine a binary classification mapping from the combined image and depth data to an excess-height decision, or a mapping from the combined image and depth data to an instance segmentation output. The output of the instance segmentation may include, for example, a list of bounding boxes and segmentation masks for when excess-height objects are present in the image.

[0106] In instances where depth data is not available, e.g., the camera does not have depth imaging capabilities, the segments of the over-height objects in the captured images can be determined in other ways. For example, a dataset captured by a depth camera at a separate robotic pick station can be used to train a neural network to perform instance segmentation based solely on color images in the present system. Alternatively, an image classifier can be used to determine whether the image contains an over-height container, and if a positive determination is made, a multi-view stereo algorithm can be implemented to generate depth information from which the over-height regions can be extracted as described above. For example, the multi-view stereo algorithm can cause the robotic manipulator to automatically move the camera and generate new depth data, e.g., a new point cloud, from images of the scene captured at one or more different poses. The over-height regions can then be extracted from the depth data, e.g., a 3D point cloud, using the methods described above.

[0107] It is also understood that any feature described in connection with any one example may also be used alone or in combination with other features described, or with one or more features of any other example, or in any combination of any other example. Moreover, equivalents and modifications not described above may also be employed without departing from the scope of the appended claims.

Claims

1. 1. A method for controlling a robotic manipulator, comprising: acquiring depth data based on an image of the container captured by a camera following placement or removal of a first object into or from the container by the robotic manipulator; determining whether a given object comprising the first object or a different second object within the container exceeds a height threshold associated with the container based on the depth data; generating a signal indicating that the container is in an over-height condition in response to determining that the given object exceeds the height threshold; and outputting a control signal based on the generated signal configured to control the robotic manipulator to manipulate the given object within the container.

2. The method of claim 1 , wherein the control signals are generated by teleoperation of the robotic manipulator.

3. The method of claim 2 , wherein the teleoperation is based on image data from a separate camera attached to the robotic manipulator.

4. The method of claim 1 , wherein the control signal is generated by an operating algorithm.

5. The method of claim 4 , wherein the manipulation algorithm is configured to generate control signals for the robotic manipulator based on image data from another camera attached to the robotic manipulator.

6. The method comprises: acquiring further depth data based on further images of the container captured by the camera following manipulation of the given object by the robotic manipulator; determining whether the given object exceeds the height threshold based on the further depth data; generating a further signal in response to determining that the given object exceeds the height threshold, the signal indicating that the container is in the excessive height condition; and outputting the further signal in a request for teleoperation of the robotic manipulator to further manipulate the given object within the container.

7. 7. The method of claim 6, wherein the method comprises outputting, based on the further generated signal, a further control signal configured to control the robotic manipulator to manipulate the given object within the container.

8. the height threshold is a first threshold, the over-height condition is a first over-height condition, the signal is a first signal, and the control signal is a first control signal, and the method includes: determining whether the given object within the container exceeds a second height threshold less than the first height threshold based on the depth data; generating a second signal indicating the container is in a second over-height condition in response to determining that the given object exceeds the second height threshold and does not exceed the first height threshold; and outputting a second control signal generated by a manipulation algorithm based on the generated second signal and configured to control the robotic manipulator to manipulate the given object within the container.

9. 10. The method of claim 8, wherein the method comprises outputting a request for teleoperation of the robotic manipulator to manipulate the given object within the container based on the generated first signal.

10. The method of claim 9 , wherein the method comprises outputting the first control signal generated by the teleoperation of the robotic manipulator.

11. 11. The method of claim 1, wherein the control signals are configured to control the robotic manipulator to re-grasp the object, move the object, and release the object into the container.

12. The method of claim 1 , wherein the control signals are configured to control the robotic manipulator to manipulate the object in the container by a non-grasping operation.

13. The method of claim 1 , wherein the camera is attached to the robotic manipulator.

14. 14. The method of any one of claims 1 to 13, wherein the camera is configured for use in an automated pick-and-place process in which the robotic manipulator is controlled to pick and place objects between selected containers from a plurality of containers including the container based on images captured by the camera.

15. The method of any one of claims 1 to 14, wherein the height threshold corresponds to an upper portion of the container.

16. 16. The method of any one of claims 1 to 15, wherein the height threshold is between 1 mm and 50 mm above the top of the container.

17. 17. The method of claim 1, wherein determining whether the object exceeds the height threshold comprises searching for points in the depth data that are located within a predetermined search area defined based on the height threshold.

18. The method of claim 17 , wherein the search area is defined based on dimensions of the container.

19. 19. A method according to any preceding claim, wherein the signal indicating that the container is in the over-height condition comprises position data representative of the position of the given object.

20. The method of claim 19 , wherein the position data represents a position of the given object relative to the camera or the robotic manipulator.

21. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 20.

22. 22. A computer readable data carrier having stored thereon the computer program of claim 21.

23. A control system for a robotic manipulator, said control system comprising one or more controllers configured to carry out the method of any one of claims 1 to 20.

24. 24. A robotic packing system comprising the control system of claim 23 and the robotic manipulator for packing objects.

Images