Pick and Place Robot System

Abstract

A robotic system for picking randomly shaped and sized objects from a continuously moving stream of bulk materials, e.g., a three-dimensional stream of bulk materials, and individually placing the objects in a queue on a guide or directly on a sorter. The pick-and-place robot has a robot actuator for moving a gripper with a controllable gripping configuration of its gripping members (e.g., four suction cups) to adapt the gripper to various objects. A control system processes a three-dimensional image of objects upstream of the pick-and-place robot's position, identifies distinct objects in the three-dimensional image, and selects an object to be grasped based on parameters of the identified distinct objects determined from the three-dimensional image. Based on the size and shape of the selected object to be grasped, the gripping configuration of the gripper is adjusted to fit the surface of the object to be grasped for optimal grasping. The robot actuator, e.g., a gantry-type robot actuator, is then controlled to move the gripper to a position to grip the object, then move the gripper with the gripped object to a target position, release the grip on the object at a target orientation, and place the object on a guide or directly on a sorter. Images of the object after it has been placed, along with the object properties determined from the three-dimensional image, can be used as input to machine learning to improve the pick-and-place performance of the robot system online (e.g., to improve algorithms online for selecting which objects to pick and for selecting an appropriate gripping configuration for the object).

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to a robotic system. Specifically, the present invention relates to a pick-and-place robotic system for picking up objects from a bulk load of objects, particularly from a three-dimensional bulk load of objects, and placing the objects, e.g., individually and oriented, at a target location, particularly onto a guide to a sorting machine or directly onto an operating sorting machine. The robotic system is suitable for handling mail, parcels, packages, soft bags, flexible / non-rigid bags, polybags, items handled in warehouse distribution, and items handled in mail order distribution centers, etc. [Background technology]

[0002] Background of the Invention Sorting machines, such as for sorting mail and / or parcels, typically include a sorter system for transporting items at a constant speed to a discharge location, where the objects are accepted into the sorter and discharged from the sorter at a given discharge location, such as according to a code on the individual objects.

[0003] Objects are often guided into the sorter from several guides that serve to accept the objects at one end, accelerate the objects, and deliver them to an empty location on the sorter (e.g., onto an empty crossbelt or tilt tray element). The guides accelerate the objects to a speed whose directional component parallel to the sorter speed is equal to, or at least approximately equal to, the sorter speed. Items may also be manually loaded onto the guides, i.e., a human picks individual items from an infeed conveyor, e.g., from a bulk load, and places them individually, oriented, and onto the guides. In sorter systems without automated guides, this sorter guidance involves humans performing the rather uncomfortable task of manually guiding objects into the sorter.

[0004] Pick-and-place robotic systems can replace humans picking up objects and placing them on guides or directly on sorters. However, the task of picking up objects that are random in size, shape, and texture and that are placed in bulk on a continuously moving feeder is a complex task. In particular, picking and placing such objects from the bulk stack with a high success rate and at the high throughput required for regular mail and parcel conveyors is complex for a robot. Summary of the Invention [Problem to be solved by the invention]

[0005] Summary of the Invention In particular, an object of the present invention can be seen as providing a reliable, high-speed robotic system for picking and placing objects from a bulk load of randomly shaped and sized objects from a conveyor (e.g., a step- or continuously moving conveyor) or a platter. [Means for solving the problem]

[0006] In a first aspect, the present invention provides a robotic system configured to pick objects from a bulk load, e.g., a three-dimensional bulk load (e.g., a continuously moving stream of bulk objects), and place the objects, e.g., individually and / or oriented, at a target location onto a guide to a sorter or directly onto the sorter, the system comprising: A pick and place robot, a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grip the object, the plurality of gripping members configured in a controllable gripping configuration; a controllable robotic actuator configured to move the controllable gripper; a pick and place robot comprising: a sensor system configured to provide an image of an object upstream of a position of the pick and place robot; a control system configured to receive the image and execute a control algorithm, the control algorithm comprising: Identifying an object according to the image; selecting which of the identified objects to grasp; controlling a gripping configuration of a plurality of gripping members in response to characteristics of the selected object determined from the image; controlling a controllable robotic actuator to move a controllable gripper to a position to grip the selected object; and controlling the controllable gripper to grip the selected object; Controlling controllable robotic actuators and controllable grippers to move the objects and release the objects at target locations, e.g., individually and / or oriented; and a sensor system configured to provide an image of the object at the target location. A control system configured as follows: A robot system is provided.

[0007] It has been found that such a robotic system can achieve a high success rate in picking up objects at high speeds and placing the objects at target locations, thereby achieving high throughput. Although the robot can be formed from relatively simple standard components, it has been found that, thanks to defined elements and a defined control system, it is possible to pick up objects from a bulk load (e.g., a three-dimensional bulk load) even when the bulk load is moving at a constant speed. Thus, picking an object from a three-dimensional bulk load of moving objects, which may even have a random, unknown shape, is an extremely complex task for a robotic system to handle effectively.

[0008] The present invention is based on the insight that combining a relatively simple input in the form of an image, preferably a three-dimensional image, of the incoming object before it arrives at the pick-and-place robot is sufficient to enable appropriate processing and control algorithms to produce a high pick-and-place success rate, despite the fact that the objects may have unknown sizes and shapes, be arranged in bulk, and even be set up to move continuously as the bulk arrives at the pick-and-place robot. It has been found that selecting the object to be picked and adjusting the gripper configuration to match the size and shape of the object to be picked provides a highly adaptable robotic system capable of accepting a wide variety of objects of different sizes and types arriving in random order. Furthermore, it has been found that the control algorithm that performs the task can be advantageously trained by a learning algorithm that optimizes pick-and-place performance, and this can even be performed continuously during normal operation of the robotic system. This allows the robot to adapt to changes in the pick-and-place task (e.g., new types of objects that it has not previously handled).

[0009] Providing an image of the object after it has been released at the target location offers several feedback possibilities (e.g., rather than simply outputting a pick-and-place success rate, feedback can also be provided to the control system that allows for improvement of the pick-and-place performance of the robotic system).

[0010] Furthermore, providing an image of the object after it is released at the target position allows immediate feedback of the robotic system's performance. Thus, any malfunction (e.g., malfunction of the gripper, the image sensor system, or the robot actuators) can be quickly detected. For example, the control system can generate an alarm signal if the image of the object at the target position indicates the occurrence of an error. Such a rapid response to malfunctions is important for a capable pick-and-place system to allow for possible shutdown and repair of the robotic system.

[0011] In particular, one embodiment of the robotic system has been tested with an incoming bulk load of objects having a constant velocity, such as 0.1-1.0 m / s, and with the incoming bulk load of objects having various sizes, shapes, and textures, resulting in a 98-100% success rate at speeds of up to 1500-2000 objects processed per hour, depending on the system layout and pick-and-place distance. Tests have been conducted with objects including rectangular boxes of various sizes, as well as plastic bags, laminated objects, and the like. Throughputs of over 2,000 objects per hour can be achieved.

[0012] The robotic system has been found to provide pick-and-place performance and reliability that allows the robotic system to be used for mail, parcels, packages, items handled in warehouse distribution, and items handled in mail order distribution centers.

[0013] A "robotic actuator" is generally understood to be a controllable device preferably having a manipulator arm with at least two joints or axes and that is controllable to move a gripper from one position in space to another position in space, i.e. essentially a robotic actuator can move a gripper to a controllable position in space.

[0014] By "singulated" it is understood that the objects are placed at a distance from each other. Preferably, the objects are also placed singulated and oriented, and by "oriented" it is understood that they are aligned with the orientation of a guide, conveyor, sorter, etc.

[0015] An "image of objects" is understood to be a sensed or measured representation of the physical configuration of an object. Preferably, the image has sufficient detail to allow, with appropriate processing, identification or classification of a single object from a bulk of objects. The image may be a visual image (e.g., acquired by a two-dimensional or three-dimensional camera). However, other techniques may be used as well (e.g., using a laser scanner or other scanning technology that provides images, such as by non-visual sensing or measurement techniques).

[0016] In general, "image" and "providing an image" should be understood to include any representation of such an image and any manner of providing an image. In particular, an image may be provided as digital image data (e.g., data captured using a two-dimensional or three-dimensional sensor or sensor system known in the art).

[0017] "3D bulk" is understood as commonly used in the art, i.e., a bulk of objects that are randomly arranged relative to each other in three dimensions; thus, the bulk of objects may in particular include objects arranged next to each other on top of each other in any random configuration.

[0018] While the pick and place robots described throughout this specification are suitable for picking objects from a three-dimensional bulk load of objects, it should be understood that the robots may also pick objects from a two-dimensional bulk load of objects or from a single object.

[0019] Preferred embodiments and features are described below.

[0020] In a preferred embodiment, the controllable robot actuator comprises a cartesian-type robot actuator, such as a gantry-type robot actuator. It has been found that such a type of robot actuator is suitable for placement with its main horizontal extension axis extending from a gripping area on an infeed conveyor transporting a continuously moving stream of objects to a target location within a target area on an associated sorter or a lead-in to the sorter. The cartesian-type (e.g., gantry-type) robot actuator is preferably configured to move a controllable gripper in a three-dimensional coordinate system (X, Y, Z), thereby enabling the gripper to navigate within a confined space. Preferably, the gripper angle can also be controlled, thereby providing at least four degrees of freedom. Preferably, the robot actuator is controllable to move the gripper along a predetermined trajectory within the confined space, for example, enabling the gripper to navigate to grasp an object in the space between two tall objects. Most preferably, the gripper can be controlled to roll, pitch, and yaw to further improve navigation. Such grippers have proven suitable for effectively grasping objects from a three-dimensional bulk load of objects.

[0021] In some embodiments, the robotic system is configured to individually and / or align and place the objects at the target location onto a guide to the sorter (e.g., onto a conventional guide component or a simple band conveyor that serves as a guide to the sorter). In some embodiments, the controllable robotic actuators are configured to accelerate the objects toward the continuously moving sorter after they are picked up, and individually and / or align and place the objects at the target location directly onto the sorter (e.g., onto a crossbelt or tilted tray) at or near the sorter speed.

[0022] The sensor system can, in principle, be based on any type of sensor technology capable of detecting or measuring physical property data that can be processed to provide an image, most preferably a three-dimensional image. The sensor system is preferably configured to provide at least a two-dimensional image (e.g., a regular two-dimensional black-and-white, gray-tone, or color photograph) with high resolution to enable accurate identification of the contours of objects in the image. Preferably, the sensor system is configured to provide a three-dimensional image, for example, in the form of a two-dimensional image with additional height information. High-resolution (e.g., 1-2 mm accuracy) three-dimensional image information can increase the likelihood of identifying features by image processing, which allows identification of distinct objects and further properties or features of the objects. Furthermore, height information in the image is preferable for precise control of a robot actuator that moves a gripper along a planned path to a position for gripping the object. The three-dimensional image can be provided by a three-dimensional camera, such as at least one of a three-dimensional line camera system (combining a two-dimensional line camera with motion), a time-of-flight three-dimensional camera, and a stereo three-dimensional camera. Other techniques may be used, such as a regular two-dimensional camera followed by image processing algorithms that generate height information based on the two-dimensional image, thus generating a substantially three-dimensional image.

[0023] In a preferred embodiment, the sensor system comprises a two-dimensional or three-dimensional camera located at a fixed position above the stream of moving objects. Preferably, the sensor system is upstream of the position of the pick and place robot and is configured to provide a static image covering a fixed area remote from the pick and place robot.

[0024] Based on the images, the control algorithm may identify and classify the objects. Preferably, the step of identifying the objects includes analyzing the images to distinguish between single objects in a bulk load of objects, thereby identifying a single object in the bulk load as a basis for determining whether to initially pick that object. Furthermore, the objects may be classified with respect to various parameters based on the analysis of the images. For example, for each identified object, the center of gravity may be determined, which influences where to position the gripping members of the gripper to grip the object with the highest possible success rate.

[0025] In some embodiments, a sensor is configured to detect the height of an object after it has been picked up by the controllable gripper. In particular, a control system is connected to the sensor to receive information indicative of the object's height, and the control system is configured to control the controllable gripper and the controllable robot actuator to release their grip on the object at a height above the target location in response to the information indicative of the object's height. This allows the robotic system to gently place small objects of unknown heights on the target location without dropping the object from too high above the target location or forcing an object of an unexpected height onto a guide or sorter. For example, this solves the problem of picking an object from a pile of objects when the object's height is unknown, based on a picking height above a feed conveyor or platter. Furthermore, the height sensor prevents damage to the gripper.

[0026] The sensor configured to provide an image of the object after it has been placed in the target location can be any type of technology capable of providing an image of the object, as previously described for the sensor system providing the initial image. For example, such a sensor may comprise a two-dimensional or three-dimensional camera located on the pick-and-place robot or at a fixed location separate from the pick-and-place robot. This allows for an evaluation of how the object is being placed relative to the target location and, therefore, the overall pick-and-place performance. Furthermore, the orientation and other characteristics or features of the object can be compared by comparing it with the initial image, i.e., a portion of the image including the object of a measurement before the particular object was picked up. This can be used in machine learning feedback to algorithms in the control system, or in standard feedback solutions, to improve pick-and-place performance. For example, to simply discard grasped objects with features or characteristics that prove difficult to pick and place correctly, or to primarily select objects with features or characteristics that result in a high success rate. This is relevant for systems with a continuous stream of incoming objects, but in other systems sensors are placed to monitor objects in the pick area in order to control the feeder in stages to ensure that all objects have been picked by the robot before the feeder moves another bulk load of objects for picking by the robot.

[0027] In particular, the control system may be configured to compare an image of the object after it has been placed in the target position with the object in the image of the object provided by the sensor system. Furthermore, the control system may be configured to process the image of the object after it has been placed in the target position to determine at least the position of the object relative to the target position (e.g., to determine the position and orientation of the object relative to the target position). The control system may be configured to compare at least the position of the object with the target position and determine whether the position of the object deviates from the target position by more than a predetermined threshold (e.g., the control system may be configured to generate an output in response to the determination of whether the position of the object deviates from the target position by more than a predetermined threshold). In particular, the control system may be configured to provide an output indicative of the pick-and-place performance of the robotic system in response to images of the plurality of objects after they have been placed in the target positions. Thus, such an output may allow a user to continuously evaluate the pick-and-place performance (e.g., the output may be used to indicate an error (e.g., a failure of a component of the robotic system) by generating an alarm if pick-and-place performance degrades).

[0028] In some embodiments, the control system is configured to provide the image of the object after it has been placed at the target location as feedback to a control algorithm, preferably a machine learning algorithm. In particular, the control system may be configured to provide the image of the object after it has been placed at the target location as feedback to a portion of the control algorithm that selects which of the identified objects to grasp in response to the image. Additionally or alternatively, the control system may include a learning algorithm configured to learn characteristics in images of objects with high or low success rates to be placed at the target location based on multiple images of the object after it has been placed at the target location, and to select which of the identified objects to grasp in response to the images in response to identified objects having similar characteristics to the objects with high or low success rates to be placed at the target location. In particular, the control system may be programmed to select for grasping an identified object with a high success rate to be placed at the target location, and further to avoid selecting for grasping an identified object with similar characteristics to the objects with a low success rate to be placed at the target location. The characteristics may include one or more of orientation, size, and type identification. The control algorithm may include at least one algorithm portion including an artificial intelligence algorithm and / or a neural network algorithm or similar adaptive algorithm for processing images of the object after it has been placed in the target location in order to train the control algorithm in selecting which of the identified objects in the image to grasp in order to improve the pick-and-place performance of the robotic system.

[0029] In a preferred embodiment, the controllable gripper comprises: a base portion configured to be mounted on a robotic actuator; at least two arms attached to the base portion, each arm comprising: a gripping member configured at or near a distal end of the arm, the gripping member configured to engage an object to grip the object; at least two arms configured to be slidable along their lengths relative to a base portion actuated by a controllable actuator to allow the arms to be controllably adjusted with respect to the position of the gripping member relative to the base portion; Equipped with To allow the gripping member to form a variety of gripping configurations, at least in terms of size, the at least two arms are configured to be slidable in different directions relative to the base portion.

[0030] Such grippers are highly adaptable in terms of gripping objects of different sizes and shapes and can be easily transitioned by electric, pneumatic, or hydraulic actuators from a small, compact version for gripping small objects to a larger gripping configuration with the arms fully extended for gripping larger objects. In particular, the base portion can be formed with compact dimensions, and via a slidable arm configuration, for example, four arms, can be formed to be very compact in the folded state while still being able to provide a large gripping configuration with the arms fully extended that can grip large objects. The compact dimensions enable the gripper to navigate not only bulk loads of objects but also three-dimensional bulk loads to grip a small object between two larger, taller objects.

[0031] In a preferred embodiment, the controllable gripper comprises four elongate arms attached to a base portion, the four arms configured to be slidable in different directions relative to the base portion to allow the four gripping members to form a variety of gripping rectangle sizes. Preferably, each arm has a suction cup attached on or near its distal end and is so positioned relative to one another to allow gripping on a surface that forms, for example, a plane parallel to an axis along the length of the arm. A controllable vacuum system is preferably connected to apply vacuum to the suction cups, and the control system is configured to control the controllable vacuum system to control when vacuum is applied to one or more suction cups to grip an object and when the vacuum is discontinued to release the object.

[0032] The control system may be configured to control a separate controllable actuator for each of a plurality of arms of the controllable gripper to grip the selected object in response to the image. Alternatively, the control system may be configured to control a single controllable actuator common to all arms. Still further, the control algorithm may be configured to control at least two controllable actuators for control of a plurality of arms of the controllable gripper to grip the selected object in response to the image, such as one actuator arm for controlling the position of one set of two arms and another actuator arm for controlling the position of another set of two arms.

[0033] In a preferred embodiment, the base portion is connected to the robot via a controllable rotation element such that the base portion can undergo controllable rotation about an axis of rotation. ActuatorPreferably, the base portion is further attached to a robot actuator via a controllable tilt element such that the base portion is controllable tilting about a tilt axis orthogonal to said rotation axis. Preferably, a control algorithm is configured to control said controllable rotation and said controllable tilt of the base portion of the controllable gripper to grip a selected object in response to said image.

[0034] In a specific embodiment, the controllable actuator of the controllable gripper comprises a controllable electric motor connected to actuate at least one of the arms by rotation applied via a cable connection. In particular, the controllable electric motor is mounted at a position above a rotation element connecting the base portion and the robot actuator. In particular, the controllable electric motor is mounted at a position above a tilting element connecting the base portion and the robot actuator.

[0035] Alternatively, or in addition to, a suction cup, the controllable gripper may have one or more other types of gripping members (e.g., finger-type elements configured to contact the sides of an object to grip the object).

[0036] To provide additional gripping force for heavy objects, the controllable gripper may have at least one suction cup for gripping the object, said at least one suction cup being attached to a fixed position on the base portion, such as a central portion of the base portion.

[0037] To increase flexibility with regard to gripping configurations, one or more arms of the controllable gripper have telescopic elements.

[0038] The control system may be configured to control the controllable gripper to cause the gripping members to form a predetermined gripping configuration in preparation for gripping an object. In particular, the control system may be configured to control the controllable gripper to cause the gripping members to assume the predetermined gripping configuration while the controllable robotic actuator moves the gripper to a predetermined position for gripping an object. This helps save time for adjusting the gripping configuration, especially when changing from a small gripping configuration to a large gripping configuration or from a large gripping configuration to a small gripping configuration.

[0039] The robotic system preferably comprises a control system configured to control the controllable robot actuators and the controllable gripper to grasp the selected object to be picked in response to a plurality of inputs determined from the image. In particular, the plurality of inputs includes one, preferably a plurality, of information regarding the object's shape, the object's horizontal boundaries, the object's size, the object's orientation, the object's top surface curvature, and the object's surface roughness, and the control algorithm processes the plurality of inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robot actuators and the controllable gripper to grasp the object. In particular, the plurality of inputs includes more than one (e.g., all) of information regarding the object's shape, the object's horizontal boundaries, the object's size, the object's orientation, the object's top surface curvature, and the object's surface roughness. Preferably, the control algorithm is configured to control the grasping configuration of the plurality of grasping members in response to the plurality of inputs. In particular, the control algorithm may be configured to control the tilt of the controllable gripper in response to a detected tilt of the surface of the object to be grasped. In particular, the control algorithm may be configured to control the rotation of the controllable gripper relative to the controllable robot actuator in response to a detected orientation of the object to be grasped.

[0040] Most preferably, the plurality of inputs includes information about the top surface of the object. This is especially preferred when the gripping members are suction cups. In particular, the information about the top surface of the object includes information about one or more of the location of wrinkled regions, the location of flat surface portions, and the angle and orientation of the slope of the top surface. In particular, the control algorithm may be configured to control the controllable robot actuators and the controllable gripper to avoid positioning a gripping member to engage with an object in a wrinkled region of the object's surface. In particular, the control algorithm may be configured to control the controllable robot actuators and the controllable gripper to position at least one gripping member in a region of the object having a flat surface. If the object is detected to have a rectangular shape, or at least a rectangular top surface, the control algorithm may be configured to control the controllable robot actuators and the controllable gripper to position a gripping member (e.g., a suction cup) proximate to three corners, preferably all four corners, of the top surface of the object.

[0041] The control system is preferably configured to provide the image of the object after being placed at the target position as feedback to a control algorithm to improve the success rate of pick-and-place. In particular, the control algorithm may provide the feedback regarding the processing of the plurality of inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robotic actuators and the controllable gripper to grasp the object. In particular, the control system may be configured to process the images of the object after being placed together with the plurality of inputs according to a learning algorithm, and accordingly modify the control algorithm regarding the processing of the plurality of inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robotic actuators and the controllable gripper to grasp the object.

[0042] In a preferred embodiment, the control algorithm comprises: analyzing the image to extract different parameters indicative of identified objects; calculating a score value for each of the plurality of identified objects as a function of the plurality of different parameters according to a predetermined scoring algorithm; The score values ​​are compared, and an object to be grasped is selected according to the result of the comparison. The apparatus is configured to select which of the identified objects to grasp according to an object selection algorithm configured to:

[0043] In particular, the plurality of different parameters indicative of each of the plurality of identified objects may include at least one of the following: a distance between the object and the current position of the gripper, a distance between the object and a target position where the object is to be placed, a texture of the object's top surface, a curvature of the object's top surface, a slope of the object from the top surface, a height of the object's top surface compared to the heights of surrounding objects, a dimension of the object, a type of the object, a shape of the object, and a quality of a portion of the image covering the object. Specifically, the control algorithm may be configured to assign, for each object, a parameter value for each of the plurality of different parameters according to a predetermined table, and the control algorithm is configured to calculate an overall score value for each object according to the assigned parameter values. The object to be grasped is then selected as the object having the best overall score value among the identified objects.

[0044] The control system is preferably configured to provide images of the objects after being placed in the target positions as feedback to the control algorithm for determining which of the identified objects to grasp. In particular, the control system may include a learning algorithm configured to modify one or more parameters in the predetermined scoring algorithm based on multiple images of the objects after being placed in the target positions to improve pick-and-place performance. The learning algorithm may specifically be configured to modify one or more of the parameter values ​​for each of the multiple different parameters according to the predetermined table to improve pick-and-place performance. In particular, the learning algorithm may be configured to modify one or more weighting coefficients in the predetermined scoring algorithm to improve pick-and-place performance. The learning algorithm may include an artificial intelligence algorithm and / or a neural network or another type of machine learning algorithm.

[0045] In some embodiments, the robotic system includes a second sensor system, such as a camera, located on or above the pick-and-place robot, where the second sensor system is configured to provide images of the objects in the gripping area. In particular, the control system is configured to compare the images of the objects in the gripping area with images of the objects at a location upstream of the gripping area to detect whether one or more objects have changed position relative to the surface of the feed conveyor. This allows the robot to respond if one or more objects move (e.g., an object falls off the bulk load) en route from the location where the upstream image was provided to the robot's gripping area. Thus, in some cases, the robot may need to change its picking strategy because an object selected for pick based on the upstream image becomes difficult to pick due to changes in the bulk load. Thus, based on the new image, the control system may be configured to select another object for pick based on the image of the object in the gripping area. In other cases, only small changes occur to the bulk object (e.g., the top surface of the object to be grasped has changed angle compared to its position in the upstream image), in which case the control system can adapt the robot actuators and grippers accordingly to grasp the object based on the image in the grasping area.

[0046] In some embodiments, the robotic system includes a plurality of pick and place robots, such as 2-10, configured to be positioned along an infeed conveyor, each of the plurality of pick and place robots: a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grip the object, the plurality of gripping members configured in a controllable gripping configuration; a controllable robotic actuator in a cartesian or gantry configuration and configured to move the controllable gripper; The present invention includes a plurality of pick and place robots.

[0047] Such a series of pick-and-place robots may provide increased capabilities compared to a single robot. In particular, the multiple pick-and-place robots may be identical, each having a control system as well as separate sensors that provide upstream images and, in some cases, sensors that provide images at the gripping area. However, some or all of the multiple pick-and-place robots may specifically receive data from the same upstream image. In particular, this upstream image may then be compared with an image provided immediately upstream of each pick-and-place robot or group of robots, or with an image provided at the gripping area of ​​each pick-and-place robot, in order to identify changes in the bulk load of objects from the initial upstream image of the bulk load and respond accordingly if changes occur that may require changes in which objects are picked and / or how the objects selected for picking are gripped.

[0048] In a second aspect, the present invention provides a sorter system comprising: a conveyor configured to transport objects of various shapes and sizes in bulk, such as a conveyor configured to move continuously; a sorter configured to receive the objects; a first robotic system according to a first aspect configured to pick up objects from a conveyor and place the objects (e.g., individually and orientated) onto a guide to a sorter or directly onto the sorter; A sorting machine system is provided.

[0049] The first robotic system may be configured to place objects directly onto the sorter as it moves at a constant speed, or onto a guide to the sorter (e.g., a guide in the form of a conveyor band configured to move objects onto the sorter perpendicular to or at another angle to the direction of movement of the sorter).

[0050] In some embodiments, the conveyor and the sorter are positioned adjacent to one another, the conveyor has a first side facing the first side of the sorter, and the controllable robotic actuator includes a gantry or cartesian robotic actuator positioned with a first support (e.g., on the floor) at a position on a second side of the conveyor and a second support (e.g., on the floor) at a position on the second side of the sorter. This configuration allows the cartesian or gantry robotic actuator to place a picked object directly onto the sorter at a target location.

[0051] In some embodiments, the second robotic system according to the first aspect is positioned downstream of the conveyor relative to the first robotic system. This can increase throughput. Additional robotic systems can be added downstream of the second robotic system to further increase capacity. Some of these multiple robotic systems can share components (e.g., at least part of the sensor system and control system).

[0052] In some embodiments, the sorter system includes a plurality of pick and place robots, such as 2-10, configured to be positioned along an infeed conveyor, each of the plurality of pick and place robots: a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grip the object, the plurality of gripping members configured in a controllable gripping configuration; a controllable robotic actuator in a cartesian or gantry configuration and configured to move the controllable gripper; Specifically, the pick and place robots may be configured to individually place objects onto a second conveyor (e.g., a conveyor parallel to the feed conveyor), the second conveyor configured to transfer the objects to one or more guides, and the one or more guides configured to deposit the objects onto the sorter.

[0053] In a third aspect, the present invention provides use of a robotic system according to the first aspect for processing objects including at least one of mail, parcels, packages, items handled in warehouse distribution, and items handled in mail order distribution centers.

[0054] In a third aspect, the present invention provides use of a sorter system according to the second aspect for processing objects including at least one of mail, parcels, packages, items handled in warehouse distribution, and items handled in mail order distribution centers.

[0055] In a fourth aspect, the present invention provides a method for picking objects from a bulk load of objects and placing the objects (e.g., individually and / or oriented) at a target location onto a guide to a sorter or directly onto the sorter, the method comprising: providing a robot comprising a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grasp the object, the gripping members being configured into a controllable gripping configuration; providing a controllable robotic actuator configured to move a controllable gripper; providing an image of an object in a stream of moving objects upstream of a location of the robot; processing the image to identify objects within the image; Step a) selecting which of the identified objects to grasp; determining a characteristic of the selected object according to the image; Step b) controlling the gripping configuration of the gripping members in response to said properties of the selected object; Step c) controlling the controllable robot actuator to move the controllable gripper to a position to grasp the selected object; controlling a controllable gripper to grasp a selected object; Controlling the controllable robotic actuators and the controllable grippers to move the objects and release the objects (e.g., individually and / or oriented) at the target locations; providing an image of an object at a target location; A method is provided, comprising:

[0056] In some embodiments, the method includes grasping an object from a bulk load of moving objects (e.g., a continuously moving stream of bulk objects). In other embodiments, the method includes grasping an object from a bulk load of stationary objects, such as the robot emptying a bulk load of stationary objects and then accepting a bulk load of another object.

[0057] In some embodiments, a sensor provides information about the position of the object after it has been placed in the target location (e.g., by providing an image (e.g., a two-dimensional or three-dimensional photograph) of the target location after the object has been braced). In particular, the method may further include feeding back the information about the position of the object after it has been placed to modify one or more of steps a), b), and c) to improve pick-and-place performance of the robotic system. In particular, the method may include continuously applying a learning algorithm to incoming data about the information about the position of the object after it has been placed in the target location. In particular, properties or characteristics determined about the object from the image are used in the learning algorithm to modify one or more of steps a), b), and c) to improve pick-and-place performance of the robotic system.

[0058] Each individual aspect of the invention may be combined with any of the other aspects. These and other aspects of the invention will become apparent from the following description in connection with the illustrated embodiments.

[0059] BRIEF DESCRIPTION OF THE DRAWINGS The present invention will now be described in more detail with reference to the accompanying drawings, which illustrate one way of practicing the invention and are not to be construed as a limitation on other possible embodiments encompassed within the scope of the appended claims. [Brief explanation of the drawings]

[0060] [Figure 1] FIG. 1 illustrates a block diagram of an embodiment of a robotic system. [Figure 2] FIG. 1 shows a block diagram of another robotic system embodiment. [Figure 3] 1 shows a gantry-type robot with a gripper having suction cups with an adjustable configuration. [Figure 4a] A preferred gripper is shown having an adjustable gripper configuration and rotation and tilt mechanisms that allow the gripper to be used with a cartesian or gantry style robotic actuator. [Figure 4b] A preferred gripper is shown having an adjustable gripper configuration and rotation and tilt mechanisms that allow the gripper to be used with a cartesian or gantry style robotic actuator. [Figure 5a] 1 shows an example of a series of robots working together to pick objects from an infeed conveyor and place them on a guide to a sorter. [Figure 5b] Another example is shown of a series of robots working together to pick objects from an infeed conveyor and place them on a simple band for guidance into a sorter. [Figure 5c] 1 shows an example of a series of robots working together to pick objects from an infeed conveyor and place the objects directly onto a sorter. [Figure 6] 1 illustrates steps of a method embodiment. [Figure 7] This is an example of a pick-and-place robot. [Figure 8] Another example of a pick-and-place robot. DETAILED DESCRIPTION OF THE INVENTION

[0061] Detailed Description of the Embodiments 1 and 2 show an embodiment of a robotic system comprising a pick-and-place robot configured to pick objects G_O from a continuously moving feed conveyor FC transporting a stream of bulk objects BLK and place the separated objects S_O at a target position TA onto a guide section I1 to a sorter SRT.

[0062] The robot system comprises a pick-and-place robot RA, G, a sensor system CM, CM2, and a control system CS with a processor system configured to execute a control algorithm. The pick-and-place robot RA, G has a controllable robot actuator RA that is responsible for moving a controllable gripper G to grasp an object. The embodiment of Figure 1 and the embodiment of Figure 2 have essentially the same elements but differ in terms of the level of detail and with respect to the robot actuator RA. However, both embodiments will be described as a whole below.

[0063] In FIG. 1, the robot actuator RA is a gantry-type actuator. That is, the robot actuator RA has a fixed part with a set of elongated elements attached to the ground at one end adjacent to the feed conveyor FC and the other end adjacent to the guide section I1, for moving a controllable gripper G between a gripping area GA, which picks up an object G_O on the feed conveyor FC, and a target area TA on the guide section I1, allowing the movable part of the robot actuator RA to move along the elongated elements. The robot actuator RA has a movable part that is responsible for moving along the elongated elements by a controllable actuator, and the movable part further has a controllable actuator that is responsible for moving the gripper G perpendicular to the elongated elements and a further actuator that is responsible for moving the gripper G vertically. Thus, taken together, the robot actuator RA can move the gripper G to a position in space covering the gripping area GA and the target position or target area TA. In FIG. 2, the robot actuator RA is shown as a robot manipulator arm with at least three joints or axes. The joints may be revolute joints or a combination of hinges and revolute joints. In a preferred variant, ,B Bots Actuator comprises a base from which the manipulator arm extends. In particular, the base may be fixedly mountable to the floor or ground between the feed conveyor FC and the sorter SRT. Preferably, the robotic manipulator arm has an extension that enables it to place the item in a target area TA that is at least 1 m horizontally away from the gripping area GA where the item was picked up.

[0064] The controllable gripper G shown in FIG. 2 has multiple gripping members M1, M2, two of which are visible. Preferably, the gripper G has four suction cups M1, M2 configured to engage the surface of the object G_O and grip it upon application of suction or vacuum to the suction cups M1, M2. The suction cups have been found to be capable of gripping a wider variety of objects G_O, ranging from small objects such as clothing wrapped in a relatively thin, fluffy plastic bag to large rectangular cardboard boxes weighing several kilograms, regardless of their surface texture, shape, and orientation. The suction cups M1, M2 of the controllable gripper G are configured on adjustable arms A1, A2 attached to the base portion B. This allows for a controllable gripping configuration of the suction cups M1, M2 in FIG. 2, shown as a variable distance D between the suction cups M1, M2. This controllable gripping configuration allows the geometry of the suction cups M1, M2 to be adapted depending on the object G_O to be gripped. This allows the gripper G to adapt the distance D to accommodate picking up large items as well as small items, and further adapts the suction cup configuration to the most preferred way of positioning the four suction cups M1, M2 on the object G_O for optimal gripping. Optimal gripping is important so that the robot can process the object G_O at high speed without losing its grip on the object G_O as it moves at high acceleration toward the target position TA. The preferred gripper G is compact, which allows the suction cups M1, M2 to have the longest distance D between them in the most compact configuration, i.e., with the shortest distance D between the suction cups M1, M2, for gripping a small object between two taller objects, while in the maximum configuration for gripping a large object, i.e., with the arms A1, A2 fully extended, the suction cups M1, M2 have the longest distance D between them. One preferred four-suction cup gripper G is described below.

[0065] Gripper G is preferably mounted to robot actuator RA such that it can rotate and tilt relative to its attached position on robot actuator RA, thereby allowing gripper G to tilt and rotate to match the orientation and inclination of the top surface of any object in bulk load BLK in order to obtain an optimum gripping force on object G_O.

[0066] The basic input to the pick-and-place robots RA, G, CS is a sensor system with a three-dimensional camera CM mounted in a fixed position, providing a three-dimensional image IM of an image area IMA upstream of the position of the pick-and-place robot RA, G. The image area IMA preferably covers the entire width of the feed conveyor FC and at least a length sufficient to cover the longest object expected in the flow of bulk objects BLK. One still photograph IM may be taken at a fixed time interval. The time interval is selected to be small enough, at least in relation to the length of the selected image area IMA and the speed of the feed conveyor FC, so that all objects are covered by the image IM. Depending on the selected image processing, the series of images IM may be provided at high speed. The time interval between images is preferably selected depending on the speed of the flow of arriving objects. However, it may be preferable for the images to overlap (e.g., with an overlap of 20-80%, such as about 40-60%). In a specific embodiment, two or about two images per second are provided.

[0067] It has been found that such a three-dimensional image IM of an incoming bulk object BLK provides sufficient input to the accurate control system CS of the pick-and-place robot RA, G without the need to mount cameras or other sensors on the moving parts of the robot actuators RA or grippers G. The image area IMA is preferably located upstream of the gripping area GA by at least a minimum distance, depending on several parameters, such as the speed of the incoming object flow, the image processing time, and the reaction time of the robot actuators RA and grippers G. In specific embodiments, a distance of 0.5 m to 2.0 m (e.g., 0.8 m to 1.5 m) may be preferred. The basic concept for controlling the pick-and-place robot RA, G to pick up an object is that the provided image IM and the known speed of the feed conveyor FC are sufficient to determine the exact time and position of the object to be grasped in the gripping area GA, which are then used in the control system CS to determine control signals G_C, C_RA for controlling the grippers G and the robot actuators RA, respectively.

[0068] While various types of cameras CM exist, it is preferable that the CM be capable of providing high-quality three-dimensional images that are capable of accurately identifying shapes to enable identification of distinct objects within the bulk load BLK, and with height dimensions that are accurate enough to enable accurate navigation of the gripper G to grasp the object G_O selected for grasping. The camera CM may be a three-dimensional line camera, a time-of-flight three-dimensional camera, and a stereo three-dimensional camera, among others. Furthermore, it should be understood that two-dimensional cameras may also be used, in which case the height dimension of the three-dimensional image IM may be calculated based on image processing of the two-dimensional photograph, or may be obtained by alternative techniques (e.g., a separate height sensor located separately from the camera CM).

[0069] A height sensor HS (e.g., a camera or other type of sensor) is positioned to detect the height of the grasped object G_O. Because only an image IM taken from a position above the bulk load BLK of objects is available, it is possible to grasp a small object G_O from a position above a larger object. Thus, the actual height of the grasped object G_O is generally unknown at the time of grasping, since only the surface height of the upper object is known. The height sensor HS detects the height of the grasped object G_O, and this information is used in controlling the gripper G and robot actuators RA to place the object G_O, or more specifically, to determine the height above the target area TA at which the object G_O is released. This ensures that even fragile objects G_O can be handled without damaging the object G_O or even potentially misplacing it due to releasing it from high above the target area TA. The height sensor can be positioned near the grasping area GA, near the target area TA, or between the grasping area GA and the target area TA. The height sensor HS may be a light bar array, a light beam sensor, a camera, or the like.

[0070] The control system CS receives the three-dimensional image IM and executes a control algorithm using several elements to generate control signals C_G, C_RA for controlling the controllable gripper G and the robot actuator RA, respectively. First, the three-dimensional image is processed according to an object identification algorithm I_O in order to identify in the image IM distinct objects of the incoming bulk goods BLK. In particular, this algorithm attempts to identify candidate objects to be picked by image processing techniques that preferably utilize the three-dimensional information in separating single objects from the bulk goods BLK. At this stage, several parameters can be determined for each identified object that will be used in the next stage, image processing techniques as known in the art.

[0071] Next, based on the identified potential objects to be picked, a selection algorithm S_O_G is executed to select which object to grasp next. The selection algorithm S_O_G preferably extracts, based on an analysis of the three-dimensional image IM, a number of different parameters indicative of the identified objects, and based on the parameters enables which object to grasp based on some predetermined criterion or balance of such criteria (e.g., estimated speed of grasping the object for high overall efficiency, and estimated success rate of grasping and placing the object). A non-exhaustive list of parameters that may be determined based on the image IM indicative of each of the identified objects is as follows: 1) The distance between the object and the current position of the gripper G. This can be taken into account because it takes time to move the gripper G if the object is far from the current position. 2) The distance between the object and the target position TA where the object is to be placed, which is preferably considered in connection with 1) again with respect to the time required to move the gripper from the exact gripping position within the gripping area GA to the target position TA. 3) The texture of the top surface of the object. This can be important as the texture can significantly affect the success rate of gripping the object; for example, a smooth surface across the top surface of the object may be preferable compared to a non-smooth texture. This depends on the type of gripping member, but a smooth texture is preferred for suction cup grippers. 4) The curvature of the object's top surface, which can also be important with respect to the success rate of gripping the object, as a curved top surface can be difficult to grip even with a suction cup gripper. 5) The inclination of the top surface of the object. This can be taken into consideration, for example, if the top surface is inclined away from the current position of the gripper G, and / or because it can be difficult to grasp an object with an inclined top surface, and in the case of a box-shaped object, the object can be considered to be slippery when grasped. 6) The height of the top surface of the object compared to the height of surrounding objects, which can be important for assessing whether the gripper G can enter the space between two tall objects to grasp the object, which can take at least a significant amount of time. 7) Object size: For various reasons, it may be desirable to prioritize either small or large objects. 8) Object Type: If the type of object can be identified, it may be desirable to prioritize that object, or vice versa. 9) Object Shape: It may prove preferable to prioritize objects of particular shapes over other objects (e.g., box-shaped objects may be prioritized over round objects). 10) The quality of the part of the image IM covering the object: if, for some reason, poor image quality is obtained for some objects, it may be possible to abandon prioritizing such objects (for example, this may be because the object has moved when the image IM is provided, which makes the exact position and orientation of the object in the grasping area GA unknown).

[0072] In particular, the selection algorithm S_O_G for selecting which object to grasp next may include calculating a score value for each of the identified objects as a function of selected two or more of parameters 1)-10) according to a predetermined score algorithm that involves balancing various parameters to achieve a desired balance of efficiency versus success rate. The score values ​​determined for each of the objects are then compared, and the next object to be grasped is selected as the object having the best score value.

[0073] After selecting the next object to be grasped, a control signal C_G is generated by an algorithm section D_GCF for determining the gripping configuration of the plurality of gripping members M1, M2 based on the characteristics of the selected object to be grasped, determined from the 3D image IM. These characteristics have preferably already been determined by the object selection algorithm S_O_G, as explained above. In particular, the shape, specifically the shape of the top surface of the selected object, is used to determine the control signal C_G for controlling the actuators of the gripper G to adjust the lengths of the arms A1, A2 to determine the appropriate distance D for the suction cups M1, M2. In particular, for a gripper G with four elongated arms A1, A2 attached to a base section B, the four arms A1, A2 are controlled to provide a configuration in which the four suction cups M1, M2 form a gripping rectangle sized to match the shape and size of the top surface of the object to be grasped. This may be determined based on several parameters to provide the best possible position of each of the suction cups M1, M2 on the top surface of the object to be grasped based on various knowledge, such as placing the suction cups M1, M2 near the corners of a rectangular flat surface. In particular, a non-exhaustive list of possible inputs to the algorithm D_GCF for determining the grasping configuration is as follows: 1) Information about the shape of an object. 2) The horizontal boundaries of the object. 3) The size of the object. 4) Object orientation. 5) The upper surface curvature of the object. 6) Surface roughness of the object.

[0074] All or some of the above inputs 1)-6) can be processed in the gripping configuration algorithm D_GCF to control the positions of the arms A1, A2 of the gripper G to determine the configuration of suction cups M1, M2 that best suits the object to be gripped. The exact configuration, i.e., the distance D between the suction cups M1, M2, whether only two or preferably four, can be designed to enable the best gripping depending on the above parameters 1)-6). In this case, the overall principle is to provide a gripping configuration that attempts to place the suction cups M1, M2 toward the boundary on a flat surface, whether it can be selected to provide a small gripper configuration for large objects or, even if the object has an irregular shape, a clearly defined but smaller flat surface area (which may be suitable for all suction cup M1, M2 placements, despite this fact). For irregular objects, the selection of the preferred suction cup location, and therefore the gripping configuration, is a compromise. A smaller gripping configuration may be preferable for best gripping by suction cups M1, M2 on curved surfaces. For small objects, and assuming the object is also lightweight, the gripping configuration can be selected to best match the surface roughness (e.g. to avoid placing suction cups M1, M2 in the creases of an object that is a plastic foil covering a garment).

[0075] In particular, in order to save time, if the algorithm D_GCF selects the gripper configuration to be used, the control system CS sends a control signal C_G to the actuators of the gripper G, which serves to actuate the arms A1, A2 to move them to the determined position before or at the same time as the robot actuators RA move the gripper G into the gripping area, so that when the gripper G reaches the object and is rotated and tilted to best suit for gripping the object in the gripping area GA, the suction cups M1, M2 are already in the desired configuration for quickly gripping the object.

[0076] Next, after determining the gripping configuration of the gripper G, the robot motion control algorithm D_RM determines where and how to control the robot actuator RA to move the robot actuator RA to position the gripper G for gripping the selected object. This can be determined from some of the previously mentioned parameters determined for the selected object (e.g., inputs 1) through 6) of the algorithm D_GCF for determining the gripper configuration, such as the tilt of the object's top surface). Furthermore, taking into account the object's speed of movement in the transport direction of the feed conveyor FC, it is calculated exactly where and when in space within the gripping area GA the suction cups M1 and M2 should contact the object's surface to grip it. The robot motion control algorithm D_RM then sends control signals C_RA to control the movement of the robot actuator RA, including appropriate control signals for the actuators responsible for tilting and rotating the gripper G. The suction cups M1 and M2 are then activated by controlling a vacuum system connected to the suction cups M1 and M2 to apply suction to grip the object.

[0077] By keeping track of the object orientation based on the 3D image IM and similarly the parameters already determined above, the control system CS controls the controllable robot actuators RA to move the gripper G and the object to a target position within the target area TA and control the tilt and rotation of the gripper G to release the individual object at the target position (i.e., aligned with the direction of movement of the guide 11). Based on input from the height sensor HS, it determines a release height above the target area TA, i.e., the height at which the vacuum suction on the suction cups M1, M2 is released.

[0078] In the embodiment shown in FIGS. 1 and 2 , a second camera CM2 is set up to cover the target area TA to provide an image IM2 of the object after it has been released by the gripper G at the target position within the target area TA. This allows the image IM2 to be used as feedback to a feedback algorithm FB in the control system CS, so that appropriate image processing can determine whether the object has been successfully picked and placed at the target position (e.g., by comparing the image IM2 with an initial image IM provided by the camera CM before picking) to determine whether the object has been successfully picked and placed as a properly positioned and aligned object S_O. This can be used to calculate a success rate SR for picking and placing the object. The success rate SR can be output to a user and indicate the success rate SR as the percentage of objects successfully processed within a certain period of time. The success rate SR can be further used by the control algorithm to detect possible fault alarms (e.g., by calculating statistics based on the success rate SR over time). If a sudden drop in the success rate SR is detected, there may be a fault in the system, and an alarm can be generated.

[0079] Additionally, the feedback algorithm FB may use the image IM2 as input to the machine learning portion of the feedback algorithm FB. In particular, the feedback algorithm FB may include an artificial intelligence algorithm and / or a neural network-based algorithm that may analyze some of the previously mentioned inputs or parameters determined in the initial three-dimensional image IM for the object. This may be used to annotate the resulting image IM2 after placing the object, so that the learning algorithm learns specific features of successfully processed objects and features of objects that may prove difficult to process. With the goal of gradually improving pick-and-place performance, the feedback algorithm FB may use machine learning to adapt the object selection algorithm S_O_G and the gripper configuration determination algorithm D_GCF based on experienced pick-and-place failures and successes. This may be done with the robot system before placing the robot in normal operation, but it may be desirable for the learning algorithm to operate online while the robot system is in normal operation. This allows the robot system to be more adaptive and gradually adapt to improved performance, for example, when various parameters (e.g., speed of incoming bulk objects BLK, type of object) change. This allows the robotic system to flexibly adapt to high performance under changing operating conditions, such as handling new types of objects.

[0080] As an example, the learning algorithm portion of the feedback algorithm FB may be able to modify one or more parameters in the above-mentioned scoring algorithm of the graspable object selection algorithm S_O_G. Thus, based on multiple images IM2 of the object after it has been placed at the target position TA, the graspable object selection algorithm S_O_G may be adapted to improve the overall performance of the robot system. Specifically, the learning algorithm may modify weighting coefficients in the scoring algorithm or provide feedback to the artificial intelligence system to improve pick-and-place performance. In particular, some features or combinations of features may be found to result in very poor pick-and-place performance, so the graspable object selection algorithm S_O_G may be designed to completely refuse to pick such objects, instead leaving them on the feed conveyor FC for manual handling.

[0081] As another example, the learning algorithm portion of the feedback algorithm FB may be able to modify one or more parameters in the gripping configuration determination algorithm D_GCF. Specifically, for a particular type of object having a particular feature or combination of features, the learning algorithm may modify one or more parameters in the processing of inputs used to determine the gripper configuration, such as determining the position of the Suction cups M1, M2 relative to the detected boundary of the object.

[0082] The feedback algorithm FB may further be configured to provide feedback to the object identification algorithm I_O and the algorithm D_RM that determines the robot's movements (e.g., reducing the movement speed for moving a particular grasped object G_O that may be known to be dropped in nature).

[0083] In some embodiments, the robotic system includes a second sensor system, such as a two-dimensional or three-dimensional camera, located on or above the pick-and-place robot RA, G, providing an image of the objects in the gripping area. This allows the control system CS to compare the image of the objects in the gripping area GA with the image IM of the objects at a position IMA upstream of the gripping area GA to detect whether one or more objects have changed position relative to the surface of the feed conveyor FC. For example, objects may move when an object G_O is picked up by the gripper G in the gripping area GA, or the object may have been displaced (e.g., such an object may have fallen off the top of the bulk load of objects BLK on its way from the upstream position IMA to the gripping area). Using this knowledge, the control system CS can L The system CS can adjust the control of the pick-and-place robots RA, G accordingly. Furthermore, the images of the objects in the gripping area GA can also be used by the control system to detect malfunctions or abnormal functions of the robot system. Thus, if the pick-and-place performance is assessed to be outside the range expected for normal functioning (e.g., if some objects in a row are placed far from their target positions), the control system CS can generate an alarm signal to enable possible repair of the robot, etc.

[0084] In some embodiments, the speed of the feed conveyor FC may be controlled in at least two steps, such as between "stop" and "normal speed." In particular, the speed of the feed conveyor FC may be controlled in multiple steps between "stop" and maximum speed, such as 2 to 10 steps between "stop" and maximum speed. It may be preferable to control the speed of the feed conveyor FC in response to an estimate of the expected pick-and-place throughput of the pick-and-place robot RA, G for the incoming bulk load of objects. By making the speed of the feed conveyor FC variable, the control system CS can ensure that the pick-and-place robot RA, G can process an adequate amount of the arriving objects. In particular, the speed of the feed conveyor FC may be controlled to enable the pick-and-place robot RA, G to process all arriving objects BLK, or at least the speed of the feed conveyor FC may be controlled to enable the pick-and-place robot RA, G to process at least a predetermined percentage of the arriving objects. Specifically, the feed conveyor FC may be controlled in a "stop-and-go" manner to allow the feed conveyor FC to transfer a bulk load of objects BLK to the gripping area GA, then stop to allow the pick-and-place robot RA, G to pick and place a predetermined percentage of the bulk load of objects BLK (e.g., all of the objects in the bulk load BLK), after which the feed conveyor FC is controlled to "go" to allow another bulk load of objects BLK to arrive at the gripping area GA for processing by the pick-and-place robot RA, G. Alternatively, the speed of the feed conveyor FC may be controlled gradually in time between multiple speed steps to ensure that the pick-and-place robot RA, G can process a predetermined percentage of the objects without stopping the feed conveyor FC. In some cases, it may be acceptable for the pick-and-place robot RA, G to process only 5-50% of the objects arriving if there is one or more pick-and-place robots downstream of the feed conveyor FC and / or if there is a manual processing station following.In such a case, the speed of the feed conveyor FC may be controlled by the control system CS to ensure a predetermined overall throughput of the pick and place system.

[0085] Specifically, the control system CS may be configured to estimate the time required to complete pick-and-place operations for one or more objects in the bulk load BLK before the bulk load BLK arrives at the gripping area GA. Such time may be estimated taking into account the identity of one or more incoming objects, e.g., as being “easy,” i.e., not time-consuming, or “difficult,” i.e., time-consuming, to process, which may be based on machine learning and / or artificial intelligence inputs. The control system may then control the speed of the feed conveyor FC so that the pick-and-place robot RA, G has time to process an acceptable number of objects G_O to be gripped.

[0086] To increase pick-and-place efficiency, the control system CS may increase the speed of the feed conveyor FC when a space is detected between two bulk objects BLK to ensure that the idle time of the pick-and-place robot RA, G is reduced. Because the robot actuator has a limit on the movement speed for moving the gripper, it is possible to optimize the utilization of the pick-and-place robot by controlling the speed of the feed conveyor FC. If an object arrives at an unfavorable position and time for the pick-and-place robot, the speed of the feed conveyor FC can be controlled to more optimally synchronize the robot's picking tasks and increase the robot's utilization rate.

[0087] The sorter SRT may include a plurality of support surfaces configured to move along a track in a transport direction, the support surfaces defining empty spaces for receiving and transporting objects or items. However, it should be understood that the sorter SRT may be of a type such as a tilt tray sorter, a crossbelt sorter, a tote-based sorter, a pusher sorter, a shoe sorter, or a pop-up sorter. In particular, the sorter SRT may be a closed-loop type sorter SRT. In particular, the sorter SRT may be configured to transport objects or items at a speed of at least 0.4 m / s (e.g., 0.5 to 1.0 m / s, such as 1.0 to 1.5 m / s, such as 1.5 to 2.0 m / s, more than 2.0 m / s, etc.). In particular, the sorter SRT may be configured to transport objects or items at a constant speed.

[0088] The sorter SRT may be part of a sorter system further comprising a plurality of discharge sections configured to receive items from the sorter SRT, the discharge sections being configured at different locations adjacent to the sorter SRT, such that the sorter system is configured to discharge items from the sorter SRT at selected discharge locations according to identification codes associated with individual items. For example, such identification codes may be barcodes, postal codes, ID tags, RFID tags, etc. By scanning the identification codes of the items, the sorter system can sort the items accordingly.

[0089] The guidance unit I1 is configured to receive an object or item S_O at one end and accelerate the object or item to deliver it to an empty space on the sorter SRT at a relative speed to the speed of the empty space on the sorter SRT, said relative speed being less than a predetermined threshold.

[0090] 3 shows a preferred robot actuator RA in the form of a gantry-type robot actuator with an adjustable gripper G having four suction cups (here shown with a gripped object). A set of horizontal elements forming a track for a first controllably movable cart CT1 is configured to be controllably actuated to move in a horizontal direction X. A second controllably movable cart CT2 is configured to be controllably actuated to move in another horizontal direction Y on the track of the first cart CT1.

[0091] Gripper G is mounted on a member fixed to the second cart CT2 and is controllably movable in a vertical direction Z to allow height adjustment of gripper G. Gripper G is shown mounted on this member by controllable actuator elements that allow controllable rotation about a vertical rotation axis RT_a. Furthermore, the gripper is mounted on this member by controllable actuator elements that allow controllable tilt about a vertical tilt axis TL_a.

[0092] The dimensions of the various elements of the robot actuator RA can be easily adapted to the possible X, Y, Z distances required for the pick-and-place robot, and the strength of the various elements can also be adapted to the load of the object to be processed. Various types of actuators for actuation in the X, Y, Z directions can be used, as are known in the art of gantry-type robots.

[0093] Figures 4a and 4b show two views of an embodiment of a preferred gripper G having a controllable gripping configuration of four gripping members in the form of suction cups M1, M2, M3, M4. In Figure 4a, the gripper G is shown in a fully extended gripper configuration, i.e., with suction cups M1, M2, M3, M4 at a maximum distance from each other and therefore suitable for gripping large objects. In Figure 4b, the gripper is shown in a fully compressed gripper configuration, i.e., with suction cups M1, M2, M3, M4 at a minimum distance from each other and therefore suitable for gripping small objects.

[0094] Base portion B provides for mounting on a robot actuator by a controllable tilt element for tilting about tilt axis TL_a and a controllable rotation element for rotating about rotation axis RT_a (also seen in FIG. 3 ). Suction cups M1, M2, M3, M4 are attached near the distal end of respective elongate arms A1, A2, A3, A4 attached to base portion B. Each arm A1, A2, A3, A4 is actuated by a controllable actuator and configured to slide along its length relative to base portion B, thereby allowing controllable adjustment of the arms A1, A2, A3, A4 with respect to the position of the suction cups M1, M2, M3, M4 relative to base portion B. The suction cups are aligned so that their suction contacts form a plane, and the arms A1, A2, A3, A4 are configured to slide to move along axes Dx, Dy parallel to this plane.

[0095] In particular, it can be seen that arms A1 and A2 are configured to slide along the same axis Dx, with portions of the arms configured to slide within one another. Arms A3 and A4 are similarly configured to slide within one another and are configured to slide along axis Dy. Thus, the four arms A1, A2, A3, and A4 are configured to extend in four different, mutually orthogonal directions within a single plane. Depending on the type of actuation selected, all four arms A1, A2, A3, and A4 may be actuated separately, albeit requiring separate actuators, to allow for high flexibility in terms of gripping configurations, or the arms A1, A2, A3, and A4 may be actuated together, for example, in pairs. Alternatively, two actuators can be used to actuate arms A1, A2, A3, and A4 in two pairs A1, A2 and A3, and one actuator can be used to actuate all four arms A1, A2, A3, and A4, allowing for only limited variations in gripping configurations. It may be preferable that the actuators (e.g., electric motors) for the arms be mounted above the tilt and rotation points to allow for a reduction in the weight of gripper G. In one embodiment, a rotation cable may transmit rotational force from the motor to actuate arms A1, A2, A3, A4 by a gear mechanism inside base portion B. If the lengths of all four arms A1, A2, A3, A4 are individually adjustable, various gripping configurations can be shaped to fit optimal gripping for irregular objects.

[0096] Such a gripper G is highly adaptable yet compact because even the compact base portion B provides highly adaptable gripping configurations, from a very compressed, compact configuration that takes up minimal space other than the dimensions of the base portion B itself, to a fully extended gripping configuration that may be used to handle larger objects. The gripper G may be controlled to provide a compressed gripping configuration that allows it to handle a larger object at one time and then immediately enter the space between two larger objects to grip a smaller object. This allows for high adaptability in handling bulk objects.

[0097] 5a-5c show various systems of pick-and-place robots R1, R2, R3, and R4 that cooperate to pick objects from a bulk load of objects BLK transported on a feed conveyor FC and place them directly onto guides I1, I2, I3, and I4 or onto empty spaces on a sorter SRT moving at a constant speed. When the capacity of a single pick-and-place robot is insufficient to handle all incoming objects BLK, it becomes necessary to use multiple pick-and-place robots. To solve this problem, several pick-and-place robots R1, R2, R3, and R4, as already described, can be positioned downstream of each other. Objects not picked by the first robot R1 are then input to the next robot R2 in line. For special objects or when there are peaks in incoming objects BLK that cannot be handled by the last robot R4, a manual pick-and-place process at a location downstream of the last robot R4 in line may be possible.

[0098] 5a shows a system with four robots R1, R2, R3, R4 arranged in two sets R1, R2 and R3, R4. The first set of gantry-type robots R1, R2 cooperate to place objects on a first guide station I1 to a sorter SRT, while the second set of gantry-type robots R3, R4 cooperate to place objects on a second guide station I2 to a sorter SRT. The first set of robots R1, R2 share a single 3D camera CM that provides input of objects BLK; the control systems for the first set of robots R1, R2 may already determine the objects to be processed by each robot R1, R2, or the robots R1, R2 may have separate control systems but may have the possibility to share a camera CM for providing 3D image input. One robot R1 picks an object from one gripping area GA on the feed conveyor FC and places it on a target area TA1 on the first guide section I1, while a second robot R2 picks an object from another gripping area on the feed conveyor FC and places it on another target area TA2 on the first guide section I1. Similarly, a second set of robots R3, R4 work together to pick up objects left on the feed conveyor FC by the first set of robots R1, R2 and place the objects on their respective target locations on the second guide section I2.

[0099] Figure 5b shows a system with four gantry-type robots R1, R2, R3, R4 arranged in two sets R1, R2 and R3, R4. The first set of robots R1, R2 cooperate to pick and place objects based on a common 3D camera input, as in Figure 5a, while the robots R1, R2 place objects onto respective simplified conveyor band guides I1, I2, which carry objects perpendicular to the empty space above the sorter SRT. Similarly, the second set of robots R3, R4 cooperate to place objects onto respective simplified conveyor band guides I3, I4. This means that, in contrast to the configuration of Figure 5a, there is no contention between the robots in a set of cooperating robots R1, R2 to place objects on the same guide; rather, in Figure 5b, all four robots R1, R2, R3, R4 pick objects on their respective non-overlapping gripping areas on the feed conveyor FC and place them on their respective target areas on separate guides I1, I2, I3, I4.

[0100] FIG. 5c illustrates yet another system with four gantry-type robots R1, R2, R3, and R4 arranged in two sets R1, R2, and R3, R4, which cooperate to pick up objects from a single feed conveyor FC, as described in FIGS. 5a and 5b. However, in FIG. 5c, each robot R1, R2, R3, and R4 is configured to grasp an object in a grasping area GA on the feed conveyor FC and place the object directly in an empty target area TA on the sorter SRT. In the illustrated embodiment, the sorter SRT and the feed conveyor FC are positioned parallel and adjacent to each other and move in the same direction (bold arrows) at the same or different speeds. The gantry-type robots R1, R2, R3, and R4 are mounted with one side next to the sorter SRT and the other side next to the feed conveyor FC. For example, the robots R1, R2, R3, and R4 can be mounted to the ground or other supports. In the illustrated embodiment, the gantry robots R1, R2, R3, R4 are configured with their main axes perpendicular to the direction of movement of the sorter SRT and the direction of movement of the feed conveyor FC. However, in other configurations, the gantry robots R1, R2, R3, R4 are set up with other angles between their main axes and the direction of movement of either or both of the feed conveyor FC and the sorter SRT.

[0101] The setup of Figure 5c eliminates the need for guides, which take up space and add extra complexity to the sorter system. However, the control systems of the pick-and-place robots R1, R2, R3, R4 still need to coordinate which empty space on the sorter SRT they place the object in. Furthermore, the control system is preferably configured to calculate whether the robots R1, R2, R3, R4 have enough time to reach the empty space location on the sorter SRT and deliver the object at a relative velocity with respect to the sorter velocity that is close to, or at least close to, zero.

[0102] FIG. 6 illustrates steps of a method embodiment for picking objects from a continuously moving stream of bulk objects and placing the singulated objects at a target location, either on a guide to a sorter or directly on the sorter. The method includes providing a controllable gripper (P_C_G) with a plurality of gripping members configured to engage a surface of the object to grip the object, the gripping members configured in a controllable gripping configuration. Further, providing a controllable robotic actuator (P_RA) configured to move the controllable gripper. Further, providing a three-dimensional image of objects in the moving stream of objects upstream of the location of the controllable robotic actuator (P_3DI). Next, processing the three-dimensional image (I_DO) to identify objects in the three-dimensional image that may be picked. Next, selecting which of the identified objects to grip (S_W_O). Next, determining characteristics of the selected objects according to the three-dimensional image (D_P_O) using image processing techniques. For example, step (D_P_O) may be determined before step (S_W_O) and before step (I_DO). Next, C_G_CF controls the gripping configuration of the gripping members according to said characteristics of the selected object. Next, C_RA controls the controllable robot actuators to move the controllable grippers into positions to grip the selected object. Next, G_O controls the controllable grippers to grip the selected object (e.g., by applying a vacuum if the gripping members are suction cups). Finally, R_O controls the controllable robot actuators and the controllable grippers to move the objects and individually align and release them at a target position onto a guide to a sorter or directly onto the sorter. After releasing the objects, P_I_TP provides an image of the objects at the target position (e.g., by a 2D or 3D camera).

[0103] A further step may include providing an image of the target location to determine whether and how the processed object was successfully placed at the target location, which the machine learning algorithm may provide as input to continually improve the performance of the robotic system by modifying parameters of one or more of steps ID_O, S_W_O, and C_G_CF.

[0104] It should be understood that, in principle, any type of object or item can be handled by the described robotic system. That is, the objects or items can be of various shapes, sizes, and have various surface characteristics. In particular, the stream of objects or items arriving at the infeed conveyor FC can include at least one of mail, parcels, luggage, items handled in warehouse distribution, and items handled in mail-order distribution, such as shoes, clothing, and textiles. In particular, the robotic system is designed to handle objects or items with a maximum weight of 1 to 100 kg (e.g., 1 to 10 kg, e.g., a maximum weight of 2 to 3 kg). In particular, objects or items with a maximum weight of 2 to 3 kg can be picked up and moved at high speed even by a medium-sized robot. It should be understood that the robotic system can alternatively be designed to handle objects heavier than 100 kg.

[0105] It will be appreciated that the functions of the control system are preferably performed by a processor system, which may be a computerized controller including a digital processor executing control algorithms implemented in software so as to allow the system's functionality to be easily updated and adapted, for example, by changes in the configuration of sorters and guides, and by including more robots in the system that need to be controlled to work together most effectively to process the flow of incoming objects or items.

[0106] In some embodiments, the control system may be implemented by a programmable logic controller (PLC). The processor may be a dedicated robot control processor, or may be implemented as part of or share a processor responsible for controlling the sorter. This allows adding one or more robots to an existing sorter system with minimal additional hardware for controlling the robots; in such implementations, program code for controlling the robots may be implemented purely as processor-executable program code. Similarly, the processor may be implemented as part of or share a processor responsible for controlling one or more guides for transferring items to the sorter. Furthermore, the processor may be implemented as part of or share a processor responsible for controlling an infeed conveyor, which may be advantageous if the robot is intended to separate items arriving on the infeed conveyor before picking them up for placement in the guide or sorter. Yet another variation may have separate robot controls with interfaces to a common machine controller for controlling the sorter, guides, and infeed conveyor. The machine controller may then have an interface to a system-wide controller, which may have an interface to a higher level control (e.g., a warehouse management system (WMS)).

[0107] The control system may be implemented with much of its functionality implemented as computer program code, and indeed the program code may be partially or fully integrated into existing systems for controlling sorters and guidance systems that are based on manual picking and movement from an infeed conveyor to a guidance station. However, it may be preferable for the control system to have two or more separate processors (e.g., separate processors responsible for performing at least some of the image processing required for three-dimensional images).

[0108] Embodiments of the robotic system have been tested and have demonstrated the ability to process up to 1500-2000 objects per hour with a 100% pick-and-place success rate when bulk loads of randomly shaped, sized, and textured objects arrive at a speed of 0.1-1.0 m / s on an infeed conveyor. In certain variations, throughputs of over 2000 objects per hour can be achieved.

[0109] 7 illustrates an example of a multi-pick-and-place robot system, shown here as two robots R1, R2 in a gantry configuration arranged along a feed conveyor FC and configured to cooperate to pick objects from their respective gripping areas on the feed conveyor FC and place the individualized, spaced-apart objects on a destination conveyor TC. The destination conveyor TC transfers the individualized, spaced-apart objects to a guidance system (not shown) that automatically guides the objects to a sorter. In this way, the pick-and-place robots R1, R2 indirectly provide objects for guidance to the sorter, and the primary task of the pick-and-place robots R1, R2 is to individualize the objects in a spaced-apart and preferably aligned orientation on the destination conveyor TC.

[0110] The robots R1, R2 may be as described above, with the gripper configured to rotate and tilt and the four suction cups controllably configured to adapt to the shape and size of the arriving objects. The robots R1, R2 may share a single upstream vision system, or the robots R1, R2 may have separate upstream vision systems to provide images of the bulk load of objects actually arriving upstream of each robot. Preferably, each robot R1, R2 is configured to individually identify the arriving bulk load of objects and determine which object to pick from the three-dimensional bulk load of arriving objects. Each robot R1, R2 may also have a separate vision system configured to provide images of the object after it has been placed on a target location on a target conveyor.

[0111] In the illustrated example, the feed conveyor FC and the destination conveyor TC are arranged parallel to each other and are configured to transport objects in opposite directions (indicated by the black arrows).

[0112] FIG. 8 shows another example of a multi-pick-and-place robot system. Here, eight gantry-type pick-and-place robots R are configured along a common feed conveyor FC to pick objects from their respective gripping areas on the feed conveyor FC and pair with them to place the individual objects onto four parallel guides I1, I2, I3, and I4, which transport and guide the objects onto a sorter (not shown). As with FIG. 7, the eight robots may be as described above, with grippers configured to rotate and tilt and four suction cups controllably configured to adapt to the shape and size of arriving objects. The eight robots may share a single upstream vision system, or each may have a separate upstream vision system to provide images of the bulk load of objects actually arriving upstream of each robot.

[0113] In summary, the present invention provides a robotic system for picking randomly shaped and sized objects from a continuously moving stream of bulk (e.g., three-dimensional) objects and individually placing the objects in a queue on a guide or directly on a sorter. The pick-and-place robot has a robot actuator for moving a gripper in a controllable gripping configuration of its gripping members (e.g., four suction cups) to adapt the gripper to various objects. A control system processes a three-dimensional image of objects upstream of the pick-and-place robot's position, identifies distinct objects in the three-dimensional image, and selects an object to be grasped based on parameters of the identified distinct objects determined from the three-dimensional image. Based on the size and shape of the selected object to be grasped, etc., the gripping configuration of the gripper is adjusted to fit the surface of the object to be grasped for optimal grasping. A robot actuator (e.g., a gantry-type robot actuator) is then controlled to move the gripper to a position to grip the object, then move the gripper together with the gripped object to a target position, release the grip on the object at the target pose, and place the object onto a guide or directly onto a sorter. Images of the object after it has been placed, along with the object properties determined from the three-dimensional image, can be used as input to machine learning to improve the pick-and-place performance of the robot system online (e.g., to improve algorithms online for selecting objects to pick and selecting an appropriate gripping configuration for the object).

[0114] Several embodiments are described below.

[0115] E1. 1. A robotic system configured to pick an object from a load of objects and place the object at a target location onto a guide to a sorter or directly onto the sorter, the system comprising: A pick and place robot, a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grip the object, the plurality of gripping members configured in a controllable gripping configuration; a controllable robotic actuator configured to move the controllable gripper; a pick and place robot comprising: a sensor system configured to provide an image of an object upstream of a position of the pick and place robot; a control system configured to receive the image and execute a control algorithm, the control algorithm comprising: Identifying an object according to the image; selecting which of the identified objects to grasp; controlling a gripping configuration of a plurality of gripping members in response to characteristics of the selected object determined from the image; controlling a controllable robotic actuator to move a controllable gripper to a position to grip the selected object; and controlling the controllable gripper to grip the selected object; Controlling a controllable robot actuator and a controllable gripper to move an object and release the object at a target location A control system configured as follows: A robot system comprising:

[0116] E2. The robot system of E1, wherein the controllable robot actuators include cartesian type robot actuators, such as gantry type robot actuators.

[0117] E3. The robotic system of E1 or E2, wherein the controllable robotic actuator is configured to accelerate the object toward the continuously moving sorter after it has been picked up, and to individually place the object directly onto the sorter at a target location at or near the sorter speed.

[0118] E4. The robot system of any one of E1-E3, wherein the sensor system is upstream of the position of the pick and place robot and configured to provide a static image covering a fixed area remote from the pick and place robot.

[0119] E5. The robot system of any one of E1 to E4, wherein the sensor system is configured to provide a three-dimensional image.

[0120] E6. The robotic system of any one of E1 to E5, wherein the sensor system includes a camera system positioned at a fixed position above the stream of moving objects.

[0121] E7. A robot system according to any one of E1 to E6, comprising a sensor configured to sense the height of an object after it has been picked up by the controllable gripper, the control system being connected to the sensor and configured to receive information indicative of the height of the object, and the control system being configured to control release of grip of the object at a height above a target position in response to the information indicative of the height of the object.

[0122] E8. The robotic system of any one of E1 to E7, comprising a sensor configured to provide an image of the object after it has been placed at the target location.

[0123] E9. The robotic system of E8, wherein the control system is configured to compare an image of the object after it has been placed at the target location with the object in the image of the object provided by the sensor system.

[0124] E10. The robot system of E9, wherein the control system is configured to process the image of the object after it has been placed at the target position to determine at least the position of the object relative to the target position, and the control system is configured to compare at least the position of the object with the target position and determine whether the position of the object deviates from the target position by more than a predetermined threshold.

[0125] E11. A robot system as described in E9 or E10, wherein the control system is configured to generate an output in response to the comparison between an image of the object after it has been placed at the target position and the object in the image of the object provided by the sensor system.

[0126] E12. A robot system according to any one of E9 to E11, wherein the control system is configured to provide an output indicative of the pick-and-place performance of the robot system in response to images of the plurality of objects after they have been placed in the target positions.

[0127] E13. The robotic system of any one of E9 to E12, wherein the control system is configured to provide the image of the object after it has been placed at the target position as feedback to a control algorithm.

[0128] E14. A robotic system as described in E13, wherein the control system includes a learning algorithm configured to learn characteristics in images of objects that have a high or low success rate of being placed in the target position based on multiple images of the object after it has been placed in the target position.

[0129] E15. A control system is configured to select which of the identified objects to grasp in response to the learning algorithm. 、E 15. The robot system according to claim 14.

[0130] E16. The robotic system of E14 or E15, wherein the characteristics include one or more of orientation, size, and type identification information.

[0131] E17. The robotic system of any one of E14 to E16, wherein the control algorithm includes at least one algorithm portion including an artificial intelligence algorithm and / or a neural network algorithm for processing images of the object after it has been placed in the target location to train one or more portions of the control algorithm to improve pick-and-place performance of the robotic system.

[0132] E18. A controllable gripper a base portion configured to be mounted on a robotic actuator; at least two arms attached to the base portion, each arm comprising: a gripping member configured at or near a distal end of the arm, the gripping member configured to engage an object to grip the object; at least two arms configured to be slidable along their lengths relative to a base portion actuated by a controllable actuator to allow the arms to be controllably adjusted with respect to the position of the gripping member relative to the base portion; Equipped with At least two arms are configured to be slidable in different directions relative to the base portion to allow the gripping member to form a variety of gripping configurations at least in terms of size. 、E The robot system according to any one of 1 to E17.

[0133] E19. The robotic system of E18, comprising four elongated arms attached to a base portion, the four arms configured to be slidable in different directions relative to the base portion to enable the four gripping members to form various gripping rectangle sizes.

[0134] E20. The robotic system of E18 or E19, wherein the control algorithm is configured to control a separate controllable actuator for each of a plurality of arms of a controllable gripper to grasp a selected object in response to the image.

[0135] E21. The robotic system of E18 or E19, wherein the control algorithm is configured to control a single controllable actuator to actuate multiple arms of a controllable gripper to grasp a selected object in accordance with the image.

[0136] E22. The robotic system of E18 or E19, wherein the control algorithm is configured to control at least two controllable actuators for control of multiple arms of a controllable gripper to grasp a selected object in accordance with the image.

[0137] E23. The robotic system of any one of E18 to E22, wherein each arm has a suction cup attached to or near the distal end of the arm.

[0138] E24. The robotic system of E23, comprising a controllable vacuum system connected to apply vacuum to the suction cups, the control system being configured to control the controllable vacuum system to control when to apply vacuum to the suction cups to grasp an object and when to discontinue the vacuum to release the object.

[0139] E25. The base portion is connected to the robot via a controllable rotation element such that the base portion can undergo controllable rotation about a rotation axis. Actuatorand the base portion is further attached to a robot actuator via a controllable tilt element such that the base portion can be controllably tilted about a tilt axis.

[0140] E26. A control system is configured to control the controllable gripper to cause the gripping members to form a predetermined gripping configuration in preparation for gripping the object. 、E The robot system according to any one of E18 to E25.

[0141] E27. The robot system of any one of E1 to E26, wherein a control algorithm of the control system is configured to control the controllable robot actuators and the controllable gripper to grasp the selected object to be picked in response to a plurality of inputs determined from the image, the plurality of inputs including at least one of information regarding the shape of the object, the horizontal boundary of the object, the size of the object, the orientation of the object, the top surface curvature of the object, and the surface roughness of the object, and the control algorithm processes the plurality of inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robot actuators and the controllable gripper to grasp the object.

[0142] E28. The robotic system of E27, wherein the control algorithm is configured to control the gripping configuration of the plurality of gripping members in response to the plurality of inputs.

[0143] E29. The robot system of E27 or E28, wherein the control algorithm is configured to control the tilt and rotation of the controllable gripper in response to the detected tilt and rotation of the object to be grasped.

[0144] E30. The robot system of any one of E27 to E29, wherein the plurality of inputs includes information about the top surface of the object regarding one or more of the location of wrinkled areas, the location of flat surface portions, the angle and orientation of the slope of the top surface.

[0145] E31. The robot system of E30, wherein the control algorithm is configured to control the controllable robot actuator and the controllable gripper to position at least one gripping member in a region of the object having a flat surface.

[0146] E32. A control algorithm is configured to control the controllable robot actuators and the controllable gripper to position the gripping members near all four corners of the object when the object is detected to have a flat surface with a rectangular shape. 、E The robot system according to any one of 27 to E31.

[0147] E33. The robot system of any one of E13 or E4 or E27 to E32, wherein the control system is configured to process the images of the object after being placed together with the inputs according to a learning algorithm, and accordingly modify a control algorithm with respect to processing the inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robot actuators and the controllable gripper to grasp the object.

[0148] E34. The control algorithm is analyzing the image to extract different parameters indicative of identified objects; calculating a score value for each of the plurality of identified objects as a function of the plurality of different parameters according to a predetermined scoring algorithm; The score values ​​are compared, and an object to be grasped is selected according to the result of the comparison. The robot system according to any one of E1 to E33, wherein the robot system is configured to select which of the identified objects to grasp in accordance with an object selection algorithm configured as follows:

[0149] E35. The robot system of E34, wherein the plurality of different parameters indicative of each of the identified plurality of objects include at least one of the following: a distance between the object and a current position of the gripper, a distance between the object and a target position at which the object is to be placed, a texture of a top surface of the object, a curvature of a top surface of the object, a slope of the object from the top surface, a height of the top surface of the object compared to the height of surrounding objects, a dimension of the object, a type of object, a shape of the object, and a quality of a portion of the image covering the object; and wherein a control algorithm is configured to assign, for each object, a parameter value for each of the plurality of different parameters according to a predetermined table, and wherein the control algorithm is configured to calculate an overall score value for each object in response to the assigned parameter values.

[0150] E36. The robot system of any one of E13 or E14 and E34 or E35, wherein the control system is configured to provide images of the object after it has been placed in the target position as feedback to the control algorithm for the step of determining which object of the identified plurality of objects to grasp, and the control system includes a learning algorithm configured to modify one or more parameters in the predetermined scoring algorithm based on the images of the object after it has been placed in the target position to improve pick-and-place performance.

[0151] E37. The robotic system of any one of E1 to E36, configured to accept the bulk objects as a continuously moving stream of bulk objects and configured to grasp a selected object from the moving bulk objects.

[0152] E38. The robotic system of any one of E1-E36, configured to accept a bulk load of objects and, after picking and placing all of the objects, accept another bulk load of objects.

[0153] E39. The robot system of any one of E1 to E38, configured to individually place objects at target locations.

[0154] E40. The robot system of any one of E1 to E39, configured to place an object in a target position with an aligned orientation.

[0155] E41. A conveyor for transporting objects of various shapes and sizes in bulk, such as a conveyor configured to move continuously; a sorter configured to receive the singulated objects; A first robotic system according to any one of E1 to E39, configured to pick up from the conveyor and place the singulated objects onto a guide to a sorter or directly onto the sorter; A sorting machine system comprising:

[0156] E42. The sorter system of E41, wherein the first robotic system is configured to place objects directly onto the sorter as the sorter moves at a constant speed.

[0157] E43. The sorter system of E41, wherein the first robotic system is configured to place objects on a guide to the sorter when the sorter moves at a constant speed.

[0158] E44. The sorter system of any one of E41 to E43, wherein the conveyor is configured to move continuously.

[0159] E45. A sorter system described in any one of E41 to E44, wherein the conveyor and sorter are arranged adjacent to each other, the conveyor has a first side facing the first side of the sorter, and the controllable robot actuator comprises a gantry-type or orthogonal-type robot actuator arranged with a first support portion located on the second side of the conveyor and a second support portion located on the second side of the sorter.

[0160] E46. A sorter system as described in any one of E41 to E45, comprising a second pick and place robot system as described in any one of E1 to E40, wherein the second robot system is positioned downstream of the conveyor compared to the first robot system.

[0161] E47. A method of using the robotic system of any one of E1 to E40 to process objects including at least one of mail, parcels, luggage, items handled in warehouse distribution, and items handled in mail order distribution centers.

[0162] E48. To process objects including at least one of mail, parcels, packages, items handled in warehouse distribution, and items handled in mail order distribution centers. E A method for using a sorting machine system according to any one of E41 to E46.

[0163] E49. 1. A method for picking an object from a bulk object, such as a continuously moving stream of bulk objects, and placing the object at a target location onto a guide to a sorter or directly onto the sorter, the method comprising: providing a controllable gripper comprising a plurality of gripping members configured to engage a surface of an object to grip the object, the gripping members being configured into a controllable gripping configuration; providing a controllable robotic actuator configured to move a controllable gripper; providing an image of an object in the stream of moving objects upstream of a location of the controllable robotic actuator; processing the image to identify objects within the image; selecting which of the identified objects to grasp; determining a characteristic of the selected object according to the image; controlling a gripping configuration of the gripping members in response to said properties of the selected object; controlling a controllable robotic actuator to move a controllable gripper to a position to grasp a selected object; controlling a controllable gripper to grasp a selected object; controlling controllable robotic actuators and controllable grippers to move the objects and release the objects (e.g., individually and / or oriented) at the target locations onto a guide to the sorter or directly onto the sorter; A method comprising:

[0164] E50. The method of E48, comprising controlling a controllable gripper to grasp a selected object from the bulk objects while the bulk objects are moving.

[0165] E51. The method of any one of E48 and E49, comprising controlling a controllable gripper to grasp a selected object from a stationary bulk object.

[0166] While the present invention has been described with reference to specified embodiments, it should not be construed as being limited in any way to the presented examples. The scope of the present invention should be interpreted in light of the appended claims. In connection with the claims, the terms "comprising" or "comprises" do not exclude other possible elements or steps. Furthermore, the use of terms such as "a" or "an" should not be construed as excluding a plurality. Furthermore, the use of reference signs in the claims for illustrated elements should not be construed as limiting the scope of the present invention. Furthermore, individual features recited in different claims may in some cases be advantageously combined, and the recitation of these features in different claims does not exclude that a combination of features is not possible or advantageous.

Claims

1. 1. A robotic system arranged to receive objects in a continuously moving bulk load (BLK) as a stream of objects in the bulk load (BLK), and configured to pick selected objects from the moving bulk load (BLK), such as a three-dimensional bulk load, and place the objects on a guidance section (I1) to a sorter (SRT) or directly on the sorter (SRT), the robotic system comprising: A pick and place robot, a controllable gripper (G) comprising a plurality of gripping members (M1, M2) configured to engage a surface of the object to grip the object, the plurality of gripping members (M1, M2) being configured in a controllable gripping configuration; a pick and place robot comprising a controllable robot actuator (RA) configured to move said controllable gripper (G); an upstream sensor system (CM) configured to provide an image (IM) of an object upstream of the position of the pick and place robot; a control system (CS) configured to receive said image (IM) and to execute a control algorithm, said control algorithm comprising: Identifying an object (I_O) according to the image (IM), Selecting which of the identified objects to grasp (S_O_G); controlling (D_GCF) the gripping configuration of the plurality of gripping members (M1, M2) in response to properties of the selected object determined from the image (IM); controlling the controllable robot actuator (RA) to move the controllable gripper (G) to a position (GA) for gripping the selected object, and controlling the controllable gripper (G) to grip the selected object (G_O); controlling the controllable robot actuator (RA) and the controllable gripper (G) to move the object (G_O) and release the object (G_O) at a target position (TA); A control system (CS) configured as follows: a downstream sensor system arranged to provide an image of the object after it has been placed at the target location (TA), the control system is configured to provide the image of the object after it has been placed at the target location as feedback to the control algorithm; the control system includes a learning algorithm configured to learn, based on a plurality of images of the object after it has been placed at the target location, features in the images provided by the upstream sensor system of objects that have a high or low success rate of being placed at the target location; the control system is configured to improve pick-and-place performance of the robotic system by modifying one or more parameters related to control of the selection and / or gripping configuration based on learning results of the learning algorithm. Robot system.

2. 2. The robotic system of claim 1, wherein the control system is configured to compare the image of the object after it has been placed at the target location with the object in the image of the object provided by the upstream sensor system.

3. 3. The robotic system of claim 1, wherein the control system is configured to generate an output in response to a comparison between the image of the object after being placed at the target location and the object in the image of the object provided by the upstream sensor system.

4. 4. The robotic system of claim 1, wherein the control system is configured to provide an output indicative of a pick-and-place performance of the robotic system in response to images of a plurality of objects after they have been placed at the target locations.

5. the characteristics include one or more of orientation, size, and type identification information; The robot system according to any one of claims 1 to 4.

6. 6. The robotic system of claim 5, wherein the control algorithm includes at least one algorithm portion including an artificial intelligence algorithm and / or a neural network algorithm for processing images of the object after it has been placed at the target location to train one or more portions of the control algorithm to improve pick-and-place performance of the robotic system.

7. the control algorithm of the control system is configured to control the controllable robot actuators and the controllable gripper to grasp a selected object to be picked in response to a plurality of inputs determined from the image of the object provided by the upstream sensor system, the plurality of inputs including at least one of information regarding a shape of the object, a horizontal boundary of the object, a size of the object, an orientation of the object, a top surface curvature of the object, and a surface roughness of the object; the control algorithm processes the plurality of inputs according to a predetermined algorithm to arrive at control parameters for controlling the controllable robotic actuators and the controllable gripper to grasp the object, the control algorithm being configured to control a grasping configuration of the plurality of grasping members in response to the plurality of inputs; 7. The robotic system of claim 2, wherein the control system is configured to process the images of the object after being placed together with the inputs according to a learning algorithm, and to accordingly modify the control parameters in relation to the processing of the inputs according to the predetermined algorithm to arrive at control parameters for controlling the controllable robot actuators and the controllable gripper to grasp the object.

8. 8. The robotic system of claim 1, further comprising a sensor configured to sense a height of the object after it has been picked up by the controllable gripper, the control system being connected to the sensor and configured to receive information indicative of the height of the object, and the control system being configured to control a release of grip of the object at a height above the target position in response to the information indicative of the height of the object.

9. the controllable gripper a base portion configured to be mounted on the robotic actuator; four elongated arms attached to the base portion, each arm comprising: a suction cup configured at or near a distal end of the arm, the suction cup configured to engage the object to grasp the object; four elongated arms configured to be slidable along axes actuated by controllable actuators to allow the arms to be controllably adjusted in terms of the position of the suction cup relative to the base portion; Equipped with the four arms are configured to be slidable in different directions relative to the base portion to allow the four suction cups to form various gripping rectangle sizes; the control algorithm is configured to control at least two controllable actuators for controlling the four arms of the controllable gripper to grip the selected object in response to the image of the object provided by the upstream sensor system; the base portion is attached to the robotic actuator via a controllable rotation element such that the base portion can controllably rotate about a rotation axis, and the base portion is further attached to the robotic actuator via a controllable tilt element such that the base portion can controllably tilt about a tilt axis; The robot system according to any one of claims 1 to 8.

10. 10. The robotic system of claim 1, wherein the control system is configured to control the controllable gripper to cause the gripping members to form a predetermined gripping configuration in preparation for gripping the object.

11. a second sensor system configured to provide an image of the object in the gripping area (GA); the second sensor system comprises a sensor, such as a camera, located on or above the pick-and-place robot (RA); The robot system according to any one of claims 1 to 10.

12. a conveyor (FC) configured to move continuously for transporting objects of various shapes and sizes in bulk; a sorter (SRT) configured to receive the singulated objects; a first robot system (R1) according to any one of claims 1 to 11, configured to pick up objects from the conveyor (FC) and place the separated objects on a guide (I1) to the sorter (SRT) or directly on the sorter (SRT); A sorting machine system comprising:

13. 12. Use of the robotic system of any one of claims 1 to 11 to process objects including at least one of mail, parcels, packages, items handled in warehouse distribution, and items handled in mail order distribution centers.

14. 1. A method for picking selected objects from a moving bulk load (BLK), such as a three-dimensional bulk load, arranged to receive the bulk objects as a continuously moving stream of bulk objects (BLK), and placing the objects by a control system onto a guide (I1) to a sorter (SRT) or directly onto the sorter (SRT), said method comprising: providing a controllable gripper (P_C_G) comprising a plurality of gripping members configured to engage a surface of the object to grip the object, the gripping members being configured into a controllable gripping configuration; providing a controllable robot actuator (P_RA) configured to move the controllable gripper; providing an image of a moving stream of objects upstream of the position of the controllable robotic actuator (P_3DI); processing the image to identify objects in the image (I_DO); Selecting which of the identified objects to grasp (S_W_O); determining a property of the selected object in response to the image (D_P_O); - controlling the gripping configuration of the gripping members in response to the properties of the selected object (C_G_CF); controlling the controllable robot actuator to move the controllable gripper to a position to grip the selected object (C_RA); controlling the controllable gripper to grasp a selected object from among the objects in the moving bulk load (G_O); controlling the controllable robot actuators and the controllable grippers to move the object and release it at a target position onto the guide to the sorter or directly onto the sorter (R_O); acquiring an image of the object after it has been placed at the target location (TA) from a downstream sensor system arranged to provide an image of the object after it has been placed at the target location (TA); Including, the control system is configured to provide the image of the object after it has been placed at the target location as feedback to a control algorithm; the control system includes a learning algorithm configured to learn, based on a plurality of images of the object after it has been placed at the target position, characteristics in the images of the moving stream of objects that have a high or low success rate of being placed at the target position; the control system is configured to improve pick and place performance by modifying one or more parameters related to the selecting and / or controlling of the gripping configuration based on learning results of the learning algorithm. method.

Images