Systems and techniques for robot-operated bin-picking

Abstract

Systems and techniques for robotic bin-picking are disclosed. In one example, a method includes receiving, by a processing unit, 3D scan information representing a bin containing multiple parts. The method further includes prioritizing, by the processing unit, a selected part of the multiple parts for picking by a robot. The method further includes signaling, by the processing unit, an indication of the selected part to a robot controller and instruction for a robot controlled by the robot controller to pick up the selected part from the bin.

Description

SYSTEMS AND TECHNIQUES FOR ROBOT-OPERATED BIN-PICKING TECHNICAL FIELD

[0001] This disclosure generally relates to the technical field of robotics. BACKGROUND

[0002] Robotics technology refers to the design, construction, operation, and use of automated systems (“robots”) to perform a variety of tasks. The deployment of robots to perform many such tasks improves the accuracy, efficiency, and scalability of these operations. Robots are often programmable by way of programmable circuitry and / or fixed-function circuitry, and in many use case scenarios, can be programmed to carry out sequences of operations automatically.

[0003] Robots have been deployed in many industrial settings, such as in the fields of industrial manufacturing, abrasives, automotive manufacturing, hydraulics, dispensing technology, additive manufacturing, pharmaceuticals, manufacturing, electronics foundries, and many others. Often, the operation of these robots is controlled via a computing device to mechanical device interface, with operation instructions and (optionally) parameters being supplied via the computing device. In the context of industrial manufacturing, one common end use application of robots is random 3D bin picking (or just “bin picking”) as a potential solution for part singulation. SUMMARY

[0004] According to existing bin picking solutions, individual objects of interest are identified inside a bin of randomly distributed objects using a vision system in a robotic workcell. From these identified objects, one is selected for further processing, for instance, by way of picking and placing by the robot. A typical use case would present the robot with a random box of injection-molded parts, and the robot is tasked with ordering the parts into nests for downstream assembly. Common examples of vision systems applicable to 3D bin picking are monocular and stereo vision cameras (e.g., as available via Intel® RealsenseTM), projected light scanners (e.g., as available via KeyenceTM3D vision systems for bin picking), and laser-based depth scanners. The vision system typically generates a 3D scan of the bin (e.g. a 3D point cloud or depth image with or without color), and discrete objects are identified inside the scanned point cloud, conventionally by pattern matching features extracted from a template component or CAD model or using a machine learning model such as an instance segmentation network. The hardware required for a bin picking system istypically general purpose, so the approach is flexible when compared to dedicated part feeding solutions such as bowl feeders. Several commercially available systems offer some combination of the full suite of bin picking sub-tasks, including physical simulation, part identification and selection, collision-free path planning, and robot control.

[0005] Such commercially available solutions present the end user with a significant trade-off between implementation ease and overall system optimization. For instance, robot paths generated by the bin picking controller are often inefficient when compared to those generated at the robot controller itself, as the bin picking controller may not have full access to control schemes such as blended moves required for fast, smooth robot motion. Additionally, general-purpose algorithms (e.g., “empty the bin from top to bottom” algorithms) that do not factor in the work area as a whole typically result in sub-optimal processing cycle times. Also, the stateless nature of commercially available bin picking systems do not provide a way to factor in the history of previous cycles to the planning of the current cycle and can easily get trapped into wasteful loops in repeated failed attempts to grab the same part over and over again. The technical problems described above give bin picking a reputation of being slow and only useful in a narrow range of industrial applications. A commonly accepted norm / rule for state-of-the-art bin picking systems is to expect roughly single digit picks per minute.

[0006] In one example, this disclosure is directed to a method. The method includes receiving, by a processing unit, 3D scan information representing a bin containing multiple parts. The method further includes prioritizing, by the processing unit, a selected part of the multiple parts for picking by a robot. The method further includes signaling, by the processing unit, an indication of the selected part to a robot controller and instruction for a robot controlled by the robot controller to pick up the selected part from the bin.

[0007] In another example, the disclosure is directed to a system. The system includes means for receiving 3D scan information representing a bin containing multiple parts. The system further includes means for prioritizing a selected part of the multiple parts for picking by a robot. The system further includes means for signaling an indication of the selected part to a robot controller and instruction for a robot controlled by the robot controller to pick up the selected part from the bin.

[0008] In another example, the disclosure is directed to a device. The system includes interface hardware configured to receive 3D scan information representing a bin containing multiple parts. The device further includes processing circuitry configured to prioritize a selected part of the multiple parts for picking by a robot. The processing circuitry is further configured to signal, via the interface hardware, an indication of the selected part to a robot controller andinstruction for a robot controlled by the robot controller to pick up the selected part from the bin.

[0009] This disclosure is directed to systems and techniques that address the technical problems described above. Aspects of this disclosure broaden the scope of bin picking applications into what would commonly be considered high-speed pick and place, such as applications requiring a pick rate on the order of one pick per second. BRIEF DESCRIPTION OF DRAWINGS

[0010] FIG.1 is a diagram illustrating aspects of existing bin picking systems and techniques.

[0011] FIG.2 is a diagram showing a part-nesting task performed by existing bin picking systems.

[0012] FIG.3 is a diagram illustrating an example of a high-fidelity simulation of a bin-picking workcell according to aspects of this disclosure.

[0013] FIG.4 illustrates a grid constructed over a volume of the bin.

[0014] FIG.5 is a diagram illustrating a schematic of a failed-attempt-based approach of this disclosure.

[0015] FIG.6 is a table illustrating a process for selection of parts according to aspects of this disclosure.

[0016] FIG.7 is a screenshot illustrating representative tools available in a commercially available system to prioritize parts are limited and not necessarily correlated to cycle time. DETAILED DESCRIPTION

[0017] FIG.1 is a diagram illustrating aspects of existing bin picking systems and techniques. FIG.1 illustrates a typical example of a 3D bin picking workcell, which consists of a 3D vision system, a bin of randomly distributed parts, and a robot for processing the parts.

[0018] FIG.2 is a diagram showing a part-nesting task performed by existing bin picking systems. The example shown in FIG.2 is a demonstration cell for pick and place of injection molded parts.

[0019] FIG.3 is a diagram illustrating an example of a high-fidelity simulation of a bin-picking workcell according to aspects of this disclosure. Systems of this disclosure, aspects of which are shown in FIG.3, include an additional processing unit inserted between the vision system and the robot controller. This added processing unit may store and may execute a suite of novel algorithms which replace the part scoring and prioritization algorithms generally used in commercially available bin-picking controllers. Systems of this disclosure use the vision system to acquire 3D scans, identify a list of pickable objects, and provide them to the new processing unit, in some instances with associated data (e.g. match quality data) that can befactored into prioritization. The added processing unit of this disclosure incorporates functionalities for prioritizing and choosing the specific parts for the robot to act on, by weighting information from the vision system with on-board information to improve and optimize workcell throughput.

[0020] One of the technical improvements provided by the added processing unit of this disclosure is the added capability to rapidly factor in high-fidelity cycle time estimates of robot motion on the set of identified parts (e.g., for high-speed application, one of the main concerns should be choosing the part which minimizes the expected processing time of the upcoming cycle, not necessarily some geometric position of the bin). The cycle time estimates can be obtained in one of several ways.

[0021] According to a first way, a high-accuracy simulation tool (usually the specific robot manufacturer’s simulation tool) is run on the added processing unit to directly estimate the cycle time for each available part. These simulations may be run in parallel processes to speed up computation time. According to a second way, the same simulation tool is used offline to regress the parameters of a prediction model (e.g. a multi-layer perceptron network or other function approximator) which takes as input the available part pose and outputs the expected cycle time for choosing that part. According to a third way, a cycle time prediction model is initialized using either a heuristic (e.g., a pick and place path length) or the above- described simulation method, but is then continually updated using actual cycle time data collected from the workcell itself. The above-listed techniques of this disclosure provide the functionality to optimize part selection directly against cycle time, which is one of the main concerns for high-speed applications.

[0022] In the example illustrated in FIG.3, pick poses are randomized at each cycle. The simulation logs the randomly generated pose and associated cycle time such that the data can be used to train a cycle time estimation model. An identical dataset can be collected on the actual workcell and used to train or update the parameters of the cycle time estimation model.

[0023] Another technical improvement provided by the added processing unit of this disclosure is the introduction of a notion of “state” to the bin. As an example, if the robot attempts to grasp a part but fails, the system may store and retain this result and include that history as an input to its future-executed decision-making operations either by purposefully retrying or deciding to avoid picking that part for some number of upcoming cycles. One technique of maintaining this information is to build / construct a map of the bin and update the map as needed.

[0024] FIG.4 illustrates a grid constructed over a volume of the bin. The grid of FIG.4 is an example of a simple map that the systems of this disclosure may use for state-guided decisionmaking as described above. In the example of FIG.4, each node of the grid is assigned a score related to the likelihood of a pick in that region to succeed. The map could be initialized using a heuristic, such as by assigning each node of the map the probability that any pick succeeds based on historical data from the cell. In one such example, if overall, the cell succeeds in picking 9 out of 10 parts, the map could be initialized uniformly with 0.9 at all nodes. In this example, every time the robot attempts to pick up a part, the map is updated based on the success or failure of that attempt. In some examples, a failure could lower the score of nearby nodes to the part, and a very low score in a region could indicate a tangled nest of parts such that the entire region of the bin should be avoided for the time being and / or for some subsequent time period. In the FIG.4 example section of a bin mapped using a grid, each grid section maintains a probabilistic score of how likely a pick inside the grid region will succeed. In the particular use-case scenario shown in FIG.4, the second and third (from the top) squares of the leftmost column and the fourth and fifth (from the top) squares in the second column from the left form an area that is a region highly unlikely to succeed, for example, a region containing tangled parts.

[0025] FIG.5 is a diagram illustrating a schematic of a failed-attempt-based approach of this disclosure. According to this approach (which is an alternative to the grid map approach described above), the systems of this disclosure may maintain a list of pick positions that failed. For each proposed next pick, the distance to each previous failed pick is determined, and if that distance is within some bound, the previous failed pick is counted as a neighbor for the current pick. The number of neighbors is then used as input to a sigmoid function that outputs the probability of pick success based on the number of neighboring failures. Further, each failure can be rated with a decay factor, such that failures are eventually disregarded over time. A successful pick could reset neighboring failures, as successful picks are likely to clear tangled parts in the nearby region. Again, in the example shown in FIG.5, a map of failed pick locations is maintained. Each failed pick has a region of impact indicated by the illustrated spheres. For example, the proposed pick in the top corner of the bin has a high proximity score due to its proximity to previous failed picks. The weight of the impact can decay over time (or a so-called “forgetting factor”). For each proposed pick, the proximity score is used to estimate the probability of a successful pick, for instance, using a sigmoid function as in the lower right that outputs a probability from 0 to 1. As the proximity score increases, the probability of a successful pick decreases. The sigmoid function parameters can be fit to historical cell data.

[0026] Irrespective of the prediction technique employed, once the systems of this disclosure have the estimated cycle times and probability of pick success for each detected object, thesystems may execute techniques of this disclosure to balance these objectives in one or more ways. While fast picks are desirable from an efficiency standpoint, a large number or frequent occurrence of missed picks will cause the cycle time to drop, thereby introducing a data precision-driven drop in efficiency. One way to balance estimated cycle time and probability of success is to calculate the expected cycle time after the pick happens for each of the identified parts. The expectation is the probabilistic weighted average for a random variable, so it essentially yields, on average, the resulting cycle time after picking a part based on the time it will take and the probability that the pick succeeds. That is, the system may likely benefit from ignoring or disregarding the part with the fastest cycle time if the probability of successfully picking it is low or very low.

[0027] Suppose the probability of successfully picking a part is , where ranges from 0 to 1 from one of the scoring methods above. The expected cycle time for the proposed pick is then expressed by equation (1) below:, , 1… (1)

[0028] Equation (1) is used to calculate the expected cycle time for each proposed pick, and then the systems of this disclosure may choose the part that offers the best expected cycle time based on the iterative calculations. In various examples consistent with this disclosure, the systems of this disclosure may add further constraints, such as, but not limited to, ignoring or not considering for selection any part which will cause the cycle time to drop out of some threshold if the pick fails.

[0029] FIG.6 is a table illustrating a process for selection of parts according to aspects of this disclosure. In the use case illustrated by way of FIG.6, the next pick decision is made prior to having data indicating whether the current pick will succeed. As such, the embodiment illustrated in FIG.6 demonstrates that the decision can be based on a look ahead window of multiple future picks according to some aspects of this disclosure.

[0030] Again, FIG.6 illustrates an example of using expected cycle time as a decision metric for which part to pick. In the use case illustrated in FIG.1, it is unknown whether or not the current pick will succeed, and so its cycle time and probability of success are factored into the decision about the following pick. For each of the four possible outcomes (namely, (fail, fail), (succeed, fail), (fail, succeed), and (succeed, succeed)), the resulting cycle time is calculated and are weighted by the probability of that particular pair of events occurring in the expectation for cycle time in the second to last column of FIG.6. An additional constraint on the right of the minimum allowable cycle time is included, and the probabilitythat the cycle time drops below this minimum is included. The particular use case shown in FIG.6 illustrates an example in which the option with the shortest estimated time (0.5 s) is not the best choice due to its associated probability of success.

[0031] The technical solutions described above can be further extended to multi-robot systems, where two or more robots can pick from the same bin. In such implementations, the optimization processing unit of this disclosure may further improve overall cell cycle times by allocating optimized poses to each robot of the multi-robot setup and by mitigating / minimizing instances of inter-robot interference.

[0032] FIG.7 is a screenshot illustrating representative tools available in a commercially available system (in this case, the Keyence® 3D VGR) to prioritize parts are limited and not necessarily correlated to cycle time.

[0033] In contrast, the technical solutions provided by the systems of this disclosure abstract away the motion control optimization from the vision system / part identification algorithm, thereby intelligently providing the user with a more generally customizable solution that avails of and utilizes high-accuracy simulation tools and / or real process data to determine the optimal part selection. In this way, the systems of this disclosure may run at the highest available throughput rate. In this way, the systems of this disclosure provide technical benefits in comparison to previously existing systems solutions in that the algorithms that form the core function of the systems of this disclosure are designed around the goal of maximizing throughput. By comparison, the limited tools available in a previously existing representative commercial system may allow controlling part priority only by its relative position in the bin and its part overlap percentage (e.g. how buried it is under other parts).

[0034] The systems of this disclosure relatively easily ingest simulation and actual process data from the real workcell and use various artificial intelligence / machine-learning (AI / ML) algorithms to generate a suite of fast optimization routines for deciding how the robot should act for the upcoming cycle(s). This approach minimizes the amount of real-time processing needed to complete the part selection, enabling part optimization calculations to be done without impacting cycle time.

[0035] Another aspect of the systems of this disclosure is the notion of including actual process historical data in the optimization routines. There may be instances in which given poses are less likely to generate successful processing. The systems of this disclosure include the process success or failure in scoring algorithms to increase or potentially maximize the likelihood that the commanded process will be successful in the estimated amount of time.

[0036] In the present detailed description of the example embodiments, reference is made to the accompanying drawings, which illustrate specific embodiments in which the invention may bepracticed. The illustrated examples are not intended to be exhaustive of all embodiments according to the invention. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present invention. The detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0037] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood as being modified in all instances by the term “about” or “approximately” or “substantially.” Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and attached claims are approximations that can vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0038] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” encompass embodiments having plural referents, unless the content clearly dictates otherwise. As used in this specification and the appended claims, the term “or” is generally employed in its sense including “and / or” unless the content clearly dictates otherwise.

[0039] It is to be recognized that depending on the example, certain acts or events of any of the methods described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0040] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, CPUs, GPUs, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), processing circuitry (e.g., fixed function circuitry, programmable circuitry, or any combination of fixed function circuitry and programmable circuitry), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The term “processor” or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit comprising hardware may also perform one or more of the techniques of this disclosure.

[0041] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the described units, modules or components may beimplemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

[0042] The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer-readable media.

[0043] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

CLAIMS What is claimed is:

1. A method comprising: receiving, by a processing unit, 3D scan information representing a bin containing multiple parts; prioritizing, by the processing unit, a selected part of the multiple parts for picking by a robot; and signaling, by the processing unit, an indication of the selected part to a robot controller and instruction for a robot controlled by the robot controller to pick up the selected part from the bin.

2. The method of claim 1, wherein prioritizing the selected part comprises weighting respective pick priorities associated with two or more of the multiple parts based on an estimated cycle time associated with each respective part of the two or more parts.

3. The method of claim 2, further comprising calculating the respective estimated cycle times by executing a simulation tool that performs robotic pick simulations for each of the two or more parts partially or fully in parallel.

4. The method of claim 2, further comprising calculating the respective estimated cycle times by executing a simulation tool that regresses parameters of a prediction model that outputs the respective estimated cycle times.

5. The method of claim 4, wherein the prediction model comprises a multi-layer perceptron network or other function approximator.

6. The method of claim 1, wherein prioritizing the selected part comprises weighting respective pick priorities associated with two or more of the multiple parts based on state information associated with each of the two or more parts.

7. The method of claim 6, further comprising setting the state information for at least one part of the two or more parts based on a past failed attempt by the robot to pick up the at least one part.

8. The method of claim 6, further comprising setting the state information for at least one part of the two or more parts based on a past failed attempt by the robot to pick up a neighboring part of the at least one part.

9. The method of claim 7, further comprising reducing a weight associated with the state information based on a decay factor associated with the past failed attempt.

10. The method of claim 6, further comprising: Constructing a map of the bin; and Updating the state information based on pick success heuristic data associated with each region of the map.

11. The method of claim 10, wherein the map is a grid, and wherein the state information is expressed as a probability value associated with a corresponding region of the grid.

12. The method of claim 6, wherein weighting the respective pick priorities comprises: calculating a probabilistic weighted average of success for each of the two or more parts; and assigning each respective probabilistic weighted average of success to an estimated cycle time associated with the respective part.

13. The method of claim 1, wherein the robot is included in a plurality of robots configured to pick from the bin, the method further comprising selecting the robot from the plurality of robots to pick up the selected part from the bin based on a determination that reduces inter-robot interference between the plurality of robots.

14. A system comprising means for performing the operations of one or more of claims 1–13.

15. The system of claim 14, wherein the means comprises a memory and processing circuitry communicatively coupled to the memory, wherein the processing circuitry includes or is part of the processing unit.

Images