Systems and methods for robotic bin picking

Abstract

The present invention provides a method and computing system, the method and computing system including identifying one or more candidate objects for selection by a robot. A path to the one or more candidate objects may be determined based at least in part on a robot environment and at least one robot constraint. Feasibility of grasping a first candidate object of the one or more candidate objects may be verified. If the feasibility is verified, the robot may be controlled to physically select the first candidate object. If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object may be selected.

Description

[0001] Related patent applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 690,186, filed on June 26, 2018, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates generally to robots, and more particularly, to systems and methods for robotic bin picking. Background Art

[0004] Some forms of manual labor such as unloading boxes into machines one workpiece at a time, bulk parts sorting, and order fulfillment are labor-intensive. These jobs are often dangerous if the workpieces or operations are heavy, sharp, or otherwise dangerous. In an effort to address these issues, box picking robots have addressed these tedious tasks. However, robotic box picking is a particularly difficult task to manage because the amount of accuracy and precision required is often beyond the capabilities of the system. Summary of the invention

[0005] In one embodiment, a method for identifying one or more candidate objects for selection by a robot is provided. A path to the one or more candidate objects may be determined based at least in part on a robot environment and at least one robot constraint. Feasibility of grasping a first candidate object of the one or more candidate objects may be verified. If the feasibility is verified, the robot may be controlled to physically select the first candidate object. If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object may be selected.

[0006] One or more of the following features may be included. The verification may include using a robot kinematic model. The path may be at least one of a feasible path or an optimal path. The path may be determined in real time while controlling the robot. Determining the path may include using information about one or more surfaces of at least one object adjacent to the candidate object and avoiding collisions with at least one object adjacent to the candidate object. The robot or at least one of the one or more candidate objects may be displayed at a graphical user interface. The graphical user interface may allow a user to visualize or control at least one of the robot, path determination, simulation, work cell definition, performance parameter specification, or sensor configuration. The graphical user interface may allow for simultaneous creation of a program and a debugging process associated with the program. The graphical user interface may be associated with one or more of a teach pendant, a handheld device, a personal computer, or a robot. An image of an environment including one or more static objects and dynamic objects using a scanner may be provided, wherein the robot is configured to receive the image and use the image to learn the environment to determine the path and collision avoidance. Controlling the robot may include performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed position with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target according to the accuracy requirement. Controlling the robot may include presenting the first candidate object to the scanner to maximize the use of one or more features on the first candidate object to accurately locate the first candidate object. Controlling the robot may include locating and picking the first candidate object in a manner that maximizes the probability of successful physical selection. A second scan may be performed in the maximum resolution area of ​​the scanner. Determining a path to the one or more candidate objects may be based at least in part on at least one of a robot link or a robot joint limit. A shrink wrap visualization on all unselected parts and unselected surfaces other than the one or more candidate objects may be displayed at a graphical user interface. At least one of identification, determination, verification, or control may be performed using at least one of a main processor and at least one coprocessor. Determining a path to the one or more candidate objects may be based at least in part on at least one of a global path plan and a local path plan. Verifying the feasibility of grasping the first candidate object may include analyzing conditional logic associated with a user program. Verifying the feasibility of grasping the first candidate object may include verifying all path alternatives, verifying a specific path alternative, verifying any path alternative, verifying one or more abnormal paths, excluding one or more verified sections, or performing parallel verification of multiple sections of the path.

[0007] In another specific implementation, a computing system including a processor and a memory is configured to perform operations including identifying one or more candidate objects for selection by a robot. A path to the one or more candidate objects may be determined based at least in part on a robot environment and at least one robot constraint. Feasibility of grasping a first candidate object of the one or more candidate objects may be verified. If the feasibility is verified, the robot may be controlled to physically select the first candidate object. If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object may be selected.

[0008] One or more of the following features may be included. The verification may include using a robot kinematic model. The path may be at least one of a feasible path or an optimal path. The path may be determined in real time while controlling the robot. Determining the path may include using information about one or more surfaces of at least one object adjacent to the candidate object and avoiding collisions with at least one object adjacent to the candidate object. The robot or at least one of the one or more candidate objects may be displayed at a graphical user interface. The graphical user interface may allow a user to visualize or control at least one of the robot, path determination, simulation, work cell definition, performance parameter specification, or sensor configuration. The graphical user interface may allow for simultaneous creation of a program and a debugging process associated with the program. The graphical user interface may be associated with one or more of a teach pendant, a handheld device, a personal computer, or a robot. An image of an environment including one or more static objects and dynamic objects using a scanner may be provided, wherein the robot is configured to receive the image and use the image to learn the environment to determine the path and collision avoidance. Controlling the robot may include performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed position with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target according to the accuracy requirement. Controlling the robot may include presenting the first candidate object to the scanner to maximize the use of one or more features on the first candidate object to accurately locate the first candidate object. Controlling the robot may include locating and picking the first candidate object in a manner that maximizes the probability of successful physical selection. A second scan may be performed in the maximum resolution area of ​​the scanner. Determining a path to the one or more candidate objects may be based at least in part on at least one of a robot link or a robot joint limit. A shrink wrap visualization on all unselected parts and unselected surfaces other than the one or more candidate objects may be displayed at a graphical user interface. At least one of identification, determination, verification, or control may be performed using at least one of a main processor and at least one coprocessor. Determining a path to the one or more candidate objects may be based at least in part on at least one of a global path plan and a local path plan. Verifying the feasibility of grasping the first candidate object may include analyzing conditional logic associated with a user program. Verifying the feasibility of grasping the first candidate object may include verifying all path alternatives, verifying a specific path alternative, verifying any path alternative, verifying one or more abnormal paths, excluding one or more verified sections, or performing parallel verification of multiple sections of the path.

[0009] The details of one or more implementations are set forth in the accompanying drawings and the detailed description that follows.Other features and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are included to provide a further understanding of the embodiments of the present disclosure and are incorporated in and constitute a part of this specification. The accompanying drawings illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the embodiments of the present disclosure.

[0011] Figure 1 is a diagrammatic view of a robotic bin picking process connected to a distributed computing network;

[0012] Figure 2 yes Figure 1 A flowchart of a specific implementation of the robot picking process;

[0013] Figure 3 It is a bin picking system according to an embodiment of the present disclosure that is configured to run all modules on a coprocessor and to interface with a UR controller via an Ethernet connection using a real-time data exchange interface of the UR.

[0014] Figure 4 is an interface showing a bin picking system deployment diagram according to an embodiment of the present disclosure.

[0015] Figure 5 is an interface showing an embodiment of a bin picking system consistent with an embodiment according to the present disclosure.

[0016] Figure 6 is an interface of a graphical user interface illustrating a bin picking process consistent with an embodiment of the present disclosure.

[0017] Figure 7 is a graphical user interface for a bin picking process consistent with an embodiment of the present disclosure.

[0018] Figure 8 is a graphical user interface for a bin picking process consistent with an embodiment of the present disclosure.

[0019] Fig. 9 is a graphical user interface for generating a program template according to an embodiment of the present disclosure.

[0020] Fig.10 is a graphical user interface for generating a program template according to an embodiment of the present disclosure.

[0021] Fig.11 is a graphical user interface for generating a program template according to an embodiment of the present disclosure.

[0022] Fig.12 is a graphical user interface that allows configuration of EOAT according to an embodiment of the present disclosure.

[0023] Fig.13 is a graphical user interface that allows configuration of tool collision shapes according to an embodiment of the present disclosure.

[0024] Fig.14 is a graphical user interface that allows box configuration according to an embodiment of the present disclosure.

[0025] Fig.15 is a graphical user interface that allows bin registration according to an embodiment of the present disclosure.

[0026] Fig.16 is a graphical user interface that allows configuration of box collision shapes according to the present disclosure.

[0027] Fig.17 is a graphical user interface that allows configuration of artifacts and loading of artifact models according to an embodiment of the present disclosure.

[0028] Fig.18 is a graphical user interface that allows configuration of workpiece collision shapes according to an embodiment of the present disclosure.

[0029] Fig.19 is a graphical user interface that allows verification of artifact detection according to an embodiment of the present disclosure.

[0030] Fig. 20 is a graphical user interface that allows rescanning of location configurations according to an embodiment of the present disclosure.

[0031] Fig.21 is a graphical user interface that allows configuration of crawl hierarchies and / or crawl selection criteria according to an embodiment of the present disclosure.

[0032] Fig. 22 is a graphical user interface that allows configuration of crawl hierarchies and / or crawl selection criteria according to an embodiment of the present disclosure.

[0033] Fig.23 is a graphical user interface that allows adding and / or arranging grabs according to an embodiment of the present disclosure.

[0034] Fig.24 is a graphical user interface that allows training of grab and place according to an embodiment of the present disclosure.

[0035] Fig.25 is a graphical user interface that allows training of placement positions and offsets according to an embodiment of the present disclosure.

[0036] Fig.26 is a graphical user interface that allows training of placement positions and offsets according to an embodiment of the present disclosure.

[0037] Fig. 27 is a graphical user interface that allows configuration of grasp and release sequences according to an embodiment of the present disclosure.

[0038] Fig.28 is a graphical user interface that allows system operation according to an embodiment of the present disclosure.

[0039] Fig.29 A graphical user interface according to an embodiment of the present disclosure may allow a user to install a bin picking URCap from a USB drive or other suitable device.

[0040] Fig.30 is a graphical user interface that allows a user to configure an environment according to an embodiment of the present disclosure.

[0041] Fig.31 is a graphical user interface that allows a user to configure a sensor according to an embodiment of the present disclosure.

[0042] Fig.32 is a graphical user interface that allows a user to register sensors according to an embodiment of the present disclosure.

[0043] Fig.33 is a graphical user interface that allows a user to register sensors according to an embodiment of the present disclosure.

[0044] Fig.34 is a graphical user interface that allows a user to register sensors according to an embodiment of the present disclosure.

[0045] Fig.35 is a graphical user interface that allows a user to register sensors according to an embodiment of the present disclosure.

[0046] Fig.36 is a graphical user interface that allows a user to create a bin picking program according to an embodiment of the present disclosure.

[0047] Fig.37 is a graphical user interface showing options for generating a procedure template according to an embodiment of the present disclosure.

[0048] Fig.38 is a graphical user interface showing examples of options available to a user according to an embodiment of the present disclosure.

[0049] Fig.39 is a graphical user interface illustrating a method for setting crawling indicators according to an embodiment of the present disclosure.

[0050] Fig.40 is a graphical user interface showing an exemplary graphical user interface that allows setting up an RRT node according to an embodiment of the present disclosure.

[0051] Fig.41 is a graphical user interface that allows a user to set an original position according to an embodiment of the present disclosure.

[0052] Fig.42 is a graphical user interface that allows a user to configure a tool according to an embodiment of the present disclosure.

[0053] Fig.43 is a graphical user interface that allows a user to align registration boxes according to an embodiment of the present disclosure.

[0054] Fig.44 is a graphical user interface that allows a user to align registration boxes according to an embodiment of the present disclosure.

[0055] Fig.45 is a graphical user interface that allows a user to configure box collision shapes according to an embodiment of the present disclosure.

[0056] Fig.46 A graphical user interface allowing a user to validate a part template according to an embodiment of the present disclosure.

[0057] Fig.47 is a graphical user interface that allows a user to configure a rescan location according to an embodiment of the present disclosure.

[0058] Fig.48 is a graphical user interface that allows a user to add grabs according to an embodiment of the present disclosure.

[0059] Fig.49 is a graphical user interface that allows a user to train grabbing and placing according to an embodiment of the present disclosure.

[0060] Fig.50 is a graphical user interface that allows a user to train picking according to an embodiment of the present disclosure.

[0061] Fig.51 is a graphical user interface that allows a user to configure an EOAT signal according to an embodiment of the present disclosure.

[0062] Fig.52 is a graphical user interface that allows a user to operate the system according to an embodiment of the present disclosure.

[0063] Fig.53 is a graphical user interface that allows a user to create additional nodes according to an embodiment of the present disclosure.

[0064] Fig.54 is a flow chart illustrating an example of installation, program configuration, and bin picking operations consistent with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] Embodiments of the present disclosure relate to systems and methods for robotic bin picking. Thus, the bin picking methods included herein can allow a robot to work with a scanning system to identify parts in a bin, pick parts from the bin, and place the picked parts at a specified location.

[0066] Embodiments of the subject application may include concepts from U.S. Patent 6,757,587, U.S. Patent 7,680,300, U.S. Patent 8,301,421, U.S. Patent 8,408,918, U.S. Patent 8,428,781, U.S. Patent 9,357,708, U.S. Publication 2015 / 0199458, U.S. Publication 2016 / 0321381, and U.S. Publication 2018 / 0060459, the entire contents of each of which are incorporated herein by reference in their entirety.

[0067] See now Figure 1 , shows a robotic bin picking process 10 that may reside on and be executed by a computing device 12, which may be connected to a network (e.g., network 14) (e.g., the Internet or a local area network). Examples of computing device 12 (and / or one or more of the client electronic devices described below) may include, but are not limited to, a personal computer, a laptop computer, a mobile computing device, a server computer, a series of server computers, a mainframe computer, or a computing cloud. Computing device 12 may execute an operating system, such as, but not limited to, OS Red or customized operating systems. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the U.S., other countries, or both; Mac and OS X are registered trademarks of Apple Inc. in the U.S., other countries, or both; Red Hat is a registered trademark of Red Hat Corporation in the U.S., other countries, or both; and Linux is a registered trademark of Linus Torvalds in the U.S., other countries, or both).

[0068] As will be discussed in more detail below, the robotic bin picking process such as Figure 1 The robot bin picking process 10 may identify one or more candidate objects for selection by the robot. A path to the one or more candidate objects may be determined based at least in part on the robot environment and at least one robot constraint. The feasibility of grasping a first candidate object of the one or more candidate objects may be verified. If the feasibility is verified, the robot may be controlled to physically select the first candidate object. If the feasibility is not verified, at least one of a different grasping point for the first candidate object, a second path, or a second candidate object may be selected.

[0069] The instruction sets and subroutines of the robotic bin picking process 10, which may be stored on a storage device 16 coupled to the computing device 12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within the computing device 12. The storage device 16 may include, but is not limited to: a hard drive; a flash drive, a tape drive; an optical drive; a RAID array; a random access memory (RAM); and a read-only memory (ROM).

[0070] Network 14 may be connected to one or more secondary networks (eg, network 18), examples of which may include, but are not limited to: a local area network; a wide area network; or an intranet.

[0071] The robotic bin picking process 10 may be a standalone application that interfaces with applet / applications accessed via client applications 22, 24, 26, 28, 66. In some embodiments, the robotic bin picking process 10 may be distributed in whole or in part in a cloud computing topology. Thus, the computing device 12 and the storage device 16 may refer to a plurality of devices that may be distributed throughout the network 14 and / or the network 18.

[0072] Computing device 12 may execute a robot control application (e.g., robot control application 20), examples of which may include, but are not limited to, the robot control application 20 from Energid Technologies of Cambridge, Massachusetts. Software Development Kit and any other bin picking application or software. The robotic bin picking process 10 and / or the robotic control application 20 are accessible via the client applications 22, 24, 26, 28, 68. The robotic bin picking process 10 may be a standalone application, or may be an applet / application / script / extension that may interact with and / or execute within the robotic control application 20, components of the robotic control application 20, and / or one or more of the client applications 22, 24, 26, 28, 68. The robotic control application 20 may be a standalone application, or may be an applet / application / script / extension that may interact with and / or execute within the robotic bin picking process 10, components of the robotic bin picking process 10, and / or one or more of the client applications 22, 24, 26, 28, 68. One or more of the client applications 22, 24, 26, 28, 68 may be stand-alone applications, or may be applet / applications / scripts / extensions that may interact with and / or execute within components of the robotic bin picking process 10 and / or the robotic control application 20. Examples of client applications 22, 24, 26, 28, 68 may include, but are not limited to, applications that receive queries to search for content from one or more databases, servers, cloud storage servers, etc., text and / or graphical user interfaces, custom web browsers, plug-ins, application programming interfaces (APIs), or custom applications. The instruction sets and subroutines of the client applications 22, 24, 26, 28, 68 that may be stored on the storage devices 30, 32, 34, 36 coupled to the client electronic devices 38, 40, 42, 44 may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into the client electronic devices 38, 40, 42, 44.

[0073] Storage devices 30, 32, 34, 36 may include, but are not limited to: hard drives; flash drives, tape drives; optical drives; RAID arrays; random access memory (RAM); and read-only memory (ROM). Examples of client electronic devices 38, 40, 42, 44 (and / or computing device 12) may include, but are not limited to, personal computers (e.g., client electronic device 38), laptop computers (e.g., client electronic device 40), smart / data-enabled cellular phones (e.g., client electronic device 42), notebook computers (e.g., client electronic device 44), tablet computers (not shown), servers (not shown), televisions (not shown), smart televisions (not shown), media (e.g., video, photo, etc.) capture devices (not shown), and dedicated network devices (not shown). Client electronic devices 38, 40, 42, 44 may each execute an operating system, examples of which may include, but are not limited to OS Red Mobile, ChromeOS, Blackberry OS, Fire OS or custom operating systems.

[0074] One or more of the client applications 22, 24, 26, 28, 68 may be configured to implement some or all of the functionality of the robotic bin picking process 10 (and vice versa). Thus, the robotic bin picking process 10 may be a pure server-side application, a pure client-side application, or a hybrid server-side / client-side application that is cooperatively executed by one or more of the client applications 22, 24, 26, 28, 68 and / or the robotic bin picking process 10.

[0075] One or more of the client applications 22, 24, 26, 28, 68 may be configured to implement some or all of the functionality of the robot control application 20 (and vice versa). Thus, the robot control application 20 may be a pure server-side application, a pure client-side application, or a hybrid server-side / client-side application that is cooperatively executed by one or more of the client applications 22, 24, 26, 28, 68 and / or the robot control application 20. Since one or more of the client applications 22, 24, 26, 28, 68, the robotic picking process 10, and the robotic control application 20, employed individually or in any combination, can implement some or all of the same functions, any description of implementing such functions via one or more of the client applications 22, 24, 26, 28, 68, the robotic picking process 10, the robotic control application 20, or a combination thereof, and any described interactions between the client applications 22, 24, 26, 28, 68, the robotic picking process 10, the robotic control application 20, or a combination thereof that implement such functions should be taken merely as examples and not as limitations on the scope of the present disclosure.

[0076] The users 46, 48, 50, 52 may access the computing device 12 and the robotic bin picking process 10 directly or indirectly (e.g., using one or more of the client electronic devices 38, 40, 42, 44) through the network 14 or through the secondary network 18. In addition, the computing device 12 may be connected to the network 14 through the secondary network 18, as shown by the dashed connecting line 54. The robotic bin picking process 10 may include one or more user interfaces, such as a browser and a text or graphical user interface, through which the users 46, 48, 50, 52 can access the robotic bin picking process 10.

[0077] Various client electronic devices may be coupled to network 14 (or network 18) directly or indirectly. For example, client electronic device 38 is shown as being directly coupled to network 14 via a hardwired network connection. Additionally, client electronic device 44 is shown as being directly coupled to network 18 via a hardwired network connection. Client electronic device 40 is shown as being wirelessly coupled to network 14 via a wireless communication channel 56 established between client electronic device 40 and a wireless access point (i.e., WAP) 58, which is shown as being directly coupled to network 14. WAP 58 may be, for example, an IEEE 800.11a, 800.11b, 800.11g, 800.11b-1, 800.11c, 800.11d, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1, 800.11d-1 and / or Bluetooth TM (including Bluetooth TM The client electronic device 42 is shown as being wirelessly connected to the network 14 via a wireless communication channel 60 established between the client electronic device 42 and a cellular network / bridge 62, which is shown as being directly connected to the network 14. In some specific implementations, the robotic system 64 can be wirelessly connected to the network 14 via a wireless communication channel 66 established between the client electronic device 42 and the cellular network / bridge 62, which is shown as being directly connected to the network 14. Storage devices 70 can be connected to the robotic system 64 and can include, but are not limited to: hard drives; flash drives, tape drives; optical drives; RAID arrays; random access memory (RAM); and read-only memory (ROM). Users 72 can access the computing device 12 and the robotic picking process 10 directly or indirectly through the network 14 or through the secondary network 18 (e.g., using the robotic system 64).

[0078] Some or all of the IEEE 800.11x specifications may use the Ethernet protocol and carrier sense multiple access and collision avoidance (ie, CSMA / CA) for path sharing. Various 800.11x specifications may use, for example, phase shift keying (ie, PSK) modulation or complementary code keying (ie, CCK) modulation. Bluetooth TM (including Bluetooth TM Wireless Low Energy (WLAN) is a telecommunications industry specification that allows, for example, mobile phones, computers, smartphones, and other electronic devices to interconnect using short-range wireless connections. Other forms of interconnection (e.g., Near Field Communication (NFC)) may also be used.

[0079] See also Figures 2 to 54, and in some embodiments, the robotic bin picking process 10 may generally include identifying one or more candidate objects for selection by the robot 200. A path to the one or more candidate objects may be determined based at least in part on the robot environment and at least one robot constraint 202. The feasibility of grasping a first candidate object of the one or more candidate objects may be verified 204. If the feasibility is verified, the robot may be controlled to physically select the first candidate object 206. If the feasibility is not verified, at least one of a different grasping point for the first candidate object, a second path, or a second candidate object may be selected 208.

[0080] As used herein, the term "Actin Viewer" may refer to a graphical user interface, "Actin" may refer to robot control software, and "UR" may refer to "Universal Robots". Any use of these specific companies and products is provided by way of example only. Thus, any suitable graphical user interface, robot control software, and devices / modules may be used without departing from the scope of this disclosure.

[0081] In some embodiments, a bin picking system (e.g., bin picking system 64) may include a robotic arm (e.g., a Universal Robots UR5 from Universal Robots, etc.), a controller, a gripper, sensors, and a coprocessor (e.g., to run computationally expensive operations based on perception and task planning). However, it should be understood that the bin picking system may include additional components and / or may omit one or more of these exemplary components within the scope of the present disclosure.

[0082] In some embodiments, and see also Figure 3 , the picking system (e.g., the picking system 64) can be configured to run all modules on the coprocessor and use the real-time data exchange interface of the UR to interface with the UR controller through, for example, an Ethernet connection. The software application can be built by custom plug-ins for one or more graphical user interfaces such as the "Actin Viewer" from Energid Technologies. In some embodiments, the sensor can be any suitable sensor (e.g., a 3D sensor). In some embodiments, the picking system (e.g., the picking system 64) can be configured to run some modules on at least one coprocessor and some modules on the UR controller. In some embodiments, all modules can run on the UR controller.

[0083] In some embodiments, the coprocessor may include a core processor and a graphics card. The operating system and compiler may be of any suitable type. The coprocessor may include multiple external interfaces (e.g., Ethernet to a UR controller, USB 3.0 to a camera, HDMI to a projector, etc.). These specific devices and systems and other devices and systems described throughout this document are provided only by way of example.

[0084] In some embodiments, a universal robot UR5 may be used in the bin picking system 64. The controller may be unmodified. For example, the suction cup end of arm tool (EOAT) may be connected to the controller via, for example, a 24VDC digital output channel. However, it should be understood that any EOAT may be used on any robotic arm within the scope of the present disclosure.

[0085] In some embodiments, any scanner can be used. This can be a structured light sensor, and third-party integration is possible. Together with the SDK, the scanner can be used with an application that can be used to create a workpiece mesh template.

[0086] In some embodiments, a bin picking application (e.g., bin picking application 20) may be configured to run on a coprocessor of a bin picking system (e.g., bin picking system 64) in place of the GUI-based Actin viewer described above. For example, the user interface may be moved to the controller and teach pendant via bin picking capabilities. As used herein, "capabilities" may generally refer to robot capabilities, accessories, or peripherals. A "UR" cap may refer to a cap from "Universal Robotics" or an assignee of the present disclosure. In one example, C ++ A capability daemon can be run on the controller to enable communication with the coprocessor via RTI Connext DDS. Figure 4 An exemplary deployment is shown.

[0087] In some embodiments, an industrial PC (IPC) can be used for the coprocessor. The coprocessor can host the relevant files for picking the box together with the box picking application, including STEP files for EOAT, boxes, and workpieces. Users can load these files onto the coprocessor via USB or over the network.

[0088] In some embodiments, the bin picking application can run on the coprocessor and perform all computationally expensive tasks, including artifact detection and motion planning. The application can be built using the Actin SDK and can be linked to the key library required for bin picking. In one example, RTI Connext DDS 5.3.1 can be used to communicate with the URCap running on the UR controller. However, it should be understood that various configurations are possible within the scope of the present disclosure. In some embodiments and as will be discussed in more detail below, the target object or artifact can be detected from the point cloud data. In one example, the API can be used to interface with the sensor. In another example, an open cascade can be used to convert a STEP file into a mesh file required to generate an Actin model and point cloud of the bin picking system components. In some embodiments, the bin picking URCap may include a Java component that forms a user interface on the UR teach pendant and a daemon for communicating with the coprocessor. For example, the daemon can be built on the Actin library and linked to, for example, RTI Connext DDS 5.3.1.

[0089] In some embodiments, the bin picking system may include multiple stages. These stages may include, but are not limited to: installation; calibration and alignment; application configuration; and bin picking operation.

[0090] In some embodiments, the bin picking system can be configured. For example, the robot, sensor, and gripper can all be physically installed and calibrated during this stage of operation. Sensor calibration can be performed to identify intrinsic and extrinsic parameters of the camera and projector. Alignment of the sensor to the robot can be performed using a 3D printed alignment object consisting of an array of spheres. For example, the target workpiece can be easily detected and it can define the robot coordinate system relative to which the workpiece pose is estimated. The installation, calibration, and alignment parameters can be saved to a file on the coprocessor.

[0091] In some embodiments, the bin picking program configuration phase is the phase where the user configures the bin picking system to perform bin picking operations with a given workpiece and place or secure. The user may first load or create a new program configuration. Creating a new program may include, but is not limited to, configuring tools, workpiece templates, and bins, and then training for grabbing and placing.

[0092] During the picking operation phase, the user can trigger the picking system to perform the picking or stop, and monitor the progress. The picking system can run automatically and scan the box before each picking attempt. In some embodiments, there are two expected user roles for the picking system, which may include a user role and a developer role. The user can interact with the picking system through a graphical user interface (for example, programming experience may not be required). Developers can extend the picking software to include new sensor support, new grippers, new pose estimation (matcher) algorithms, new boundary generators, and new grab script selectors. Users can perform various tasks, and developers can perform other tasks.

[0093] In some embodiments, the bin picking software can be implemented in custom plug-ins to the Actin viewer. These custom plug-ins can include, but are not limited to: perceptionPlugin, taskExecutionPlugin, and urHardwarePlugin.

[0094] In some embodiments, the perceptionPlugin can interface with the taskExecution plugin through the perception system class. This class is a member of the perception module and consists of three main class interfaces: sensor, matcher, and boundary generator.

[0095] In some embodiments, the sensor interface may include the following methods and may be implemented by the sensor class to interface with the scanner.

[0096]

[0097] In some embodiments, the matcher interface includes the following methods and is implemented by the matcher class to utilize the SDK pose estimation utility.

[0098]

[0099] In some embodiments, the boundary generator interface includes the following method, and is implemented by the height field generator.

[0100]

[0101] In some embodiments, the Task Plan Evaluator class quickly evaluates the expected grabs via various metrics. This class is located in the Task Planning module and includes a core interface called EcBaseTaskPlantric.

[0102] In some embodiments, the TaskPlantric interface includes the following methods and can be implemented by the HeightTaskPlanMetric, which scores the grasping script based on its height in the box (the highest point in the box gets the highest score) and the AngleTaskPlanMetric, which scores the grasping script based on how vertical the grasp is (vertical grasping angles achieve the maximum score, and grasping angles requiring movement from the bottom of the table achieve the minimum score).

[0103]

[0104] In some embodiments, the Picking URCap can use the URCap SDK to create template programs that closely follow the patterns and conventions of the native UR ​​task wizards such as "Pallet" and "Find Cargo". The configuration elements can be divided into two main groups: those configuration elements that are common to the picking system setup are placed in the installation node, and those configuration elements specific to the specific picking application are placed in the program node created by the picking template. The runtime status can be displayed through the native program node highlighting mechanism provided by the UR program execution and through display elements located on the main picking sequence node.

[0105] In some embodiments, the overall design of the UI may follow the above-mentioned bin picking use cases. The bin picking URCap design may be presented relative to each use case. For each UI element, a screenshot may be provided along with a list of use cases in which the element participates. Use cases are discussed in further detail below.

[0106] See now Figure 5 , provides an implementation of a bin picking system. The bin picking system installation can be started by connecting the coprocessor to the UR controller with an Ethernet cable. The user then turns on the coprocessor, which automatically launches the bin picking application. First, the user can transfer the bin picking URCap to the UR controller and install it through the setup robot page.

[0107] See now Figure 6 , provides a graphical user interface that complies with the bin picking process. URCap creates a bin picking node on the Installation tab. The user can select the node and view the status page. The status page shows LED style indicators for the status of required components including the URCap daemon, coprocessor, and sensors. If a problem is detected, an error message may be written to the UR log and visible on the Log tab.

[0108] See now Figure 7, providing a graphical user interface that complies with the picking process. Next, the user can select the Environment tab to configure the workspace obstacles. In this tab, the user can load, create, edit and / or save a set of shapes that define all obstacles in the workspace that can be avoided during the picking operation. Three shape types are supported: spheres, capsules, and tablets. However, many other shape types are also within the scope of the present disclosure. The user can load and save collision shapes from files on the picking system.

[0109] See now Figure 8 , provides an additional graphical user interface that complies with the bin picking process. The user can select the Sensor tab and choose the sensor type and configure the parameters. These parameters can be used to tune the sensor and this page can be revisited when in the testing and tuning phase.

[0110] See now Fig. 9 , provides a graphical user interface for generating program templates. Users can configure a bin picking UR program (.urp) by following the steps and use cases below. Users first generate a template bin picking program tree and click on the root node.

[0111] See now Fig.10 , provides a graphical user interface for generating procedure templates. The user can edit basic procedure options by selecting the Basic tab. This includes options to complete or not complete rescans, check for collisions in boxes, etc. Fig.11 As shown, the user can select the Advanced tab and edit additional parameters. This can include a collision detection radius for unpicked workpieces.

[0112] See now Fig.12 , provides a graphical user interface that allows configuration of EOAT. The user can configure EOAT by first clicking on the "Tools" node in the program tree.

[0113] See now Fig.13 , provides a graphical user interface that allows configuration of tool collision shapes. Tool collision shapes are configurable in an editor that is similar to the editor used for environment collision shapes. Tools and shapes are constantly rendered, and the user can rotate and zoom to view the shape while editing it.

[0114] See now Fig.14 , provides a graphical user interface that allows bin configuration. Users can configure a bin by clicking on the "bin" node in the program tree.

[0115] See now Fig.15, provides a graphical user interface that allows bin registration. The bin can be registered relative to the base of the robot. The user can first define a UR feature plane to avoid contact with the EOAT TCP on the three corners of the bin. This plane can then be selected in the bin node "Registration Plane" drop-down menu.

[0116] See now Fig.16 , provides a graphical user interface that allows configuration of the box collision shape. Next, use a dialog similar to the environment node, tool node, and artifact node to configure the box's collision shape.

[0117] See now Fig.17 , provides a graphical user interface that allows configuration of workpieces and loading of workpiece models. Users can configure the workpiece to be picked by clicking on the "Part Template" node in the program tree. Users can load workpiece CAD models from files on the picking system. CAD models can be converted to mesh files for rendering and point clouds for pose detection. Users can view the workpiece template in the rendering window to verify that the workpiece template has been loaded and converted correctly.

[0118] See now Fig.18 , provides a graphical user interface that allows configuration of workpiece collision shapes. The user can configure the collision shapes of the workpiece. These shapes are used to detect and avoid collisions between the workpiece and the environment after the workpiece is picked.

[0119] See now Fig.19 , provides a graphical user interface that allows verification of part inspections. The user can verify part configurations by adding parts to a bin and then triggering scans and inspections to find matches. Inspection results can be rendered and displayed in a list.

[0120] See now Fig. 20 , provides a graphical user interface that allows for the rescan position configuration. The user can then set the robot's rescan position. This is the position that can be used for training grip points and for rescanning when picking (if that option is enabled).

[0121] See now Figure 21 to Figure 22 , provides a graphical user interface that allows configuration of grab hierarchies and / or grab selection criteria. The user can configure a grab hierarchy, including grab criteria, grab points and offsets, and subsequent drop points and offsets. The grab selection criteria define how the program chooses which grabs to use when possible. The user can select grab criteria from a list and edit parameters for each grab metric.

[0122] See now Fig.23, provides a graphical user interface that allows grabs to be added and / or arranged. The user can add and arrange grabs in a hierarchical structure. The grab list defines the priority order used when evaluating grabs. Grabs can be added and removed by clicking on the Add Grab and Remove Grab buttons. Grabs can be selected in the list by clicking on them. Selected grabs can be moved up or down in the list with the buttons provided.

[0123] See now Fig.24 , provides a graphical user interface that allows training of grab and place. The user can train grab and place by clicking on the grab node in the program tree on the left and following the grab page tabs from left to right. Each grab page allows the user to 1) define the grab position relative to the workpiece, 2) define the grab offset used when approaching the workpiece, 3) define the placement position relative to the robot base, and 4) define the placement offset used when approaching the placement position. The user can give a unique name to each grab by clicking on the "Name" field. The user can set the grab pick position by following the steps shown in the dialog on the "Pick Position" tab. The pick position may refer to the point on the workpiece surface where the EOAT will be attached. The user can click on the first button to move the robot to the teach position (rescan position). Next, the user can place the workpiece in the gripper and click on the second button to trigger the scan. The workpiece pose relative to the EOAT can be recorded and saved as a grab position. Then, the user can switch to the pick offset tab and set the offset value.

[0124] See now Fig.25 , provides a graphical user interface that allows training of pick positions and offsets. The user can train the workpiece pick positions and offsets by following the Pick Positions and Pick Offsets tabs.

[0125] See now Fig.26 , provides a graphical user interface that allows training of placement positions and offsets. The user can train the workpiece placement positions and offsets by following the "Placement Position" and "Placement Offset" tabs.

[0126] See now Fig. 27 , provides a graphical user interface that allows configuration of grip and release sequences. Users can add program structure nodes to the grip and release sequence folders to define the EOAT actions to be taken to actuate the EOAT. The default nodes in each sequence may include setup and wait nodes. These folders may be locations where users can add EOAT-specific nodes (including those provided by other URCaps).

[0127] See now Fig.28, provides a graphical user interface that allows system operation. Users can now test, tune, and run programs. To view the picking system status information, users can click on the "Picking Sequence" node in the program tree. This node page displays a rendered view of the picking system and a point cloud overlay of the scanned and detected parts. Users can run the program using the standard UR start pause and stop buttons. Program operation can be reset by clicking the stop button and then the start button. Users can monitor the picking system status by viewing the "Picking Sequence" node page. The selected grab can be rendered in the "Current View" window and its ID will be displayed on the left side of the window.

[0128] In some embodiments, the graphical user interface may allow the user to set up the robot. Once the Set Up Robot option is selected, such as Fig.29 The graphical user interface shown allows the user to install a bin picking URCap from a USB drive or other suitable device. The user can select "URCap" and "+" to load the URCap file. The robot can be restarted after installation.

[0129] See now Fig.30 , provides a graphical user interface that allows the user to configure the environment. In this example, the user can select "Environment" and then create and save collision shapes. For example, sphere - 1 point, capsule - 2 points, lozenge - 3 points, etc. In some embodiments, points can be defined in a variety of ways. Some of these may include, but are not limited to, setting from feature points, setting from robot positions, manually setting, etc.

[0130] See now Fig.31 , provides a graphical user interface that allows the user to configure the sensor. In some embodiments, the user can select a sensor from a drop-down menu and configure its settings.

[0131] See now Figure 32 to Figure 35 , provides a graphical user interface that allows the user to register the sensor. In some embodiments, the sensor can be registered to determine its pose offset relative to the base of the robot. The user can select the "Start Wizard" option to begin. Fig.33 A graphical user interface and the option to secure registration markers to the holder are shown. The registration markers can be 3D printed plastic spheres or hemispheres that can be mounted directly to the holder. Fig.34 A moving robot is depicted to place registration markers at different locations within the scan area. The registration markers can be facing directly toward the sensor. The user can select the "Add Sample" option to record each step. After a few samples, the registration error can be less than, for example, 2 mm. In some embodiments, more than 10 samples can be used. Fig.35 , the registration markers can be removed from the holder and the "Done" option can be selected to complete the registration.

[0132] See now Fig.36 , provides a graphical user interface that allows the user to create a bin picking program. The user can select the "Program" option and choose "Blank Program" to create a new task. Fig.37 , the option to generate a program template is provided. Here, the user can select the "Structure" and "URCap" options before selecting "Pick Bin". This inserts the Pick Bin program template into the program tree. Fig.38 shows examples of options available to the user, and Fig.39 shows one method for setting a fetch metric. A fetch metric defines how the program chooses which fetches to use when possible. Fig.40 An exemplary graphical user interface that allows for setting up RRT nodes is shown. The RRT nodes can be configured to provide path planning guidance to the robot to pick up parts in difficult locations in a bin (e.g., near walls, corners, etc.). The RRT nodes can be set to be a certain distance from the pick location of difficult workpieces. In some embodiments, the robot may only need to move along a straight line to pick up a workpiece without significantly changing its pose or encountering singularities.

[0133] See now Fig.41 , provides a graphical user interface that allows the user to set the home position. The user can select the "Home Position" option in the program tree and then select "Set Home Position". The user can then follow the instructions on the teach pendant to move the robot to the desired home position.

[0134] See now Fig.42 , provides a graphical user interface that allows the user to configure the tool. The user can select the "Tool" option in the program tree and set the tool center point by manually typing in the coordinates and orientation. The user can also be given the option to load an object file.

[0135] See now Fig.43 , provides a graphical user interface that allows the user to register the box. The user can select the "base" option as the registration plane and the "teach" option as the box type. The pointer can be mounted to the end effector.

[0136] See now Fig.44 , provides a graphical user interface that allows the user to register the box. The user can use a pointer to contact four points on the interior of each box wall to perform registration. In some embodiments, the teaching points can be extended. A side definition diagram can be provided to register each side. Once the registration is completed, the LED indicator can be switched.

[0137] See now Fig.45, provides a graphical user interface that allows the user to configure the box collision shape. The user can select the "default shape" option to define the collision shape of the box based on the registration. In some embodiments, the user can change the size of the collision shape.

[0138] See now Fig.46 , provides a graphical user interface that allows the user to verify the part template. The user can select the "Scan" option to scan the workpiece in the box. In some embodiments, the picking system can attempt to match the point cloud with the part template.

[0139] See now Fig.47 , provides a graphical user interface that allows the user to configure the rescan position. The user can select the "Rescan Position" option in the program tree and select "Set Rescan Position". Once the robot moves to the desired rescan position, the user can select "OK".

[0140] See now Fig.48 , provides a graphical user interface that allows a user to edit a grab list. In some embodiments, the grab list may define a priority order to use when evaluating grabs. Grabs may be added and removed by selecting "Add Grab" or "Remove Grab". Selected grabs may be moved up or down in the list with buttons, as shown.

[0141] See now Fig.49 , provides a graphical user interface that allows the user to view the crawl wizard. The user can select a new crawl node in the program tree or select Next to access the crawl wizard. The user can change the crawl name under the Options tab.

[0142] See now Fig.50 , provides a graphical user interface that allows the user to train picking. The user can select the "Teach Picking Approach" option and move the robot to the approach position. The approach position should not be in the part template collision zone. The user can select the "OK" option to record the position and continue to set up other positions.

[0143] See now Fig.51 , provides a graphical user interface that allows the user to configure the EOAT signal. In some embodiments, the standard UR setting node can be used to trigger a digital or analog output to actuate the EOAT. The user can delete or add nodes under each sequence.

[0144] See now Fig.52 , provides a graphical user interface that allows the user to operate the bin picking system. The user can display the point cloud and detected parts. The user can run the program using the UR start and pause buttons.

[0145] See now Fig.53, provides a graphical user interface that allows a user to train a pallet loading sequence. In a pallet loading sequence, the picking program iterates through a list of placement locations, placing each subsequent part at a different location as specified by the pallet loading pattern.

[0146] In some embodiments, the bin picking system described herein can be implemented with a series of sensors or a single sensor model with different lenses, but a single sensor model that will cover the entire operating range can also be used. The product can operate in a volume of, for example, 10×10×10 cm to, for example, 1.2×0.9×0.8 meters (H×W×D). Resolution and accuracy specifications can be met at the worst case location within the volume.

[0147] In some embodiments, the resolution and accuracy may vary with the box size. The implementation may use multiple sensor models or configurations to cover the entire volume. If a box outside the sensor's field of view does affect the performance of the picking system, the software can detect and report the error. The sensor may be mounted above the box, on an arm, or in any suitable location.

[0148] In some embodiments, there may be enough space above the bin between the sensor and the top of the pick volume for the robot to operate without affecting the cycle time. There may be enough space above the bin for the operator to dump more parts into the bin. The distance between the sensor and the bin can vary by ±10% or ±10 cm, whichever is greater. Similarly, the sensor can tolerate ±10° variation in sensor mounting around the x-axis, y-axis, or x-axis as long as the entire bin is still visible.

[0149] In some embodiments, assuming that the sensor does not move after alignment, the sensor may not require a precise position to meet specifications. After the unit is configured and calibrated, the sensor can be considered fixed. The picking system can tolerate temporary obstructions between the sensor and the box. Temporary obstructions may include operators, refill boxes, mixing rods, etc. "Tolerant" may indicate that the picking system can retry picking for a reasonable amount of time and will only generate an error after multiple retries or elapsed time. For both configurations, obstructions that cause force limitations can be detected and forced to retry.

[0150] In some embodiments, the box picking system can be used for boxes of arbitrary shapes, such as cardboard boxes, cylindrical barrels, kidney-shaped bowls, etc. For boxes of roughly parallelepiped shape, programming may not require a CAD model of the box. If a CAD model is required, if the box has slight differences from the CAD model, such as a warped cardboard box, a plastic box with cracks, a wooden crate with missing slats, the box picking system can still function with the desired performance. Operation may not require the main sensor axis to be perpendicular to the top plane or bottom plane of the box. This allows the box to be tilted or the sensor to be placed imprecisely.

[0151] In some embodiments, the setup may require scanning an empty box. The setup may be agnostic to the box size and shape. Preferably, the box can even change between picks, such as from a plastic tote to a cardboard box, without affecting the system operation. The picking system can be used for cardboard boxes with open flaps. The picking system can work when there is no box, such as if the parts are in a pile. The picking system can also be used as a 2D picker, for example where the parts are evenly arranged on a flat surface. The picking system can be used for workpieces as small as 1×1×0.1cm and as large as 30×30×30cm. The resolution and accuracy can vary with the size of the workpiece. The picking system can accept a CAD model of the workpiece and / or can also work with a point cloud of the workpiece.

[0152] In some embodiments, the bin picking system can be used for workpieces that are very thin or very narrow in one or two dimensions (i.e., as thin as a sheet of metal or having an aspect ratio of a wire), but still meet the requirement that the workpiece be rigid. The bin picking system can work even if there are foreign objects or misshapen workpieces in the bin. These workpieces can be avoided and not picked up. The bin picking system can implement multiple types of pickable workpieces in the same bin. If this is the case, the bin picking system is able to programmatically specify the type of desired workpiece before starting to pick. The bin picking system can also work with vacuum pickers and mechanical grippers. The mechanical gripper may include an inner gripper and an outer gripper. The gripper may be combined with the identification of a part that has sufficient clearance for the gripper without colliding with adjacent parts.

[0153] In some embodiments, the bin picking system is capable of accepting a CAD model of the end effector. The bin picking system may also work with a point cloud of the end effector. The bin picking system may have selectable options to avoid collisions between the end effector and the bin or non-clamped workpieces. When collision avoidance with adjacent workpieces is selected, the gripper, robot, and any clamped workpieces should not contact other workpieces during clamping. This means that path planning can search for a certain degree of clearance around the target workpiece. The bin picking system may allow multiple pick points or grasps for a given workpiece to be defined. If multiple pick points or grasps for different workpieces are definable, instructions for which fixture to use are available to the control program. If multiple pick points or grasps for different workpieces are definable, there may be a hierarchy of clamping preferences.

[0154] In some embodiments, the bin picking system may generate a signal or return a warning when no pickable parts are visible. The bin picking system may distinguish between "no parts visible" and "parts visible but not pickable". The bin picking system may also signal that the bin is "almost empty". The picking operation may allow the robot to block the view of the bin during picking.

[0155] In some embodiments, the bin picking system may include a signaling or error return mechanism to the calling program. The bin picking system may have a "reasonable" range of error resolution, for example, may include a mode where "part not found" is not an error but a state: the sensor periodically rescans the area and waits for the workpiece to arrive. The sensor may also be mounted in a fixed position above the bin or on a robotic arm. The sensor may be tolerant of small vibrations, such as may be present on a factory floor.

[0156] In some embodiments, the sensor may operate with target reliability in an environment where both overhead lighting and task lighting may exist and where robots, people passing by, and other machines may cast different shadows. "Ambient light" may be fluorescent, LED fluorescent, incandescent, indirect natural light, etc., that is, it may contain a narrow spectral band or may be a wide spectrum. The picking system may include the ability to programmatically change the projection pattern to allow for future enhancements. The picking system may be insensitive to workpiece surface texture. The picking system may exclude the use of parts with significant specular reflections. The picking system may exclude the use of boxes with significant specular reflections. The picking system may be insensitive to the contrast with the background (because the background is more of the same workpiece type, by definition, there will be low contrast). The picking system may exclude the operation of transparent parts. The picking system may allow parts to have a certain degree of translucency. In some embodiments, the picking system may exclude the operation of transparent boxes or translucent boxes. The picking system can be used for boxes that are not precisely placed and boxes that move between cycles.

[0157] The bin picking system allows a moderately skilled UR programmer to generate a bin picking program in eight hours (excluding the program portions outside of bin picking, such as final part placement, signaling to operators, other operations, etc.). The bin picking system enables offline bin picking program development to minimize the impact on production throughput. Previously trained part types can be called up and new bin picking programs created in one hour. The bin picking system can use wizards or other interactive tools to generate programs.

[0158] In some embodiments, the bin picking system may be executed on the UR controller, or if a second image processing computer is present, it may be executed on that computer. In some embodiments, the bin picking system (e.g., bin picking system 64) may allow bin picking programs to be generated based on simulations on one of the two computers mentioned above or on a separate computer. The bin picking system may be a URCap-compatible application. If multiple sensor models or variants are used, the configuration and programming software may operate using all sensor types. If multiple sensor models or variants are used, the configuration and programming software may automatically detect which sensor type is being used.

[0159] In some embodiments, the bin picking system may include a vision mechanism to verify the position of the clamped workpiece relative to the clamp and compensate for any offset in the position of the workpiece. If arbitrary box shapes are supported, programming may require a CAD model of the box. The bin picking system may work using a general description of the end effector (e.g., length, width, breadth). Checking for collisions between the end effector and an unclamped workpiece may be user-selectable. The bin picking system may allow a general area of ​​a pick point to be defined.

[0160] The placement training process may include the following steps: 1) Offline: Teach the robot to pick up a workpiece and present it to the sensor for scanning. Record both the end effector pose and the workpiece pose. 2) Offline: Teach the robot to place the workpiece at its destination, recording the end effector pose. 3) Online: Pick up a workpiece and present it to the sensor for scanning using the same robot pose as in step 1, recording the end effector pose and the workpiece pose. 4) Online: Place the workpiece at its destination using the information collected in the previous steps.

[0161] In some embodiments, placement accuracy can be dominated by three main sources: 1) robot kinematic model calibration, 2) sensor calibration and alignment, and 3) workpiece pose estimation. These three tasks determine the coordinate system transformation that defines the robot end effector pose, sensor pose, and workpiece pose in a common coordinate system. The final workpiece placement can be calculated based on these transformations.

[0162] In some embodiments, checking for collisions between the end effector and an unclamped workpiece may be user selectable. In some embodiments, path planning may search for a certain degree of clearance around the target workpiece. Resolution and accuracy specifications may be met at the worst case position within the box.

[0163] In some embodiments, there may be enough room above the box for the operator to pour more parts into the box. Typically, this means that there may be room for a similar sized refill box to be rotated above the box until the box size is 40 cm deep (i.e., there is an upper limit to the size of the refill box). In some embodiments, operation may not require the main sensor axis to be perpendicular to the top plane or bottom plane of the box. This allows the box to be tilted or the sensor to be placed imprecisely. In some embodiments, operation may not require the box to be horizontal. If not combined in the sensor, the processor can be combined with the UR processor in the UR controller housing. Any separate software that generates a point cloud from the sensor can support all sensors in the product line.

[0164] In some embodiments, obstructions that cause force limitations can be detected and forced to retry. The picking system can generate a signal or return a warning when no pickable parts are visible. The picking system can use a wizard or other interactive tool to generate a program. In some embodiments, the picking application can be a URCap-compatible application. The picking system may include an option for returning a six-dimensional offset to the calling program instead of performing a placement operation. The picking system is able to programmatically specify the type of desired workpiece before starting to pick. The picking system may include a signaling or error return mechanism to the calling program. The settings may be agnostic to the box size and shape. In some embodiments, the picking system is able to accept a CAD model of the workpiece. In some embodiments, the picking system may allow the generation of a picking program based on a simulation on one of the two computers mentioned above or on a separate computer. The picking system can allow temporary obstructions between the sensor and the box. Temporary obstructions may include operators, refill boxes, mixing rods, etc.

[0165] In some embodiments, the bin picking system can work with a vacuum picker and a mechanical gripper. In some embodiments, the bin picking system can be used for workpieces as small as 1×1×0.1 cm and as large as 30×30×30 cm. However, it should be understood that any size of workpiece or object can be used within the scope of the present disclosure.

[0166] See now Fig.54 , a flowchart illustrating an example of a bin picking operation consistent with an embodiment of the present disclosure is provided. For example, in some embodiments, the robotic bin picking process 10 may identify a list 200 of candidate workpieces or objects to be picked. As described above, workpieces may typically include objects that can be manipulated (e.g., grasped, picked, moved, etc.) by a robot. In some embodiments, the list may be sorted based on one or more indicators. The indicators may include the likelihood of successful picking, the likelihood of successful placement, and / or the suitability for placement in a particular location. As described above and in some embodiments, a bin picking system (e.g., a bin picking system 64) may include a scanning system (e.g., one or more sensors and / or scanners) configured to identify parts in a bin.

[0167] In some embodiments, the robotic bin picking process 10 may determine a path 202 to one or more candidate objects based at least in part on the robot environment and at least one robot constraint. For example, the robotic bin picking process 10 may define a path to a candidate object or workpiece taking into account one or more aspects, including but not limited to the workpiece shape, the environment, the bin, the end of the arm tool, and / or the robot link / joint limitations / constraints. In some embodiments, the path may be a feasible path, an optimal path, or both. For example, a feasible path may generally include possible paths to a workpiece, while an optimal path may generally include a path optimized for one or more attributes (e.g., shortest time, least adjustments in the robot arm, etc.). In some embodiments, the path may be determined dynamically in real time as the candidate workpiece is picked.

[0168] In some embodiments, the sensor may be a 3D sensor. In some embodiments, the sensor may be a 2D sensor. Rescanning may be performed in areas of the sensing volume where the sensor resolution is greatest. The sensor (e.g., a scanner) may also provide a data set describing the perceived environment including static and dynamic objects. In some embodiments, the robotic bin picking process 10 may use the data set to learn the environment to determine paths and / or avoid collisions.

[0169] In some embodiments, the robotic bin picking process 10 may verify the feasibility of grabbing a first candidate object among one or more candidate objects 204. For example, the robotic bin picking process 10 may attempt to verify the feasibility of grabbing a candidate object or workpiece on the list by simulating a pick and place operation faster than real time 204. In some embodiments, the simulation may include using a robot kinematic model. In some embodiments, the simulation may include a model of the environment surrounding the robot. The environment may include static objects and dynamic objects (e.g., moving objects). In some embodiments, these objects may include a machine represented by a kinematic model that updates its state based at least in part on sensor feedback. In some embodiments, one or more objects may be modeled as dynamic obstacles based on point cloud data from a sensor. The point cloud may be transformed into a voxel grid, a height field, or a grid representing the perceived outer surface of the object. Although examples for using simulation to verify the feasibility of grabbing the first candidate object have been discussed above, it should be understood that the feasibility of grabbing the object may be verified in other ways within the scope of the present disclosure.

[0170] In some embodiments, if the feasibility is verified, the robotic bin picking process 10 may control the robot to physically select the first candidate object 206. For example, if the verification passes, the robotic bin picking process 10 may control the robot to pick up the candidate workpiece.

[0171] In some embodiments, if feasibility is not verified, the robotic bin picking process 10 may select at least one of a different gripping point for the first candidate object, a second path, or a second candidate object 208. For example, if verifying the feasibility of grasping the first candidate object 204 fails, the robotic bin picking process 10 may select at least one of: a different gripping point for the same candidate workpiece, a different path, and / or a different candidate workpiece on the list (e.g., a lower ranked object on the list). In some embodiments, selecting a different gripping point, a different path, and / or a different candidate object may include simulating the feasibility of different gripping points, different paths, and / or different candidate objects, as described above.

[0172] In some embodiments and as described above, determining a path 202 to one or more candidate objects may include using information about one or more surfaces of at least one object adjacent to the candidate object, and avoiding collisions with at least one object adjacent to the candidate object. Thus, when determining the path of the candidate object, the robotic picking process 10 may use information about the surfaces of objects surrounding the candidate workpiece to avoid collisions with objects surrounding the candidate workpiece. For example, in some embodiments, information about one or more surfaces of at least one object adjacent to the candidate object is collected as part of identifying the candidate object. In some embodiments, identifying the candidate object 200 may include distinguishing the candidate object from one or more adjacent objects, which may include collecting information about the adjacent objects. In some embodiments, the robotic picking process 10 may generate a simplified model of the workpiece based on the outer surface of the workpiece.

[0173] In some embodiments, controlling the robot 206 may include performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed position with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target according to the accuracy requirement. For example, the robot may pick up a candidate workpiece and move it to a placement location that may be a machine. The machine may have a fixed position with a higher accuracy requirement. Therefore and in order to improve the placement accuracy, the robot picking process 10 may scan the picked workpiece (e.g., rescan), manipulate the workpiece and position it on the machine. The rescanning operation may use the same sensor / scanner as the sensor / scanner used to position the workpiece, or use an additional sensor / scanner. In some embodiments, a second scan of the candidate object may be performed in the area of ​​maximum resolution of the scanner. Although the placement target or placement location has been described as a machine in the above example, it should be understood that the placement target is not limited to a machine and may be any target for placing a candidate object within the scope of the present disclosure.

[0174] In some embodiments, controlling the robot 206 may include presenting the first candidate object to the scanner so as to maximize the use of one or more features on the first candidate object to accurately locate the first candidate object. For example, the robotic bin picking process 10 may present the workpiece to the sensor / scanner so as to maximize the use of features on the workpiece to accurately locate the workpiece. In some embodiments, the robotic bin picking process 10 may position and pick the workpiece in a manner that maximizes the probability that the workpiece can be successfully physically selected or picked up rather than maximizing the accuracy of the pick.

[0175] In some embodiments, the robotic bin picking process 10 may display at least one of the robot or one or more candidate objects at a graphical user interface (GUI), wherein the graphical user interface allows a user to visualize or control at least one of the robot, path determination, simulation, work unit definition, performance parameter specification, or sensor configuration. For example, the robotic bin picking process 10 may display a GUI that can be used to operate the bin picking system. As described above and in some embodiments, displaying the GUI may include, but is not limited to, providing the user with path determination, simulation, work unit definition, performance parameter specification, model import and export, sensor configuration, etc. In some embodiments, the GUI may allow for simultaneous creation of programs and debugging of created programs. The GUI may also allow bin picking program commands to be mixed with other robot control commands.

[0176] In some embodiments, the robotic bin picking process 10 can display a shrink wrap visualization at the graphical user interface on all unselected parts and unselected surfaces except for one or more candidate objects. This display can help the programmer determine whether the trained grip is suitable for picking up the workpiece given the surrounding objects.

[0177] In some embodiments and as described above, the GUI may be located on any suitable device, including but not limited to on a teach pendant, on a handheld device, on a personal computer, on the robot itself, etc. In some embodiments, the GUI may draw the information it displays from multiple sources, such as from the robot controller and from a processor separate from the robot controller. In some embodiments, the GUI may direct user input to one or more destinations, such as to the robot controller and / or a processor separate from the robot controller. In some embodiments, the user of the GUI may or may not be aware of the existence of multiple data sources or destinations.

[0178] In some embodiments, a main processor and at least one coprocessor may be used to perform at least one of identifying one or more candidate objects, determining a path to one or more candidate objects, verifying the feasibility of grabbing a first candidate object, and / or controlling a robot. In some embodiments and as described above, the robot picking process 10 may be configured to stream a GUI from a coprocessor to a robot teach pendant. In this way, the robot picking process 10 may run a GUI application on the coprocessor, which may include a 3D rendered view of the robot and the work unit, and then stream an image of the GUI to the teach pendant for display. In some embodiments, user touch events may be streamed from the teach pendant to the coprocessor to interact remotely with the GUI application.

[0179] In some embodiments, determining a path 202 to one or more candidate objects may be based at least in part on at least one of the following: global path planning and local path planning. For example, the robotic picking process 10 may utilize global path planning, local path planning, or a combination of the two. As used herein, global path planning can generally help find a collision-free path when local planning is not possible. Local planning can be similar to a gradient descent algorithm in situations where it may be blocked in a local solution. This may happen if there are many obstacles in the environment. The local planning method of the robotic picking process 10 may include real-time control with collision avoidance optimization. For example, it can operate quickly, but may not always explore solutions throughout the robot's entire workspace. In contrast, global path planning via the robotic picking process 10 can be configured to search for solutions throughout the workspace.

[0180] In some embodiments, verifying the feasibility of grabbing the first candidate object 204 may include analyzing the conditional logic associated with the user program. As described above and in some embodiments, in a bin picking application, the user may need to define various system features and develop a user program for picking and placing parts. In this way, the robot bin picking process 10 can try to ensure successful end-to-end robot motion in a constrained environment, taking into account the changing start (pick) and end (place) robot positions and multiple alternative paths defined by the conditional logic in the user program. When executing the user program, the robot bin picking process 10 can repeatedly perform three main tasks: perception (i.e., identifying parts in the box by using sensors); verification (i.e., identifying which parts can be picked up and then placed by the robot according to the rules specified in the user program given the environmental constraints); and movement (i.e., performing robot motion on the verified parts according to the rules specified in the user program). During the verification task, the robot bin picking process 10 can determine the robot motion that needs to be performed in order to pick up and place parts before actually performing the motion. Therefore, the robot bin picking process 10 can avoid the situation when the robot is stuck in the middle of the motion due to some environmental or robot flexibility constraints.

[0181] In some embodiments, verifying the feasibility of grabbing the first candidate object 204 may include at least one of verifying all path alternatives, verifying a specific path alternative, verifying any path alternative, verifying one or more abnormal paths, excluding one or more verified sections, or performing parallel verification of multiple sections of the path. For example, to verify all path alternatives, the user program may have conditional logic where the robot is expected to take different paths based on some conditions that are unknown at the time of verification. For example, if a part needs to be inspected by a camera after it is picked, the inspection results determine whether the part is placed in, for example, placement location 1 or, for example, placement location 2. To ensure successful movement, the verification logic of the robotic bin picking process 10 can confirm both alternatives before the part can be moved.

[0182] In order to verify specific path alternatives, the user program may have conditional logic where the robot can be expected to take different paths based on some conditions known at the time of verification. For example, the user program may define the robot movement based on how to pick up the part (i.e., how the robot holds the part). During pallet loading, the part can be placed in one of several known positions, and the program iterates over these positions in a predictable pattern. In these cases, the conditions that determine the possible alternative paths are known at the time of verification. In order to ensure successful movement, it may only be necessary to analyze the movements specified in some branches of the conditional flow in the user program. In fact, it may be harmful to analyze all code paths in these cases because it will take longer because those path segments that cannot be taken based on the conditional logic in the user program should not prevent the robot from moving, regardless of whether they can be verified.

[0183] To validate any alternate path, the user program may define several path alternatives, any of which are acceptable. For example, during pallet loading, a part or object may be placed in any of several known locations. In this case, validation would need to consider multiple path options specified by the program until it finds one that works.

[0184] To validate one or more abnormal paths, the robot may take one or more paths due to abnormal conditions. For example, if a part or object fails to attach to the robot gripper during picking, the robot bin picking process 10 may direct the robot back to the starting position. If the robot encounters excessive forces resisting its motion while picking the part, the robot bin picking process 10 may direct the robot back to the starting position. In these cases, validation may be required to confirm the feasibility of these paths even if they are not explicitly specified in the user program flow.

[0185] In order to exclude one or more verified sections, the user can choose to exclude some sections of the program flow from being verified. For example, one or more code paths may contain motion types that cannot be verified. In some embodiments, the user can choose to perform verification to optimize performance. In these cases, verification can be conditionally not performed.

[0186] In some embodiments, the robotic bin picking process 10 can perform parallel verification of multiple sections of the path. For example, in order to optimize performance, multiple sub-sections of the path can be verified in parallel.

[0187] As described above, the present invention provides a method and a corresponding device consisting of various modules that provide functions for performing the steps of the method. The modules can be implemented as hardware, or can be implemented as software or firmware for execution by a computer processor. Specifically, in terms of firmware or software, the present invention can be provided as a computer program product including a computer-readable storage structure that embodies computer program code (i.e., software or firmware) thereon for execution by a computer processor.

[0188] It should be understood that the above-described arrangements are merely exemplary applications of the principles of the present invention. Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the scope of the present disclosure.

Claims

1. A method for robot box picking, the method comprising: identifying one or more candidate objects for selection by the robot; determining a path to the one or more candidate objects based at least in part on a robot environment and at least one robot constraint, wherein determining the path includes determining that the one or more candidate objects are placeable, wherein the at least one robot constraint includes at least one of a robot link or a robot joint limit; Verifying the feasibility of grasping a first candidate object among the one or more candidate objects, wherein verifying the feasibility includes finding a collision-free path, wherein the collision-free path includes a path after the object is placed; as well as If the feasibility is verified, controlling the robot to physically select the first candidate object; If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object is selected. The method of claim 1 , wherein the validating comprises using a robot kinematics model. The method of claim 1 , wherein the path is at least one of a feasible path or an optimal path. The method of claim 1 , wherein the path is determined at least in part in real time while controlling the robot. 5 . The method of claim 1 , wherein determining the path comprises using information about one or more surfaces of at least one object adjacent to the candidate object and avoiding collision with the at least one object adjacent to the candidate object.

6. The method according to claim 1, further comprising: At least one of the robot or the one or more candidate objects is displayed at a graphical user interface, wherein the graphical user interface allows a user to visualize or control at least one of the robot, path determination, simulation, work cell definition, performance parameter specification, or sensor configuration.

7. The method of claim 6, wherein the graphical user interface allows for simultaneous creation of a program and a debugging process associated with the program.

8. The method of claim 6, wherein the graphical user interface is associated with one or more of a teach pendant, a handheld device, a personal computer, or the robot.

9. The method according to claim 6, further comprising: At the graphical user interface, a visualization is displayed on all unselected components and unselected surfaces other than the one or more candidate objects.

10. The method according to claim 1, further comprising: A scanner is used to provide an image of the environment including one or more static objects and dynamic objects, wherein the robot is configured to receive the image and use the image to learn the environment to determine the path and collision avoidance.

11. The method of claim 1 , wherein controlling the robot comprises performing a second scan of the first candidate object, moving the first candidate object to a placement target having a fixed position with an accuracy requirement, manipulating the first candidate object and delivering the first candidate object to the placement target according to the accuracy requirement.

12. The method of claim 11, wherein the second scan is performed in a maximum resolution area of ​​the scanner.

13. The method of claim 1, wherein controlling the robot comprises presenting the first candidate object to a scanner to maximize use of one or more features on the first candidate object to accurately locate the first candidate object.

14. The method of claim 1, wherein controlling the robot comprises positioning and picking the first candidate object in a manner that maximizes a probability of successful physical selection.

15. The method of claim 1, wherein at least one of identifying, determining, verifying, or controlling is performed using at least one of a main processor and at least one co-processor.

16. The method of claim 1, wherein determining a path to the one or more candidate objects is based at least in part on at least one of: a global path plan, a local path plan, a robot link, or a robot joint limit.

17. The method of claim 1, wherein verifying the feasibility of grabbing the first candidate object comprises analyzing conditional logic associated with a user program.

18. The method of claim 17, wherein verifying the feasibility of grabbing the first candidate object comprises at least one of verifying all path alternatives, verifying a specific path alternative, verifying any path alternative, verifying one or more exception paths, excluding one or more verified segments, or performing parallel verification of multiple segments of the path.

19. The method according to claim 1, further comprising: The first candidate object and the second candidate object to be picked up by the robot are configured based at least in part on one or more models corresponding to the first candidate object and the second candidate object, wherein the first candidate object and the second candidate object are of different types.

20. A method for robotic bin picking, the method comprising: identifying one or more candidate objects for selection by the robot; determining a collision-free path to the one or more candidate objects based at least in part on the robot environment and at least one robot constraint, wherein determining the collision-free path comprises determining how to avoid collisions with the robot, wherein the collision-free path comprises a path after the objects are placed; Verifying the feasibility of grabbing a first candidate object among the one or more candidate objects; as well as If the feasibility is verified, controlling the robot to physically select the first candidate object; If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object is selected.

21. A method for robotic bin picking, the method comprising: identifying one or more candidate objects for selection by the robot; determining a collision-free path to the one or more candidate objects based at least in part on the robot environment and at least one robot constraint, wherein determining the path comprises analyzing the one or more candidate objects, the robot, and a gripper associated with the robot, wherein the collision-free path comprises a path after the objects are placed; Verifying the feasibility of grabbing a first candidate object among the one or more candidate objects; as well as If the feasibility is verified, controlling the robot to physically select the first candidate object; If the feasibility is not verified, at least one of a different grasping point of the first candidate object, a second path, or a second candidate object is selected.

Images