Robotic bin picking system and method using advanced scanning technology

Abstract

A method and system for programming pick and place of a workpiece is provided. Embodiments may include associating the workpiece with an end effector attached to a robot and scanning the workpiece while the workpiece is associated with the end effector. Embodiments may also include determining a pose of the workpiece relative to the robot based at least in part on the scan.

Description

[0001] Cross-references to related patent applications

[0002] This application claims the benefit of U.S. Non-Provisional Application Serial No. 16 / 520,007, filed on July 23, 2019. The entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to robots, and more particularly to robotic bin picking systems and methods. Background Art

[0004] Some forms of manual labor, such as unloading workpieces from bins into machines, bulk sorting, and fulfilling orders, are labor intensive. These jobs are often dangerous if the workpieces or operations are heavy, sharp, or hazardous. To address these issues, bin picking robots have been handling these tedious jobs. However, robotic bin picking tasks are particularly difficult because the amount of accuracy and precision required often exceeds the capabilities of the system. Bin picking typically involves finding the part in a bin, picking it, moving it to its destination, and then placing it down. Traditional robot programming typically involves programmatically specifying a series of robot poses. "Programming by example," "lead-in programming," or "teach programming" typically means guiding the robot through a series of poses. Summary of the invention

[0005] In one specific implementation, a method for programming pick and place of a workpiece is provided. The method may include associating the workpiece with an end effector attached to a robot, and scanning the workpiece while the workpiece is associated with the end effector. The method may further include determining a pose of the workpiece relative to the robot based at least in part on the scan.

[0006] One or more of the following features may be included. The method may include picking up a workpiece by a robot, wherein picking up the workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece. The method may further include placing the workpiece, wherein placing the workpiece includes determining the robot pose and inferring the pose of the workpiece from the robot pose. The method may also include placing the workpiece, wherein placing the workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece. The method may further include displaying the pose of the workpiece relative to the robot at a graphical user interface to allow visual verification. In some embodiments, scanning and determination can be performed when the robot is in a fixed position. The method may further include allowing a user to adjust the pose of the workpiece relative to the robot by entering one or more coordinates. The method may also include displaying the adjusted pose of the workpiece at a graphical user interface for verification.

[0007] In another specific implementation, a system for programming pick and place of a workpiece is provided. The system may include an end effector attached to a robot and configured to associate a workpiece therewith. The system may also include a scanner configured to scan the workpiece when the workpiece is associated with the end effector. The system may further include a computing device configured to determine a pose of the workpiece relative to the robot based at least in part on the scan.

[0008] One or more of the following features may be included. The robot may be configured to pick up a workpiece, wherein picking up the workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece. The robot may be configured to place a workpiece, wherein placing the workpiece includes determining the robot pose and inferring the pose of the workpiece from the robot pose. The robot may be configured to place a workpiece, wherein placing the workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece. The computing device may be configured to display the pose of the workpiece relative to the robot at a graphical user interface to allow visual verification. Scanning and determination may be performed when the robot is in a fixed position. The graphical user interface may be configured to allow a user to adjust the pose of the workpiece relative to the robot by entering one or more coordinates. The graphical user interface may be configured to display the adjusted pose of the workpiece at the graphical user interface for verification.

[0009] The details of one or more implementations are set forth in the accompanying drawings and the detailed description that follows. Other features and advantages will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are included to provide a further understanding of the embodiments of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the embodiments of the present disclosure.

[0011] Figure 1 is a diagrammatic view of a robotic bin picking process coupled to a distributed computing network;

[0012] Figure 2 yes Figure 1 A flowchart of a specific implementation of a robot bin picking process;

[0013] Figure 3 A bin picking system according to an embodiment of the present disclosure is configured to run all modules on a coprocessor and interact with a UR controller through an Ethernet connection using a real-time data exchange interface from the UR.

[0014] Figure 4 is an interface showing a deployment diagram of a bin picking system according to an embodiment of the present disclosure.

[0015] Figure 5 is a graphical user interface that allows a user to set a starting location according to an embodiment of the present disclosure.

[0016] Figure 6 is a graphical user interface that allows a user to configure a tool according to an embodiment of the present disclosure.

[0017] Figure 7 is a graphical user interface that allows a user to register bins according to an embodiment of the present disclosure.

[0018] Figure 8 is a graphical user interface that allows a user to register bins according to an embodiment of the present disclosure.

[0019] Fig. 9 is a graphical user interface that allows a user to configure a container collision shape according to an embodiment of the present disclosure.

[0020] Fig.10 is a graphical user interface that allows a user to validate a component template according to an embodiment of the present disclosure.

[0021] Fig.11 is a graphical user interface that allows a user to configure a rescan location according to an embodiment of the present disclosure.

[0022] Fig.12 is a graphical user interface that allows a user to add a grip according to an embodiment of the present disclosure.

[0023] Fig.13 is a graphical user interface that allows a user to train grasping and placement according to an embodiment of the present disclosure.

[0024] Fig.14 is a graphical user interface that allows a user to train picking according to an embodiment of the present disclosure.

[0025] Fig.15 is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0026] Fig.16 is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0027] Fig.17 is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0028] Fig.18 is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0029] Fig.19is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0030] Fig. 20 is a graphical user interface that allows a user to set picking rules according to an embodiment of the present disclosure.

[0031] Fig.21 is a graphical user interface that allows a user to set placement rules according to an embodiment of the present disclosure.

[0032] Fig. 22 is a graphical user interface that allows a user to set placement rules according to an embodiment of the present disclosure.

[0033] Fig.23 is a graphical user interface that allows a user to set placement rules according to an embodiment of the present disclosure.

[0034] Fig.24 is a graphical user interface that allows a user to set placement rules according to an embodiment of the present disclosure.

[0035] Fig.25 is a graphical user interface that allows a user to set placement rules according to an embodiment of the present disclosure.

[0036] Fig.26 is a graphical user interface according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Embodiments of the present disclosure relate to robotic bin picking systems and methods. Thus, the bin picking methods included herein may allow a robot to work with a scanning system to identify parts in a bin, pick parts from the bin, and place the picked parts at a specified location. Compared to the traditional robotic programming examples discussed above, the embodiments of the robotic bin picking process described herein are configured to focus on the relationship of the workpiece to the robot end effector, rather than solely on the robot. The workpiece can be located anywhere in the bin, and the final workpiece placement can be achieved using many different robot poses.

[0038] Embodiments of the subject application may include concepts from U.S. Patent No. 6,757,587, U.S. Patent No. 7,680,300, U.S. Patent No. 8,301,421, U.S. Patent No. 8,408,918, U.S. Patent No. 8,428,781, U.S. Patent No. 9,357,708, U.S. Patent Publication No. 2015 / 0199458, U.S. Patent Publication No. 2016 / 0321381, U.S. Patent Publication No. 2018 / 0060459, and U.S. Patent Application Nos. 16 / 453,197 and 16 / 378,343, the entire contents of each of which are incorporated herein by reference in their entirety.

[0039] Reference now Figure 1 , shows a robotic bin picking process 10 that may reside on and be executed by a computing device 12 that may be connected to a network (e.g., network 14) (e.g., the Internet or a local area network). Examples of computing device 12 (and / or one or more of the client electronic devices mentioned below) may include, but are not limited to, a personal computer, a laptop computer, a mobile computing device, a server computer, a series of server computers, a mainframe computer, or a computing cloud. Computing device 12 may execute an operating system, such as, but not limited to, or a custom operating system. (Microsoft and Windows are registered trademarks of Microsoft Corporation in the U.S. or other countries, or in the U.S. and other countries; Mac and OS X are registered trademarks of Apple Inc. in the U.S. or other countries, or in the U.S. and other countries; Red Hat is a registered trademark of Red Hat Corporation in the U.S. or other countries, or in the U.S. and other countries; and Linux is a registered trademark of Linus Torvalds in the U.S. or other countries, or in the U.S. and other countries.

[0040] As will be discussed in more detail below, the robotic bin picking process, such as Figure 1 to Figure 2 A robotic bin picking process 10 may be used to program the picking and placement of a workpiece. The process may include associating 200 the workpiece with an end effector attached to a robot, and scanning 202 the workpiece while the workpiece is associated with the end effector. The process may further include determining 204 a pose of the workpiece relative to the robot based at least in part on the scan.

[0041] The instruction sets and subroutines of the robotic bin picking process 10, which may be stored on a storage device 16 coupled to the computing device 12, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within the computing device 12. The storage device 16 may include, but is not limited to: a hard drive; a flash drive, a tape drive; an optical drive; a RAID array; a random access memory (RAM); and a read-only memory (ROM).

[0042] Network 14 may be connected to one or more auxiliary networks (eg, network 18), examples of which may include, but are not limited to, a local area network; a wide area network; or an intranet, for example.

[0043] The robotic bin picking process 10 may be a standalone application that interacts with applets / applications accessed via client-side applications 22, 24, 26, 28, 68. In some embodiments, the robotic bin picking process 10 may be distributed in whole or in part in a cloud computing topology. In this manner, the computing device 12 and the storage device 16 may refer to multiple devices, which may also be distributed throughout the network 14 and / or the network 18.

[0044] Computing device 12 may execute a robot control application (e.g., a robot control application), examples of which may include, but are not limited to, the robot control application from Energid Technologies of Cambridge, Massachusetts. Software Development Kit and any other bin picking application or software. The robotic bin picking process 10 and / or the robotic control application may be accessed via the client-side applications 22, 24, 26, 28, 68. The robotic bin picking process 10 may be a standalone application, or may be an applet / application / script / extension that may interact with and / or execute within the robotic control application, a component of the robotic control application, and / or one or more of the client-side applications 22, 24, 26, 28, 68. The robotic control application may be a standalone application, or may be an applet / application / script / extension that may interact with and / or execute within the robotic control application, a component of the robotic bin picking process 10, and / or one or more of the client-side applications 22, 24, 26, 28, 68. One or more of the client-side applications 22, 24, 26, 28, 68 may be stand-alone applications, or may be applet / application / script / extension that may interact with and / or execute within the robotic bin picking process 10 and / or the robotic control application, and / or may be a component of the robotic bin picking process and / or the robotic control application. Examples of client-side applications 22, 24, 26, 28, 68 may include, but are not limited to, applications that receive queries to search for content from one or more databases, servers, cloud storage servers, etc., text and / or graphical user interfaces, custom web browsers, plug-ins, application programming interfaces (APIs), or custom applications. The instruction sets and subroutines of the client-side applications 22, 24, 26, 28, 68 that may be stored on the storage devices 30, 32, 34, 36 coupled to the client electronic devices 38, 40, 42, 44 may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into the client electronic devices 38, 40, 42, 44.

[0045] Storage devices 30, 32, 34, 36 may include, but are not limited to: hard drives; flash drives, tape drives; optical drives; RAID arrays; random access memory (RAM); and read-only memory (ROM). Examples of client electronic devices 38, 40, 42, 44 (and / or computing device 12) may include, but are not limited to, personal computers (e.g., client electronic device 38), laptop computers (e.g., client electronic device 40), smart / data-enabled cellular phones (e.g., client electronic device 42), notebook computers (e.g., client electronic device 44), tablet computers (not shown), servers (not shown), televisions (not shown), smart televisions (not shown), media (e.g., videos, photos, etc.), capture devices (not shown), and dedicated network devices (not shown). Client electronic devices 38, 40, 42, 44 may each execute an operating system, examples of which may include, but are not limited to, Mobile, ChromeOS, Blackberry OS, Fire OS, or custom operating systems.

[0046] One or more of the client-side applications 22, 24, 26, 28, 68 may be configured to implement some or all of the functionality of the robotic bin picking process 10 (and vice versa). Thus, the robotic bin picking process 10 may be a pure server-side application, a pure client-side application, or a server-side / client-side hybrid application that is cooperatively executed by one or more of the client-side applications 22, 24, 26, 28, 68 and / or the robotic bin picking process 10.

[0047] One or more of the client-side applications 22, 24, 26, 28, 68 may be configured to implement some or all of the functionality of the robot control application (and vice versa). Thus, the robot control application may be a pure server-side application, a pure client-side application, or a server-side / client-side hybrid application that is cooperatively executed by one or more of the client-side applications 22, 24, 26, 28, 68 and / or the robot control application. Since some or all of the same functionality may be implemented using one or more of the client-side applications 22, 24, 26, 28, 68, the robotic bin picking process 10, and the robotic control application, either alone or in any combination, any description of implementing such functionality through one or more of the client-side applications 22, 24, 26, 28, 68, the robotic bin picking process 10, the robotic control application, or a combination thereof, and any described interactions between the client-side applications 22, 24, 26, 28, 68, the robotic bin picking process 10, the robotic control application, or a combination thereof for implementing such functionality should be used merely as examples and not as limitations on the scope of the present disclosure.

[0048] The users 46, 48, 50, 52 may access the computing device 12 and the robotic bin picking process 10 directly or indirectly through the network 14 or through an auxiliary network (e.g., using one or more of the client electronic devices 38, 40, 42, 44). In addition, the computing device 12 may be connected to the network 14 through an auxiliary network, as shown by the dashed connecting line 54. The robotic bin picking process 10 may include one or more user interfaces, such as a browser and a text or graphical user interface, through which the users 46, 48, 50, 52 can access the robotic bin picking process 10.

[0049] Various client electronic devices may be coupled to network 14 (or network 18) directly or indirectly. For example, client electronic device 38 is shown as being directly coupled to network 14 via a hardwired network connection. Additionally, client electronic device 44 is shown as being directly coupled to network 18 via a hardwired network connection. Client electronic device 40 is shown as being wirelessly coupled to network 14 via a wireless communication channel 56 established between client electronic device 40 and a wireless access point (i.e., WAP) 58, which is shown as being directly coupled to network 14. WAP 58 may be, for example, an IEEE 800.11a, 800.11b, 800.11g, 800.11d, 800.11e, 800.11f, 800.11g ... and / or Bluetooth tm (including Bluetooth tmThe client electronic device 42 is shown as being wirelessly coupled to the network 14 via a wireless communication channel 60 established between the client electronic device 42 and a cellular network / bridge 62, which is shown as being directly coupled to the network 14. In some implementations, the robotic system 64 may be wirelessly coupled to the network 14 via a wireless communication channel 66 established between the client electronic device 42 and the cellular network / bridge 62, which is shown as being directly coupled to the network 14. Storage devices 70 may be coupled to the robotic system 64 and may include, but are not limited to: hard drives; flash drives, tape drives; optical drives; RAID arrays; random access memory (RAM); and read-only memory (ROM). A user 72 may access the computing device 12 and the robotic bin picking process 10 directly or indirectly through the network 14 or through an auxiliary network (e.g., using the robotic system 64).

[0050] Some or all of the IEEE 800.11x specifications may use the Ethernet protocol and carrier sense multiple access / collision avoidance (ie, CSMA / CA) for path sharing. For example, various 800.11x specifications may use phase shift keying (ie, PSK) modulation or complementary code keying (ie, CCK) modulation. Bluetooth tm (including Bluetooth tm Low Power Consumption (NFC) is a telecommunications industry specification that allows electronic devices such as mobile phones, computers, smartphones, and other electronic devices to interconnect using short-range wireless connections. Other forms of interconnection (e.g., Near Field Communication (NFC)) may also be used.

[0051] In some embodiments, a bin picking system (e.g., a bin picking system) may include a robotic arm (e.g., a Universal Robots UR5 from Universal Robots, etc.), a controller, a gripper, sensors, and a coprocessor (e.g., to run computationally expensive operations from perception and task planning). However, it should be understood that a bin picking system may include additional components and / or may omit one or more of these exemplary components within the scope of the present disclosure.

[0052] In some embodiments, and also with reference to Figure 3, a bin picking system (e.g., a bin picking system) can be configured to run all modules on a coprocessor and interact with a UR controller using a real-time data exchange interface from UR, for example, via an Ethernet connection. The software application can be built with custom plug-ins for one or more graphical user interfaces, such as "Actin Viewer" from Energid Technologies. In some embodiments, the sensor can be any suitable sensor (e.g., a 3D sensor). In some embodiments, a bin picking system (e.g., a bin picking system) can be configured to run some modules on at least one coprocessor and some modules on a UR controller. In some embodiments, all modules can run on a UR controller.

[0053] As used herein, the term "Actin viewer" may refer to a graphical user interface, "Actin" may refer to robot control software, and "UR" may refer to "Universal Robots". Any use of these specific companies and products is provided by way of example only. Thus, any suitable graphical user interface, robot control software, and devices / modules may be used without departing from the scope of this disclosure.

[0054] In some embodiments, the coprocessor may include a core processor and a graphics card. The operating system and compiler may be of any suitable type. The coprocessor may include multiple external interfaces (e.g., an Ethernet interface connected to a UR controller, a USB 3.0 interface connected to a camera, an HDMI interface connected to a projector, etc.). These specific devices and systems and other devices and systems described in this document are provided only as examples.

[0055] In some embodiments, a Universal Robots UR5 can be used in a bin picking system. The controller can be unmodified. For example, the suction cup end of an end-of-arm tool (EOAT) can be connected to the controller via, for example, a 24VDC digital output channel. However, it should be understood that any EOAT can be used on any robot within the scope of the present disclosure.

[0056] In some embodiments, any scanner can be used. This can be a structured light sensor, and third-party integration can be implemented. Along with the SDK, the scanner may come with an application that can be used to create a workpiece mesh template.

[0057] In some embodiments, a bin picking application (e.g., bin picking application 20) may be configured to run on a coprocessor of a bin picking system (e.g., bin picking system) in place of the GUI-based Actin Viewer discussed above. For example, the user interface may be moved to the controller and teach pendant via a bin picking cap. As used herein, a "cap" may generally refer to a robot function, accessory, or peripheral. A "UR" cap may refer to a cap that is commercially available from Universal Robots or the assignee of the present disclosure. In one example, C ++ Cap Daemon can run on the controller to enable communication with the coprocessor via RTI Connext DDS. Figure 4 An exemplary deployment is shown.

[0058] In some embodiments, the bin picking system may include multiple stages. These stages may include, but are not limited to: installation; calibration and alignment; application configuration; and bin picking operation.

[0059] In some embodiments, a bin picking system can be configured. For example, the robot, sensor, and gripper can all be physically installed and calibrated at this stage of operation. Sensor calibration can be performed to identify the intrinsic and extrinsic parameters of the camera and projector. Sensor and robot alignment can be performed using a 3D printed alignment object consisting of an array of spheres. For example, the target workpiece can be easily detected and it can define a robot coordinate system related to the workpiece pose estimation. The installation, calibration, and alignment parameters can be saved to a file on the coprocessor.

[0060] In some embodiments, the bin picking program configuration phase is the phase where the user configures the bin picking system to perform bin picking operations and placement or fixturing for a given workpiece. The user may first load or create a new program configuration. Creating a new program may include, but is not limited to, configuring tools, workpiece templates, and bins, and then training gripping and placement.

[0061] During the bin picking operation phase, the user can trigger the bin picking system to perform bin picking or stop, and monitor progress. The bin picking system can run automatically and scan the bin before each attempt to pick up. In some embodiments, there are two expected user roles for the bin picking system, which may include a user role and a developer role. The user can interact with the bin picking system through a graphical user interface (for example, programming experience may not be required). Developers can extend the bin picking software to include new sensor support, new grippers, new pose estimation (Matcher) algorithms, new boundary generators, and new grip script selectors. Various tasks can be performed by users, and other tasks can be performed by developers.

[0062] Reference now Figure 5, provides a graphical user interface that allows the user to set the starting position. The user can select the "Start Position" option in the program tree and then select "Set Start Position". The user can then follow the instructions on the teach pendant to move the robot to the desired starting position.

[0063] Reference now Figure 6 , provides a graphical user interface that allows the user to configure the tool. The user can select the "Tool" option in the program tree and set the tool center point by manually entering the coordinates and orientation. The user can also be given the option to load an object file.

[0064] Reference now Figure 7 , provides a graphical user interface that allows the user to register the bin. The user can select the "Base" option as the registration plane and then the "Teach" option as the bin type. A pointer can be mounted to the end effector.

[0065] Reference now Figure 8 , a graphical user interface is provided that allows the user to register the bins. The user can use a pointer to touch four points inside each bin wall to register. In some embodiments, the teach points can be scattered. A side definition graphic can be provided to register each side. Once the registration is complete, the LED indicator can be toggled.

[0066] Reference now Fig. 9 , provides a graphical user interface that allows the user to configure the collision shape of the container. The user can select the "default shape" option to define the collision shape of the container based on the registration. In some embodiments, the user can change the size of the collision shape.

[0067] Reference now Fig.10 , provides a graphical user interface that allows the user to verify the part template. The user can select the "Scan" option to scan the workpiece in the bin. In some embodiments, the bin picking system can attempt to match the point cloud with the part template.

[0068] Reference now Fig.11 , provides a graphical user interface that allows the user to configure the rescan position. The user can select the "Rescan Position" option in the program tree and then select "Set Rescan Position". Once the robot has moved to the desired rescan position, the user can select "OK".

[0069] Reference now Fig.12 , provides a graphical user interface that allows the user to edit the list of grips. In some embodiments, the list of grips may define a priority order to be used when evaluating grips. Grips may be added and removed by selecting "Add grip" or "Remove grip". Selected grips may be moved up and down the list using buttons, as shown.

[0070] Reference now Fig.13 , provides a graphical user interface that allows the user to view the grip wizard. The user can select a new grip node in the program tree or select "Next" to access the grip wizard. The user can change the grip name under the "Options" tab.

[0071] Reference now Fig.14 , provides a graphical user interface that allows the user to train picking. The user can select the "Teach Pick Approach" option and move the robot to a pick approach position. This approach position should not be in the part template collision zone. The user can select the "OK" option to record the position and continue to set other positions.

[0072] Reference now Figures 15 to 26 , embodiments of a robotic bin picking process 10 illustrate various graphical user interfaces that may be used to program a robot by scanning a workpiece that may be associated with and / or attached to its end effector, and determining a pose of the workpiece relative to the robot based at least in part on the scan. Fig.15 An exemplary graphical user interface that can be used to begin the process of adding a pick rule or a place rule is shown.

[0073] In some embodiments, and as Fig.16 As shown, the robotic bin picking process 10 may include associating a workpiece with an end effector attached to the robot. In operation, a user may select a workpiece and teach picking by attaching the workpiece to an end effector or tool associated with the robot. It should be noted that the workpiece may be associated with the end effector using any suitable method, including but not limited to magnetized attachment, suction attachment, etc. Once the workpiece is associated with the end effector, the user may select a "teach" option. The offset from the tool center point ("TCP") may also be displayed and / or edited by the user at the graphical user interface.

[0074] In some embodiments, and as Fig.17 As shown, the robotic bin picking process 10 may include scanning the workpiece while the workpiece is associated with the end effector. Thus, the robotic arm and / or the end effector may move so that the scanner can scan the workpiece that has been associated with the end effector. The user may then verify at the graphical user interface that the workpiece is associated or attached to the end effector as expected. If the user has verified the alignment, he or she may select a scan option. If the alignment requires further adjustment, it may be canceled and one or more adjustments may be made. If the scan option is selected, the graphical user interface may indicate that scanning is in progress, such as Fig.18 shown.

[0075] In some embodiments, the robotic bin picking process 10 may determine the pose of the workpiece relative to the robot based at least in part on scanning. Fig.19 As shown, the pose of the workpiece relative to the robot or end effector can be displayed to the user at a graphical user interface. This can allow visual verification of the workpiece and how it is associated with the end effector and the X, Y and Z offsets from the TCP, such as Figures 19 to 20 The X, Y, and Z coordinate systems may be shown at the graphical user interface and may be adjusted based on user preferences. The process may allow the user to adjust the pose of the workpiece relative to the robot by entering and / or editing coordinates. The adjusted pose of the workpiece may be displayed at the graphical user interface for verification.

[0076] In some embodiments, the robotic bin picking process 10 may include picking up a workpiece using a robot and / or an end effector. This may include specifying one or more poses of the robot relative to the pose of the workpiece.

[0077] In some embodiments, the robotic bin picking process 10 may include placing a workpiece, wherein placing the workpiece includes determining a robot pose and inferring a pose of the workpiece from the robot pose. Placing the workpiece may also include specifying one or more poses of the robot relative to the pose of the workpiece. In some embodiments, the workpiece may be scanned and the determination performed while the robot is in a fixed position.

[0078] Reference now Figure 21 to Figure 25 , provides an exemplary graphical user interface that can be used according to one or more placement rules. Fig.21 As shown, placement rules can be taught by attaching the workpiece to the end effector and selecting a "teach" option from the graphical user interface. To prepare for scanning, the user may need to move the robot and / or the end effector to present the workpiece and end effector to the scanner. In some cases, the end effector can be controlled using one or more controls accessible through the graphical user interface, however, manual presentation is also within the scope of the present disclosure. Once the workpiece and end effector are visible to the scanner, scanning can be initiated, such as Figure 21 to Figure 23 shown.

[0079] Reference now Fig.24 , provides an exemplary graphical user interface showing the scan results. In this particular example, the scan results show the end effector position and specific X, Y, and Z coordinates. The robotic bin picking process 10 allows the user to verify the relationship of the workpiece to the end effector and also provides the ability to perform a rescan if necessary. Fig.25 An exemplary graphical user interface is shown in which a user or system can define a desired placement location. In some embodiments, upon selecting the "next" option, the end effector and workpiece can be moved to the desired placement location. Fig.26 A graphical user interface showing both tool position coordinates and joint position coordinates is shown.

[0080] Embodiments of the present disclosure may allow pick and place to not be programmed as a sequence, but rather to start with forming a pose of the workpiece relative to the robot and work forward or backward from that initial starting point. The relationship may be established by scanning the workpiece while it is held by the end effector. The end effector may be rigidly attached to the robot so this may also form a pose of the workpiece relative to the robot. The remainder of the pick may be specified by specifying a numerical offset of the robot position relative to the formed workpiece / robot pose. The remainder of the place may be specified by reading the robot pose, inferring the workpiece pose from the robot pose, and specifying a numerical offset of the robot position relative to the formed workpiece pose.

[0081] Therefore, the embodiments of the robotic bin picking process 10 described herein can provide many advantages over existing methods. For example, by starting with a scan each time, pick and place can be independently programmed and debugged. The relationship between the workpiece and the end effector can be optimized before scanning, thereby removing errors caused by component movement, deformation of the suction cup during picking (or deformation of the elastic pad on the mechanical gripper), unexpected offsets, etc. The relationship between the workpiece and the end effector can be visually verified after scanning. This method can be considered as programming "from the middle", working backward to pick and working forward to place. This cuts any error accumulation chain in half. Programming the pick can be done without moving the robot at all, thereby reducing the workload and removing errors caused by human vision establishing the robot position. The embodiments included herein can improve the accuracy of workpiece pose detection by presenting significant features, scanning in the best part of the scanning space, scanning the workpiece independently of other workpieces or environmental debris, etc.

[0082] In some embodiments, the placement training process may include the following steps: 1) Offline: Teach the robot to pick up a workpiece and present it to the sensor for scanning. Record both the end effector pose and the workpiece pose. 2) Offline: Teach the robot to place the workpiece at its destination, recording the end effector pose. 3) Online: Pick up the workpiece and present it to the sensor for scanning using the same robot pose as step 1, recording the end effector pose and the workpiece pose. 4) Online: Place the workpiece at its destination using the information collected in the previous steps.

[0083] In some embodiments, placement accuracy can be driven by three main sources: 1) robot motion model calibration, 2) sensor calibration and alignment, and 3) workpiece pose estimation. These three tasks determine the coordinate system transformations that define the robot end effector pose, sensor pose, and workpiece pose and in a common coordinate system. The final workpiece placement can be calculated based on these transformations.

[0084] In some embodiments and as described above, the GUI can be located on any suitable device, including but not limited to on a teach pendant, on a handheld device, on a personal computer, on the robot itself, etc. In some embodiments, the GUI can draw its display information from multiple sources, such as from the robot controller and from a processor separate from the robot controller. In some embodiments, the GUI can direct user input to one or more destinations, such as to the robot controller and / or a processor separate from the robot controller. In some embodiments, the user of the GUI may or may not be aware of the existence of multiple data sources or destinations.

[0085] As described above, the present invention provides both a method and a corresponding device consisting of various modules that provide functionality for performing the steps of the method. The modules may be implemented as hardware, or may be implemented as software or firmware for execution by a computer processor. Specifically, in the case of firmware or software, the present invention may be provided as a computer program product that includes a computer-readable storage structure that embodies computer program code (i.e., software or firmware) thereon for execution by a computer processor.

[0086] It should be understood that the above-described arrangements are merely illustrative of the application of the principles of the present invention. Numerous variations and alternative arrangements may be devised by those skilled in the art without departing from the scope of the present disclosure.

Claims

1. A method for picking and placing a workpiece, the method comprising: configuring a work cell having a scanner and a robot so that a first workpiece can be presented to the scanner by the robot for scanning by the scanner; Determine the robot’s position and posture; attaching the first workpiece to an end effector, the end effector being attached to the robot; moving the robot so that the first workpiece attached to the end effector is in a position to be scanned by the scanner; scanning the workpiece with the scanner while the workpiece is attached to the end effector; inferring a workpiece posture according to the robot posture; determining a pose of the workpiece relative to the robot based at least in part on the scan; as well as A second workpiece is picked up and placed, via the robot, based on the robot pose, the workpiece pose, and the pose of the workpiece relative to the robot, wherein the second workpiece is similar to the first workpiece.

2. The method according to claim 1, in, Picking up the second workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece.

3. The method according to claim 1, in, Positioning the second workpiece includes placing the second workpiece at a destination.

4. The method according to claim 1, in, Placing the second workpiece includes specifying one or more poses of the robot relative to the pose of the workpiece.

5. The method according to claim 1, further comprising: The pose of the workpiece relative to the robot is displayed at a graphical user interface to allow visual verification.

6. The method of claim 1, wherein scanning and determining are performed while the robot is in a fixed position.

7. The method according to claim 1, further comprising: The user is allowed to adjust the pose of the workpiece relative to the robot by inputting one or more coordinates.

8. The method according to claim 7, further comprising: The adjusted pose of the workpiece is displayed at a graphical user interface for verification.

9. A system for picking and placing a workpiece, the system comprising: a scanner configured to create a scan of a first workpiece when the first workpiece is attached to the end effector; a robot having the end effector attached to the robot, the robot being configured to move so that the first workpiece can be positioned to be scanned by the scanner while attached to the end effector; a work cell configured with the scanner and the robot so that the first workpiece can be presented to the scanner by the robot to be scanned by the scanner; A computing device, the computing device being configured to: Determine the robot’s position and posture; inferring a workpiece posture according to the robot posture; determining a pose of the workpiece relative to the robot based at least in part on the scan; as well as The robot is further configured to pick up and place a second workpiece based on the robot posture, the workpiece posture, and the posture of the workpiece relative to the robot, wherein the second workpiece is similar to the first workpiece.

10. The system of claim 9, wherein the robot is further configured to specify one or more poses of the robot relative to the pose of the workpiece to facilitate picking up the second workpiece.

11. The system of claim 9, wherein the robot is further configured to place the second workpiece at a destination.

12. The system of claim 9, wherein the robot is further configured to specify one or more poses of the robot relative to the pose of the workpiece to facilitate placement of the second workpiece.

13. The system of claim 9, wherein the computing device is further configured to display the pose of the workpiece relative to the robot at a graphical user interface to allow visual verification.

14. The system according to claim 13, wherein: The graphical user interface is configured to allow a user to adjust the pose of the workpiece relative to the robot by inputting one or more coordinates. 15 . The system of claim 14 , wherein the graphical user interface is configured to display the adjusted pose of the workpiece at the graphical user interface for verification.

16. The system of claim 9, wherein scanning and determining are performed while the robot is in a fixed position.

Images