Robotic tool and method of operation
Patent Information
- Application Number
- JP2022190679
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-08-02
- Filing Date
- 2022-11-29
- Publication Date
- 2024-11-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Robots often struggle with grasping objects in non-optimal positions or poses, lacking the agility and adaptability required for complex tasks, leading to inefficiencies and increased resource consumption.
A robotic system that selects and connects different end effector tools, such as standard fixed, fixed-angle, and adjustable head tools, to optimize object grasping and transfer plans, reducing resource consumption and error rates by planning transfers for multiple objects simultaneously.
The system enhances the robotic system's ability to grasp and transfer objects efficiently by minimizing tool change times and reducing errors, improving overall task performance and completion time.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical Field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 109,870, filed November 5, 2020, which is incorporated herein by reference in its entirety.
[0002] Technical Field The present technology is directed generally to robotic systems, and more particularly to robotic tools configured to grasp and hold objects. [Background technology]
[0003] Robots (e.g., machines configured to automatically or autonomously perform physical actions) are now widely used in many fields. For example, robots can be used to perform various tasks (e.g., manipulating or transporting objects), such as in manufacturing, packaging, conveying, and / or shipping. In performing tasks, robots can replicate human actions, thereby replacing or reducing human intervention otherwise required to perform dangerous or repetitive tasks. Robots often lack the sophistication necessary to replicate the human acuity and / or adaptability required to perform more complex tasks. For example, robots sometimes have difficulty grasping object(s) in certain suboptimal positions or poses. Therefore, there is a need for improved robotic systems and techniques for transporting objects using a set of grasping tools. [Brief explanation of the drawings]
[0004] [Figure 1] FIG. 1 illustrates an exemplary environment in which a robotic system transports an object, in accordance with one or more embodiments of the present technology. [Figure 2] FIG. 1 is a block diagram illustrating a robotic system in accordance with one or more embodiments of the present technology. [Figure 3]FIG. 1 is a top view of a robotic system in accordance with one or more embodiments of the present technology. [Figure 4A] FIG. 1 is a diagram of an exemplary transfer unit in accordance with one or more embodiments of the present technology. [Figure 4B] FIG. 1 is a diagram of an exemplary toolset in accordance with one or more embodiments of the present technology. [Figure 5A] FIG. 1 is a diagram of a standard capture scenario, in accordance with one or more embodiments of the present technology. [Figure 5B] FIG. 10 is an illustration of an angled grasping scenario in accordance with one or more embodiments of the present technology. [Figure 6A] FIG. 1 is a diagram of a standard release scenario, in accordance with one or more embodiments of the present technology. [Figure 6B] FIG. 10 is an illustration of a first angled release scenario in accordance with one or more embodiments of the present technology. [Figure 6C] FIG. 10 illustrates a second angled release scenario in accordance with one or more embodiments of the present technology. [Figure 7] FIG. 1 is an exemplary timing diagram in accordance with one or more embodiments of the present technology. [Figure 8] 1 is a flow diagram for operating a robotic system in accordance with some embodiments of the present technology. DETAILED DESCRIPTION OF THE INVENTION
[0005] Described herein are systems and methods for selecting, modifying, and using end effector tool sets. For example, a robotic system may include a transport robot configured to selectively connect to end effector tools. The robotic system can select and connect different end effector tools to grasp and transport objects. The robotic system can access and choose from a tool set. In some embodiments, the tool set may include standard fixed tools, fixed-angle tools, and / or flexible-head tools. A standard fixed tool may have an end effector rigidly attached to a tool arm. A standard fixed tool can be configured to grip objects with a general orientation or top surface that is orthogonal to the orientation of the tool arm. A fixed-angle tool can be configured to have a non-orthogonal angle between the tool arm and the gripper head / interface. A flexible or adjustable-head tool can be configured so that the relative angle / orientation between the tool arm and the gripper head / interface can be adjusted depending on the pose of the object.
[0006] In some embodiments, the robotic system may select an end effector tool based on simulating and / or planning the transfer of the object according to the selected tool, deriving metrics for the transfer (e.g., cost and / or benefit), accumulating a set of plans for a set of objects of interest, and / or selecting a combination / order of plans that optimizes corresponding metrics (e.g., total transfer time and / or estimated loss / error rate). Additionally or alternatively, the robotic system may implement the selected plan in parallel with derivation of plans for subsequent objects.
[0007] By tailoring plans and tool usage to a set of tools (e.g., multiple selectable tools) and / or to a group of objects (e.g., instead of based on a single object when it is detected), reduced resource consumption and / or fewer errors are realized. In some embodiments, the planning and validation process can be completed in a shorter time (e.g., 1 second or less) than the time required to execute / implement the plan (e.g., 1-5 seconds) and / or the time required to change tools (e.g., 5-10 seconds). Thus, by planning transfers for a set of objects (e.g., rather than planning for one object at a time), the robotic system can derive a set of plans that minimizes actions with the highest costs (e.g., time / duration), such as tool change operations.
[0008] In the following, numerous specific details are set forth to provide a thorough understanding of the presently disclosed technology. In other embodiments, the technology presented herein may be practiced without these specific details. In other instances, well-known features, such as particular functions or routines, are not described in detail to avoid unnecessarily obscuring the present disclosure. References herein to "an embodiment," "one embodiment," or the like mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases herein do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments illustrated in the figures are merely illustrative representations and are not necessarily drawn to scale.
[0009] For clarity, some details describing structures or processes that are well known and often associated with robotic systems and subsystems, but that may unnecessarily obscure some important aspects of the disclosed technology, are not included in the following description. Moreover, although the following disclosure illustrates several embodiments of different aspects of the technology, some other embodiments may have different configurations or different components than those described in this section. Thus, the disclosed technology may have other embodiments that have additional elements or that do not have some of the elements described below.
[0010] Many embodiments or aspects of the present disclosure described below may take the form of computer-executable or controller-executable instructions, including routines executed by a programmable computer or controller. Those skilled in the art will appreciate that the disclosed technology can be implemented in computer or controller systems other than those shown and described below. The technology described herein may be embodied in a special-purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Accordingly, the terms "computer" and "controller," as used generally herein, refer to any data processor, including Internet appliances and handheld devices (including palmtop computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers may be displayed on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer-executable or controller-executable tasks may be stored on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable medium, including tangible, non-transitory computer-readable media.
[0011] The terms "coupled" and "connected," along with their derivatives, may be used herein to describe a structural relationship between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless otherwise clearly indicated by context, the term "coupled" may be used to indicate that two or more elements are in direct or indirect contact with each other (with other intervening elements therebetween), that two or more elements cooperate or interact with each other (e.g., in a causal relationship such as signal transmission / reception or function call), or both.
[0012] Suitable environment 1 is a diagram of an exemplary environment in which a robotic system 100 transports an object, in accordance with one or more embodiments of the present technology. The robotic system 100 may include and / or communicate with one or more units (e.g., robots) configured to perform one or more tasks. Aspects of tool selection and tool use may be performed or carried out by various units.
[0013] In the example shown in FIG. 1 , robotic system 100 may include an unloading unit 102, a transfer unit 104 (e.g., a palletizing robot and / or a piece picker robot), a transport unit 106, a loading unit 108, or a combination thereof, in a warehouse or logistics / shipping hub. Each unit in robotic system 100 may be configured to perform one or more tasks. Tasks may be combined in sequence to perform operations to achieve a goal, such as unloading objects from a truck or van and storing them in a warehouse, or unloading objects from a storage location and preparing them for shipment. In another example, a task may include placing objects at a target location (e.g., on a pallet and / or inside a bin / cage / box / case). As described below, the robotic system may derive a plan for placing and / or stacking objects (e.g., placement location / orientation, order for transferring objects, and / or corresponding motion plan). Each unit may be configured to perform a series of actions (e.g., by operating one or more components therein) according to one or more of the derived plans to perform the tasks.
[0014] In some embodiments, a task may include manipulating (e.g., moving and / or reorienting) a target object 112 (e.g., one of a package, box, case, cage, pallet, etc. corresponding to the task being performed), such as moving the target object 112 from a start location 114 to a task location 116. For example, the unloading unit 102 (e.g., a devanning robot) may be configured to transfer the target object 112 from a location in a conveying machine (e.g., a truck) to a location on a conveyor belt. The transfer unit 104 may also be configured to transfer the target object 112 from one location (e.g., a conveyor belt, a pallet, or a bin) to another location (e.g., a pallet, a bin, etc.). In another example, the transfer unit 104 (e.g., a palletizing robot) may be configured to transfer the target object 112 from a supply location (e.g., a pallet, a pickup area, and / or a conveyor) to a destination pallet. Upon completion of the operation, the transport unit 106 may transfer the target object 112 from the area assigned to the transfer unit 104 to the area assigned to the loading unit 108, and the loading unit 108 may transfer the target object 112 from the transfer unit 104 to a storage location (e.g., a location on a shelf) (e.g., by moving a pallet carrying the target object 112). More details regarding the tasks and associated actions are provided below.
[0015] For illustrative purposes, robotic system 100 is described in the context of a shipping center; however, it will be understood that robotic system 100 may be configured to perform tasks in other environments / for other purposes, such as manufacturing, assembly, packaging, healthcare, and / or other types of automation. It will also be understood that robotic system 100 may include other units not shown in FIG. 1 , such as manipulators, service robots, modular robots, etc. For example, in some embodiments, robotic system 100 may include a depalletizing unit for transferring objects from a cage cart or pallet onto a conveyor or other pallet, a container switching unit for transferring objects from one container to another, a packaging unit for wrapping objects, a sorting unit for grouping objects according to one or more characteristics of the objects, a piece-picking unit for manipulating (e.g., sorting, grouping, and / or transporting) objects differently according to one or more characteristics of the objects, or combinations thereof.
[0016] The robotic system 100 can include and / or be coupled to physical or structural members (e.g., robotic manipulator arms) connected by joints for movement (e.g., rotational and / or translational displacement). The structural members and joints can form kinematic chains configured to manipulate end effectors (e.g., grippers) configured to perform one or more tasks (e.g., grasping, spinning, welding, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 can include actuation devices (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members around or at the corresponding joints. In some embodiments, the robotic system 100 can include transport motors configured to transport corresponding units / chassis from location to location.
[0017] The robotic system 100 may include sensors configured to obtain information used to perform tasks such as manipulating structural members and / or transporting a robotic unit. The sensors may include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the state, status, and / or position of one or more structural members / joints thereof) and / or one or more physical characteristics of the surrounding environment. Some examples of sensors include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.
[0018] In some embodiments, for example, the sensors may include one or more imaging devices (e.g., visual and / or infrared cameras, 2D and / or 3D imaging cameras, distance measuring devices such as lidar or radar, etc.) configured to detect the surrounding environment. The imaging devices may generate a representation of the detected environment, such as a digital image and / or a point cloud, that may be processed via machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications). The robotic system 100 may process the digital image and / or point cloud to identify the target object 112, the start position 114, the task position 116, the pose of the target object 112, or a combination thereof.
[0019] To manipulate a target object 112, the robotic system 100 can capture and analyze images of a designated area (e.g., a pick-up location, such as in a truck or on a conveyor belt) to identify the target object 112 and its start location 114. Similarly, the robotic system 100 can capture and analyze images of another designated area (e.g., a drop-off location for placing an object on a conveyor, a location for placing an object in a container, or a location on a pallet for stacking purposes) to identify a task location 116. For example, the imaging device can include one or more cameras configured to generate images of the pick-up area and / or one or more cameras configured to generate images of the task area (e.g., the drop-off area). As described below, based on the captured images, the robotic system 100 can determine the start location 114, the task location 116, the associated pose, the packing / placement plan, the transfer / packing sequence, and / or other processing results.
[0020] In some embodiments, for example, the sensors may include position sensors (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of the robotic system 100. The robotic system 100 may use the position sensors to track the position and / or orientation of the structural members and / or joints during the performance of a task.
[0021] Robot System 2 is a block diagram illustrating a robotic system 100 in accordance with one or more embodiments of the present technology. In some embodiments, for example, the robotic system 100 (e.g., one or more of the units and / or robots described above) can include electronic / electrical devices such as one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input / output devices 208, one or more actuation devices 212, one or more transport motors 214, one or more sensors 216, or combinations thereof. The various devices can be coupled to each other via wired and / or wireless connections. For example, the robotic system 100 can include a bus such as a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) Standard 1394 bus (also known as "Firewire"). Also, for example, the robotic system 100 may include bridges, adapters, processors, or other signal-related devices to provide wired connections between devices. Wireless connections may be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., Wireless Fidelity (WIFI)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near Field Communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and / or other wireless communication protocols.
[0022] The processor 202 may include a data processor (e.g., a central processing unit (CPU), a special-purpose computer, and / or an on-board server) configured to execute instructions (e.g., software instructions) stored in a storage device 204 (e.g., computer memory). In some embodiments, the processor 202 may be included in a separate / standalone controller operably coupled to other electronic / electrical devices shown in FIG. 2 and / or the robotic unit shown in FIG. 1. The processor 202 may execute program instructions for controlling / interfacing with other devices, thereby causing the robotic system 100 to perform actions, tasks, and / or operations.
[0023] The storage device 204 may include a non-transitory computer-readable medium on which program instructions (e.g., software) are stored. Some embodiments of the storage device 204 may include volatile memory (e.g., cache and / or random access memory (RAM)) and / or non-volatile memory (e.g., flash memory and / or magnetic disk drive). Other embodiments of the storage device 204 may include a portable memory drive and / or a cloud storage device.
[0024] In some embodiments, the storage device 204 can be used to further store and provide access to processing results and / or predetermined data / thresholds. For example, the storage device 204 can store master data including descriptions of objects (e.g., boxes, cases, and / or products) that can be manipulated by the robotic system 100. In one or more embodiments, the master data can include registration data for each such object. The registration data can include the dimensions, shape (e.g., templates for possible poses and / or computer-generated models for recognizing the object in various poses), color scheme, image, identification information (e.g., barcodes, quick response (QR) codes, logos, etc., and / or their expected locations), expected weight, other physical / visual characteristics, or a combination thereof, of the objects expected to be manipulated by the robotic system 100. In some embodiments, the master data can include operation-related information about the objects, such as the center of gravity (CoM) location or an estimate thereof for each object, expected sensor measurements (e.g., force, torque, pressure, and / or adhesion) corresponding to one or more actions / manipulations, or a combination thereof.
[0025] The communications device 206 may include circuitry configured to communicate with external or remote devices over a network. For example, the communications device 206 may include a receiver, a transmitter, a modulator / demodulator (modem), a signal detector, a signal encoder / decoder, a connector port, a network card, etc. The communications device 206 may be configured to send, receive, and / or process electrical signals according to one or more communications protocols (e.g., Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system 100 may use the communications device 206 to exchange information between units of the robotic system 100 and / or with systems or devices external to the robotic system 100 (e.g., for purposes of reporting, data collection, analysis, and / or troubleshooting).
[0026] The input / output devices 208 may include user interface devices configured to communicate information with and / or receive information from a human operator. For example, the input / output devices 208 may include a display 210 and / or other output devices (e.g., speakers, haptic circuitry, or haptic feedback devices, etc.) for conveying information to a human operator. The input / output devices 208 may also include control or receiving devices, such as a keyboard, mouse, touchscreen, microphone, user interface (UI) sensors (e.g., a camera for receiving movement commands), wearable input devices, etc. In some embodiments, the robotic system 100 may use the input / output devices 208 to interact with a human operator in performing actions, tasks, operations, or combinations thereof.
[0027] In some embodiments, a controller (e.g., a separate electronic device) can include the processor 202, the storage device 204, the communication device 206, and / or the input / output device 208. The controller can be a standalone component or part of a unit / assembly. For example, each unloading unit, transfer assembly, transport unit, and loading unit of the system 100 can include one or more controllers. In some embodiments, a single controller can control multiple units or standalone components.
[0028] The robotic system 100 may include physical or structural members (e.g., robotic manipulator arms) connected by joints for movement (e.g., rotational and / or translational displacement). The structural members and joints may form a kinematic chain configured to manipulate end effectors (e.g., grippers) configured to perform one or more tasks (e.g., grasping, spinning, welding, etc.) depending on the use / operation of the robotic system 100. The robotic system 100 may include actuation devices 212 (e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and / or reorient) the structural members around or at corresponding joints. In some embodiments, the robotic system 100 may include a transport motor 214 configured to transport a corresponding unit / chassis to various locations. For example, the actuation devices 212 and transport motors may be connected to or part of a robot arm, linear slide, or other robotic component.
[0029] The sensors 216 can be configured to obtain information used to perform tasks such as manipulating structural members and / or transporting the robotic unit. The sensors 216 can include devices configured to detect or measure one or more physical characteristics of the robotic system 100 (e.g., the state, status, and / or position of one or more structural members / joints thereof) and / or one or more physical characteristics of the surrounding environment. Some examples of the sensors 216 can include contact sensors, proximity sensors, accelerometers, gyroscopes, force sensors, strain gauges, torque sensors, position encoders, pressure sensors, vacuum sensors, etc.
[0030] In some embodiments, for example, sensors 216 may include one or more imaging devices 222 (e.g., visual and / or infrared cameras, two-dimensional and / or three-dimensional imaging cameras, distance measuring devices such as lidar or radar, etc.) configured to detect the surrounding environment. Imaging devices 222 may include cameras (including visual and / or infrared cameras), lidar devices, radar devices, and / or other distance measuring or detection devices. Imaging devices 222 may generate representations of the detected environment, such as digital images and / or point clouds, that may be processed via machine / computer vision (e.g., for automated inspection, robotic guidance, or other robotic applications).
[0031] To manipulate the target object 112, the robotic system 100 (e.g., via the various circuits / devices described above) can capture and analyze images of a designated area (e.g., a pickup location, such as in a truck or on a conveyor belt) to identify the target object 112 and its start location 114. Similarly, the robotic system 100 can capture and analyze images of another designated area (e.g., a drop-off location for placing the object on a conveyor, a location for placing the object in a container, or a location on a pallet for stacking purposes) to identify a task location 116. For example, the imaging device 222 can include one or more cameras configured to generate images of the pickup area and / or one or more cameras configured to generate images of the task area (e.g., a drop-off area). As described below, based on the captured images, the robotic system 100 can determine the start location 114, the task location 116, the associated pose, the packing / placement plan, the transfer / packing sequence, and / or other processing results.
[0032] In some embodiments, for example, sensors 216 may include position sensors 224 (e.g., position encoders, potentiometers, etc.) configured to detect the position of structural members (e.g., robotic arms and / or end effectors) and / or corresponding joints of robotic system 100. Robotic system 100 may use position sensors 224 to track the positions and / or orientations of structural members and / or joints during the performance of a task. Robotic system 100 may use the detected positions, tracked positions, tracked orientations, etc. from sensors 216 to derive tracking data representing the current and / or a series of past positions of target object 112 and / or structural members of FIG. 1 .
[0033] 3 is a top view of a robotic system 100 in accordance with one or more embodiments of the present technology. In some embodiments, the robotic system 100 can include a system manager 302, a planner 304, and / or a robotic arm 306. The system manager 302 and / or the planner 304 can implement or include one or more circuits (e.g., processor 202, storage device 204, communication device 206, etc.) shown in FIG.
[0034] The system manager 302 may include mechanisms (e.g., devices and / or software applications) configured to manage the overall operation of one or more task stations and / or corresponding robotic units. For example, the system manager 302 may include a facilities management system for a warehouse, shipping hub, or the like. In managing the overall operation, the system manager 302 may receive an order input 312 (e.g., a customer request for a set of objects accessible by the robotic system 100). The system manager 302 may derive the interaction / control between various tasks, task stations, and / or corresponding robots, the associated sequence or timing, etc., to collect the objects listed in the order input 312. The system manager 302 may further interact with the robotic arm 306 to perform / execute tasks.
[0035] Planner 304 may include mechanisms (e.g., devices, software applications / functions, or a combination thereof) configured to derive fine-grained control for operating one or more robots or components therein. Planner 304 may derive detailed steps, such as robot motion plans and / or protocols or sequences of communication with other subsystems, to operate one or more robotic units and accomplish tasks determined by system manager 302.
[0036] To operate the robotic arm 306, the robotic system 100 may use the planner 304 to derive a transfer plan 314 corresponding to a path for moving one or more items from the start location 114 in FIG. 1 to the task location 116 in FIG. 1. For example, the robotic system 100 may acquire one or more images of the start location 114 and the task location 116 via the imaging device 222 in FIG. 2. The robotic system 100 may process the images to identify or recognize ordered objects and / or their poses inside a bin (e.g., a container having at least two vertical walls) at the start location 114. Similarly, the robotic system 100 may use the images of the task location 116 to derive or identify placement positions for each of the target objects at the start location 114. The planner 304 can derive a transfer plan 314 including paths and / or corresponding commands, settings, timing, etc. for operating the robot arm 306 (e.g., its actuation device 212, FIG. 2 and / or transport motor 214, FIG. 2 ) to transfer the target object between corresponding locations. In some embodiments, the planner 304 may iteratively derive the transfer plan 314 by starting from a placement location and iteratively deriving next incremental positions as it moves toward the destination. The next incremental positions can be tested positions that satisfy one or more predetermined rules, for example, to avoid collisions, minimize distance / time, or the like.
[0037] The robotic system 100 can implement the transfer plan 314 and operate the robotic arm 306 accordingly to transfer one or more target objects identified in the order input 312. For example, the system manager 302 can interact with the robotic arm 306, such as by communicating the path and / or commands / settings of the transfer plan 314 to the robotic arm 306. The robotic arm 306 can execute the received information to perform a task.
[0038] As an illustrative example, the system manager 302 can interact with one or more robots (e.g., transport units such as automated guided vehicles (AGVs), conveyors, and / or subsystems) to access a container (e.g., including a start bin 322) from a storage area in which the ordered objects are stored. In response, the robotic system 100 can operate the transport units to transport the start bin 322 to the start location 114 of the robotic arm 306. Similarly, the robotic arm 306 may place a target container 324 (e.g., a packaging box or a destination bin) at a task location 116. Alternatively, the task location 116 may correspond to a drop-off or placement location on a conveyor or robot (e.g., an AGV). Once the start location 114 and task location 116 are ready, the system manager 302 can interact with the robotic arm 306 according to the transfer plan(s) 314. Thus, the robotic arm 306 can grasp the target object(s) 112 (e.g., one or more of the objects specified in the order input 312) and transfer it / them from the start location 114 to the task location 116. For example, the robotic arm 306 can pick the ordered objects from the start bin(s) 322 at the start location 114 and place them in the target container(s) 324 at the task location 116 and / or in the specified location to fulfill the order.
[0039] In some embodiments, the robotic system 100 may utilize a set of tools (e.g., specialized end effectors) to perform different tasks using the same robot and / or improve performance of a given task. For example, the robotic system 100 may selectively connect the robotic arm 306 to a gripper, a welder, or a cutter to perform the corresponding functions according to the assigned task. The robotic system 100 may also selectively connect the robotic arm 306 to a pincher gripper or a vacuum gripper depending on the physical characteristics of the object of interest and / or the physical characteristics of its surrounding environment (e.g., the relative positions of other objects, the availability of approach paths, etc.).
[0040] In some embodiments, as described in more detail below, the robotic system 100 may selectively connect the robotic arm 306 to grippers having different angled interfaces depending on the pose of the target object. For example, the robotic system 100 may select a first tool when the target object is placed flat on the bottom of a bin and exhibits a relatively horizontal top surface and / or relatively vertical surrounding surfaces. Alternatively, the robotic system 100 may select a second tool when the target object has non-parallel surfaces or is posed at an angle relative to the lateral plane, such as when the target object is resting on an uneven or non-planar contact point. The second tool may include a contact or gripping interface configured to grasp such angled objects.
[0041] When utilizing a set of tools, the system manager 302 may provide a target selection 313 to the planner 304 to identify the selected tools and / or target objects 112 for one or more of the tasks / objects. The planner 304 may derive a transport plan 314 according to the target selection 313 and corresponding feedback 318. For example, the feedback 318 may include a transport plan 314 for transporting the target object with the specified tool when the derivation is successful (e.g., meeting one or more threshold conditions, such as to avoid a collision or to meet a minimum error estimate). The feedback 318 may also include an error message when the derivation is unsuccessful, such as when the specified tool is inappropriate for capturing and / or transporting the target object (e.g., by causing a collision event). In an alternative embodiment, the planner 304 may select a tool and derive a corresponding transport plan 314 without interacting with the system manager 302 regarding the tool selection.
[0042] When multiple tools are available, the robotic system 100 may derive and evaluate multiple tasks or actions as a single set, instead of processing each task or action separately. Alternatively or additionally, the robotic system 100 may adjust the sequence or timing of derivation / evaluation, tool changes, and plan implementation to improve or maximize the efficiency (e.g., overall completion time) of the series of tasks. Because there are differences in tradeoffs and benefits between the derivation process, the tool change process, and the execution of actions, deriving and evaluating individual tasks and actions as a set can improve overall performance. For example, the amount of time required to derive and evaluate the transfer plan 314 (e.g., 1 second or less) can be shorter than the amount of time required to change tools (e.g., 5 seconds or more) and / or shorter than the amount of time required to execute the plan with the robotic arm 306 (e.g., 1-5 seconds). In some embodiments, a tool change can take longer than the average or maximum amount of time required to complete the transfer of an item.
[0043] To derive and evaluate a set of tasks, the robotic system 100 (e.g., the system manager 302 and / or the planner 304) may determine tool-based object groups 316, which identify or group items according to the appropriate tool for that operation. For example, the robotic system 100 may derive tool-based object groups 316 based on the pose of the target objects in the starting bin 322 (e.g., using the angle or orientation of the exposed surface relative to one or more predetermined horizontal / vertical lines or planes). The robotic system 100 may evaluate different sequences or combinations of transfer plans 314 according to a set of predetermined rules that consider the cost of utilizing different tools. In one or more embodiments, the robotic system 100 may calculate a cost, such as an estimated transfer time and / or an estimated error probability, associated with each transfer plan 314. As described above, the robotic system 100 may eliminate transfer plans that cause collisions and / or have an error probability (e.g., potential loss) above a predetermined threshold, resulting in a plan that uses an appropriate tool to grasp the object. The robotic system 100 can derive different sequences of the set of determined plans and include or plan tool change operations accordingly. For each sequence, the robotic system 100 can calculate an overall compensation metric for evaluation (e.g., overall completion time or estimated completion time considering likely error rates). For implementation, the robotic system 100 can select the sequence that minimizes the overall compensation metric.
[0044] In some embodiments, the robotic system 100 can improve overall task performance by controlling the timing and parallel execution of plans, tool changes, and plan execution. For example, the robotic system 100 may identify tools already attached to the robot arm 306 when the objects are identified or when the start bin 322 is placed at the start location 114. The robotic system 100 can identify a first group of target objects that can be grasped with the existing tools. The robotic system 100 can select one of the objects in the group and derive a corresponding plan. Once the first plan is complete, the robotic system 100 can execute the first plan in parallel with the derivation of a second plan for this group. Once planning for the first group of objects is complete, the robotic system 100 can derive a plan for the second group of objects in parallel with (e.g., simultaneously with) the execution of the first plan set and / or tool change operations.
[0045] In another exemplary embodiment, the system manager 302 may identify an initial tool already attached to the robot arm 306. The system manager 302 may interact with the planner 304 to identify and plan a first object that can be grasped and transferred using the attached tool. During execution of the first plan, the system manager 302 may interact with the planner 304 to create a plan for a second object. If derivation fails, the system manager 302 may select and interact to create a plan for a different object in an iterative manner in parallel with the execution of the previous plan. If derivation of an existing tool is successful, the corresponding object and plan may then be put into execution. If any of the remaining target objects are not suitable for the currently connected tool, the robotic system 100 may plan to perform a tool change operation at the end of the ongoing or previous plan execution. In an alternative embodiment, the robotic system 100 may derive and evaluate the feasibility and tradeoffs of all available tools for each target object. The robotic system 100 may analyze the full set of derivations when determining a preferred sequence.
[0046] Tool examples 4A is a diagram of an exemplary transfer unit (e.g., a robotic arm 306) in accordance with one or more embodiments of the present technology. The robotic arm 306 may be the transfer unit 104 of FIG. 1 (e.g., a piece picker or bin picker robot) or a portion thereof. The robotic arm 306 may include an arm portion 402 configured to move an end effector (e.g., a gripper) through an operating space. The arm portion 402 may include a set of structural members (e.g., beams, columns, etc.), a set of joints between the structural members, and / or a corresponding set of actuators / motors configured to move the set of structural members about the joints.
[0047] An end effector tool 404 can be attached to the arm portion 402, such as at its distal end. The end effector tool 404 can include a tool connector 412 (e.g., a selective locking / attachment mechanism) configured to interface with and attach the tool to the arm portion 402. The tool connector 412 can be structurally connected to or integral with a tool arm 414 having an end effector attached to an opposing end. In the example shown in FIG. 4A , the end effector can include a gripper configured to grasp an object for manipulation (e.g., transfer or displacement through space). The gripper end effector can include a gripper head 416 that houses or facilitates a gripping interface 418 (e.g., a set of suction cups for a vacuum-based gripper). The gripping interface 418 can be used to create an adhesive force or mechanism (e.g., a vacuum) that attaches the object of interest to the gripper head 416 and / or the robot arm 306.
[0048] 4B is a diagram of an exemplary toolset 430 in accordance with one or more embodiments of the present technology. Toolset 430 may comprise tools available to robotic system 100 of FIG. 1 to perform one or more tasks. In some embodiments, toolset 430 may include a standard fixed gripping tool 440, a fixed-angle gripping tool 450, and / or an adjustable gripping tool 460. While not shown in FIG. 4B , it should be understood that toolset 430 may include other tools, such as finger-based pinch grippers, grippers with different types of suction / contact interfaces, different categories of end effectors (e.g., non-gripping end effectors for welding, cutting, etc.), etc.
[0049] The standard fixed gripping tool 440 can be configured to grip an object that lies relatively flat compared to the bottom of the start bin 322. For example, the standard fixed gripping tool 440 can include a laterally oriented gripping interface 418 for gripping a laterally oriented top surface of the object. The joint connecting the tool arm 414 and the gripper head 416 can be fixed with the tool arm 414 extending perpendicular to the suction interface or parallel to the pincher interface.
[0050] Fixed-angle gripping tool 450 can be configured to grip objects that are positioned at an angle relative to the bottom of start bin 322. For example, fixed-angle gripping tool 450 can include suction gripping interface 418 and tool arm 414 configured at a non-orthogonal angle to grip a non-lateral top surface of an object. Angled joint 452 can fixedly connect tool arm 414 and gripper head 416 with the structures forming a corresponding non-orthogonal angle or orientation.
[0051] The adjustable gripping tool 460 can be configured to adjust the orientation of the gripper head 416 relative to the tool arm 414. For example, the adjustable gripping tool 460 can include an orientation control mechanism 464, such as a manipulator arm, a set of cables, an actuator, a motor, or the like, configured to adjust the orientation of the gripper head relative to the tool arm 414 about a rotatable joint 462. Thus, the robotic system 100 can operate the orientation control mechanism 464 to adjust the attitude or orientation of the gripping interface 418 depending on the orientation of the object of interest or one or more portions thereof (e.g., the exposed top surface).
[0052] Tool usage examples 5A is a diagram of a standard grasping scenario 500 in accordance with one or more embodiments of the present technology. The standard grasping scenario 500 may be for grasping an object 502 in a flat pose. The object 502 in a flat pose may correspond to a target object having an orientation (e.g., the bottom of the object) parallel to the bottom surface of the start bin 322. Thus, the standard grasping scenario 500 may be consistent with a standard fixed grasping tool 440. For example, the robotic system 100 of FIG. 1 may operate the robotic arm 306 of FIG. 3 to position the standard fixed grasping tool 440 directly above the object 502 in a flat pose and lower the end effector to grasp the object.
[0053] FIG. 5B is a diagram of an angled grasping scenario 510 in accordance with one or more embodiments of the present technology. The angled grasping scenario 510 may be for grasping an angled object 512. The angled object 512 may include an object leaning or resting on a non-flat contact point (e.g., resting along a non-horizontal surface) and / or an object with non-parallel opposing surfaces. In some embodiments, the angled object 512 may correspond to an object posed with one or more surfaces oriented along a direction / plane that is angled relative to a horizontal / vertical reference direction. Thus, the angled grasping scenario 510 may be consistent with a fixed-angle grasping tool 450. For example, the robotic system 100 of FIG. 1 can operate the robotic arm 306 of FIG. 3 to position the fixed-angle grasping tool 450 directly above the angled object 512 and lower the end effector to grasp the object.
[0054] Incidentally, a standard fixed gripping tool 440 may not be appropriate for the angled gripping scenario 510. For example, to grip the angled object 512, the standard fixed gripping tool 440 may be tilted or angled such that the gripping interface 418 is oriented parallel to the connecting portion of the angled object 512. Lowering the standard fixed gripping tool 440 at an angle may result in a collision event (indicated by an "X" in FIG. 5B ) between the tool and / or the robot arm 306 and the start bin 322.
[0055] In some embodiments, the robotic system 100 can select the fixed-angle gripping tool 450 based on the surface pose 514 of the angled object 512. For example, the robotic system 100 can process one or more images (e.g., top-view images) of the starting bin 322 and / or the angled object 512 therein captured by the imaging device 222 of FIG. 2 . The robotic system 100 can identify edges depicted in the images based on an edge detection mechanism (e.g., a Sobel filter). The robotic system 100 can identify each continuous surface depicted in the images based on determining the connection and / or relative orientation between a set of edges and / or recognizing the shape, color, and / or design between the edges. The robotic system 100 can map the surfaces to a three-dimensional image (e.g., a depth map) and use the depth indication to calculate one or more slopes of each surface. Using the calculated slope(s), the robotic system 100 can derive the surface pose 514 of each surface. The robotic system 100 can determine an angled object 512 based on comparing the surface pose 514 to a threshold that represents a horizontal or flat surface.
[0056] 6A is a diagram of a standard release scenario 600, in accordance with one or more embodiments of the present technology. The standard release scenario 600 may be for releasing a grasped object (e.g., target object 112 of FIG. 1 ) from a standard fixed gripping tool 440. The standard release scenario 600 may be for placing the grasped object (e.g., target object 112 of FIG. 1 ) at a corresponding placement location in task location 116 and / or in target container 324 of FIG. 3 . The standard release scenario 600 may be consistent with a flat placement of the grasped object and / or an upright orientation of the standard fixed gripping tool 440.
[0057] FIG. 6B is a diagram of a first angled release scenario 610 in accordance with one or more embodiments of the present technology. The first angled release scenario 610 may be for releasing a grasped object (e.g., target object 112 of FIG. 1 ) from a fixed-angle gripping tool 450. The first angled release scenario 610 may be for placing the grasped object (e.g., target object 112 of FIG. 1 ) at a corresponding placement position in task location 116 and / or a corresponding placement position in target container 324 of FIG. 3 . The first angled release scenario 610 may correspond to a flat placement pose 612 for the grasped object. Thus, the robotic system 100 of FIG. 1 can match the fixed-angle gripping tool 450 to an angled tool release pose 614 (e.g., a non-vertical or tilted orientation of the tool arm therein). The tool release pose 614 may be consistent with the angled joint 452 of FIG. 4B of the fixed-angle gripping tool 450.
[0058] 6C is an illustration of a second angled release scenario 620 in accordance with one or more embodiments of the present technology. The second angled release scenario 620 may be for releasing a grasped object (e.g., target object 112 of FIG. 1 ) from a fixed-angle gripping tool 450. The second angled release scenario 620 may be for placing the grasped object (e.g., target object 112 of FIG. 1 ) at a corresponding placement location in task location 116 and / or a corresponding placement location in target container 324 of FIG. 3 . The second angled release scenario 620 may correspond to an angled placement orientation 622 of the grasped object. Thus, the robotic system 100 of FIG. 1 may match the fixed-angle gripping tool 450 to an upright tool release orientation 624 (e.g., a vertical orientation of the tool arm therein). The angled placement orientation 622 may be consistent with the angled joint 452 of FIG. 4B of the fixed-angle gripping tool 450. The angled placement position 622 and the upright tool release position 624 may be for releasing a grasped object without performing the orientation operations associated with the tool release position 614 and the flat placement position 612 shown in Figure 6B. Thus, the angled placement position 622 and the upright tool release position 624 may represent a lower performance trade-off (e.g., shorter mounting time) than the tool release position 614 and the flat placement position 612. The angled placement position 622 and the upright tool release position 624 may be performed for objects having certain characteristics, such as softer and / or more flexible objects that are less susceptible to damage from impact.
[0059] Task Timing Example 7 is an exemplary timing diagram 700 in accordance with one or more embodiments of the present technology. The timing diagram 700 may represent a sequence or temporal relationship between operations and / or processes of the robotic system 100 of FIG. 1 (e.g., the system manager 302 of FIG. 3 , the planner 304 of FIG. 3 , and / or the robotic arm 306 of FIG. 3 ). For example, the timing diagram 700 may show the temporal relationship between a planning schedule 702 and an implementation schedule 704. The planning schedule 702 may represent a sequence of processes in which each process derives an instance of the transfer plan 314 of FIG. 3 , such as to transfer a target object with an assigned tool as specified by the target selection 313 of FIG. 3 . The implementation schedule 704 may represent a sequence of implementation / execution of the plan in / by the robotic arm 306.
[0060] The robotic system 100 can perform the plan derivation process and the plan implementation process in parallel to increase overall efficiency and reduce overall task execution time. In the example shown in FIG. 7 , the system manager 302 can send a first object selection (e.g., an instance of the target selection 313) that identifies object-1 to be grasped and transferred using part-1 (e.g., the standard fixed gripping tool 440 of FIG. 4B ). In response to the first object selection, the planner 304 can derive a first object plan 712 for grasping and transferring object-1 with part-1. If the derivation is successful, the planner 304 can communicate the first object plan 712 to the system manager 302 via the corresponding feedback 318 of FIG. 3 , and the system manager 302 can interact with the robot arm 306 to implement the first object plan 712. The first object transfer 732 of the implementation schedule 704 can represent the implementation of the first object plan 712 by the robot arm 306.
[0061] Additionally, if the derivation is successful, system manager 302 can send a second object selection (a new instance of target selection 313) that identifies object-2 to be grasped and transported using connected part-1. In response to the selection of the second object, planner 304 can derive a plan 714 for the second object, such as using the exemplary iterative process described above. If the second object is angled object 512 of FIG. 5B, planner 304 can return a failure status indicating that a successful plan cannot be derived, such as because of a predicted collision event when attempting to grasp the object. Planner 304 can communicate the failure status to system manager 302, such as via corresponding feedback 318. In some embodiments, system manager 302 may identify other objects of interest to consider for creating a plan using the currently connected tool. System manager 302 may identify a second object and perform planning involving selection of an updated tool (e.g., an alternative head angle for fixed-angle gripping tool 450 of FIG. 4B and / or adjustable gripping tool 460 of FIG. 4B ), such as when none of the remaining objects can be grasped with the currently connected tool and / or gripper head orientation. Using the updated tool, planner 304 may derive updated second object plan 716. Second object transfer 736 of implementation schedule 704 may represent implementation of updated second object plan 716 on robot arm 306. Various planning processes can occur in parallel with (e.g., independently of) implementation schedule 704.
[0062] If the subsequent plan is successfully derived, the system manager 302 can place its implementation in the implementation schedule 704. If the successful derivation is based on providing an updated tool to the target selection 313, such as for an updated second object plan 716, the system manager 302 can include a tool change 734 in the implementation schedule 704.
[0063] In some embodiments, the robotic system 100 (e.g., in the system manager 302 and / or the planner 304) can estimate performance metrics for various implementations. For example, the robotic system 100 can derive a first plan metric 722 for the first object plan 712 (e.g., total execution time including picking and / or placing the object), a second plan metric 726 for the updated second object plan 716, and / or a tool change metric 724 for the tool change 734 (e.g., maximum duration and / or average duration). The robotic system 100 can use the performance metric to derive a plan sequence that minimizes the combined metric. In some embodiments, the tool change metric 724 can represent a trade-off associated with switching the pose of the gripping interface 418 on the adjustable gripping tool 460 between functioning as a standard fixed gripping tool 440 and functioning as a fixed-angle gripping tool 450. The tool change indicator 724 may further include or account for compensation (e.g., changes in time and / or failure estimates) associated with achieving the tool release posture 614 (e.g., additional operations compared to a standard release from a standard fixed gripping tool). The tool change indicator 724 may also include or account for compensation associated with changes in drag or air resistance due to changes in surface area relative to the direction of travel, such as due to an angled grip, and / or additional operations to reduce drag.
[0064] As an illustrative example, the robotic system 100 can derive a planning schedule 702 for transferring multiple objects from a start location (e.g., a bin) to a destination location (e.g., a different bin or conveyor). The robotic system 100 can derive estimated performance metrics for transferring one or more objects using corresponding tools. The estimated metrics can account for any speed changes, additional operations, and / or adjusted picking / dropping operations or poses for different tools. In some embodiments, the estimated metrics can further account for an expected error rate associated with the corresponding object and tool combination. For example, the estimated metrics may include a transfer time increased by a calculated amount to account for repair actions to be deployed if an error (e.g., piece loss) occurs. This additional adjustment can be calculated based on weighting the average duration of the repair actions by the error rate associated with the object and tool combination and / or the corresponding movement setting (e.g., speed). Optionally, the robotic system 100 may weigh (1) any negative adjustments due to utilizing a suboptimal tool to transfer the object against (2) the cost of changing the tool to transfer the object using the optimal tool and the impact of changing the tool on transferring other objects. That is, the robotic system 100 may evaluate the total cost of different combinations of motion plans and select one that minimizes the cost of transferring one or more or all of the objects of interest. In some embodiments, the estimated metric may be expressed as the number of picks per minute or per hour using the same robotic unit.
[0065] Operation flow FIG. 8 is a flow diagram of a method 800 for operating a robotic system (e.g., robotic system 100 of FIG. 1 ) in accordance with one or more embodiments of the present disclosure. Method 800 can be implemented using one or more of the devices described above, such as system manager 302 of FIG. 3 , planner 304 of FIG. 3 , and / or robotic arm 306 of FIG. 3 . Method 800 can be implemented using one or more of the components described above, such as processor 202 of FIG. 2 , storage device 204 of FIG. 2 , etc. Method 800 can be for planning and executing a task (e.g., transferring an object) using a set of tools (e.g., toolset 430 of FIG. 4B ). As described in more detail below, method 800 can accommodate planning and executing tasks in parallel and / or processing objects in groups according to appropriate tools to minimize tool changes 734 of FIG. 7 .
[0066] At block 802, the robotic system 100 may receive an order for a set of items (e.g., order input 312 in FIG. 3). For example, the robotic system 100 may receive the order input 312 from a customer or a requesting warehouse (e.g., via communication device 206 in FIG. 2). The order input 312 may also correspond to an internally generated order to repackage or regroup items for storage, such as to rearrange / combine objects to reduce storage containers. Thus, the robotic system 100 may identify a set of objects that need to be moved from a storage location to a different location (e.g., a shipping container) to fulfill the order.
[0067] At block 804, the robotic system 100 may coordinate access to the ordered object. The robotic system 100 may identify the storage location(s) of the ordered object (e.g., via the system manager 302 of FIG. 3). For example, the robotic system 100 may compare the order input 312 to a record of stored / accessible objects. Consequently, the robotic system 100 identifies the bin and / or object corresponding to the order input 312, as shown in block 805. The robotic system 100 may determine the storage location and / or identifier of the container in which the ordered object is stored.
[0068] The robotic system 100 may use the determined location and / or container identifier to coordinate access to the ordered object. For example, the robotic system 100 may directly operate one or more transport units (e.g., AGVs and / or conveyors) to transport the target container(s) from the storage location(s) to the start location 114, FIG. 1 , for the robotic arm 306, FIG. 3 . The robotic system 100 may similarly operate a transport unit to transport the target container 324, FIG. 3 , from its storage location to the task location 116, FIG. 1 . Additionally or alternatively, the robotic system 100 may interact with or communicate with one or more subsystems (e.g., a storage access system) to place the targeted container(s) and / or target container 324 at their respective locations.
[0069] At block 806, the robotic system 100 may acquire image data. For example, the robotic system 100 may use the imaging device 222 of FIG. 2 to acquire two-dimensional and / or three-dimensional images depicting the storage container in a start position. As a result, the acquired image data may depict one or more of the ordered objects stored within the storage container in real time.
[0070] At block 808, the robotic system 100 may identify a group of objects (e.g., target objects or ordered objects) within the storage container. The robotic system 100 may process the acquired image data to detect or recognize objects within the start bin 322. For example, the robotic system 100 may identify surfaces based on line detection (e.g., via a Sobel detection mechanism) and the connections / arrangements between the detected lines. The robotic system 100 may compare the depicted image or a portion thereof (e.g., a portion within a surface) with the surface image data in the master data. Additionally or alternatively, the robotic system 100 may compare the dimensions of the identified surface with the dimensions of the object stored in the master data. The robotic system 100 may identify the object based on a match between the depicted image and / or measured dimensions and corresponding images and / or predetermined dimensions stored in the master data.
[0071] The robotic system 100 may compare the set of identified objects in the starting bin 322 with the targeted or expected portion of the order input 312. In response, the robotic system 100 may find the targeted objects in the starting bin 322. The robotic system 100 may further process the targeted objects to classify / group the objects according to the toolset 430, thereby determining the tool-based object groups 316 of FIG. 3. For example, the robotic system 100 may determine whether each of the targeted objects is the flat pose object 502 of FIG. 5A or the angled object 512 of FIG. 5B. The robotic system 100 may calculate one or more slopes using the depth indicators associated with the depicted surfaces. The robotic system 100 may use the calculated slopes to derive the surface pose 514 of FIG. 5B for each of the targeted objects. Surface pose 514 may be used to classify or group objects, such as flat pose object 502, angled object 512, or other such objects, according to the tool dedicated or assigned to manipulate the corresponding object.
[0072] Robotic system 100 may further categorize / group objects according to various end effectors and / or other characteristics corresponding to the intended task. For example, robotic system 100 may group objects according to structural rigidity and / or exterior material (e.g., box, plastic packaging, bag, etc.) associated with different types of grippers (e.g., different types of contact interfaces such as suction grippers, suction cup sizes / positions, finger-based grippers, etc.), overall shape, etc. Robotic system 100 may also group objects that are bonded together and therefore require a cutting tool separately from other unbonded objects. To this end, this grouped object may correspond to tool-based object group 316.
[0073] As an illustrative example, the robotic system 100 can determine at least a first set of objects (e.g., a set of objects 502 in a flat pose) and a second set of objects (e.g., a set of angled objects 512) based on image data depicting the objects in the starting bin 322. The first set of objects can have one or more aspects (e.g., poses) corresponding to characteristics of the first tool (e.g., the angle / orientation of the gripping interface 418), and the second set of objects can have one or more aspects corresponding to characteristics of the second tool (e.g., the angle / orientation of the gripping interface 418). The flat pose objects 502 can each have a top surface parallel to the bottom surface of the starting bin 322, thereby matching with the standard fixed gripping tool 440. The angled objects 512 can each have a top surface pose that forms an angle with respect to the bottom surface of the starting bin 322, thereby matching with the fixed-angle gripping tool 450.
[0074] At block 810, the robotic system 100 may track and / or update the connected tools. The robotic system 100 may dynamically detect tools connected to the robot arm 306 using internal mechanisms (e.g., radio frequency identification (RFID) circuitry, hard-coded identifiers, etc.). Additionally or alternatively, the robotic system 100 may track tool change operations and tool selections to track connected tools.
[0075] When tracking and / or updating the connected tools, the robotic system 100 can access or determine the toolset available to the robotic system 100. For example, the robotic system 100 may access predetermined data regarding the tools and / or corresponding characteristics (e.g., the pose of the gripping interface 418) of the toolset 430 available to the robotic arm 306. Additionally or alternatively, the robotic system 100 can dynamically (e.g., in real time) track the tools being used or connected to the robotic units. As a result, the robotic system 100 can determine the remaining tools (e.g., tools not connected to any of the other robotic units) and the tools available for access / connection. The robotic system 100 can further access predetermined or previously measured data to determine the tool change indicator 724 in FIG. 7 (e.g., a cost, such as time) associated with switching between tools (e.g., disconnecting from one tool and connecting to a different tool) when manipulating a target object. In some embodiments, the tool change indicator 724 may represent a trade-off associated with changing the orientation of the gripping interface 418 of the adjustable gripping tool 460, such as by manipulating the orientation control mechanism 464 of FIG.
[0076] Also, in block 810, the robotic system 100 may update the connected tool, such as in response to a status determination (e.g., decision block 820) described below. For example, if a new tool is needed to manipulate / transfer the object according to the status determination, the robotic system 100 may operate the robot arm 306 to disconnect the connected tool and connect a new tool appropriate for the subsequent transfer. The robotic system 100 may update tracked tool information. In some embodiments, the robotic system 100 may schedule a tool update (e.g., tool change 734) and update tracked information when the tool change 734 is implemented.
[0077] At block 812, the robotic system 100 may select one or more objects for planning. The robotic system 100 may select from a group corresponding to tools connected to the robotic arm 306. Within the selected group, the robotic system 100 may repeatedly select one object for transfer. The robotic system 100 may select based on a set of predetermined selection rules for first selecting higher and / or closer objects, etc.
[0078] At block 814, the robotic system 100 may derive one or more plans (e.g., transfer plans 314) for the selection (e.g., via the planner 304). For example, the system manager 302 may provide the planner 304 with target selections 313 according to the selected objects and / or connected tools. In some embodiments, the robotic system 100 may provide a list of objects for a group. In response, the planner 304 may derive one or more transfer plans for each object. If the plan derivation is successful, the planner 304 may provide the resulting transfer plan 314 to the system manager 302 via feedback 318. Alternatively, if the plan derivation fails (e.g., due to a possible collision caused by an improper tool assignment), the planner 304 may provide feedback 318 reporting an unsuccessful status (e.g., indicating an improper match between the assigned tool and the target object).
[0079] Thus, the robotic system 100 can derive a set of transfer plans 314 for transferring objects of interest depending on the tool. In some embodiments, for example, the robotic system 100 can follow a process for planning transfers based on tool change indicators 724 (e.g., according to a set of rules and / or architecture to minimize overall transfer cost / time). In systems where tool changes have average / maximum indicators greater than the indicators of the planning process and / or plan execution, the robotic system 100 may plan transfers to minimize tool changes. As an example, the robotic system 100 can plan and schedule execution for applicable objects / groups using the connected tool before developing a plan for another group of objects. In some embodiments, the robotic system 100 can iteratively perform the selection and derivation process and implement the derived plans accordingly. In other embodiments, the robotic system 100 can derive plans for one or more groups of objects and arrange / order the plans according to a set of rules to minimize the total cost.
[0080] In deriving the plan, the robotic system 100 may calculate metrics (e.g., plan metrics 720 in FIG. 7 ), as shown in block 816. For example, the robotic system 100 may calculate an estimated length of time for each transfer plan based on the calculated distance traveled, the number and / or type of operations, corresponding settings, etc.
[0081] In the example shown in FIG. 8 , the robotic system 100 may derive a set of plans based on repeatedly selecting and planning an object in one group before processing another group. At decision block 818, the planner 304 may determine whether the plan derivation was successful and provide corresponding feedback 318 to the system manager 302. If the plan derivation was not successful, the robotic system 100 may determine whether any objects remain in the currently processed group of objects, as shown in decision block 820. If objects remain in the currently selected / processed group, flow may proceed to block 812, where the robotic system 100 may select the next object in the group. If no more objects remain in the group (e.g., the tool-based object group 316 is empty), flow may proceed to block 810, where the robotic system 100 may update the connected tool (e.g., by performing a tool change 734). The robotic system 100 may select and connect a new tool to the tool depending on the remaining objects in the bin and the corresponding tool(s). In conjunction with a change of tool, the robotic system 100 can derive a plan for another group associated with the new tool.
[0082] In some embodiments, once the planner 304 derives a plan, the robotic system 100 can derive and / or update a sequence for executing the plan. For example, the system manager 302 may schedule the execution of the plan, such as by queuing the plan at the system manager 302 or the robotic arm 306.
[0083] In one or more alternative embodiments, the robotic system 100 may derive multiple plans per object (e.g., one for each available tool). At block 822, the robotic system 100 may derive a sequence / combination of plans based on the corresponding plan index and the tool change index 724. The robotic system 100 may select and schedule the sequence with the lowest overall cost (e.g., total execution / transfer time).
[0084] As an illustrative example of blocks 808-822, the robotic system 100 may identify a first tool (e.g., standard fixed gripping tool 440) connected to the robotic arm 306 before transferring the set of targeted objects in the start bin 322. The robotic system 100 may determine and select a first set of objects (e.g., an instance of the tool-based object group 316, such as object 502 in a flat pose) corresponding to the first tool. The robotic system may iteratively derive a first set of plans for operating the robotic arm 306 to transfer the first set of objects. In some embodiments, the robotic system 100 may derive the first set of plans based on selecting objects in the set, deriving test plans for the selected objects, and determining whether the test plans are feasible. After deriving the first set of plans, the robotic system 100 may schedule a tool change 734 to follow the last plan in the first set. Consequently, after transferring the first set of objects, the robotic system 100 can schedule a second tool (e.g., fixed-angle gripping tool 450) to be connected to the robotic arm 306. The robotic system 100 can continue deriving a plan for the second set of objects.
[0085] At block 824, the robotic system 100 may implement the scheduled plan (e.g., via the system manager 302). That is, the robotic system 100 may operate the robotic arm 306 to transfer objects according to the derived transfer plan and its scheduled order. This plan may be implemented in parallel with (e.g., independently of) a subsequent derivation process. For example, the system manager 302 may provide the target selection 313 to the planner 304 to initiate execution of a first plan with the robotic arm 306 immediately before or after initiating plan derivation for a second object. Thus, while the robotic arm 306 is transferring the first object, derivation processing for the second object, or any subsequent objects, may be occurring. In this manner, the robotic system 100 may continuously derive and implement motion plans until the starting location or set of targeted objects is exhausted, as shown in block 826.
[0086] In some embodiments, the robotic system 100 can include in the transfer plan an action to reduce tension for the orientation control mechanism 464 of the adjustable grasping tool 460. The robotic system 100 can include a corresponding command and / or setting (1) after the corresponding object is brought into contact with at least a portion of the grasping interface and / or (2) before grasping the corresponding object. Thus, during implementation, the grasping interface 418 can move and / or rotate to increase the contact area with the exposed surface of the target object. In other words, the grasping interface 418 can rotate or adjust about the rotatable joint 162 to match the surface pose 514 of the target object. The reduction in tension can occur before or during actuation of the grasping interface 418 (e.g., a suction cup) to improve the grip established between the grasping interface 418 and the target object.
[0087] conclusion The above detailed description of embodiments of the disclosed technology is not intended to be exhaustive or to limit the disclosed technology to the precise form disclosed above. While specific embodiments of the disclosed technology have been described above for illustrative purposes, those skilled in the art will recognize that various equivalent modifications are possible within the scope of the disclosed technology. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps or use systems having blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of ways. Also, while processes or blocks are sometimes shown as being performed in serial order, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Furthermore, any specific numbers described herein are merely examples, and alternative embodiments may use different values or ranges.
[0088] These and other modifications can be made to the disclosed technology in light of the above detailed description. While the detailed description sets forth specific examples and the best mode contemplated for the disclosed technology, the disclosed technology, no matter how detailed the above description appears in the text, can be practiced in many ways. Details of the system, while encompassed by the technology disclosed herein, may vary considerably in particular implementations. As noted above, a particular term used when describing a particular feature or aspect of the disclosed technology should not be construed as meaning that the term has been redefined herein to be limited to the particular characteristic, feature, or aspect of the disclosed technology with which it is associated. Accordingly, the present invention is not limited, except as by the appended claims. In general, the terms used in the following claims should not be construed to limit the disclosed technology to the specific examples disclosed herein, unless such terms are expressly defined in the Detailed Description section above.
[0089] While certain aspects of the invention are presented below in particular claim forms, applicants contemplate various aspects of the invention in any number of claim forms, and accordingly, applicants reserve the right to pursue additional claims after filing this application in order to pursue such additional claim forms in either this application or any continuing application.
Claims
[Claim 1] 1. A method for operating a robotic system, comprising: identifying a toolset including two or more tools, each uniquely configured to manipulate an object, each tool configured for selective attachment to a robotic arm; acquiring image data representative of a set of targeted objects at a starting position, the image data depicting a representation of two or more objects in the set of targeted objects; grouping the two or more objects according to a set of physical properties and / or a current pose thereof, the set of physical properties and / or the current pose corresponding to the unique configuration of the two or more tools in the toolset; deriving a set of plans, including at least a first plan and a second plan, based on the image data, the set of plans (1) for operating a robot arm to transfer at least a portion of the set of targeted objects from the start location to a target location, and (2) incorporating various usages of tool changes; determining one or more tool change indicators for the derived plan, each tool change indicator representing a cost associated with removing a previous tool from the robot arm and installing a subsequent tool therein; calculating a compensation index for each plan in the set of plans, the calculated compensation index accounting for (1) the tool change index and / or (2) transport time associated with picking, transporting, and placing at least the portion of the set of targeted objects; selecting a plan from the set of plans based on the cost index, the selected plan corresponding to a lowest cost index associated with transporting at least the portion of the set of targeted objects; A method comprising:
