Methods and computational systems for performing robot motion planning and storage bin detection

Abstract

This disclosure relates to a method and computational system for performing robot motion planning and storage detection. A method and computational system for transferring objects between a source storage and a destination storage are provided. The computational system is configured to operate using a combination of pre-planned and image-based trajectories to improve the speed and reliability of object transfer. The computational system is configured to capture image information of the storage and use the captured information to modify or adjust the pre-planned trajectory to improve performance.

Description

[0001] This application is a divisional application of the invention patent application filed on May 3, 2022, with application number 202280004888.9 and entitled "Method and Computational System for Performing Robot Motion Planning and Storage Detection".

[0002] Cross-reference to related applications

[0003] This application claims the benefit of U.S. Application No. 17 / 733,024, filed April 29, 2022, entitled “METHOD AND COMPUTING SYSTEM FOR PERFORMING ROBOT MOTION PLANNING AND REPOSITORY DETECTION”, and U.S. Provisional Application No. 63 / 183,685, filed May 4, 2021, entitled “ROBOTIC SYSTEM FOR ADJUSTING ROBOT TRAJECTORY”, the entire contents of each of which are incorporated herein by reference. Technical Field

[0004] This technology is generally directed toward robotic systems, and more specifically toward systems, processes and techniques for performing object size measurement and / or minimum feasible region detection. Background Technology

[0005] With their ever-improving performance and decreasing costs, many robots (e.g., machines configured to perform physical actions automatically / autonomously) are now widely used in a variety of different fields. For example, robots can be used to perform various tasks (e.g., manipulating or transferring objects through space) in manufacturing and / or assembly, packing and / or packaging, transportation and / or shipping. In performing tasks, robots can replicate human actions, thereby replacing or reducing additional human intervention required for performing dangerous or repetitive tasks. Summary of the Invention

[0006] In one embodiment, a computing system is provided. The computing system includes a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector and a camera attached to the robotic arm; at least one processing circuit configured to perform the following steps to transfer an object from a source storage to a destination storage when the robot is within an object transfer range including a source storage and a destination storage. These steps include outputting a first destination storage proximity command to cause the robotic arm to approach the destination storage in such a manner that the camera is pointed at the destination storage; receiving image information describing the destination storage, wherein the image information is generated by the camera after the first destination storage proximity command is executed; performing a storage detection operation based on the image information, the storage detection operation determining at least one of the following: storage structure information describing the structure of the destination storage, storage pose information describing the pose of the destination storage, or storage content information describing the presence of one or more objects in the destination storage; and performing the operation after the camera has generated an image describing the destination storage. Following the information, a source storage proximity command is output to bring the robotic arm close to the source storage; after the source storage proximity command is executed, an object pickup command is output to bring the end effector device to pick up an object from the source storage; after the object pickup command is executed, a second destination storage proximity command is output to bring the robotic arm close to the destination storage; after the second destination storage proximity command is executed, an object placement command is output to bring the end effector device to place the object in the destination storage, wherein the object placement command is generated based on the result of a storage detection operation, wherein the result includes at least one of storage structure information, storage posture information, or storage content information.

[0007] In one embodiment, a robot control method is provided for transferring an object from a source storage repository to a destination storage repository. The method is operable by at least one processing circuit via a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector device and a camera attached to the robotic arm. The method includes outputting a first destination storage repository approach command to cause the robotic arm to approach the destination storage repository in such a manner that the camera is pointed at the destination storage repository; receiving image information describing the destination storage repository, wherein the image information is generated by the camera after the first destination storage repository approach command is executed; performing a storage repository detection operation based on the image information, the storage repository detection operation determining at least one of the following: storage repository structure information describing the structure of the destination storage repository, storage repository pose information describing the pose of the destination storage repository, or storage repository content information describing the presence of one or more objects in the destination storage repository; and the method continues after the camera has generated image information describing the destination storage repository. After the information is received, a source storage approach command is output to bring the robotic arm close to the source storage; after the source storage approach command is executed, an object pick-up command is output to bring the end effector device to pick up an object from the source storage; after the object pick-up command is executed, a second destination storage approach command is output to bring the robotic arm close to the destination storage; after the second destination storage approach command is executed, an object placement command is output to bring the end effector device to place the object in the destination storage, wherein the object placement command is generated based on the result of the storage detection operation, wherein the result includes at least one of storage structure information, storage posture information, or storage content information.

[0008] In an embodiment, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium is configured with executable instructions for implementing a robot control method for transferring an object from a source repository to a destination repository. This method is operable by at least one processing circuit via a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector device and a camera attached to the robotic arm. The method includes outputting a first destination repository approach command to cause the robotic arm to approach the destination repository in such a manner that the camera is pointed at the destination repository; receiving image information describing the destination repository, wherein the image information is generated by the camera after the first destination repository approach command is executed; performing a repository detection operation based on the image information, the repository detection operation determining at least one of the following: repository structure information describing the structure of the destination repository, repository pose information describing the pose of the destination repository, or repository content information describing the presence of one or more objects in the destination repository; and performing the image information describing the destination repository generated by the camera. After the information is received, a source storage approach command is output to bring the robotic arm close to the source storage; after the source storage approach command is executed, an object pick-up command is output to bring the end effector device to pick up an object from the source storage; after the object pick-up command is executed, a second destination storage approach command is output to bring the robotic arm close to the destination storage; after the second destination storage approach command is executed, an object placement command is output to bring the end effector device to place the object in the destination storage, wherein the object placement command is generated based on the result of the storage detection operation, wherein the result includes at least one of storage structure information, storage posture information, or storage content information. Attached Figure Description

[0009] Figure 1A-1D A system consistent with the embodiments herein is shown for performing or facilitating the definition of a minimum feasible scope.

[0010] Figure 2A-2G A block diagram is provided illustrating a computing system configured to perform or facilitate the definition of a minimum feasible scope, consistent with the embodiments herein.

[0011] Figures 3A-3F An environment in which a minimum feasible scope can be defined is shown according to an embodiment of this document.

[0012] Figure 4 A flowchart illustrating a method for defining a minimum feasible scope according to embodiments herein is provided.

[0013] Figures 5A-5C The motion planning trajectory of the robotic arm, consistent with the embodiments described herein, is shown.

[0014] Figure 6 A library template consistent with the embodiments herein is shown. Detailed Implementation

[0015] This document describes systems and methods for robotic systems with coordinated transfer mechanisms. Robotic systems configured according to embodiments described herein (e.g., integrated systems of devices, each performing one or more specified tasks) autonomously perform integrated tasks by coordinating the operations of multiple units (e.g., robots).

[0016] The techniques described in this paper offer technological improvements to existing computer-based image recognition and robot control fields. These improvements provide overall speed, reliability, and accuracy enhancements in robot trajectory planning operations. The robot trajectory planning operations described in this paper involve a combination of pre-planned trajectories and adjusted trajectories to accomplish robot tasks (grasping and moving objects) with greater speed, reliability, and accuracy.

[0017] In particular, the technique described herein improves robotic systems by allowing for an intelligent combination of pre-planned and adjusted trajectories. Pre-planned trajectories can be used to allow a robotic arm to move rapidly through loading / unloading areas without requiring extensive computation during movement. However, when the source and destination of objects differ, relying strictly on pre-planned trajectories can have the disadvantage of reduced accuracy and reliability. Image analysis-adjusted trajectories can provide more accurate and reliable object picking and placement at the cost of speed. The system and method described herein combine pre-planned trajectories and image-aided trajectories to improve speed, accuracy, and reliability compared to using either method alone. The pre-planned trajectory can be used to position the robotic arm close to the target location, while the image-aided trajectory can be used to adjust or fine-tune the final trajectory for picking up or placing objects. Therefore, the method and system described herein provide a technical solution to the technical problems arising in the field of computer-aided robot control.

[0018] In the following, specific details are set forth to provide an understanding of the currently disclosed technology. In the embodiments, the technology described herein may be practiced without including every specific detail disclosed herein. In other instances, well-known features such as specific functions or routines are not described in detail to avoid unnecessarily obscuring this disclosure. References to “embodiment,” “an embodiment,” etc., in this specification mean that a particular feature, structure, material, or characteristic being described is included in at least one embodiment of this disclosure. Therefore, the appearance of such phrases in this specification does not necessarily refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

[0019] For clarity, certain details describing structures or processes well-known and commonly associated with robotic systems and subsystems, but which may unnecessarily obscure some important aspects of the disclosed technology, are not set forth in the following description. Furthermore, although the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments may have different configurations or different components than those described in this section. Therefore, the disclosed technology may have other embodiments with or without certain elements among those described below.

[0020] Many embodiments or aspects of this disclosure described below can take the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. Those skilled in the art will understand that the disclosed technology can be implemented or practiced on computer or controller systems other than those shown and described below. The technology described herein can be embodied in a dedicated computer or data processor specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Therefore, as commonly used herein, the terms “computer” and “controller” refer to any data processor and can include internet tools and handheld devices (including handheld computers, wearable computers, cellular or mobile phones, multiprocessor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers can be presented on any suitable display medium, including a liquid crystal display (LCD). Instructions for performing computer or controller executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive, a USB device, and / or other suitable media.

[0021] The terms “coupling” and “connection”, and their derivatives, are used herein to describe structural relationships between components. It should be understood that these terms are not intended to be synonyms. Rather, in certain embodiments, “connection” can be used to indicate that two or more elements are in direct contact with each other. Unless explicitly stated otherwise in the context, the term “coupling” can be used to indicate that two or more elements are in direct or indirect contact with each other (through other inserting elements between them), or that two or more elements cooperate or interact with each other (e.g., as in a causal relationship, such as for signal transmission / reception or for function invocation), or both.

[0022] Any reference to image analysis performed via a computational system herein can be performed based on or using spatial structure information, which may include depth information describing corresponding depth values ​​relative to various locations of selected points. Depth information can be used to identify objects or estimate how objects are arranged in space. In some cases, spatial structure information may include or be used to generate a point cloud describing the positions of one or more surfaces of an object. Spatial structure information is merely one form of possible image analysis, and other forms known to those skilled in the art can be used according to the methods described herein.

[0023] Figure 1A A system 1500 for performing object detection or, more specifically, object recognition is illustrated. More specifically, system 1500 may include a computing system 1100 and a camera 1200. In this example, camera 1200 may be configured to generate image information that describes or otherwise represents the environment in which camera 1200 is located, or more specifically, the environment within the field of view (also referred to as the camera field of view) of camera 1200. The environment may be, for example, a warehouse, manufacturing plant, retail space, or other location. In such cases, the image information may represent objects located in such locations, such as boxes, cabinets, crates, or other containers. System 1500 may be configured to generate, receive, and / or process image information, such as by distinguishing between various objects in the camera field of view using the image information, to perform object recognition or object registration based on the image information, and / or to perform robot interaction planning based on the image information, as discussed in more detail below (the terms “and / or” and “or” are used interchangeably in this disclosure). Robot interaction planning may be used, for example, to control a robot in a location to facilitate robot interaction between the robot and containers or other objects. The computing system 1100 and the camera 1200 can be located in the same place or can be far apart from each other. For example, the computing system 1100 can be part of a cloud computing platform hosted in a data center far from a warehouse or retail space and can communicate with the camera 1200 via a network connection.

[0024] In an embodiment, camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and / or a 3D camera. For example, Figure 1BA system 1500A (which may be an embodiment of system 1500) is shown, comprising a computing system 1100 and cameras 1200A and 1200B, both of which may be embodiments of camera 1200. In this example, camera 1200A may be a 2D camera configured to generate 2D image information comprising or forming a 2D image describing the visual appearance of the environment in the camera's field of view. Camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) configured to generate 3D image information comprising or forming spatial structure information about the environment in the camera's field of view. This spatial structure information may include depth information (e.g., a depth map) describing corresponding depth values ​​relative to various positions of camera 1200B, such as positions on the surfaces of various objects in the field of view of camera 1200. These positions in the camera's field of view or on the surfaces of objects may also be referred to as physical positions. In this example, depth information can be used to estimate how objects are spatially arranged in three-dimensional (3D) space. In some cases, spatial structure information can include or can be used to generate point clouds that describe the positions of objects on one or more surfaces within the field of view of the camera 1200B. More specifically, spatial structure information can describe various positions on the structure of the object (also referred to as object structure).

[0025] In an embodiment, system 1500 may be a robot operating system for facilitating robot interaction between the robot and various objects in the environment of camera 1200. For example, Figure 1C The robot operating system 1500B is shown, which can be Figure 1A and Figure 1B An embodiment of system 1500 / 1500A. Robot operating system 1500B may include computing system 1100, camera 1200, and robot 1300. As described above, robot 1300 can be used to interact with one or more objects (such as boxes, crates, cabinets, or other containers) in the environment of camera 1200. For example, robot 1300 may be configured to pick up containers from one location and move them to another. In some cases, robot 1300 may be used to perform depalletizing operations in which a set of containers or other objects are unloaded and moved to, for example, a conveyor belt. In some implementations, camera 1200 may be attached to robot 1300, such as a robotic arm 3320 attached to robot 1300. In some implementations, camera 1200 may be separate from robot 1300. For example, camera 1200 may be mounted to the ceiling or other structure of a warehouse and may remain fixed relative to that structure.

[0026] In an embodiment, Figure 1A-1CThe computing system 1100, also referred to as a robot controller, can be formed or integrated into the robot 1300. The robot control system can be included in system 1500B and can be configured to generate commands for the robot 1300, such as robot interaction movement commands for controlling robot interactions between the robot 1300 and containers or other objects. In such embodiments, the computing system 1100 can be configured to generate such commands based on image information, for example, generated by camera 1200. For example, the computing system 1100 can be configured to calculate motion planning based on image information, where the motion planning may be intended for, for example, grasping or otherwise picking up objects. The computing system 1100 can generate one or more robot interaction movement commands to execute the motion planning.

[0027] In this embodiment, the computing system 1100 may form part of a vision system. The vision system may be a system that generates, for example, visual information describing the environment in which the robot 1300 is located, or alternatively or additionally, describing the environment in which the camera 1200 is located. The visual information may include the 3D and / or 2D image information discussed above, or some other image information. In some cases, if the computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above, or it may be separate from the robot control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. This information may be output to the robot control system, which may receive such information from the vision system and perform motion planning and / or generate robot interactive movement commands based on that information. Further information regarding the vision system is described in detail below.

[0028] In one embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 via a direct connection (such as a connection provided via a dedicated wired communication interface (such as an RS-232 interface, a Universal Serial Bus (USB) interface) and / or via a local computer bus (such as a Peripheral Component Interconnect (PCI) bus)). In another embodiment, the computing system 1100 may communicate with the camera 1200 and / or the robot 1300 via a network. The network may be any type and / or form of network, such as a Personal Area Network (PAN), a Local Area Network (LAN) (e.g., an intranet), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), or the Internet. The network may utilize different technologies and protocol layers or stacks, including, for example, Ethernet, Internet Protocol Suite (TCP / IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Optical Networking) protocol, or SDH (Synchronous Digital Hierarchy) protocol.

[0029] In embodiments, the computing system 1100 may communicate information directly with the camera 1200 and / or with the robot 1300, or it may communicate via an intermediate storage device or more generally via an intermediate non-transitory computer-readable medium. For example, Figure 1D System 1500C, which may be an embodiment of system 1500 / 1500A / 1500B, is illustrated. System 1500C includes a non-transitory computer-readable medium 1400, which may be external to computing system 1100 and may act as an external buffer or a storage library for storing, for example, image information generated by camera 1200. In such an example, computing system 1100 may retrieve or otherwise receive image information from non-transitory computer-readable medium 1400. Examples of non-transitory computer-readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. Non-transitory computer-readable media may be formed, for example, computer floppy disks, hard disk drives (HDDs), solid-state drives (SDDs), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), and / or memory sticks.

[0030] As described above, camera 1200 can be a 3D camera and / or a 2D camera. A 2D camera can be configured to generate 2D images, such as color or grayscale images. A 3D camera can be, for example, a depth-sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and / or 3D camera may include image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide-semiconductor (CMOS) sensors. In embodiments, the 3D camera may include a laser, a LiDAR device, an infrared device, a light / dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a RADAR detector, or any other device configured to capture depth information or other spatial structure information.

[0031] As described above, image information can be processed by computing system 1100. In embodiments, computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, laptop computer, etc.), a smartphone, a tablet computing device, and / or any other computing system. In embodiments, any or all of the functions of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer) or may include multiple computing devices.

[0032] Figure 2A A block diagram illustrating an embodiment of a computing system 1100 is provided. The computing system 1100 in this embodiment includes at least one processing circuitry 1110 and a non-transitory computer-readable medium (or media) 1120. In some cases, the processing circuitry 1110 may include a processor (e.g., a central processing unit (CPU), a dedicated computer, and / or an onboard server) configured to execute instructions (e.g., software instructions) stored on the non-transitory computer-readable medium 1120 (e.g., computer memory). In some embodiments, the processor may be included in a separate / independent controller operatively coupled to other electronic / electrical devices. The processor may implement program instructions to control other devices / interface with other devices, thereby enabling the computing system 1100 to perform actions, tasks, and / or operations. In embodiments, the processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller (“PLC”), an application-specific integrated circuit (“ASIC”), a programmable gate array (“PGA”), a field-programmable gate array (“FPGA”), any combination thereof, or any other processing circuitry.

[0033] In embodiments, the non-transitory computer-readable medium 1120, which is part of the computing system 1100, may be an alternative to or addition to the intermediate non-transitory computer-readable medium 1400 discussed above. The non-transitory computer-readable medium 1120 may be a storage device, such as an electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device, or any suitable combination thereof, for example, such as a computer floppy disk, hard disk drive (HDD), solid-state drive (SDD), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, any combination thereof, or any other storage device. In some cases, the non-transitory computer-readable medium 1120 may include multiple storage devices. In some implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some cases, the non-transitory computer-readable medium 1120 may store one or more model templates for performing object recognition operations. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuitry 1110, cause the processing circuitry 1110 to perform one or more methods described herein.

[0034] Figure 2BA computing system 1100A is depicted, which is an embodiment of the computing system 1100 and includes a communication interface 1130. The communication interface 1130 can be configured, for example, to receive signals from... Figure 1A-1D Image information generated by camera 1200. Image information can be received via the intermediate non-transitory computer-readable medium 1400 or network discussed above, or via a more direct connection between camera 1200 and computing system 1100 / 1100A. In an embodiment, communication interface 1130 can be configured to communicate with… Figure 1C The computing system 1100 communicates with the robot control system 1300. If the computing system 1100 is external to the robot control system, the communication interface 1130 of the computing system 1100 can be configured to communicate with the robot control system. The communication interface 1130 may also be referred to as a communication component or communication circuit, and may include, for example, communication circuitry configured to perform communication via wired or wireless protocols. As an example, the communication circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, etc. Controller, PCI bus controller, any other communication circuit or combination thereof.

[0035] In an embodiment, such as Figure 2C As shown, the non-transitory computer-readable medium 1120 may include storage space 1122 configured to store one or more data objects discussed herein. For example, the storage space may store model templates, robotic arm movement commands, and any additional data objects that the computing system 1100B may need to access.

[0036] In an embodiment, the processing circuitry 1110 may be programmed by one or more computer-readable program instructions stored on a non-transitory computer-readable medium 1120. For example, Figure 2D A computing system 1100C is illustrated, which is an embodiment of computing systems 1100 / 1100A / 1100B, wherein processing circuitry 1110 is programmed by one or more modules including an object recognition module 1121, a motion planning module 1129, and an object manipulation module 1126. The motion planning module 1129 (and any other modules associated with the computing system 1100C) can access trajectory information 1128 and template information 1127 as needed.

[0037] In embodiments, the object recognition module 1121 may be configured to acquire and analyze image information, as discussed throughout this disclosure. The methods, systems, and techniques concerning image information discussed herein may be used with the object recognition module 1121. As discussed herein, the object recognition module may be used for both object recognition and library recognition.

[0038] Motion planning module 1129 can be configured to plan and execute robot movements. For example, motion planning module 1129 can derive various placement positions / orientations, calculate corresponding motion plans, or combinations thereof, for grasping and moving objects. Motion planning module 1129 can access and update trajectory information 1128. Trajectory information 1128 may include pre-planned initial trajectory information that can be accessed and updated by motion planning module 1129. The motion planning module can also store adjusted trajectory information. Motion planning module 1129 can access and update template information 1127, including object template information and library template information (both source and destination), as discussed in more detail below. The methods, systems, and techniques discussed herein regarding robotic arm movement and trajectory can be performed by motion planning module 1129. The methods, systems, and techniques discussed herein regarding models and templates can be performed by motion planning module 1129.

[0039] The object manipulation module 1126 can be configured to plan and execute object manipulation activities of the robotic arm, such as grasping and releasing objects, and to execute robotic arm commands to help and facilitate such grasping and releasing.

[0040] refer to Figure 2E-2G 3A, explains the methods related to the object recognition module 1121 that can be executed for image analysis. Figure 2E and 2F Example image information associated with the image analysis method is shown, while Figure 3A An example robotic environment associated with the image analysis method is illustrated. References in this document relating to image analysis performed by a computational system can be performed based on or using spatial structure information, which may include depth information describing corresponding depth values ​​relative to various locations of selected points. The depth information can be used to identify objects or estimate how objects are arranged in space. In some cases, the spatial structure information may include or be used to generate a point cloud describing the positions of one or more surfaces of an object. Spatial structure information is merely one form of possible image analysis, and other forms known to those skilled in the art can be used according to the methods described herein.

[0041] As an example, Figure 2E The first set of image information, or more specifically, 2D image information 2600, as described above, is depicted by camera 3200 ( Figure 3A The (shown) generation and can be represented Figure 3AThe objects 3530, 3520, 3510 and the storage libraries 3110 / 3120 / 3130 / 3140 / 3150 are described. More specifically, the 2D image information 2600 can be a grayscale or color image and can describe the appearance of the objects 3530, 3520, 3510 and the storage libraries 3110 / 3120 / 3130 / 3140 / 3150 from the viewpoint of the camera 3200. In an embodiment, the 2D image information 2600 can correspond to a single color channel (e.g., red, green, or blue channel) of a color image. The 2D image information 2600 can represent the appearance of the respective camera-facing surfaces of the objects 3530, 3520, 3510 and the storage libraries 3110 / 3120 / 3130 / 3140 / 3150. Figure 2E In the example, 2D image information 2600 may include corresponding portions 2000A / 2000B / 2000C / 2000D / 2550, also referred to as image portions, which represent the corresponding surfaces of an object imaged by camera 3200. For example, portions 2000A / 2000B / 2000C / 2000D may represent an object such as a box, while portion 2550 may represent a portion such as a tray for stacking or arranging boxes. Figure 2E In the 2D image information 2600, each image portion 2000A / 2000B / 2000C / 2000D / 2550 can be an image range, or more specifically, a pixel range (if the image is formed by pixels). Each pixel in the pixel range of the 2D image information 2600 can be characterized as having a position described by a set of coordinates [U,V] and can have values ​​relative to the camera coordinate system or some other coordinate system, such as... Figure 2E and 2F As shown. Each pixel may also have an intensity value, such as a value between 0 and 255 or between 0 and 1023. In another embodiment, each pixel may include any additional information associated with pixels of various formats (e.g., hue, saturation, intensity, CMYK, RGB, etc.).

[0042] As described above, in some embodiments, the image information can be all or part of an image, such as 2D image information 2600. For example, the computing system 1100 can be configured to extract image portion 2000A from the 2D image information 2600 to obtain image information associated only with the corresponding object. For example, the computing system 1100 can be based on the 2D image information 2600 and / or Figure 2FThe 3D image information 2700 shown extracts image portions 2000A by performing an image segmentation operation. In some implementations, the image segmentation operation may include detecting the image location where the physical edges of objects (e.g., the edges of a box) in the 2D image information 2600 appear and using such image locations to identify image portions (e.g., 5610) limited to representing a single object within the camera's field of view (e.g., 3210).

[0043] Figure 2F An example is depicted where the image information is 3D image information 2700. More specifically, 3D image information 2700 may include, for example, depth maps or point clouds indicating corresponding depth values ​​at various locations on one or more surfaces (e.g., top surface or other outer surfaces) of the imaged object / repository. In some implementations, image segmentation operations for extracting image information may involve detecting the image locations where the physical edges of objects (e.g., edges of boxes) in the 3D image information 2700 appear and using such image locations to identify image portions (e.g., 2730) limited to representing a single object within the camera's field of view (e.g., 3000A).

[0044] The corresponding depth values ​​can be relative to the camera 3200 that generates the 3D image information 2700, or they can be relative to some other reference point. In some implementations, the 3D image information 2700 may include a point cloud comprising the corresponding coordinates of various structural positions of objects within the camera's field of view (e.g., 3210). Figure 2F In the example, the point cloud may include a corresponding set of coordinates describing the position on the corresponding surface of the imaged object / repository. The coordinates may be 3D coordinates, such as [XYZ] coordinates, and may have values ​​relative to the camera coordinate system or some other coordinate system. For example, 3D image information 2700 may include a set of positions 27101-2710 indicating the physical position on the surface of the corresponding object (2000D). n The first part 2710 (also referred to as the image part) is the corresponding depth value. Furthermore, the 3D image information 2700 may also include second, third, and fourth parts 2720, 2730, and 2740, respectively. These parts can then indicate that they can be represented by 27201-2720 corresponding to objects 2000B, 2000A, and 2000C, respectively. n 27301-2730 n and 27401-2740 n This represents the corresponding depth values ​​for a set of locations. This includes locations 27501-2750. nThe fifth part 2750 can correspond to object 2550. These figures are merely examples, and any number of objects with corresponding image parts can be used. Similarly, the acquired 3D image information 2700 can, in some cases, be a portion of the first set of 3D image information 2700 generated by the camera. Figure 2E In the example, if the acquired 3D image information 2700 represents Figure 3A If the first object is 3000A, then the 3D image information 2700 can be reduced to only the reference image portion 2710.

[0045] Figure 2G An example 3D image information 3700 of an object storage library is shown. The 3D image information 3700 may include a point cloud having a first image portion 3710 representing the storage library and a second image portion 3830 representing objects located in the storage library. Figure 2G This is merely an example, and the library / object consistent with this embodiment can take many forms. Figure 2G In the image, 3D image information is captured in the "front" view, where camera 3200 has been positioned directly in front of the storage without any significant angle.

[0046] In an embodiment, as part of acquiring image information, an image normalization operation may be performed by the computing system 1100. The image normalization operation may involve transforming an image or portion of an image generated by the camera 3200 to generate a transformed image or portion of an image. For example, if the acquired image information may include 2D image information 2600, 3D image information 2700, or a combination of both, it may undergo an image normalization operation to attempt to alter the image information in terms of viewpoint, object pose, and lighting conditions. Such normalization may be performed to facilitate a more accurate comparison between image information and model (e.g., template) information, as discussed in more detail below. Viewpoint may refer to the pose of an object relative to the camera 3200, and / or the angle at which the camera 3200 is observing the object when the camera 3200 generates an image representing the object.

[0047] For example, image information can be generated during object recognition operations, where the target repository or object is within the camera's field of view. When the target repository or object has a specific pose relative to the camera, camera 3200 can generate image information representing the target repository or object. For example, the target repository or object may have a pose in which its side surfaces are perpendicular to the optical axis of camera 3200. In such an example, the image information generated by camera 3200 can represent a specific viewpoint, such as a side or front view of the target object. In some cases, the camera's optical axis can be configured to be at an angle to the target repository or object. In such cases, the viewpoint can be an angled or tilted viewpoint. In some cases, when camera 3200 generates image information during object recognition operations, the image information can be generated under specific lighting conditions, such as illumination intensity. In such cases, the image information can represent a specific illumination intensity, illumination color, or other lighting conditions.

[0048] In an embodiment, image normalization may involve adjusting an image or portion of an image of a scene generated by a camera to better match the viewpoint and / or lighting conditions associated with information from a model template. The adjustment may involve transforming the image or portion to generate a transformed image that matches at least one of the object pose or lighting conditions associated with the visual descriptive information of the model template.

[0049] Viewpoint adjustment can involve processing, rolling, and / or shifting an image of a scene so that the image represents the same viewpoint as the descriptive information in the model template. For example, processing may include changing the color, contrast, or lighting of an image; rolling the scene may include changing the size, dimensions, or scale of the image; and shifting the image may include changing its position, orientation, or rotation. In example embodiments, processing, rolling, and / or shifting can be used to alter the orientation and / or size of objects in an image of the scene to match or better correspond to the visual descriptive information of the model template. If the model template describes a front view (e.g., a side view) of an object, the image of the scene may be rolled so that it also represents a front view of an object in the scene.

[0050] In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. When the processing circuitry or other hardware components are executing a module or computer-readable instructions, the module and computer-readable instructions can be described as performing various operations or tasks.

[0051] This disclosure relates to the picking and placing operations of a robotic arm within a loading / unloading (transfer) area. The robotic arm can be configured to grasp objects (e.g., boxes, containers, etc.) from a source storage and move the objects to a destination storage. In embodiments, the transfer operation can be assisted by image analysis and trajectory planning. In embodiments, image analysis can be assisted by model template comparison.

[0052] In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or code configured to cause the processing circuitry 1110 to perform one or more functional tasks. When the processing circuitry or other hardware components are executing a module or computer-readable instructions, the module and computer-readable instructions can be described as performing various operations or tasks.

[0053] Figures 3A-3F An example environment in which robot trajectory planning and execution can be used is shown.

[0054] Figure 3A A system 3000 (which may include a computing system 1100 and a robot 3300) is depicted. Figure 1A-1D The system 3000 is configured to operate in an environment similar to embodiments of system 1000 / 1000A / 1000B / 1000C. Robot 3300 may include a robotic arm 3320, an end effector device 3330, and one or more cameras (or optical recognition devices) 3310A / 3310B. The end effector device may include various robotic tools, such as grippers, suction cups, claws, graspers, etc., configured for object interaction. Cameras 3310A / 3310B may be embodiments of camera 1200 and may be configured to generate image information representing the scene within the camera's field of view. System 3000 may be configured to operate in an object transfer area 3100, arranged to facilitate the transfer of objects between storage units such as storage units 3110, 3120, 3130, 3140, and 3150. System 3000 is configured to plan, facilitate, and / or execute the transfer of objects (e.g., boxes, cabinets, manufacturing components, parts, and / or other physical items) between storage units in the object transfer area 3100.

[0055] like Figure 3A As shown, the conveyor-type source storage 3150 can bring objects to the object transfer area 3100, which has one or more destination storage units 3110-3140, which can be shelf-type storage units, cabinet-type storage units, or any other suitable storage units, which can be used for, for example, organizing or temporarily storing objects. An example of a shelf-type storage unit is shown below. Figure 3B , Figure 3C , Figure 3Eand Figure 3F As shown. For example, one or more shelves may include storage shelves, manufacturing shelves, etc. In an embodiment, shelves in the shelf area can be moved into or out of the shelf area. For example, shelves can be lifted and transported into the shelf area by an automated guided vehicle (AGV) 3900, such as... Figure 3C and 3E As shown below, system 3000 may include robot 3300 configured to place objects into or on a destination storage vault, wherein robot 3300 may retrieve the object from a source storage vault (such as a conveyor or another shelf). In some cases, objects on conveyor-type storage vault 3150 may be reached from different object transfer areas (e.g., depalletizing area 3100A), such as... Figure 3D As shown in the diagram. In the depalletizing area, another robot or computing system consistent with the embodiments herein can pick up objects from a pallet (e.g., a source storage) and place them on a conveyor (e.g., a destination storage). Thus, depending on the context, the same physical structure can be used as both a source storage and a destination storage.

[0056] Figure 3B Other aspects of the environment in which the system 3000 can operate are shown. For example... Figure 3B As shown, the object transfer area 3100 may include a conveyor-type storage vault 3150 (e.g., as a source vault), one or more shelf-type storage vaults 3110 and 3130, and one or more cabinet-type storage vaults 3120 and 3140. Objects 3520 and 3510 can reach the object transfer area 3100 via storage vault 3150. In an embodiment, storage vault 3110 (which stores object 3530) and storage vault 3150 can each serve as a source vault 3610, while storage vault 3130 serves as a destination vault 3630. In a further embodiment, depending on the object transfer context, each of these storage vaults can serve as a source vault or a destination vault, or both. A storage vault can serve as a destination vault to receive certain objects, or it can serve as a source vault whereby the same objects can be retrieved and subsequently moved to a new storage vault.

[0057] Figure 3C This illustrates another aspect of the environment in which the system 3000 can operate. Figure 3C An object transfer area 3100 is shown, comprising two transmitter-type repositories 3150 and 3160. In an embodiment, the first transmitter-type repositories 3150 can be used as a source repository, while the second transmitter-type repositories 3160 can be used as a destination repository.

[0058] Figure 3D This illustrates another aspect of the environment in which the system 3000 can operate. Figure 3D Multiple object transfer areas 3100A and 3100B are shown, connected to each other via a conveyor-type storage library 3150. The conveyor-type storage library 3150 can serve as a destination storage library for object transfer area 3100A and a source storage library for object transfer area 3100B. For example, objects can be transported to object transfer area 3100A by a transport truck 3800 or other vehicles or transportation methods (train, shipping container, etc.). Therefore, robot 3300A can be operated to perform, for example, a depalletizing process and transfer objects to conveyor-type storage library 3150 as the destination storage library. Robot 3300B can be operated to transfer objects from object transfer area 3100B. Figure 3D Multiple autonomous ground vehicles (AGVs) 3900A / B / C are further shown. These AGVs 3900A / B / C can be configured to transport one or more of rack-type storage warehouses 3610, 3640, and 3650, thereby allowing them to move between one object transfer area and another.

[0059] Figure 3E Further aspects of the environment in which the system 3000 can operate are shown. Specifically, Figure 3E A shelf-type storage unit 3110 is shown, which contains objects 3530 positioned for transfer by an AGV 3900A.

[0060] Figure 3F Further aspects of the environment in which the system 3000 can operate are shown. Specifically, Figure 3E A rack-type storage unit 3110 is shown, divided into multiple storage shelves 36151, 36152. Each storage shelf 3615 may have an enclosure divided into multiple storage compartments or units 36121, 36122, 36123, and 36124. In the rack-type storage unit 3110, each storage unit 3612 includes one or more unit rails 3617. 1-4 In one embodiment, the compartment or unit 3612 may be provided by track 3617. 1-4 definition.

[0061] This disclosure relates to performing, facilitating, and / or planning the transfer of objects from a source repository to a destination repository. Figure 4 A flowchart is depicted for an example method 4000 for performing, facilitating, and / or planning the transfer of objects from a source repository to a destination repository.

[0062] In an embodiment, method 4000 may be, for example, Figure 2A-2D The computing system 1100 or Figure 3AThe method may be executed by a computing system 1100 of the 3C, or more specifically by at least one processing circuit 1110 of the computing system 1100. In some cases, at least one processing circuit 1100 may execute method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). For example, the instructions may cause the processing circuit 1110 to execute... Figure 2D One or more of the modules shown can perform method 4000. For example, in an embodiment, steps related to object / repository identification, such as operations 4004, 4006, etc., can be performed by object identification module 1121. For example, in an embodiment, steps related to motion and trajectory planning, such as operations 4002, 4008, 4012, etc., can be performed by motion planning module 1129. For example, in an embodiment, steps related to object placement and handling, such as operations 4010, 4014, etc., can be performed by object manipulation planning module 1126. In some embodiments, motion planning module 1129 and object manipulation planning module 1126 can operate collaboratively to define and / or plan trajectories involving motion and object manipulation.

[0063] The steps of method 4000 can be used to implement a specific sequence of robot trajectories for performing a specific task. As a general overview, method 4000 can be operated to cause system 3000 to transfer an object from a source repository to a destination repository. This transfer operation may include the operation of robot 3300 according to a pre-planned trajectory, which is updated and / or refined based on various operations occurring during the operation.

[0064] For example, the robotic system can generate a pre-planned trajectory. This pre-planned trajectory can be generated independently of the storage facility detection operation (described below) or any other operation utilizing image information. Therefore, even without image information, the computing system 3000 can have sufficient information about the possible locations of shelves, conveyors, or other source or destination storage facilities to at least generate an initial, pre-planned trajectory. Although in some cases the pre-planned trajectory may not be accurate enough to ensure that the end effector device will be able to correctly retrieve an object from a specific cell on the shelf and / or place an object into that specific cell, a coarse trajectory may be sufficient to allow the end effector device to approach the shelf.

[0065] For a variety of reasons, pre-planned trajectories may lack sufficient accuracy. In some examples, in the case of a rack storage warehouse, the computing system may lack information about which racks and units are currently occupied. In another example, in the case of a mobile rack storage warehouse transported by an AGV 3900, the AGV 3900 may fail to position the racks at the exact intended location. In yet another example, in the case of a mobile rack storage warehouse transported by an AGV 3900, varying tire inflation levels may cause the rack storage warehouse to be at a different height or angle than expected. In other examples, for instance, regarding conveyor-type storage warehouses, objects may stop in different places or with different orientations / postures / heights / positions. Each of these examples (not exhaustive) represents a situation where the object source or destination is located close to the intended location but not precisely within it. Therefore, it is advantageous to use pre-planned trajectories to approach the source / destination and to modify the trajectory (e.g., based on object recognition operations) to complete object picking or placement operations.

[0066] The pre-planned trajectory can be adjusted based on the results of the storage rack detection operation and the object detection operation, wherein the adjusted trajectory is accurate enough to ensure that the end effector device will be able to correctly retrieve objects from and / or place objects into specific cells of the shelf. The robotic system can use image information to perform the storage rack detection operation, which will be discussed in more detail below. The storage rack detection operation may involve positioning a camera, which can be attached to a robotic arm, in front of the shelf and generating image information representing at least a portion of the shelf. The storage rack detection operation can use the image information to determine more accurate information about the shelf, and this information can be used to control the robot's movement to interact with the shelf. In embodiments, the storage rack detection operation may also use image information captured by cameras not positioned on the robotic arm performing object placement (e.g., a fixed camera positioned throughout the transfer area and / or mobile cameras located on separate and different robotic arms).

[0067] The process of generating an initial pre-planned trajectory and then adjusting a portion of that trajectory offers technical advantages such as improved start time for robot processing, increased reliability and accuracy of pick / place operations, and reduced computational load involved in controlling robot movement. For example, using the pre-planned trajectory allows the robot to begin moving earlier (e.g., to move to a location for reservoir detection), which in turn allows the robot to begin processing objects from the source reservoir earlier. Furthermore, while a portion of the trajectory can be adjusted based on the results of the reservoir detection operation, the remainder of the coarse trajectory can remain unchanged. This remainder can therefore be reused to place additional objects from the source reservoir. This reuse of a portion of the pre-planned trajectory reduces the total computational load involved in robot processing operations such as removing objects from the source reservoir and placing them into one or more destination reservoirs.

[0068] In method 4000, the robot system can be configured and operated to control the robot's movement to retrieve objects from a specific source repository and / or place objects into a specific destination repository. The robot system can be configured to adjust the robot's pre-planned or predicted trajectory, or more specifically, the pre-planned or predicted trajectory of the end effector device, wherein the adjusted trajectory improves the accuracy and reliability of the end effector device picking up objects from and / or placing objects on / in the repository. Adjustment of the pre-planned trajectory can be based on performing repository / object detection, which may involve using image information to more accurately determine the structure of the repository, the location of the repository, and / or the position of the object on the repository.

[0069] In one embodiment, the predicted trajectory enables the robot to move between a source and a destination for picking up an object from the source and transporting it to the destination. In another embodiment, the predicted trajectory may be used as part of the movement. In yet another embodiment, system 3000 may include the ability to execute multiple different predicted trajectories. The movement of robot 3300 within the predicted trajectory may be referred to herein as a movement loop.

[0070] The description of method 4000 refers to movement loop A and movement loop B. The movements in these loops may include movements for also positioning a camera (e.g., a handheld camera attached to a robotic arm or other camera) in front of the storage container, such that the camera can generate image information representing the storage container, where the image information is used for storage container / object detection operations.

[0071] In a type of movement loop (referred to as loop A), such as Figure 5A and 5BAs depicted, the robot 3300 in object transfer area 3100 can retrieve objects from a fixed source store (e.g., a conveyor) and place them on a destination store of a certain type. In movement loop A, the source store is fixed, and no store detection operation is required.

[0072] In another type of movement loop (called loop B), such as Figure 5A and 5C As depicted, robot 3300 can retrieve objects from a mobile source storage (e.g., a movable shelf) and move the objects to a shelf-type destination storage. Therefore, movement cycle B can include additional commands to refine the robot trajectory with respect to the mobile source storage.

[0073] Movement cycles A and B are provided by way of example only. The robot can be controlled to have other movement cycles involving moving objects between other storage units used as sources or destinations. For example, a conveyor-type destination storage unit may result in a rejection area or another object transfer area. A rejection area can be used as a temporary storage area for rejected objects (such as damaged or unrecognized objects by the computing system). For example, the robot can move such objects directly from a conveyor-type source storage unit or from another shelf-type source storage unit to a conveyor-type destination storage unit. In some cases, some or all of the shelves may be moved into or out of the shelf area by one or more AGVs.

[0074] The descriptions of move loops A and B should be understood as descriptions of operations involving reservoir / object detection, which can be used to refine, adjust, and otherwise modify a pre-planned trajectory or create a new trajectory. Move loops requiring different combinations of source and destination reservoirs may require different combinations and / or different orders of the operations disclosed herein without departing from the scope of this disclosure.

[0075] The following describes method 4000 specifically regarding move loop A and move loop B. As described below, move loop A (such as...) Figure 5B (as shown) and movement loop B (as shown) Figure 5C(As shown) Many similar operations can be shared. In the following discussion, operations that may be specific to one or another movement cycle are interpreted as such. However, the steps, processes, and operations of method 4000 can be equivalently applied to other movement cycles. Method 4000 can be executed via a computing system including a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector device and having a camera attached to the robotic arm. As discussed herein, the computing system may also include at least one processing circuit configured to perform method steps for transferring objects from the source storage to the destination storage when the robot is within an object transfer range including a source storage (e.g., a conveyor or a first shelf) and a destination storage (e.g., a second shelf, cabinet, etc.).

[0076] Method 4000 may begin with or otherwise include operation 4002, wherein a computing system (or its processing circuitry) is configured to output a first destination storage proximity command to cause a robotic arm to approach the destination storage in such a manner that the camera is pointed at the destination storage. Figure 5A , 5B As shown in Figure 5C, the computing system outputs a first target storage approach command, which causes the robotic arm to move, for example, via a target storage approach trajectory (e.g., trajectory A1 or B1) to a position where a camera (or multiple cameras) can image the target storage. This operation allows the computing system to acquire images (e.g., image information) of the target storage using the cameras (or multiple cameras) and thus obtain information about the target storage for future robotic arm movements. The target storage approach trajectory can be calculated to begin from any position / configuration where the robotic arm is currently positioned. The first target storage approach command can be calculated based on the first target storage approach trajectory.

[0077] At least one processing circuit may be further configured to calculate a first target storage approach trajectory in a manner not based on any storage detection operation or other operation using an image or image information of the specific storage to which the object will be transported. The first target storage approach trajectory is a sub-trajectory followed by the robotic arm to approach the target storage, and the first target storage approach command is generated based on the first target storage approach trajectory. The first target storage approach trajectory may be calculated based on an estimated position of the target storage location within the object transfer range. In embodiments, this estimated position or the first target storage approach trajectory may be stored in the memory of a computer system, for example, in embodiments that include a non-moving target storage location. In embodiments, the same first target storage approach trajectory may be accessed multiple times from memory without modification, adjustment, or alteration to execute multiple first target storage approach commands at different times across spans of minutes, hours, days, weeks, and / or months.

[0078] In an embodiment, the destination storage unit is a frame forming a set of shelves and units, as described above, and can be moved by vehicles (e.g., AGVs) to different locations within and outside the object transfer range, as well as to locations within the object transfer range. In such a case, at least one processing circuit is also configured to determine the estimated location of the destination storage unit based on the vehicle trajectory followed or intended to be followed by the vehicle.

[0079] Method 4000 may include operation 4004, wherein a computing system (or its processing circuitry) receives image information describing a target storage library. As discussed above, the image information is generated or captured by the camera (or multiple cameras) after a first target storage library proximity command is executed and a camera is positioned for viewing or imaging the target storage library. Receiving the image information may also include any of the methods or techniques described above related to object recognition, such as the generation of spatial structure information (point cloud) regarding the imaged storage library.

[0080] In this embodiment, the robotic arm may be limited in its movement, and therefore may not be able to position the camera at a horizontal or frontal viewpoint when capturing image information, and / or may not be able to position the camera far enough away from the storage unit to capture the entire storage unit. This may be due to the limited accessibility of the robotic arm and / or the tight spacing around the robotic arm. As a result, in some cases, image information may show the shelves in a rotated, tilted, and / or off-angle manner, and / or only a portion of the shelves may be captured. In such cases, the image information may be normalized, as discussed above.

[0081] In an embodiment, method 4000 includes operation 4006, wherein a computing system performs a storage detection operation based on image information, the storage detection operation determining at least one of the following: storage structure information describing the structure of a target storage, storage pose information describing the pose of the target storage, or storage content information describing the presence of one or more objects in the target storage. The storage content information may also include information about the pose and type of objects present in the target storage. As discussed above, the image information of the target storage is obtained by the computing system. The computing system then operates to analyze or process the image information to determine one or more of the storage structure information, storage pose information, and / or storage content information.

[0082] In an embodiment, the computing system can perform a storage detection operation while it is also executing one of the source storage approach command (operation 4008) and the object pickup command (operation 4010). As discussed above, the method discussed herein allows the robotic system to operate faster and with less latency. Therefore, while the robotic arm is executing the sub-trajectory associated with the storage approach command and the object pickup command, the computing system can perform some or all of the calculations associated with the storage detection operation 4006. Therefore, it may not be necessary for the robotic arm to pause while executing the described trajectory.

[0083] Image information may include all or at least a portion of the storage library within the camera's field of view. The computational system can use the image information to more accurately determine the physical structure of the storage library and / or objects on it. The structure can be determined directly from the image information, and / or by comparing the image information generated by the camera with, for example, a model storage library template and / or a model object template.

[0084] The storage database detection operation 4006 may include a variety of optional steps and / or operations to improve system performance. The steps and / or operations for improving system performance are discussed below.

[0085] In this embodiment, the library detection operation can be operated using a predefined library pattern (also referred to herein as a template). The processing circuitry can be configured to determine whether image information satisfies predefined template matching conditions when compared with a predefined library model describing the library structure. The predefined template matching conditions represent a threshold, probability, or degree of matching between the predefined library model and a portion of the image information. The predefined library model can be, for example, an image template, against which the image information can be compared to determine the presence, alignment, and / or orientation of libraries within the second image information. Figure 6 As shown, image information representing the structure of storage 3110 can be compared with one or more predefined storage models 6110 and 6111 to identify a type of storage and thus access the stored storage structure information associated with that storage type. Comparison with predefined storage models allows processing circuitry to improve and accelerate further processing by using predefined (e.g., already processed) template information at points where the image information aligns with the predefined storage model, as discussed further below. In an embodiment, image information can be compared with multiple predefined storage models to identify predefined storage models in which the image information satisfies predefined template matching conditions. Processing circuitry can generate template matching information indicating whether, how much, and in what manner the image information satisfies the predefined template matching conditions. The template matching information obtained during the storage detection operation can be used in subsequent operations, as discussed below, for example, in an object placement operation.

[0086] In an embodiment, the storage detection operation 4006 may include the identification of features that can be used, for example, in template matching. This may involve comparing identified features in the image information with templates for the storage and / or the target object. For example, if the computing system determines that the target object is located in a specific area / cell of the storage, it may attempt to crop the image information to extract the specific portion corresponding to that cell and use the extracted portion for template matching. In some cases, such an area may be determined based on known information about the setup of a particular storage. For example, if the storage is transported to the area by an AGV, the movement of the AGV may have a possible error of, for example, 2 cm. In this example, the AGV may remain directly under the shelf. Therefore, the AGV and the storage may be located at most 2 cm to the left or 2 cm to the right of the estimated location of the storage. This expected error can provide the computing system with an expectation of where the object or cell may appear in the image information, or more specifically, an expectation of the image area in which the object or cell may appear. As a result, the computing system can search for image features it may expect within the image area or portion. If the computing system determines that the AGV has an error in its movement accuracy (e.g., 2 cm), it can expand the image area in which it searches for image features to address this potential error. For example, it can expand the search area by 2 cm in one or more directions. When the camera generates image information, the computing system can also determine the distance between the camera and the shelf based on the error. If the AGV is expected to have a significant error between its intended and actual positions, the computing system can position the camera further away from the shelf and the AGV to ensure that the relevant units on the shelf fall within the camera's field of view.

[0087] The computational system may expect the shelf structure or shelf surface geometry to appear in the image information, such as horizontal layers. The system can search for straight lines that can represent the edges of that layer. If the system expects objects of a specific object type on the shelf (if the shelf is the source shelf), it can search for image features that match a template associated with that object type. More specifically, the system can search for features representing the shelf geometry within an image region or portion. To perform template matching, the system can attempt to overlap or align edges or other recognized features appearing in the image information region or portion with edges in a predefined storage model.

[0088] Image information can describe corresponding depth values ​​for multiple surface locations on a target memory model. These multiple surface locations may include, for example, a first surface location closer to the camera than a second surface location. Processing circuitry can be configured to use the depth values ​​to determine whether the image information satisfies predefined template matching conditions. For example, the matching level or degree between the image information and the predefined memory model can assign more weight to the matching degree of the first location with the predefined memory model than to the matching degree of the second location. That is, a location closer to the camera may have a greater impact on the matching process than a location farther from the camera. The matching level or degree exceeding the defined conditions can be all or part of the predefined template matching conditions.

[0089] In an embodiment, the computing system can assign higher confidence or importance to certain points or locations in the scene captured by the image information, and can give those points more weight when performing template matching, or more generally when determining the physical structure of a shelf or target object. The confidence assigned to a point can be based on its distance from the camera, and / or whether the point is viewed directly by the camera or at an angle. For example, if the camera is angled to the front of the shelf, such that its optical axis is tilted to the front or side of the shelf, some points on the shelf may be closer to the camera relative to the distance between the camera and other points on the shelf. Those points closer to the camera may be assigned a higher confidence level.

[0090] In some embodiments, the confidence levels for various points can be assigned based on the camera angle. For example, the camera may have a more frontal view of the first point and a more tilted or angled view of the second point. That is, the first point and the surrounding portion of the front or side of the shelf may be more perpendicular to the camera or its optical axis, while the second point and the surrounding portion of the front or side of the shelf may be more tilted and less perpendicular to the camera. In these examples, the calculation system may assign increased confidence to the first point based on its more perpendicular view relative to the camera, while the second point may have decreased confidence based on its more tilted view relative to the camera. In some cases, the relative confidence levels based on the camera angle may be reversed.

[0091] When such points with different confidence levels are part of or represent features used in template matching, the computational system can place more weight on points with higher confidence that match certain template features, and less weight on points with lower confidence that match template features.

[0092] In embodiments, the computing system may perform template matching in a hierarchical manner, with multi-level alignment. Multi-level alignment may include the alignment of image information and one or more model templates at different resolutions or structural levels. For example, higher-level alignment may include the alignment of broad features of the image information with one or more model templates, while lower-level alignment may include the alignment of more specific features of the image information with one or more model templates. Although three alignment levels have been discussed, more or fewer alignment levels may be used in various embodiments.

[0093] In an embodiment, the library detection operation may include aligning a broad set of features and a library model based on image information at a first alignment level. The library detection operation may include determining how the image information is aligned with a first library model (first alignment level) that describes the elements of a broad library structure. For example, the first library model may describe the framework of a library structure. As described above, the processing circuitry can identify a library model that matches the image information based on predefined template matching conditions.

[0094] The highest level of alignment, Level 1, can be used for global alignment that may have a higher tolerance for error, while the next level (one or more levels) of alignment can be used for finer alignment that may have a lower tolerance for error. Level 1 alignment may involve a template representing all or most of the structure of the shelving. In some cases, a storage model at the Level 1 alignment level may be a high-level template (first storage model) with a lower amount of detail / granularity. For example, a high-level template may describe features such as the outer beams or outer walls that form the shelving. In high-level alignment, the computational system may attempt to overlap points in the image information with features in this template. The computational system may perform high-level template matching to attempt to align points in the image information with the aforementioned template. High-level template matching can be used to coarsely identify global features of the shelving, such as the outer edges or outer walls of its shell. High-level alignment may result in an error of, for example, a few centimeters between the estimated location of such global features and their actual location (e.g., because it uses a high-level template with less detail), but this error may be low enough to allow the computational system to plan a trajectory with a safety margin that will avoid collisions with these global features of the shelving.

[0095] The processing circuitry can also identify a portion of image information representing a more detailed or finer structural element of the library in a second alignment level, based on the image information being at least partially aligned with a first library model (first alignment level). The identified portion of image information may include a portion of the library of interest. Elements or features of the second alignment level may represent smaller, more detailed aspects of the library. In embodiments, elements represented by the second alignment level may be considered sub-elements of the first alignment level. For example, where the first alignment level may include the framework structure of the library, the second alignment level may include shelves of the library. In examples, the second alignment level may include elements representing specific sectors or regions of the library. In examples, the second alignment level may represent a layer of the library including a sector of interest (e.g., a target sector). Thus, for example, based on the image information being at least partially aligned with a first library model, where a portion of the image information includes a layer of the target library with the target sector.

[0096] The processing circuitry can also determine, after identifying a portion of image information containing the part of interest (e.g., the target sector), how that portion of the image information is at least partially aligned with a second library model, wherein the second library model represents a second alignment level, including more detailed, higher resolution, and / or finer or smaller features about the library. In the example, as discussed above, the second library model may represent a specific sector of the library.

[0097] In an embodiment, the second-level alignment may involve the computing system magnifying and extracting a portion of the image information to focus on a specific portion of the image information corresponding to a specific layer of a shelf, and attempting to match that portion of the image information with a second storage model. The portion selected for scaling by the computing system may be based on global alignment in the first alignment level. That is, global alignment in the highest-level template matching can provide the computing system with a general indication of how the image information roughly aligns with various features of the storage, including an indication of which portion of the image information corresponds to a specific part of interest (such as a layer of a shelf or a specific unit). Global alignment may include some errors, but provides sufficient accuracy for the extracted portion to reliably capture the relevant part of interest. The computing system can then use the extracted portion of the image information to compare with a second template and attempt to align the extracted portion with the second template. This second-level alignment can allow for more accurate determination of the location of a specific part of interest, such as a specific layer or track on a shelf, because it may involve a more precise template, and / or because it may involve a magnified portion of the image information focused on a specific layer or unit of the shelf, which reduces the impact of imaging noise in other parts of the image information. Because operations using end effectors to pick up target objects sitting on a pair of tracks can have low tolerance for errors, higher levels of accuracy may be required to avoid collisions between the end effector and the tracks.

[0098] In an embodiment, the hierarchical structure may include a third-level alignment for template matching. The third-level alignment may involve a third repository model that focuses on a more detailed and / or more specific structure than the first or second repository model, such as, for example, a track structure in a shelf, and may describe the track structure more precisely than the second template. That is, the third-level alignment can be used to identify and align with specific features identified within the portion of interest of the second-level alignment.

[0099] In embodiments involving shelf-type storage warehouses, the computing system can extract a portion of image information corresponding to the track structure and use the track structure to compare with a third template. For example, the computing system can use a second storage warehouse model to determine which portion of the image information corresponds to a pair of guide rails on the shelf. The computing system can extract a portion of image information corresponding to the rails and compare it with a third storage warehouse model to obtain a more accurate determination of the physical structure / layout and physical location of the pair of rails on which the target object is located (or on which the target object is to be placed)

[0100] In an embodiment, after establishing storage structure information, for example, through template matching, storage orientation information can be determined. For example, the computing system can determine whether the storage is positioned at an angle or tilted, rather than, for example, in an upright orientation. The computing system can identify features expected to be horizontal (e.g., shelves) and determine the angle at which these features are arranged relative to the horizontal.

[0101] Once the storage facility's structural and orientation information is determined (e.g., once shelf detection is established), the computing system can identify target objects on the storage facility (e.g., to determine storage facility content information). The computing system can be configured to verify whether the target object captured by the image information belongs to or is associated with a specified object type. For example, the computing system can use the image information to measure the object's dimensions or determine whether the object has expected physical characteristics, such as a lid.

[0102] The computational system can take into account the alignment of the shelves, such as rotation or angle, to provide more axis-aligned detection results for trajectory planning operations. The storage detection operation 4006 may require providing the motion planning operation with an accurate representation of the object, such as the kind used to update pre-planned trajectories.

[0103] In an embodiment, the processing circuitry may be configured to detect multiple edges of a storage library that divide the storage library into multiple extents from image information. The processing circuitry may also be configured to determine multiple imaginary bounding boxes, wherein each imaginary bounding box within the multiple imaginary bounding boxes encloses one of the multiple edges, or encloses one of the multiple extents.

[0104] In this embodiment, the computational system can generate a hypothetical bounding box around the identified target object. The object may reside entirely within the bounding box. Therefore, if motion planning can avoid collisions between the end effector and the bounding box, it will also likely ensure that the end effector does not collide with the object. The bounding box simplifies the robot's approach to and grasping of the object, and allows movement across the bounding box to move even closer to the object.

[0105] In an embodiment, the storage warehouse detection operation 4006 can generate various bounding boxes as simplified representations of the geometry of a shelf or a portion of a shelf. The bounding boxes may include one representing a target unit or object, where the bounding box represents space that the end effector device can approach and traverse, while other bounding boxes may represent areas that the end effector device should avoid (e.g., areas that may be occupied) to reduce the risk of collision.

[0106] Bounding boxes can collectively occupy each extent of a shelf. That is, a bounding box can occupy the entire space occupied by the shelf. Bounding boxes can identify spaces that end effector devices or robotic arms should avoid, and may be a simpler way to identify spaces to avoid than providing the entire geometry of the shelf, which can make motion planning overly complex and / or increase computation time and resources.

[0107] In an embodiment, the storage detection operation 4006 can determine whether the target object is at an angle relative to the horizontal orientation, as part of the storage content information. This can occur, for example, when an object pivots from one of the tracks, resulting in the target object being in a tilted and / or angled posture relative to the horizontal orientation. If the gripper attempts to pick up the object from its bottom, the tilted orientation may cause a portion of the end effector device to collide with a portion of the object. In an embodiment, the computing system can determine whether an object has fallen into the space below the target object, as part of the storage content information. Such an object may be located between the target object and the track on which the target object is positioned, and may obstruct the path of at least a portion of the end effector device.

[0108] In embodiments, image noise can affect the ability of a computational system to accurately detect edges. To address the problems caused by image noise, the computational system can use statistical elements, such as histograms of pixel intensity or depth values, to determine the location of edges or features in the image information. As discussed above, image information can be captured in a 2D image, a depth map, or a point cloud indicating the coordinates and intensities of pixels representing a target repository or object. The presence of image noise can make edge determination difficult, where pixel intensity may lack strong and clear contrast lines. For example, noise may cause pixel intensity to gradually change as an edge is approached. Using histograms or other statistical techniques can allow for accurate edge detection in noisy images. For example, the computational system can determine physical edges to correspond to edges or locations where the histogram indicates inflection points or peaks in pixel intensity.

[0109] In an embodiment, the library detection operation 4006 can compensate for image noise by inferring lost image information. For example, if the computing system is using a 2D image or point cloud representing a library, the 2D image or point cloud may have one or more missing portions due to noise. The library detection operation 4006 can be configured to infer the lost information by closing or filling gaps (e.g., by interpolation or other means).

[0110] In embodiments, the storage detection operation 4006 can be used to overcome challenges associated with a compact or crowded layout around a storage unit within the robotic arm or object transfer range, or challenges associated with limitations on the robotic arm's accessibility to poses. For example, within the object transfer range, a first storage unit may be positioned between the robotic arm and a second storage unit, making it difficult for the robotic arm's camera to reach a suitable position for imaging the second storage unit. These limitations may prevent the camera from generating image information with optimal orientation or optimal coverage. When capturing image information from a non-optimal angle or viewpoint, viewpoint adjustment techniques discussed herein can be used during the storage detection operation 4006.

[0111] The storage detection operation 4006 can be used to refine the computational system's understanding of the storage geometry, which can be used to guide the robot. For example, a more detailed storage model or map can allow the system to guide the robot around / away from collisions with the storage. In an embodiment, the storage detection operation 4006 can calculate a buffer zone or volume around the storage to prevent collisions between the robotic arm and the storage.

[0112] In an embodiment, the storage detection operation 4006 may further include determining, based on image information, whether a target sector in the target storage is unoccupied or occupied. The target sector, such as the sector in the target storage to which the object is to be transported, may be determined by processing circuitry. The target sector may be selected based on image information, based on determined storage structure information, predefined according to an operation plan, and / or determined via other means. During the storage detection operation, the processing circuitry may be configured to determine whether the target sector is occupied. If occupied, the processing circuitry may determine an alternative target sector during the storage detection operation 4006. The storage detection operation 4006 may also detect whether the object deviates from the position where the end effector typically picks up the object. Such detection can be used to determine whether the object can be picked up, or whether the object's position makes robot pickup infeasible. In such cases, the storage detection operation 4006 may indicate that a pickable object has not yet been detected.

[0113] In an embodiment, method 4000 includes operation 4008, wherein the computing system outputs a source storage proximity command to bring a robotic arm closer to the source storage. The source storage proximity command may be provided after the camera has generated image information describing the target storage. When executed, the source storage proximity command can bring the robotic arm closer to the source storage, for example, as... Figure 5B As shown, the approach trajectory follows the source repository, such as A2, B2a / B2b. The source repository approach trajectory can be calculated to begin at the endpoint of the first destination repository trajectory. The source repository approach command can be calculated based on the source repository approach trajectory.

[0114] In an embodiment, the source storage approach command can be generated as a two-part trajectory comprising a first trajectory portion and a second trajectory portion. When executed by the robot, this two-part trajectory causes the robot arm's movement toward the source storage to pause during a first time period after the first trajectory portion (e.g., at the end of B2a), and resume movement during a subsequent second time period (e.g., during B2b) to execute the second trajectory portion. During the pause, for example, during the first time period, a camera can be controlled to capture a second set of image information about the source storage. As discussed below, this second image information can be used during a second storage detection operation. As discussed above, the B-movement loop can be used at any time when the location, structure, pose, contents, or other details of the source storage are uncertain.

[0115] In an embodiment, method 4000 includes operation 4009, in which the computing system performs a second storage detection operation. The second storage detection operation may be similar to operation 4006, which is considered to be the first storage detection operation 4006. The first storage detection operation 4006 employs (first) image information captured by a camera describing a target storage. In the second storage detection operation 4009, the computing system may use second image information captured by a camera on the source storage after executing a source storage proximity command. The second image information may include information describing the source storage and may be used to determine at least one of the following: source storage structure information describing the structure of the source storage, source storage pose information describing the pose of the source storage, or source storage content information describing the presence of one or more objects in the target storage. The second image information may include information describing the object type or pose associated with the object when the object to be picked is in the source storage. In some embodiments, the second image information, object pose information, and / or source storage information may be used to generate an object pickup trajectory, as discussed below. In environments where the source repository is not fixed, performing a second repository detection operation 4009 may be advantageous.

[0116] In an embodiment, the second reservoir detection operation can be performed during a first time period (e.g., a pause after the source reservoir approaches the first portion of trajectory B2a) or during a second time period (e.g., during the second portion of the source reservoir approaches trajectory B2b). In this way, the computing system can save time by performing processing that can be used to modify the object picking trajectory using the second reservoir detection operation, while the initial phase of the object picking trajectory is being performed.

[0117] The second storage detection operation 4009 may include any or all of the steps, methods, and features of the first storage detection operation 4006, as described above, in any combination.

[0118] In an embodiment, method 4000 includes operation 4010, wherein the computing system outputs an object pickup command. The object pickup command causes the end effector device of the robotic arm to pick up an object from a source storage container. The object pickup command can be executed after a source storage container proximity command has been executed and the robotic arm is in a position to pick up or grasp an object from the source storage container. Figure 5B As shown, when executing an object picking command, the robotic arm can follow an object picking trajectory, such as trajectory A3 or B3. The object picking trajectory is the path followed by the robotic arm when picking up or grasping an object, and can be calculated, for example, starting at an endpoint near the source storage. The object picking command can be calculated based on the object picking trajectory.

[0119] In an embodiment, at least one processing circuit may also be configured to calculate a source storage approach trajectory (in operation 4008) and an object pickup trajectory (in operation 4010) based on a predefined position of the source storage fixed within the transfer range. Such a method may occur when the source storage is fixed to a predefined position within the transfer range. The predefined position of the source storage may be stored, for example, in a memory device associated with a computer system.

[0120] In this embodiment, the computing system can use information generated during the second storage detection operation to calculate the object pickup trajectory, instead of using a predefined location. For example, when the source storage is not fixed within the transfer range, using second image information to calculate the object pickup trajectory can offer advantages in accuracy.

[0121] As discussed above, in some embodiments, the computing system may capture second image information from the source store before executing the object pickup command. In such embodiments, the second image information (including information about the object pose or the source store) can be used to generate an object pickup trajectory for generating the object pickup command. As discussed above, the second image information may be obtained during a pause in movement loop B, i.e., after the first portion of the source store approach trajectory has been executed. In such embodiments, the second image information can be used to modify or change a portion of the object pickup trajectory from the initially pre-planned trajectory, or to generate a new trajectory as needed. The modified or changed portion of the trajectory may be the ending portion of the object pickup trajectory, which includes operations for picking up the object. Thus, the first portion of the initially pre-planned trajectory for object pickup can be executed according to the initial pre-plan, while the second portion of the initially pre-planned trajectory for object pickup, which occurs after the first portion, can be modified, changed, and / or altered according to the second image information. In embodiments, the entire initially pre-planned object pickup trajectory can be modified, changed, or otherwise adjusted according to the second store detection operation.

[0122] In one embodiment, method 4000 includes operation 4012, wherein the computing system outputs a second destination storage proximity command after an object pickup command has been executed, causing the robotic arm to approach the destination storage a second time. In this case, the robotic arm approaches the second destination storage while carrying / transporting an object picked up from the source storage. Figure 5B and 5C As shown, the robotic arm can follow a second destination storage approach trajectory, such as trajectories A4 and B4, which are used to approach the destination storage when executing a second destination storage approach command. The second destination approach trajectory can be calculated to begin at the endpoint of the object pickup trajectory. The second destination storage approach command can be calculated based on the second destination storage approach trajectory.

[0123] In an embodiment, the second destination storage location approach command can be generated without using image information generated during the storage location detection operation, rather than based on the storage location detection operation and / or on any image information obtained from a specific destination storage location. For example, the second destination storage location approach command can be generated based on information of the same or similar type as that used to generate the first destination storage location command (e.g., estimated location of the destination storage location within the object transfer range, vehicle trajectory information indicating the location of the destination storage location, stored trajectories, etc.).

[0124] In an embodiment, method 4000 includes operation 4014, wherein the computing system outputs an object placement command that causes the end effector device to place an object in a target storage container. The object placement command is executed after a second target storage container approach command has brought the robotic arm close to the target storage container. The object placement command is generated based on the result of a storage container detection operation, wherein the result includes at least one of storage container structure information, storage container pose information, or storage container content information. Therefore, the object placement command causes the robotic arm to follow an object placement trajectory, for example, as... Figure 5B and 5C As shown in A5 and B5, the object is placed in the target storage based on the analysis of image information captured by the camera after the first approach to the target storage. For example, the object placement trajectory can begin at the endpoint of the second target storage approach trajectory.

[0125] The generation of object placement commands may be subject to several additional constraints and / or operations. For example, in one embodiment, object placement commands may be generated based on template matching information generated during the library detection operation. In a further embodiment, object placement commands may be generated based on a portion of image information being at least partially aligned with a second library model, as discussed above regarding object library detection commands and hierarchy alignment. In a further embodiment, object placement commands may be generated based on multiple bounding boxes, as discussed above regarding object library detection commands.

[0126] In an embodiment, at least one processing circuit is configured to calculate an object placement trajectory [e.g., A5 or B5] based on the result of a storage detection operation to generate an object placement command while at least one of a source storage proximity command, an object picking command, or a second destination storage proximity command is being executed, such as trajectories A2 / B2a / B2b, A3 / B3, and A4 / B4. Thus, as previously described, the processing circuit can perform processing and / or calculations during the time period when the robotic arm is in motion to improve efficiency.

[0127] As discussed above regarding method 4000, various sub-trajectories can be part of an initially pre-planned trajectory. Various sub-trajectories can be adjusted, for example, modified or varied from the initially pre-planned trajectory. Trajectory adjustment can involve changing only a portion of the trajectory, for example, while retaining other parts of the trajectory. Trajectory adjustment can involve changing the trajectory, for example, by replacing the entire trajectory with an alternative trajectory. For example, in embodiments, the initially pre-planned trajectory discussed above can be generated based on various estimation information. In some cases, the estimation information can include a model of the source or destination storage facility (e.g., a CAD model) and / or a model of the target object. For example, the model of the destination storage facility (e.g., a shelf) can identify how many layers are in the shelf and how many units are in each layer, or more generally, it can identify the placement of objects in the shelf and / or the height of the shelf. The model of the target object can, for example, describe the shape and / or size (e.g., dimensions) of the target object.

[0128] In some cases, the estimated information in calculating the initial pre-planned trajectory may include information about the planned route of the AGV transporting the rack-type storage warehouse to the rack area within the transfer range. For example, the AGV may be programmed to follow a specific route (e.g., based on ground markings). This known route of the AGV can indicate where the AGV stops. The indicated location of the AGV can provide an estimate of the location of the racks in the rack area. In some implementations, the calculation system can track which units (one or more) of the rack have objects placed in them, and this information can be used to determine whether a particular unit in the rack is empty or occupied.

[0129] The computing system can determine or compute an initial pre-planned trajectory in various ways. For example, in one embodiment, the computing system can generate an initial pre-planned trajectory as part of an object in a moving shelving area. The computing system can cache or otherwise store the initial pre-planned trajectory so that it can be reused for other objects in the moving shelving area. In one embodiment, the computing system can determine the number of units / compartments in the shelving and the location of the compartments based on a model of the shelving. The computing system can use the location of virtual target objects and virtual objects to generate a coarse trajectory.

[0130] In embodiments, after the computing system has generated an initial pre-planned trajectory in advance, for example, based on initial information describing the location (fixed or movable) of the source and destination storage bins, the computing system can adjust the initial pre-planned trajectory based on further operations. For example, as a result of storage bin detection operation 4006, the computing system can adjust the initial pre-planned trajectory. In some implementations, the initial pre-planned trajectory can be generated based on initial information that lacks sufficient accuracy in describing the source or destination storage bin and / or in describing objects arranged in the source storage bin. For example, these potential inaccuracies might cause the end effector device to move to the wrong location on the shelf, even if it normally moves to the storage bin, and / or have the wrong orientation for picking up objects from the shelf and / or placing objects on the storage bin (e.g., incorrect roll, pitch, or yaw). Therefore, the computing system can update the initial pre-planned trajectory, especially the portion of the initial pre-planned trajectory that controls the precise movement of the end effector device to have close interaction with the storage bin or objects thereon.

[0131] For example, for the trajectory described above for movement cycle A, such as Figure 5BAs shown, the computing system can adjust the initially pre-planned trajectory to generate an object pickup trajectory, such as A5, based on the analysis of image information received after the first destination storage approach command and storage detection operation. Furthermore, the computing system may not need to update the initially pre-planned trajectory to reach the first destination storage approach trajectory, such as A1, because this sub-trajectory is used to position the camera typically in front of the storage, or more specifically in front of a designated location within the storage (e.g., a unit of the destination shelf), and the initially pre-planned trajectory provides sufficient accuracy for such a purpose. The computing system can have, for example, information about the position of the AGV used to transport the destination shelf to the shelf area (with an error margin of, for example, 2-3 cm), and can use this information to determine the approximate location of the destination shelf. This approximate location may be accurate enough because generating image information about the destination shelf can be accomplished by typically placing the camera in front of the destination shelf. The computing system may not need to place the camera in a specific location or orientation. Instead, as long as its position falls within a threshold range of the location, the camera can capture relevant information about the destination storage.

[0132] In this embodiment, the source storage approach trajectory and object pickup trajectory (e.g., A2 and A3) may not require adjustment because the location of the source storage and the object's position on the source storage can be fixed or otherwise constrained, such that the location of the source storage and the object's position on the source storage can be known by the computing system with a sufficiently high level of accuracy. For example, the source storage can be a conveyor fixed to the ground, making its location fixed. Furthermore, the conveyor can have barriers to prevent objects from moving along the conveyor. The barriers can have known positions that can be used by the computing system to accurately determine the object's position on the conveyor. Therefore, the computing system may not need to perform trajectory adjustment or conveyor / object detection on the conveyor.

[0133] In an embodiment, the computing system can make a second destination approach trajectory, such as A4, remain unchanged from the initially pre-planned trajectory. For example, the second destination approach trajectory can be used to make the end effector device approach and move closer to the destination shelf after it has picked up the target object from the conveyor. Sub-trajectory A4 can provide a sufficient level of accuracy for this approach.

[0134] In an embodiment, the computing system may adjust the initially pre-planned trajectory to reach the object placement trajectory (e.g., trajectory A5) based on the results of the storage facility detection operation. The object placement trajectory is used for close interaction with the target storage facility and may require a high level of precision to ensure that the end effector device correctly places the target object into a specific part of the storage facility, such as a designated unit on a shelf.

[0135] As mentioned above, the initially pre-planned trajectory may be based on estimations lacking sufficient accuracy in describing the destination shelf, and therefore may not provide enough precision to correctly place the object into the shelf. Therefore, the computational system can adjust the initially pre-planned trajectory based on the storage bin detection operation to achieve an object placement trajectory, such that the adjusted trajectory achieves the level of accuracy required to correctly place the object into the destination shelf. As an example, the object placement trajectory may allow the end effector device to have different endpoint positions compared to the corresponding portion of the initially pre-planned trajectory, and / or may cause the end effector device to have a different orientation at the endpoint positions compared to the corresponding portion of the initially pre-planned trajectory. Therefore, in an embodiment, when the robot moves, it can follow the initially pre-planned trajectory as a first destination storage bin approach trajectory, a source storage bin approach trajectory, an object pickup trajectory, and a second destination approach trajectory, and then execute the object placement trajectory adjusted compared to the corresponding portion of the initially pre-planned trajectory.

[0136] Additional discussion of various embodiments:

[0137] Example 1 includes a computing system, a method executed by the computing system, or a non-transitory computer-readable medium including instructions for implementing the method. The computing system includes a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector device, and a camera attached to the robotic arm, and at least one processing circuit configured to perform the following operations to transfer an object from a source storage to a destination storage when the robot is within an object transfer range including a source storage and a destination storage. The method may include outputting a first destination storage proximity command to cause the robotic arm to approach the destination storage in such a manner that the camera is pointed at the destination storage; receiving image information describing the destination storage, wherein the image information is generated by the camera after the first destination storage proximity command is executed; performing a storage detection operation based on the image information, the storage detection operation determining at least one of the following: storage structure information describing the structure of the destination storage, storage pose information describing the pose of the destination storage, or storage content information describing the presence of one or more objects in the destination storage; and performing the following operations after the camera has generated image information describing the destination storage. After the information is received, a source storage approach command is output to bring the robotic arm close to the source storage; after the source storage approach command is executed, an object pick-up command is output to bring the end effector device to pick up an object from the source storage; after the object pick-up command is executed, a second destination storage approach command is output to bring the robotic arm close to the destination storage; and after the second destination storage approach command is executed, an object placement command is output to bring the end effector device to place the object in the destination storage, wherein the object placement command is generated based on the result of the storage detection operation, wherein the result includes at least one of storage structure information, storage posture information, or storage content information.

[0138] Example 2 includes all the features of Example 1, wherein at least one processing circuit is configured to perform a storage library detection operation based on image information while at least one of a source storage library proximity command or an object picking command is being executed.

[0139] Example 3 includes all the features of any one of Examples 1 or 2, wherein at least one processing circuit is configured to calculate an object placement trajectory based on the result of a storage detection operation while at least one of a source storage proximity command, an object picking command, or a second destination storage proximity command is being executed, wherein the object placement trajectory is the trajectory followed by the end effector device to place an object in the destination storage, and wherein the object placement command is generated based on the object placement trajectory.

[0140] Example 4 includes the features of any one of Examples 1-3, wherein at least one processing circuit is configured to calculate a first target storage approach trajectory in a manner not based on the result of a storage detection operation, wherein the first target storage approach trajectory is for a robotic arm to follow in order to approach the target storage, and wherein the first target storage approach command is generated based on the first target storage approach trajectory.

[0141] Example 5 includes the features of any one of Examples 1-4, wherein at least one processing circuit is configured to determine the location of the destination storage unit based on a vehicle trajectory followed or to be followed by a vehicle when the destination storage unit is a frame forming a set of shelves and is movable in and out of the transfer range by a vehicle, wherein the location of the destination storage unit is an estimated location of the destination storage unit within the transfer range.

[0142] Example 6 includes the features of any one of Examples 1-5, wherein at least one processing circuit is configured to calculate a second target storage approach trajectory to be followed after an object picking command is executed, wherein the second target storage approach trajectory is calculated in a manner not based on the result of a storage detection operation, and wherein the object placement trajectory is calculated to begin at an endpoint of the second target storage approach trajectory.

[0143] Example 7 includes the features of any one of Examples 1-6, wherein at least one processing circuit is configured to calculate a source storage approach trajectory and an object pickup trajectory based on the predefined position of the source storage being fixed in the transfer range when the source storage is fixed in the transfer range, wherein the source storage approach trajectory is the trajectory followed by the robotic arm as it approaches the source storage, and the source storage approach command is generated based on the source storage approach trajectory, and wherein the object pickup trajectory is the trajectory followed by the end effector device as it picks up an object from the source storage, and the object pickup command is generated based on the object pickup trajectory.

[0144] Example 8 includes features of any one of Examples 1-7, wherein the storage detection operation is a first storage detection operation, and the image information is a first set of image information, wherein at least one processing circuit is configured to perform the following operations when the source storage is not fixed to the transfer range: receiving a second set of image information describing the source storage, wherein the second set of image information is generated by the camera during or after the execution of the source storage proximity command; performing a second storage detection operation based on the second set of image information; and calculating an object pickup trajectory based on the result of the second storage detection operation, wherein the object pickup trajectory is the trajectory followed by the end effector device to pick up an object from the source storage, and the object pickup command is generated based on the object pickup trajectory.

[0145] Example 9 includes the features of any one of Examples 1-8, wherein at least one processing circuit is configured to perform a second storage library detection operation by determining the object pose associated with the object based on a second set of image information when the object is in the source storage library, wherein the object picking command is generated based on the object pose.

[0146] Example 10 includes the features of any one of Examples 1-9, wherein at least one processing circuit is configured to: generate a source storage approach command in such a manner that when performed by a robot, the movement of the robotic arm toward the source storage is paused during a first time period and resumed during a subsequent second time period; generate a second set of image information by a camera during the first time period; and perform a second storage detection operation during the second time period.

[0147] Example 11 includes the features of any one of Examples 1-10, wherein at least one processing circuit is configured to perform a storage detection operation by determining that image information is at least partially aligned with a predefined storage model describing the storage structure, wherein an object placement command is generated based on the image information being at least partially aligned with the predefined storage model.

[0148] Example 12 includes the features of any one of Examples 1-11, wherein at least one processing circuit is configured to perform a storage detection operation by determining whether image information satisfies a predefined template matching condition when compared with a predefined storage model describing the storage structure, wherein at least one processing circuit is configured to generate an object placement command based on the predefined storage model in response to determining that the image information satisfies the predefined template matching condition.

[0149] Example 13 includes the features of any one of Examples 1-12, wherein the image information describes the corresponding depth values ​​of a plurality of surface locations on the target storage, and wherein at least one processing circuit is configured to determine the matching level between the image information and the predefined storage model in such a way that when the plurality of surface locations include a first surface location that is closer to the camera than a second surface location, the matching degree of the first location with the predefined storage model is assigned more weight than the matching degree of the second location with the predefined storage model.

[0150] Example 14 includes the features of any one of Examples 1-13, wherein at least one processing circuit is configured to determine that an object in the target storage will be placed in the target sector, and is configured to perform a storage detection operation by determining whether the target sector is unoccupied based on image information.

[0151] Example 15 includes the features of any one of Examples 1-14, wherein at least one processing circuit is configured to determine that an object in a target storage library will be placed in a target sector, and is configured to perform a storage library detection operation by: determining image information in a manner in which it is at least partially aligned with a first storage library model describing the storage library structure; identifying a portion of the image information, representing at least one layer of the target storage library including the target sector, based on the image information in a manner in which it is at least partially aligned with the first storage library model; and determining that the portion of the image information in a manner in which it is at least partially aligned with a second storage library model describing at least one layer of the storage library structure, wherein an object placement command is generated based on the portion of the image information in a manner in which it is at least partially aligned with the second storage library model.

[0152] Example 16 includes the features of any one of Examples 1-15, wherein the first storage model describes the framework of the storage structure, such that at least one processing circuit is configured to determine how image information is aligned with the framework described by the first storage model.

[0153] Example 17 includes the features of any one of Examples 1-16, wherein at least one processing circuit is configured to: detect from image information multiple edges of a target library dividing the target library into multiple ranges; determine multiple imaginary bounding boxes, wherein each of the multiple imaginary bounding boxes surrounds one of the multiple edges, or surrounds one of the multiple ranges, wherein an object placement command is generated based on the multiple bounding boxes.

[0154] It will be apparent to those skilled in the art that other suitable modifications and adaptations can be made to the methods and applications described herein without departing from the scope of any of the embodiments. The above embodiments are illustrative examples and should not be construed as limiting the invention to these specific embodiments. It should be understood that the various embodiments disclosed herein can be combined in combinations different from those specifically presented in the specification and drawings. It should also be understood that, depending on the example, certain actions or events of any process or method described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., all described actions or events may not be necessary for performing the method or process). Furthermore, although for clarity some features of the embodiments herein are described as being performed by a single component, module, or unit, it should be understood that the features and functions described herein can be performed by any combination of components, units, or modules. Therefore, those skilled in the art can influence various changes and modifications without departing from the spirit or scope of the invention as defined by the appended claims.

Claims

1. A computing system, comprising: A communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector and a camera attached to the robotic arm; At least one processing circuit is configured to, when the robot is within an object transfer range including a source storage and a destination storage, perform the following operations to transfer an object from the source storage to the destination storage: When the end effector is not transporting an object, it outputs a first destination storage proximity command to cause the robotic arm to approach the predetermined position of the destination storage in such a way that the camera is pointed at the destination storage; Receive image information describing the destination storage, wherein the image information is generated by the camera after a first destination storage proximity command is executed; Based on the image information, a storage library detection operation is performed, wherein the storage library detection operation calculates and / or verifies and / or updates at least one of the following: storage library structure information for describing the structure of the target storage library, or storage library pose information for describing the pose of the target storage library, or storage library content information for describing whether one or more objects exist in the target storage library. After the camera has generated the image information describing the target storage, a source storage proximity command is output to bring the robotic arm closer to the source storage. Output an object pick-up command to cause the end effector device to pick up the object from the source storage; After the object picking command is executed, a second destination storage proximity command is output to bring the robotic arm close to the destination storage. as well as After the second destination storage proximity command is executed, an object placement command is output to cause the end effector device to place the object in the destination storage, wherein the object placement command is generated based on at least one of the storage structure information or the storage posture information.

2. The computing system of claim 1, wherein the at least one processing circuit is configured to perform the storage detection operation based on the image information while at least one of the source storage proximity command or the object picking command is being executed.

3. The computing system of claim 1, wherein the at least one processing circuit is configured to calculate an object placement trajectory based on the result of the storage detection operation while at least one of the source storage proximity command, the object picking command, or the second destination storage proximity command is being executed, wherein the object placement trajectory is the trajectory followed by the end effector device to place the object in the destination storage, and wherein the object placement command is generated based on the object placement trajectory.

4. The computing system of claim 3, wherein the at least one processing circuit is configured to calculate a first target storage approach trajectory in a manner not based on the result of the storage detection operation, wherein the first target storage approach trajectory is for the robotic arm to follow in order to approach the target storage, and wherein the first target storage approach command is generated based on the first target storage approach trajectory.

5. The computing system of claim 4, wherein the at least one processing circuit is configured to calculate the location of the destination storage facility based on a vehicle trajectory followed or to be followed by a vehicle when the destination storage facility is a frame forming a set of shelves and is movable in and out of the object transfer range by a vehicle, wherein the location of the destination storage facility is an estimated location of the destination storage facility within the object transfer range.

6. The computing system of claim 4, wherein the at least one processing circuit is configured to, after the object picking command is executed, calculate a second destination storage approach trajectory to be followed. The second target storage approach trajectory is calculated in a manner not based on the results of the storage detection operation, and The object placement trajectory is calculated to begin near the endpoint of the trajectory at the second target storage location.

7. The computing system of claim 1, wherein the at least one processing circuit is configured to calculate a source storage approach trajectory and an object pickup trajectory based on the predefined position in which the source storage is fixed within the object transfer range when the source storage is fixed to a predefined position within the object transfer range. The source storage approach trajectory is the trajectory followed by the robotic arm as it approaches the source storage, and the source storage approach command is generated based on the source storage approach trajectory. The object picking trajectory is the trajectory followed by the end effector device when picking up the object from the source storage, and the object picking instruction is generated based on the object picking trajectory.

8. The computing system of claim 1, wherein the storage detection operation is a first storage detection operation, and the image information is a first set of image information. The at least one of the processing circuits is configured to perform the following operations when the source storage is not fixed to the object transfer range: Receive a second set of image information describing the source storage, wherein the second set of image information is generated by the camera during the execution of the source storage proximity command; Perform a second storage database detection operation based on the second set of image information; as well as At least a portion of the object picking trajectory is calculated based on the result of the second storage detection operation, wherein the object picking trajectory is the trajectory followed by the end effector device when picking up the object from the source storage, and the object picking command is generated based on the object picking trajectory.

9. The computing system of claim 8, wherein the at least one processing circuit is configured to perform a second storage library detection operation by determining an object pose associated with the object based on a second set of image information when the object is in the source storage library, wherein the object picking command is generated based on the object pose.

10. The computing system of claim 8, wherein the at least one processing circuit is configured to: The source storage approach command is generated in such a manner that, when executed by the robot, the movement of the robotic arm toward the source storage is paused during a first time period and resumed during a subsequent second time period. The second set of image information is generated by the camera during the first time period; and The second storage check operation is performed during the second time period.

11. The computing system of claim 1, wherein the at least one processing circuit is configured to perform the memory detection operation by determining how the image information is at least partially aligned with a predefined memory model describing the memory structure. The object placement command is generated based on the image information by aligning it at least partially with the predefined library model.

12. The computing system of claim 1, wherein the at least one processing circuit is configured to perform the memory detection operation by determining whether the image information satisfies a predefined template matching condition when compared with a predefined memory model describing the memory structure. The at least one of the processing circuits is configured to generate the object placement command based on the predefined library model in response to determining that the image information satisfies the predefined template matching condition.

13. The computing system of claim 12, wherein the image information describes corresponding depth values ​​for a plurality of surface locations on the target storage, and The at least one of the processing circuits is configured to determine the matching level between the image information and the predefined memory model by assigning more weight to the degree of matching between the first position and the predefined memory model than to the degree of matching between the second position and the predefined memory model when the plurality of surface positions include a first surface position that is closer to the camera than a second surface position.

14. The computing system of claim 1, wherein the at least one processing circuit is configured to determine that the object in the target storage will be placed in the target sector, and is configured to perform the storage detection operation by determining whether the target sector is unoccupied based on the image information.

15. The computing system of claim 1, wherein the at least one processing circuit is configured to determine that the object in the destination repository will be placed in the destination sector, and is configured to perform the repository detection operation by: The image information is determined by how it is at least partially aligned with a first repository model describing the repository structure; Based on the image information, a portion of the image information representing at least one layer of the target library including the target sector is identified by means of its at least partial alignment with the first library model; The portion of the image information is determined by aligning it at least partially with a second library model describing at least one layer of the library structure. as well as The object placement command is generated based on a portion of the image information, which is at least partially aligned with the second storage model.

16. The computing system of claim 15, wherein the first storage model describes the framework of the storage structure, such that the at least one processing circuit is configured to determine how the image information is aligned with the framework described by the first storage model.

17. The computing system of claim 16, wherein the at least one processing circuit is configured to: Detect multiple edges of the target storage library that divide the target storage library into multiple ranges from the image information; Define a plurality of imaginary bounding boxes, wherein each of the plurality of imaginary bounding boxes surrounds one of the plurality of edges, or surrounds one of the plurality of extents; as well as The object placement command is generated based on the multiple bounding boxes.

18. A robot control method for transferring an object from a source storage to a destination storage, the robot control method being operable by at least one processing circuit via a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector and having a camera attached to the robotic arm, the method comprising: When the end effector is not transporting an object, it outputs a first destination storage proximity command to cause the robotic arm to approach the predetermined position of the destination storage in such a way that the camera is pointed at the destination storage; Receive image information describing the destination storage, wherein the image information is generated by the camera after a first destination storage proximity command is executed; Based on the image information, a storage library detection operation is performed, wherein the storage library detection operation determines and / or verifies and / or updates at least one of the following: storage library structure information for describing the structure of the target storage library, or storage library pose information for describing the pose of the target storage library, or storage library content information for describing whether one or more objects exist in the target storage library. After the camera has generated the image information describing the target storage, a source storage proximity command is output to bring the robotic arm closer to the source storage. Output an object pick-up command to cause the end effector device to pick up the object from the source storage; After the object picking command is executed, a second destination storage proximity command is output to bring the robotic arm close to the destination storage. as well as After the second destination storage proximity command is executed, an object placement command is output to cause the end effector device to place the object in the destination storage, wherein the object placement command is generated based on at least one of the storage structure information or the storage posture information.

19. A non-transitory computer-readable medium configured with executable instructions for implementing a robot control method for transferring an object from a source repository to a destination repository, the robot control method being operable by at least one processing circuit via a communication interface configured to communicate with a robot having a robotic arm including or attached to an end effector and having a camera attached to the robotic arm, the method comprising: When the end effector is not transporting an object, it outputs a first destination storage proximity command to cause the robotic arm to approach the predetermined position of the destination storage in such a way that the camera is pointed at the destination storage; Receive image information describing the destination storage, wherein the image information is generated by the camera after a first destination storage proximity command is executed; Based on the image information, a storage library detection operation is performed, wherein the storage library detection operation determines and / or verifies and / or updates at least one of the following: storage library structure information for describing the structure of the target storage library, or storage library pose information for describing the pose of the target storage library, or storage library content information for describing whether one or more objects exist in the target storage library. After the camera has generated the image information describing the target storage, a source storage proximity command is output to bring the robotic arm closer to the source storage. Output an object pick-up command to cause the end effector device to pick up the object from the source storage; After the object picking command is executed, a second destination storage proximity command is output to bring the robotic arm close to the destination storage. as well as After the second destination storage proximity command is executed, an object placement command is output to cause the end effector device to place the object in the destination storage, wherein the object placement command is generated based on at least one of the storage structure information or the storage posture information.

Images