Multi-purpose robotic systems and methods for warehouse automation

Abstract

Moveable robotic arm of robotic apparatus grasps and moves objects and a conveyor system. The conveyor system comprises a first conveyor mechanism, a second conveyor mechanism coupled to the first conveyor mechanism, a lifting mechanism coupled to the first and second conveyor mechanism, and a third conveyor mechanism coupled to the second conveyor mechanism. The lifting mechanism is moveable between a plurality of heights. Also, the robotic arm, lifting mechanism, and conveyor system can move together.

Description

MULTI-PURPOSE ROBOTIC SYSTEMS AND METHODS FOR WAREHOUSE AUTOMATION REFERENCE

[0001] This application claims the benefit of U. S. Provisional Patent Application No. 63 / 553,904, filed on February 15, 2024, which is entirely incorporated herein by reference.BACKGROUND

[0002] Automation systems have been utilized in various loading / unloading applications or industries. For instance, transload and cross docking facilities often manage a large variety of shipments through their facility. Such facilities require flexible receiving areas capable of handling any kind of load, processing goods through the facility without storage, thus an automation system capable of receiving goods and loading them out quickly is critical. Third party logistics providers (3PLs) may further require a high level of flexibility in their automation solutions. The logistics industry is dynamic with changing client requirements, fluctuating demand, and evolving supply chain dynamics. 3PLs also have complex integration requirements and often have to integrate automation into existing operation. Distribution centers (e.g., retail warehouses and ecommerce fulfillment centers) may experience high degrees of seasonality, or peak times, that require scaling the capacity of the receiving dock up and down. They also deal with daily fluctuations in outbound needs, and the pressure of always meeting end-customer demands and expectations. Moreover, they often have a wide product mix that can quickly change, so having automation that can handle a larger variety of product is important.

[0003] Load and throughput are important metrics for measuring loading / unloading system performance. Current automation systems may involve robotic arms which are used to vertically and horizontally pick up and move boxes and packages. Collaborative robots are used to move boxes and packages and allow human interaction. However, due to the limitation of the motor torque at different joints, robotic arms have limited payloads which can differ depending on how the object is picked up. For example, a robot end effector may have a 25 kg pay load for objects it picks up from the top vertically. The same robot end effector may only have a 15 kg payload when picking items up from the side horizontally, due to the increased torque resulting from a longer lever arm. Current robotic systems for warehouse automation may utilize vertical lift for picking up heavy load which lacks the side picking capability or flexibility.

[0004] Additionally, increasing speed and payload improves the throughput of the robot. For example, a robot in a warehouse, or at a large shipping container for transporting numerous boxes, picks boxes to move to another location. The throughput is the number of boxes therobot can move in a specified time period. However, robotic arms have limited moving speeds. Collaborative robots interact with humans are slower moving due to the safety considerations.SUMMARYLOOO5J A need exists for an improved automation robotic system for loading or unloading boxes, packages and various objects such as robot apparatus for picking and placing objects, e.g., boxes. The present disclosure addresses the above need by providing methods and systems capable of picking up and moving objects (e.g., goods, boxes and packages etc.) with improved load capacity and automation system throughput. In particular, the robotic system or robot apparatus herein may comprise a robotic arm capable of picking up objects vertically and horizontally and moving the object to a multi -part conveyor system that is actuated autonomously to an optimal position (i.e., both vertically and horizontally) in accordance with the movement of the robotic arm thereby improving the load capacity (i.e., maximum load can be moved by the robotic arm) and throughput (e.g., number of objects can be moved by the system within a specified period of time).

[0006] In some embodiments, the robot apparatus may comprise i) a moveable robotic arm with an end effector configured to grasp and move the objects and ii) a multi -part conveyor system. The multi-part conveyor system may comprise first, second and third conveyor mechanisms and is controlled to move autonomously to an optimal position (i.e.. both vertically and horizontally) to improve the overall throughput of the robotic apparatus. In some embodiments, the robotic apparatus can also comprise a lifting mechanism coupled to the first and second conveyor mechanisms and raise and low er a height level of the first and second conveyor mechanisms. In some cases, for unpacking, the scene with the objects to be picked is scanned with a sensor(s) to identify a top row of objects to be picked. The robotic apparatus can then control the lifting mechanism so that the first and second conveyors are at a height level just below the height level of the top row of objects to be picked. The first conveyor is also generally parallel to the top row. The robotic arm can then carry the objects on the top row; onto the first (or second) conveyor, which conveys the objects to the second (or third) conveyor mechanism. The end effector of the robotic arm may pick up one object at a time or perform multi-picking, i.e., picking multiple boxes at the same time to improve efficiency. The second conveyor mechanism can change the direction of the conveyed objects such that the second conveyor mechanism conveys the picked objects to the third conveyor mechanism, which can be orientated, for example, perpendicular to the first conveyor mechanism, so that the third conveyor mechanism carries the object aw ay from where the robotic arm is picking the boxes. The process can be repeated upon all of the objects in the top row being picked and conveyedaway. Also, once all the objects in the front are conveyed away in this manner, the robot (robot base) can move horizontally such as moving forward to start convey way the next wall of objects in the same manner.

[0007] In some cases, the same robotic system, or same components of the robotic system may be utilized for a loading process with variations in the specific operations (i.e., pushing instead pulling containers by the robotic arm) or set up of the system. For instance, the loading process can operate in reverse order when packing the objects, that is: conveyed from third conveyor to second conveyor to first conveyor, which has been adjusted by the lifting mechanism to be at a height level at about where the objects are to be stacked, so that the robotic arm can easily move the objects from the first conveyor to where they are to be stacked.

[0008] The present disclosure provides methods and algorithms that can increase the speed and payload for pick and place robots. The throughput or speed for moving the objects may be increased by dynamically determining an optimal position for the first conveyor mechanism and / or the second conveyor mechanism (height and distance to the front of the object) such that most of the objects need only be moved to the first conveyor mechanism that is adjusted to be at a height level that is close to the height level of the object being picked. The pay load can be increased because most of the objects can be lifted vertically onto the first conveyor, the second conveyor or the third-conveyor or dragged from side faces onto the first conveyor or the second conveyor. As described above, when lifting an object from the side, the robot pay load drops significantly due to a distance of the object located away from the motor resulting in a large torque applied to the motor. The system herein may generate an optimal picking location for grasping the object (e.g., from the top side) unless in certain locations such as a higher location that the top side cannot be easily accessed, the robot system may grasp the object from the side and drag the box onto the conveyor.

[0009] The robotic system herein may coordinate a height of the conveyor holding the objects with motion of the mobile manipulator (robotic arm), so the load on the robot motor of the robot arm is reduced greatly compared with traditional solutions. These and other benefits that can be realized through embodiments of the present disclosure will be apparent from the description that follows.

[0010] In an aspect of the present disclosure, a robotic system for transferring objects is provided. The robotic system comprises: a conveyor system comprising a first conveyor mechanism to receive one or more objects and a lifting mechanism; a mobile manipulator system controlled to grasp and move one or more objects to the first conveyor mechanism; and a controller configured to: i) receive sensor data of an environment containing a plurality ofobjects, the conveyor system and the mobile manipulator system, ii) determine a conveyor height by identifying a number of the objects within a height threshold above the conveyor height based on the sensor data, iii) select a target object from the layer of objects and determine a grasp motion for actuating the mobile manipulator system to grasp and move the target object to the first conveyor mechanism.

[0011] In some embodiments, the conveyor system further comprises a second conveyor connecting the first conveyor mechanism with a third conveyor mechanism such that the mobile manipulator system grasps and moves the target object to the first conveyor mechanism, the second conveyor mechanism or the third conveyor mechanism. In some cases, the first conveyor mechanism and the third conveyor mechanism transfer the plurality of objects in different directions. In some embodiments, the lifting mechanism is actuated to lift the first conveyor mechanism to the conveyor height determined in ii).

[0012] In some embodiments, the controller is further configured to generate a park location for the first conveyor mechanism relative to the plurality of objects and the environment. In some cases, the conveyor system has a dimension smaller than a dimension of the environment or a length of wall of the plurality of objects, and where the park location is determined to be at a clearance distance to the plurality of objects in the horizontal plane. In some instances, the clearance distance is smaller than a threshold to prevent the plurality of objects from dropping into a clearance gap.

[0013] In some embodiments, the sensor data is processed to identify a position, orientation and shape in a three-dimensional (3D) scene for the plurality of objects. In some cases, the controller is further configured to detect a change in the 3D scene based on the sensor data.

[0014] In some embodiments, identifying the number of the objects within the height threshold above the conveyor height comprises identifying a maximum number of objects with a dropping distance relative to the first conveyor mechanism within the height threshold. In some cases, the height threshold is based at least in part on a weight of the plurality of objects. In some cases, the height threshold is configurable by a user or determined by the robotic system based on weight sensed by the robotic system.

[0015] In some embodiments, the target object is selected based at least in part on positions of the plurality of objects. In some cases, the target object has a bottom side located below the conveyor height. In some embodiments, determining the grasp motion for actuating the mobile manipulator system comprises searching, for the target object, a side from a pluralityof sides of the object for grasping. In some embodiments, the grasp motion comprises generating a pose and a path for moving an end effector of the mobile manipulator system. In some cases, the path is a non-linear path in 3D space for collision avoidance. In some embodiments, the grasp motion is generated using a model trained using reinforcement learning.

[0016] In some embodiments, the plurality of objects have variable dimensions or are not leveled. In some embodiments, the conveyor system is removably coupled to a platform of the mobile manipulator system. In some cases, the platform is actuated to move to a park location with aid of a plurality of powered wheels. In some examples, each of the plurality of powered wheels is actuated by two motors.

[0017] In another aspect, a system for transferring objects is provided. The system comprises: a front conveyor mechanism to receive one or more objects and transfer the one or more objects in a first direction; a transitional conveyor mechanism connected to the front conveyor mechanism and a side conveyor mechanism, where the side conveyor mechanism transfers the one or more objects in a second direction that is different from the first direction; a lifting mechanism controlled to move the front conveyor mechanism to a height; and a controller configured to: i) determine, based at least in part on a three-dimensional (3D) scene containing the plurality of objects, a top layer of objects from the plurality of objects to be moved to the front conveyor mechanism, the transitional conveyor mechanism or the side conveyor mechanism, ii) determine the height for the front or the transitional conveyor mechanism, wherein the height is determined at a dropping distance from the top layer of objects, where the dropping distance is within a height threshold to reduce a frequency of adjusting the height, iii) determine a park location to move a mobile base on the system utilizing an optimization algorithm.

[0018] In some embodiments, the park location is determined to be at a clearance distance to the plurality of objects in the horizontal plane. In some cases, the clearance distance is smaller than a threshold to prevent the plurality of objects from dropping into a clearance gap. In some cases, the optimization algorithm comprises maximizing a number of the objects within a reachable space of the robotic arm.

[0019] In some embodiments, the controller is further configured to determine a target object from the number of objects in the top layer for picking up according to a pre-determined rule. In some embodiments, the controller is further configured to determine a new height for the front or the transitional conveyor mechanism upon detection of a collision and wherein the new height is within the height threshold.

[0020] In some embodiments, the one or more objects are moved to the front conveyor mechanism, the second conveyor mechanism or the third conveyor mechanism by a robotic arm.

[0021] Additional aspects and advantages of the present disclosure will become readily apparent to those skilled in this art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.INCORPORATION BY REFERENCE

[0022] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings (also '‘Figure” and '‘FIG.” herein), of which:

[0024] FIG. 1 is an isometric view of a robotic system, according to various aspects of the present disclosure.

[0025] FIG. 2 is a front view of the robotic system of FIG. 1.

[0026] FIG. 3 is a side view of the robotic system of FIG. 1.

[0027] FIG. 4 is a top view of the robotic system of FIG. 1.

[0028] FIG. 5 is the robotic system of FIG. 1 with a plurality of boxes, according to various aspects of the present disclosure.

[0029] FIG. 6 is the robotic system of FIG. 1 with a plurality of boxes, according to various aspects of the present disclosure.

[0030] FIG. 7 is a representation of a partial scene to select boxes, according to various aspects of the present disclosure.

[0031] FIG. 8 is a method of operating the robotic system of FIG. 1.

[0032] FIG. 9 illustrates an example of a control algorithm for controlling the robotic system in an unloading process.

[0033] FIG. 10 shows an example of a height control algorithm for computing the box layer and / or the height of the conveyor and an example of a task planning algorithm, in accordance with some embodiments of the present disclosure.

[0034] FIG. 11 show examples of the robotic system performing the loading / unloading operations within a shipping container environment.

[0035] FIG. 12 shows a top view and front view of a scene having a front wall of boxes and a back wall of boxes.

[0036] FIG. 13 shows an example of a park location control algorithm.

[0037] FIG. 14 show examples of the robotic system performing the loading / unloading operations within a warehouse environment.

[0038] FIG. 15 shows a mobile manipulator system can be a standalone system that works with any existing systems (e.g.. flex or telescopic conveyor).

[0039] FIG. 16 shows an example of the conveyor system removably coupled to a mobile base of the mobile manipulator system.

[0040] FIG. 17 shows an example of a plurality of 3D sensors mounted to a mast supported by the mobile base.

[0041] FIG. 18 shows various examples of dynamic trajectories or paths of the end effector of the robotic system.

[0042] FIG. 19 shows various examples of picking or grasping operations.DETAILED DESCRIPTION

[0043] While various embodiments of the invention have been show n and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0044] The embodiments disclosed herein can be combined in one or more of many ways to provide improved robotic system for moving objects in warehouse, distribution center, transload and cross docking facilities, trucks, conveyors, and various other applications or industries. As described above, the present disclosure provides methods and robotic systems capable increasing speed, payload and / or the throughput of the robot system. As described above, throughput of a robotic system may refer to, the number of objects (e.g., boxes, packages, etc.) the robot system can move in a specified time period. For instance, a robot system in a warehouse, or at a large shipping container for transporting numerous boxes, picks boxes to move to another location. As an example, the robotic system herein may have specifications or capacity for payloads of 50kg with a throughput of 1000 packages per hour. The robotic system herein may have a throughput of at least 1000 packages unloads / hour, at least 2000 packages unloads / hour, at least 3000 packages unloads / hour, at least 4000 packages unloads / hour, for payloads of 50kg or more.

[0045] In some embodiments, the present disclosure provides a robot system comprising a moveable robotic arm, a height adjustable conveyor system, and a control system to adjust the height of the conveyor system and control the robot arm or robotic end effector to pick boxes. The robot system of the present disclosure may increase the payload and throughput of the system. For instance, the payload is increased due to either lifting the boxes from the top (i.e., vertical lift) or dragging the boxes from the side (i.e., side pick) to the height adjustable conveyor system. The autonomous height adjustable conveyor system and the control system may dynamically determine an optimal position (i.e., both vertically and horizontally) for the conveyor system to improve the overall throughput of the robotic apparatus. For instance, an optimal position (particularly conveyor height in the vertical direction) may be dynamically determined such that an overall distance for moving the boxes by the robotic arm is reduced thereby increasing the throughput (e g., robot arm moving speed has less impact). The conveyor system may be actuated by a lifting mechanism to move relative to the robotic arm (e.g., adjusting height). In some cases, the conveyor system and the robotic arm may be removably coupled or may not share the same mobile base or platform. For instance, the conveyor system may be removably coupled to a mobile base of the robotic arm and can traverse a larger container, workplace or warehouse while the conveyor mechanism can be elevated or lowered relative to the robotic arm. Alternatively, the conveyor system and the robotic arm may share the same mobile base or platform thereby eliminating the registration between the conveyor frame of reference and the robotic arm frame of reference. Details about the algorithms for determining the optimal position and controlling the conveyor system are described later herein.

[0046] It is to be understood that any one or more of the structures and steps as described herein can be combined with any one or more additional structures and steps of the methods and apparatus as described herein, the drawings and supporting text provide descriptions in accordance with embodiments.

[0047] While exemplary embodiments will be primarily directed at a robotic device or system for unloading boxes, containers or packages, one of skill in the art will appreciate that this is not intended to be limiting, and the devices, systems described herein may be used for moving any other objects in any other processes.

[0048] Whenever the term “at least,” “greater than,” or “greater than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “at least,” “greater than” or “greater than or equal to” applies to each of the numerical values in that series of numerical values. For example, greater than or equal to 1, 2. or 3 is equivalent to greater than or equal to 1, greater than or equal to 2, or greater than or equal to 3.

[0049] Whenever the term “no more than.” “less than,” or “less than or equal to” precedes the first numerical value in a series of two or more numerical values, the term “no more than,” “less than,” or “less than or equal to” applies to each of the numerical values in that series of numerical values. For example, less than or equal to 3. 2, or 1 is equivalent to less than or equal to 3. less than or equal to 2. or less than or equal to 1.

[0050] FIG. 1 shows an isometric view of a robotic system, according to various aspects of the present disclosure. FIG. 2 is a front view of the robotic system of FIG. 1. FIG. 3 is a side view of the robotic system of FIG. 1. FIG. 4 is a top view of the robotic system of FIG. 1. FIGs. 1-4 are described in conjunction. The robotic system 100 comprises a robotic arm 102 that is moveable in three-dimensional (3D) space. The robotic arm 102 may be moveable with up to 6 degrees of freedom, between a plurality of positions or the robotic arm 102 may be moveable with more than 6 degrees of freedom between a plurality of positions. The robotic arm 102 is coupleable to a base 112. The base 112 may include the control system. In another embodiment, the control system may be separate from the base 112 and couples to the robotic system 100. In some cases, the robot control system is configured to: sense an arrangement of the objects such as boxes or freight units in the scene based on the image; choose a freight unit based on a trained neural network; choose a starting pose of the robotic end effector 101 as a means to pick the chosen freight unit; choose a goal pose of the robotic end effector as a means to place the chosen freight unit; plan a motion of the end effector 101 from the starting pose to the goal pose; and control the motion on the robotic end effector based on the planning.

[0051] The end effector 101 may utilize any suitable mechanism such as a vacuum cup or a gripping / grabbing tool to pick up the object. FIG. 17 shows another example of the mobile manipulator system 1700 in accordance with some embodiments of the present disclosure. In some embodiments, the mobile manipulator system 1700 may comprise an omni-directional mobile base 1705 supporting a heavy-duty collaborative robot arm. An end effector 1701 such as a heavy-duty vacuum gripper may be attached to a distal end of the robotic arm for grabbing / grasping an object. The vacuum gripper 1701 may be powered via a vacuum generator resided inside the mobile base 1705. For example, an array of suction cups on the vacuum gripper may pick the boxes with vacuum force. A vacuum valve controls the release of the box when end effector / object reaches the target location to drop the box.

[0052] In particular, the vacuum gripper 1701 may access and grab the object on a side (e.g., top, front side, left / right side) that is determined by a grasping algorithm described elsewhere herein and the vacuum gripper can provide sufficient coupling force to lift or drag the object (e.g., box, package, container, etc.). In some cases, the vacuum gripper 1701 may grasp the object on a side with a pose determined by a grasp planning algorithm (e.g., grasp planning 1023 in FIG. 10). The grasp planning algorithm may determine a grasp location on the object based on information such as the box weight, size, position relative to the conveyor belt, possible collision, payload capacity7of the robotic arm, grasping capacity, and the like. Details about the grasp planning algorithm are described later herein. In some cases, the robotic system herein may determine an optimal height and dynamically adjust the height for the conveyor such that the overall vertical lifting distance may be minimized during the loading / unloading process thereby improving the throughput of the system (e.g., by reducing the delay caused by the limited robot arm moving speed).

[0053] In some cases, the vacuum gripper 1701 may provide sufficient coupling force for the robotic arm to pick up the object and move the object along a dynamically determined trajectory (e.g., linear or non-linear) in the three-dimensional (3D) space. The dynamic trajectory7may be generated by the control system based on real-time sensor data and / or a position of the conveyor system. For example, the dynamic trajectory7that the end effector moves along with may be generated for moving an object from a starting location (e.g., stacked box wall) to an ending location (e.g., a location on the conveyor belt) with non-linear path for collision avoidance purpose (e.g., sensing object / obstacles) or ensuring object safely such as by preventing objects falling onto the conveyor belt without increasing the dimension of the conveyor belt (e.g., moving objects that cannot be dragged onto the conveyor belt along a linear / straight path) while minimizing the length of the trajectory (i.e., for fast speed purpose).The capability of moving the end effector along highly non-linear trajectories beneficially allows the robotic system having a reduced footprint (e.g., smaller conveyor system, smaller working space of the robotic arm) to be able to adapt to various complex environments, narrower workplace, with improved mobility. Details about the dynamic non-linear trajectory movement and motion planning of the end effector are described later herein.

[0054] The omni-directional mobile base 1705 can be the same as the mobile base 112 as illustrated in FIG. 1. The mobile base 1705 may comprise a plurality of (e.g., four) power steering wheels. The wheels or propellers may be actuated to move the mobile base or platform to perform zero-radius turning. This beneficially allows for navigation of the robotic system within a limited workplace (e.g.. inside the narrow container environment).

[0055] In some cases, the mobile base or the mobile manipulator system may have a compact design or reduced footprint allowing it to navigate within a limited space and / or adaptive to complex environment. In some cases, the robot arm may be mounted on a rising mechanism (e.g., raising column) to further extend the reach of the robotic arm in the vertical direction (e.g., Z-axis). For instance, the base of the robotic arm may be lifted or raised in the vertical direction by the rising mechanism. The rising mechanism herein may adopt a unique design with reduced physical dimension to fit within a mobile base of the mobile manipulator system without comprising a travel distance for raising the robotic arm base.

[0056] The rising mechanism may comprise a linear-rail guided mechanism powered by a motor-driven ball screw. By using the ball-screw mechanism, the robotic platform may have a compact mechanism design without comprising the robotic working space. The ball-screw mechanism may be located within the mobile base which beneficially increases the space utilization inside the mobile base, allowing for the capacity to include additional components placement and better heat dissipation. Furthermore, the guided rails (e g., linear rail 1 and linear rail 2) increase the system rigidity significantly, allowing the robot arm to perform at higher acceleration and speed. It should be noted that the ball screw mechanism for the rising mechanism is just for illustration purpose. Various other mechanism may be employed for providing a linear rising function with a small footprint.

[0057] The robotic system may comprise a self-propelled (e.g.. via one or more propulsion units such as wheels, rotors, propellers) robotic base autonomously placing itself within a workplace (e.g., warehouse such as shown in FIG. 14 or shipping container shown in FIG. 11) or to an optimal park location with respect to the objects (e.g., front box wall). In some embodiments, a propulsion unit may include a plurality of wheels that may permit the mobilebase to roll over an underlying surface. In some embodiments, the mobile base may employ powered wheels to achieve omnidirectional movement with zero turning radius.

[0058] In some embodiments, a wheel of the mobile base may be actuated by two embedded motors: one motor controlling the yaw angle of the tire surface and the second motor for rotating the tire. In some cases, the mobile base may be equipped with a plurality of (e.g., four) such wheels beneficially allowing the mobile base to translate left and right without any turning radius. Without changing the yaw angle of the robot, the mobile base can directly move in any direction on the travelling floor. Alternatively, the mobile base may employ any other number of wheels. In some examples, two, three or four wheels may be provided which may permit the mobile base to stand stably while not moving. In some instances, stabilization may occur with aid of one or more wheels or other stabilization platforms, such as gyroscopic platforms. The wheels may vary in size or be the same size. In some cases, the wheels can have a diameter of at least about 1 cm, 2 cm, 3 cm, 4 cm, 5 cm, 8 cm, 9 cm, 10 cm, 15 cm, 20 cm, 25 cm, 30 cm, 35 cm, 40 cm, 45 cm, 50 cm, 55 cm, 60 cm, 65 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, 95 cm. 100 cm, 150 cm. or 200 cm. The wheels can have a smooth or treaded surface and may be made of any suitable materials such as rubber, synthetic rubber, steel, textiles (rayon, polyester, aramid, and nylon), fillers (carbon black, silica from sand, carbon), and the like. The wheels may also permit the mobile base to move laterally and / or rotate in place. The mobile base may be capable of making any combination of translational or rotational movement. The propulsion unit may be driven with aid of one or more actuators. For example, a motor, engine, drive train, or any other component may be provided that may aid in driving the propulsion of the mobile base.

[0059] In some cases, based on a 3D scene or 3D depth map (generated based on 3D sensor 1703), an optimal location or park location of the mobile base or the robotic system / conveyor system may be generated. The park location may be generated based on a dimension of the conveyor system / robotic system, the dimension of the robotic arm (workspace), and the 3D depth map. Real-time sensor data (e.g., imaging sensor, proximity sensor, etc.) may be collected and may be used to determine whether the robotic system or the conveyor system is in the proper location relative to the objects to be moved. The proper location, the movement speed, moving acceleration, and the movement trajectory for the mobile base may be calculated by one or more processors of the robotic platform based at least in part on the 3D depth map. Details about the algorithms for determining the park location of the conveyor belt in the horizontal plane with respect to the objects to be moved are described elsewhere herein.

[0060] Referring back to FIG. 1, the robotic system 100 comprises a conveyor system that may be coupleable to the base 102. The control system is configured to control the robot 102 and the conveyor system. The base 112 may be mobile, thereby also making the robotic arm 102 connected to it mobile. In some embodiments, the robot 102 may be a collaborative robot.

[0061] A sensor tower 114 can be coupled to the base 112. The sensor tower 114 is configured to scan the scene around the robot 102 and the base 112. The sensor tower 114 may comprise a plurality of different types of sensors, such as cameras, lidar, proximity sensors, depth cameras, etc. The sensor tower 114 sends input from the sensors to the control system, which uses the input from the sensors to control the movement of the robot arm 102, end effector 101. and the conveyor system. In another embodiment, sensors are disposed on the conveyor system, in combination with or in lieu of the sensor tower 114. The sensors may comprise the same types of sensors as the sensor tower 114.

[0062] FIG. 17 shows an example of a plurality of sensors 1703 mounted to a mast supported by the mobile base 1705. The plurality of sensors 1703 may be the same as the sensors as described above, such as including but not limited to RGB cameras, depth sensors, LIDARs. IMUs, etc. Sensor data may provide the environment information to an artificial intelligence (Al) perception system for downstream processing (e.g., motion planning for the end effector 1701, trajectory for navigating the robotic system 1700 within a warehouse, dynamic adjustment of the height of the conveyor system in accordance with the load capacity and / or objects statuses, etc.).

[0063] The one or more electronic components in the robotic system may comprise an imaging device, illumination device or sensors. In some embodiments, the imaging device may be a video camera. The imaging device may comprise optical elements and image sensor for capturing image data. The image sensors may be configured to generate image data in response to wavelengths of light. A variety of image sensors may be employed for capturing image data such as complementary metal oxide semiconductor (CMOS) or charge-coupled device (CCD). The imaging device may be a low-cost camera. In some cases, the image sensor may be provided on a circuit board. The circuit board may be an imaging printed circuit board (PCB). The PCB may comprise a plurality of electronic elements for processing the image signal. For instance, the circuit for a CCD sensor may comprise A / D converters and amplifiers to amplify and convert the analog signal provided by the CCD sensor. Optionally, the image sensor may be integrated with amplifiers and converters to convert analog signal to digital signal such that a circuit board may not be required. In some cases, the output of the image sensor or the circuitboard may be image data (digital signals) can be further processed by a camera circuit or processors of the camera. In some cases, the image sensor may comprise an array of optical sensors, RGB-D camera and the like.

[0064] The cameras or sensor though are show n on the 3D sensor 1703, the various sensors may be located on other components and locations in the robotic system. For instance, camera may also be located on the end effector (e.g., gripper) 1701 and on the conveyors to observe the box flow-out. Other sensors such as lidars 1704 may be located on the mobile base for safety (e.g., avoid collision) and for sensing proximity / distance in the environment. Other sensors such as proximity sensors may be located on the end effector or gripper 1701 to collect proximity data about various objects within a proximity to the gripper.

[0065] In some case, the sensing mast 1703 or the robotic base 1705 may be housing or comprise components configured to process image data, provide power, or establish communication with other external devices. In some cases, the communication may be wireless communication. For example, the wireless communications may include Wi-Fi, radio communications, Bluetooth, IR communications, or other types of direct communications. The illumination device may comprise one or more light sources such as a light-emitting diode (LED), an organic LED (OLED), a quantum dot, or any other suitable light source. In some cases, the light source may be miniaturized LED for a compact design or Dual Tone Flash LED Lighting.

[0066] The robotic platform herein may be able to detect one or more objects in the environment surrounding the robotic system. In the cases, the detection of the environment may comprise detecting a change in the scene to update the box library, and / or generating an obstacle map. An obstacle map may be a three-dimensional (3D) map describing positions of objects detected in the 3D space. Details about the box library construction are described later herein.

[0067] The 3D map of the environment may be constructed based on sensing data. In some cases, the sensing data is received from one or more vision sensors, including depth information for the environment. The vision sensor may comprise a camera, a video camera, a three-dimensional (3D) depth camera, a stereo camera, a depth camera, a Red Green Blue Depth (RGB-D) camera, a time-of-flight (TOF) camera, an infrared camera, a charge coupled device (CCD) image sensor, or a complementary metal oxide semiconductor (CMOS) image sensor. For example, the vision sensor can include only one camera (monocular vision sensor). Alternatively, the vision sensor can include two (binocular vision sensor) or more cameras. The vision sensors may be disposed on the robotic system such as the robotic cart, the monitor and the like. Alternatively or additionally, the vision sensors may not be disposed on the roboticsystem. For instance, the vision sensors may be disposed on the walls, ceilings or other places in the environment (e.g., container, warehouse). In embodiments where multiple vision sensors are used, each sensor can be located on a different portion of the robotic system, and the disparity7between the image data collected by each sensor can be used to provide depth information for the environment. Depth information can be used herein to refer to information regarding distances of one or more objects from the robotic system and / or sensor. In embodiments where a single vision sensor is used, depth information can be obtained by capturing image data for a plurality of different positions and orientations of the vision sensor, and then using suitable image analysis techniques (e.g., structure from motion) to reconstruct the depth information.

[0068] As mentioned elsewhere herein, an environment of the system may generally comprise an environment external to the system (e.g.. the container the system is in, within certain proximity7, etc.) or the environment that the system can move or operate within. In some cases, the camera may be a plenoptic camera having a main lens and additional micro lens array (MLA). The plenoptic camera model may be used to calculate a depth map of the captured image data. In some cases, the image data captured by the camera may be grayscale image with depth information at each pixel coordinate (i.e., depth map). The camera may be calibrated such that intrinsic camera parameters such as focal length, focus distance, distance between the MLA and image sensor, pixel size and the like are obtained for improving the depth measurement accuracy. Other parameters such as distortion coefficients may also be calibrated to rectify the image for metric depth measurement.

[0069] In some cases, the image data may be received and processed by one or more processors of the robotic system. For example, pre-processing of the capture image data may be performed. In an embodiment, the pre-processing algorithm can include image processing algorithms, such as image smoothing, to mitigate the effect of sensor noise, or image histogram equalization to enhance the pixel intensity values. Next, optical approaches as described elsewhere herein may be employed to generate a depth map of the environment. In some cases, computer vision (CV) techniques or computer vision systems may be used to process the sensing data to extract high-level understanding of the environment, object detection, object classification, extraction of the scene depth and estimation of relative positions of objects, extraction of objects’ orientation in space. For example, the CV output data may be generated using passive methods that only require images. Passive methods may include, for example, object recognition, stereoscopy, monocular shape-from-motion, shape-from-shading, and Simultaneous Localization and Mapping (SLAM). Alternatively, active methods may be utilized which may require controlled light to be projected into the target scene and the active methodsmay include, for example structured light and Time-of-Flight (ToF). In some cases, computer vision techniques such as optical flow, computational stereo approaches, iterative method combined with predictive models, machine learning approaches, predictive filtering or any non- rigid registration methods may be used to generate the descriptions of the 3D scene.

[0070] In some cases, the imaging device may be used in conjunction with other types of sensors (e.g.. proximity sensor, location sensor, positional sensor, etc.) to improve accuracy of the location information. For instance, the sensing data may further comprise sensor data from one or more proximity sensors. The proximity sensors can be any suitable type such as ultrasonic sensor (e.g., a wide angle sensor, an array sensor) or light detection and ranging (Lidar) sensor. Lidar can be used to obtain three-dimensional information of an environment by measuring distances to objects. The proximity sensors can also be disposed at the robotic system. The proximity' sensors can be located near the vision sensors. Alternatively, the proximity sensors can be situated on a portion of the robotic system different from the portions used to carry the vision sensors.

[0071] In some cases, the 3D depth map may be generated using a single modality sensor data (e.g., image data, Lidar, proximity data, etc.). Alternatively, the 3D depth map may be generated using multi-modality data. For example, the image data and 3D point cloud generated by the Lidar system may be fused using Kalman filter or deep learning model to generate a 3D map. The 3D map may then be used for autonomous navigation in the environment, constructing box library, collision avoidance and various other functions as described elsewhere herein.

[0072] The collaborative robotic arm 1702 can be the same as the robotic arm 102 as described in FIG. 1 and may be capable of picking up heavy load (e.g.. heavy boxes) while maintaining the collaborative nature, ensuring the operational safety when the system is operating collaboratively with a human worker. In some cases, safety may be provided such as by detecting collision or force applied to the system (e.g., force sensor). Alternatively, the robotic arm 1702 may have redundant degrees of freedom allowing for its elbow to be algorithmically, or passively, moved into configurations that are convenient for an operator (e.g., worker) initiate the positioning of the robotic system or other robotic instrument. In some cases, the robotic arm may comprise a plurality of j oints having redundant degrees of freedom such that the joints of the robotic arm can be driven through a range of differing configurations for a given end effector position. The redundant degrees of freedom may beneficially allow the robotic arm to be self-adjusted to an optimal configuration to avoid collision with other objects in the environment prior to or during a loading / unloading process. For example, during theunloading process, upon detection of another object on the conveyor belt passing through, the robotic arm is able to auto-adjust accordingly to avoid collision while maintain the position of the end effector, avoiding collision. In another example, when the robotic arm is actuated during picking up the object and moving to the conveyor belt, the system may detect unwanted proximity between any portion of the robotic arm and other objects surrounding it (e.g., other box) and the robotic arm may be automatically reconfigured, moved away from the box to avoid collision.

[0073] The robotic system herein may be capable of avoiding collision with moving object. In some cases, a proximity determination algorithm may be executed to determine a potential collision whether the minimum proximity violates a predetermined proximity threshold. In some cases, upon determining a violation of the predetermined proximity threshold, the system may activate an admittance controller to reconfigure a configuration of the robotic arm (e.g., by actuating one or more joints / links of the robotic arm). For example, the admittance controller may execute an admittance control algorithm which defines a proximal- most point of collision as repulsive force and reconfigure the robotic arm to increase the (minimum) distance between the robotic arm and the obstacle (e.g., human worker). An algorithm is executed to reposition the arm for the purpose of avoiding a collision prior to occurring. In some cases, the admittance control algorithm may map a sensed proximity between any portion of the robotic arm and an object to a virtual force applied on the robotic arm and calculate an output motion for the robotic arm as if an equivalent contact force was applied to the robotic arm (through contact). For instance, the admittance control algorithm comprises mapping a shortest distance between a link of an arm and obstacle to an input to a commanded task-space velocity of the arm link. The Geometric Jacobian of the arm may be used to map a task-space motion command to arm joint commands in order to achieve arm motion which increases the distance between the arm and obstacle.

[0074] Alternatively, the robotic arm 102, 1702 may not have redundancy and the collision avoidance may be provided through a motion / path planning for the end effector and objects detected in the environment. For instance, based on a 3D map with depth information, the motion and / or path for the end effector may be generated to avoid collision with the one or more other objects in the environment. FIG. 18 shows various examples of dynamic trajectories or paths of the end effector of the robotic system. The trajectories may be non-linear (i.e., nonstraight path) in 3D space for moving an object from a starting position to a target position. The trajectories or motion path of the end effector may be for moving an object from a plurality of stacked objects to a conveyor belt. The trajectories or motion path may be for the end effector(free of load) to move across the conveyor belt to reach, pick up an object or drag and move the object onto the conveyor.

[0075] The robotic system beneficially ensures object safely falling onto the conveyor belt without increasing the dimension of the conveyor belt while minimizing the length of the trajectory (i.e., for fast speed purpose). The capability of moving the end effector along highly non-linear trajectories beneficially allows the robotic system having a smaller footprint (e.g., smaller conveyor system, smaller working space of the robotic arm) to be able to adapt to various complex environments with improved mobility. In some cases, the objects such as boxes or containers may not have uniform size, not leveled, not aligned, and / or not stacked neatly. For example, as shown in FIG. 12, the example scene 1220 shows objects or boxes are not arranged in an aligned 2D wall layout. The boxes may be tilted, leaning on to each other. The boxes may also have different sizes and shapes, making a simple linear movement extremely difficult to drag out the boxes on to the conveyor. FIG. 13 shows another example 1300 where the conveyor system has a small footprint (e.g., length of the conveyor 1320) wherein a length of the conveyor is smaller than the container width 1330 such that objects 1303 cannot be dragged onto the conveyor belt along a linear / straight path without falling into the gap between the conveyor 1320 and the container 1330.

[0076] For complicated scenes, objects such as containers or boxes may not be perfectly perpendicular to each other, neatly aligned or in uniform size. To make any feasible pick and drop the boxes safely on the conveyors, the robot end effector may perform nonlinear motion. The example 1830 shows a top view of the motion paths 1831, 1832 of the end effector to drag the objects (e.g., box 1303) to the conveyor. The motion path in the 3D space may not be linear or a straight line. For instance, the conveyor mechanism may be narrower than the box wall such that to make sure that the boxes land safely on the conveyor, avoiding falling outside of the conveyor belt or falling into the gaps between the conveyors and the container wall, the robot arm may perform an “S” shaped maneuver as shown in the horizontal plane. In the vertical direction, the end effector may also move along a non-linear path as necessary to safely land the boxes on the conveyor.

[0077] The non-linear paths in the example 1810 may be for moving the end effector for collision avoidance purpose. As shown in the example, the path 1 is nonlinear for collision avoidance between the box and the conveyor. The starting point of path 1 in the vertical direction is below the height of the conveyor belt thus the end effector may lift up the box and drag it up to the conveyor belt along path 1. The path 2 is nonlinear for the collision avoidance between the robot arm and a previous box placed on the conveyor. The example 1820 showsvarious non-linear paths when the end effector or objects are located near a container wall or container ceiling.

[0078] In some cases, the non-linear paths or motion may be used to reposition or reorient the mobile manipulator system when the robot joint reaches a joint limit (e.g., to avoid singularities). The path 3 is nonlinear as the end effector / gripper is near the ceiling, and the robot joint may be close to a joint limit, and to smoothly move to another box surface, the end effector / gripper may move out, rotate (e.g., wrist rotation) and move onto the surface. Path 4 is nonlinear as the box is near the side container wall, and box may fall out of the conveyor range. The path 4 is a curved path to move the box out while avoiding collision with neighbor boxes.

[0079] As described above, the robot arm system may also perform complicated rotational movement between picks to avoid singularities and obstacles in the environment. The robotic arm movement can be highly non-linear. The robotic end effector (e.g.. robotic arm) travels a path from a start pose to a goal pose or goal position, wherein the path is determined based on a collection of samples in a physical or configuration space of the robotic system. In some cases, the path may be generated using artificial intelligence (Al) algorithms or a learning- embedded motion planning algorithm. For instance, the system may train a model operable to generate paths of the robotic end effector for sequential box handling, to determine a starting pose and a goal pose of the robotic end effector for sequential box handling, and to assess box stability during the sequential handling. In some cases, the model may be trained using reinforcement learning. The system may comprise a box stack simulation module where an ensemble of random box arrangements is generated. In some cases, the method may subsequently bias a plurality of samples of a configuration space of the robotic end effector. In many embodiments, the configuration space comprises the degrees of freedom or the full range of motion of the robotic end effector, and in many embodiments, samples of this space may be biased by the distribution generator to avoid collisions with obstacles in an environment. The algorithm further chooses a starting pose and a goal pose for the robotic end effector for a single pick and place operation. In some embodiments, the distribution generator module may provide high level human-like guidance by generating biased samples and goals to guide the motion plan. The distribution generator may be a 3D convolutional neural network (CNN), a feed-forward neural network, or other neural network or other machine learning model. The motion path planning method herein may utilize a starting pose, a goal pose, and a biased set of samples in the configuration space to generate a path for the robotic end effector for a single pick and place operation.

[0080] In some embodiments, the motion path planning method may utilize tree-based path construction methods, wherein a construction of a tree may be informed by biases assignedto configuration space sample points. In some cases, the method evaluates a stability of a configuration of boxes after a pick and place operation, assigning a score to the path generated by the motion path planning method. The method herein may evaluate a quality' of the distribution of samples of the distribution generator, similarly assigning a score for a generated path. In some embodiments, a distribution evaluator may optimize not only a path of an individual pick and place operation but also may globally optimize an entire sequential box handling process. The neural network may be a 3D CNN, feed-forward neural network, or other neural network or other machine learning model.

[0081] The distribution generator and distribution evaluator may be trained using a reinforcement learning algorithm. The training data may comprise a simulation of a random box stack, choosing a box from the randomly generated box stack, encoding of the environment (e.g., comprising obstacles and the loading and unloading areas), and annotated as a state. In some cases, after choosing a box to pick, a sample-based motion planning routine is initiated where a distribution of samples of the configuration space of the robotic end effector is encoded. A physics-based simulation assesses an effect of picking the chosen box on a remaining stack of boxes, computing a reward. In some embodiments, the reward may be based on a cycle time of a computed path, a smoothness of the computed path, or other criteria. Finally, the remaining stack of boxes may be encoded as a new state. The states and new states may be 3D voxel grids or point clouds. In some cases, the states and new states, the distribution, the reward may be defined for a specific chosen box in a particular randomly generated box stack and may be stored as training data. The distribution generator and distribution evaluator neural networks may be trained in tandem, in a reciprocating fashion using the training data as described above. The above-mentioned models may be trained off-line, and once trained, the models and parameters can be loaded into a memory of the computer system of the robotic system so that the robotic system can deploy the trained models to determine the paths when loading / unloading boxes.

[0082] Referring back to FIG. 1, the conveyor system can comprise a first conveyor mechanism 104 coupled to a second conveyor mechanism 106 and a third conveyor mechanism 108 coupled to the second conveyor mechanism 106. The conveyor mechanism may be a conveyor belt, wheel type of drive, or an omni-directional ball type. The first, second and third conveyor mechanisms can be implemented, for example, with conveyor belts. The conveyor mechanisms 104, 106, 108 are configured to move a box or package placed on one of the conveyor mechanisms to another one of the coupled conveyor mechanisms. For example, a box placed on the first (or front-facing) conveyor mechanism 104 is transported to the second (e.g.,directi on-changing) conveyor mechanism 106 and then to the third (or exiting / entering) conveyor mechanism 108. The conveyor system may operate in the reverse to load the boxes. For example, the box is on the third conveyor mechanism 108 and is transported to the second conveyor mechanism 106, and to the first conveyor mechanism 104 where the box is picked and stacked by the robotic arm 102.

[0083] The first 104 and second conveyor mechanisms 106 can be disposed on a lifting mechanism 110 so that their heights can be adjusted, particularly to adjust to an optimal height based on the height level of the boxes that the robot 102 is picking-placing so that the robot 102 can drag horizontally the boxes onto the first and / or second conveyor mechanism 104, 106. The lifting mechanism 110 may comprise a linear lift or scissor lift, for example. The lifting mechanism 110 as shown in FIGs. 1-4 is a scissor lift with a cylinder 1 18. The cylinder 118 may be a hydraulic or compressed air lift. The control system within the base is configured to control the height of the lifting mechanism 110, which controls the height of the first and second conveyor mechanisms 104, 106. It should be noted that the lifting mechanism is for illustration purpose only. Any suitable lifting mechanism can be employed. For example, the lifting mechanism may comprise forklifts or pully systems so long as the height of the conveyor can be controlled by the actuating the lifting mechanism.

[0084] In some embodiments, the second conveyor mechanism 106 may be an omnidirectional conveyor mechanism. In another embodiment, the second conveyor mechanism 106 is a direction changing conveyor mechanism. As shown, the second conveyor mechanism 106 moves packages traveling in a first direction, e.g., the conveyance direction of the first conveyor mechanism 104, to moving in a second direction, e.g., the conveyance direction of the third conveyor mechanism 108. In that connection, the second direction (e.g., the direction of the third conveyor mechanism) is perpendicular to the first direction (e.g., the direction of the first conveyor mechanism).

[0085] The third conveyor mechanism 108 is coupled to the second conveyor mechanism 106 on a first end and coupled to a supporting structure 120 on a second end. The second end is opposite the first end. The third conveyor mechanism 108 is configured to be height adjustable at the first end such that the first end moves with the lifting mechanism 110. The second end moves along a slide 116 on the supporting structure 120 to accommodate the change in height of the lifting mechanism 110. The supporting structure 120 comprises the slide 116. The slide 116 allows the third conveyor mechanism 108 to change angles.

[0086] For example, as the first end of the third conveyor mechanism 108 moves upward, the angle of the third conveyor mechanism 108 increases, and the second end movesalong the slide toward the first end. As the first end of the third conveyor mechanism moves down, the angle of the third conveyor mechanism 108 decreases and the second end of the third conveyor mechanism 108 moves away from the first end. The third conveyor mechanism 108 may form a ramped surface that changes its angle based on the height of the lifting mechanism 110.

[0087] As described above, the conveyor system may be an add-on system that is operably coupled to the mobile manipulator system. In some cases, the conveyor system may comprise a separate mobile base that is controlled to traverse the floor of the environment autonomously. For instance, the mobile base of the conveyor system may comprise powered wheels as described above allowing for omnidirectional movement of the conveyor system. Alternatively, the conveyor system may be removably coupled to the mobile base of the mobile manipulator system and once coupled, the conveyor system and the mobile manipulator system may traverse the floor together.

[0088] FIG. 16 shows an example 1600 of the conveyor system removably coupled to a mobile base of the mobile manipulator system. In some cases, the base of the conveyor system 1603 may be removably coupled to the platform 1605 of the mobile manipulator system via a coupling mechanism 1601 such as quick install / release means (e.g.. hinge, hook, magnets, spring-loaded levels, etc.). In some cases, the base of the conveyor system 1603 may be coupled to or released from the mobile base 1605 of the mobile manipulator system manually without using a tool. In some cases, the coupling mechanism may be a hinged mechanism allowing at least one-degree of freedom along the hinge axis (e.g., pitch) while constraining relative movement between the conveyor based and the mobile manipulator system base in other directi ons / orientations (e.g., translational movement, or roll and yaw orientation rotation). For instance, once coupled, the reference frame of the mobile manipulator system and the reference frame of the conveyor system is registered automatically in at least two or more orientations (e.g., roll, yaw). Such relative motion may allow the coupled system navigating across floors while adapting to uneven surfaces (e.g., ramp, bump, etc.). Alternatively, the conveyor system and the mobile manipulator system can have translational movement relative to one another and the position of the conveyor belt and the robotic arm may be calculated based on the sensor data (e.g.. collected by the 3D sensing mast).

[0089] As described above, the conveyor system may have a unique design by including at least a transitional conveyor belt connecting a front / horizontal conveyor belt and a side / ramp conveyor. The front conveyor may be located with a length along a length of a box wall and moving the boxes in a first direction, and the side conveyor may be located substantially at theside to transfer the boxes to the next location. The side conveyor may or may not have ramp depending on the application. In some cases, the heigh of the front conveyor, the transitional conveyor or the side conveyor may be adjustable as described elsewhere herein. Unlike conventional conveyor belt design lacking of the transitional conveyor (assisting in changing moving direction of the object) which typically introduces box stuck, falling, or other undesired maneuvers, the transitional conveyor (e.g., second comer conveyor 106) assist in transferring the boxes from the first front conveyor (e.g., first front conveyor lies horizontally 108) to the third ramp conveyor (e.g., 104) which may have substantially different directions (e.g., first conveyor and third conveyor form an ‘L’ shape or interest at a non-zero angle). The third ramp conveyor may transfer the boxes further down to another conveyor line. The transitional or comer conveyor serves as the transition between two box travelling directions which beneficiary ensures boxes staying on the conveyor.

[0090] It should be noted that the transitional conveyor as shown in FIGs. 1-5 are for illustration purpose only. The transitional conveyor or comer conveyor can be in any other suitable forms. For example, the transitional conveyor may be a conveyor belt as shown in FIGs. 1-5, or a conveyor with omni-directional wheels to improve transferability. In another example, the transitional conveyor may be a two directional slit belt conveyor.

[0091] As described above, the robotic system may have a compact design or reduced footprint thereby allowing for improved mobility and flexibility to adapt to various complicated scenes or environments. FIG. 11 and FIG. 14 show examples of the robotic system performing the loading / unloading operations within a container 1100 environment and within a warehouse 1400. Unlike conventional solutions that may have a conveyor belt with a dimension to fit the full size of a container (e.g., conveyor extended to full width and height of a container), the system herein may utilize a conveyor system with a length smaller than a workspace dimension (e.g., width, depth, etc) such as a width of a typical container while the mobility of conveyor system and the robotic system is improved allowing for omnidirectional movement of the system to adapt to various environments. For instance, the conveyor may not extend the full length of the standard shipping container. When fully extended vertically, the system also may not extend the full height of the standard shipping container. The smaller footprint or width allows the robot system to navigate more freely inside the container. Additionally, the improved mobility of the robotic system (e.g., powered wheels of the mobile base) may allow the system to translate on the floor (e.g., forward, backward, left and right), thereby reaching boxes that are further away without requiring a longer robot arm. As described above, the mobile base of the robotic system may comprise powered steering wheels where each wheel has two degrees offreedom. This beneficially allows the mobile base of the robotic system and / or the conveyor system to have a translational movement with zero turning radius. Additionally, without changing the yaw angle of the robot, the mobile base is capable of directly moving in any direction on the travelling floor. As shown in FIG. 11, the mobile base of the mobile manipulator system may move the mobile manipulator system left and right when reaching the boxes located to the far left and right. In some cases, the mobile base of the robotic system may move the conveyor system along with the mobile manipulator system such as forward to be at an optimal distance to the front wall of the boxes.

[0092] In some embodiments, the maximum height of the conveyor system may be lower than a height of a shipping container. Due to the scissor type lifting mechanism, the lifting height is proportional to the scissor lift width. A reduced maximum height of the conveyor system beneficially allows for a smaller footprint of the robotic system. The reduced height of the conveyor system does not comprise the reachability of the robotic system as the robot does not need to reach to the ceiling of the container. For the boxes packed all the way to the ceiling of the container, the conveyor system moves to a height that is below the bottom surface of the target box. Details about determining the optimal height of the conveyor system are described later herein.

[0093] FIG. 5 depicts an example of the robotic system of FIG. 1 with a plurality' of boxes 130 to pick-place, according to various aspects of the present invention. A plurality of boxes 130 is stacked in front of the robotic system 100, e.g., adjacent to the first conveyor mechanism 104. The robot 102 is configured to pick up the boxes 130 and move them to the first conveyor mechanism 104. The boxes 130 are then moved to the second conveyor mechanism 106 and then, after changing directions, to the third conveyor mechanism 108.

[0094] FIG. 6 also depicts an example of the robotic system of FIG. 1 with a plurality of boxes to pick and place, according to various aspects of the present invention. The plurality of boxes 130 is stacked in front of the robotic system 100. The robotic system 100 includes a fourth conveyor mechanism 122 in this depicted embodiment that is coupled to the third conveyor mechanism 108. This figure shows that the robot 102 can pull or drag the boxes horizontally - as opposed to be lifted vertically — onto the first conveyor 104, which then conveys the picked boxes one-by-one to the second, third and fourth conveyor mechanisms in sequence.

[0095] For an unloading application, once the robotic system 100 removes all objects in the front, row-by-row, the robotic system 100 can move forward to the next panel or wall of objects so that the robotic system 100 can commence conveying away the next panel or wall ofobjects, row-by-row, from top to bottom, in the same manner, with the process being repeated until all of the objects to be conveyed away are conveyed away. In a packing application, once the robotic system 100 stacks, row-by-row, from bottom to top, the object for the back panel or wall of objects, the robotic system 100 can move backward to commence stacking the next panel or wall of objects. In that connection, the robotic system 100 can be mobile and include, for example, wheels, continuous tracks, or legs for moving the robotic system 100 forward and backward.

[0096] FIG. 7 is a representation of a partial scene 200 with a plurality of boxes 202 to be picked and placed according to various aspects of the present invention. The plurality of boxes 202 may be stacked at a variety of heights such that the height levels of the boxes - from one side of the area to be picked to the other left to right in FIG. 7 (e.g., the width of the shipping container) — are not a straight line. Rather, height levels of the boxes 202 across the scene forms layers. For example, the first box 202a, the second box 202b, and the third box 202c may form the first layer 206, albeit a non-level one in the example of FIG. 7. The fourth box 202d, the fifth box 202e, and the sixth box 202f may form the second layer 208, again not level in the example of FIG. 7.

[0097] The plurality of boxes 202 may have non-uniform heights above, or relative to. the first conveyor mechanism 104. For example, the first box 202a may have a first height 204a above the first conveyor mechanism, the second box 202b may have a second height 204b above the first conveyor mechanism 104, and the third box 202c may have a third height 204c above the first conveyor mechanism 104. As shown, the height of the boxes relative to the first conveyor mechanism does not have to be the same. For example, the first height 204a is less than the second height 204b. The control system can control the height of the first conveyor mechanism 104 to reduce or minimize how far the boxes in a layer have to be dropped onto the first convey’ or mechanism 104. That is, the control system can position the first conveyor mechanism 104 just below the lowest / bottom edge of the lowest box on a layer, e.g., box 202a for the first / highest layer in the example of FIG. 7. After the box on the first layer are picked and placed, the control system can lower the first conveyor mechanism 104 to just below the lowest / bottom edge of the lowest box on the second layer, e.g., box 202d in the example of FIG.7. This process can be repeated to the bottom of the stack of boxes. It is possible that the lowest layer(s) of boxes might have to be lifted onto the first conveyor mechanism 104, but the control system can adjust the height of the first conveyor mechanism 104 to reduce how far the lowest boxes in the stack need to be lifted.

[0098] The plurality of boxes may have non-uniform heights above, or relative to, the second conveyor mechanism 106. The first 104 and second 106 conveyor mechanisms are coupled together and can be disposed by the lifting mechanism at the same height. The boxes may be disposed in front of the first conveyor or the second conveyor.

[0099] FIG. 8 is a method of operating the robotic system of FIG. 1. The method 300 is described in conjunction with the robotic system 100 of FIG. 1 and the scene 200 of FIG. 7. The method 300 may start at step 302 based on an input, e.g., a command to the robot 102 to start picking.

[0100] A sensor scans at step 304 the scene 200. The scene 200 may include the plurality of boxes or packages 202. The sensors may be included with the sensor tower 114, on the conveyor mechanisms 104, 106, and / or 108, or on both. The sensor sends the scene data to the control system of the robotic system 100. In one embodiment, there are multiple sources of inputs to scan the scene.

[0101] The control system of the robotic system 100 resolves the boxes to be picked in the scene data captured by the sensor tower 1 14 and computes at step 306 a desired height of the first conveyor mechanism 104 based on its identification of the boxes, including their location within the scene. The height of the first conveyor mechanism 104 is preferably based on the topmost layer of boxes. The topmost layer of boxes is the layer where the boxes above the first conveyor mechanism 104 are within a threshold height above the first conveyor mechanism 104. The threshold height can be the maximum height a box can be picked without damaging the box when moved to the first conveyor mechanism 104. In some cases, the height threshold (d_max) may be determined based at least in part on the weight of the object. For instance, a relationship between height threshold and the weight of the box may be obtained based on empirical data and is stored in a configuration file. The system may assign a height threshold to each layer based on the weight of an object in the scene and the configuration file. The weight of the object in the scene may be sensed by the robotic arm (e.g., torque sensor, force sensor, etc.), sensed by the conveyor system (e.g., scale), obtained from a third-party system (e.g., customer or shipping company), or inputted by the user or other sources. In some cases, the height threshold (d_max) may be desired to have a greater value to maximize the number of boxes can be picked up at one layer / height thereby decreasing the frequency of adjusting the height. The system may then adjust an initial value of the height threshold (e.g., decrease the threshold if the box is heavy) to ensure safety of the manipulating the object.

[0102] For example, the control system receives the data from the scene 200 (e.g., 3D scene data captured by the 3D sensing mast). A box can be picked if the bottom of the box isabove the height of the conveyor mechanism 104. For example, in FIG. 7, the first 202a, second 202b, and third boxes 202c can be picked because these boxes are above the height of the first conveyor mechanism 104. The fourth 202d, fifth 202e, and sixth boxes 202f cannot be picked since they are below the height of the first conveyor mechanism 104.

[0103] In some cases, the height of a box may refer to a dropping distance. The dropping distance may be related to a height of a bottom side of the box relative to the conveyor surface 104 from above. A plurality of boxes may have non-uniform heights e.g., 202a, 202c.

[0104] In some cases, while a layer of boxes that is associated with a conveyor height may be defined as boxes with a bottom side not lower than a conveyor height, the system may pick up box with a bottom side that is located below a top surface of the conveyor 104. For example, the box 202f may have a bottom surface that is positioned below the conveyor 104 at a distance 202g (i.e.. negative distance or lifting distance). The algorithms for determining the layer of box may determine whether the box 202f can be lifted up and placed onto the conveyor 104 and if so, merge the box 202f into a pick-up task for the layer. In some cases, the algorithm may be the task planning algorithm as describe elsewhere herein that determines whether to merge a box located below a conveyor height to a box layer above the conveyor height based at least in part on a weight of the box, the size and orientation of the box, and / or whether a collision exists between the robotic end effector and the conveyor for grabbing the box at a surface (e.g., top surface, front surface, etc.).

[0105] The height of the first conveyor mechanism 104 may be calculated based on the number of boxes that the robot 102 can pick in the topmost layer of boxes and the number of boxes within the threshold height. The height of each of the boxes above the conveyor mechanism is determined. For example, the second height 204b of the second box 202b may be above the threshold height and the first conveyor mechanism 104 may need to be raised before the second box 202b can be picked.

[0106] The robotic end effector may determine which box to move or the order of picking up the boxes within a layer utilizing a task planning algorithm. The robotic end effector may or may not move the boxes next to one another and may skip some boxes and may reach for boxes located below the current conveyor height. The task planning algorithm may determine the next box to pick up based at least in part on the scene (e.g., to ensure boxes not fall or slip), the moving speed of the robotic arm and / or the rising / lowering speed of the lift mechanism of the conveyor to improve an overall throughput of the system. For instance, a slower lifting speed may result in more boxes being grouped in the same layer (i.e., increased height threshold d_max) thereby reducing the lift movement frequency. As an example, the algorithm maydetermine, based at least in part on a three-dimensional (3D) scene containing the plurality of objects, a top layer of objects from the plurality of objects to maximize the number of objects in the top layer to be moved to the first conveyor mechanism and determine the height for the first conveyor mechanism, wherein the height is determined at a dropping distance from the top layer of objects to reduce a frequency of adjusting the height (by grouping / merging as many boxes to pick up as possible including boxes located below the conveyor height).

[0107] The task planning algorithm may determine the target box and generate motion planning for the end effector to pick up the box for complex scenes. In some case, the task planning algorithm may comprise a highly efficient learning-embedded motion planning algorithm for sequential box motion planning. The learning-embedded motion planning algorithm can be the same as those described above. The height control and movement of the conveyor is choreographed with the movement of the robot arm to improve efficiency and prevent collision. Details about the height control algorithm for computing the box layer or the height of the conveyor system and the task planning algorithm for generating motion planning for the end effector are describe later herein.

[0108] The control system controls the movement of the lifting mechanism 110 to the determined height. The lifting mechanism 110 moves 308 to the height determined.

[0109] The robot 102 grasps a box along the layer of boxes. The box is moved to the first conveyor mechanism 104 or the second conveyor mechanism 106. The box is then transported down the first conveyor mechanism 104 to the second conveyor mechanism 106 to the third conveyor mechanism 108. The control system determines 312 if the layer of boxes is finished.

[0110] For example, as shown in FIG. 7, the first box 202a is moved from the first layer 206 to the conveyor mechanism. The control system determines 312 if the first layer 206 is finished. The first layer 206 is not finished since the second box 202b and the third box 202c remain. The process repeats grasping and determining if the layer is finished for the second box 202b and the third box 202c. After the third box 202c is moved, the control system determines the first layer 206 is finished because no boxes remain above the first conveyor mechanism 104 to be picked.

[0111] The control system determines 314 whether there are more boxes remaining in the scene 200. For example, in FIG. 7 there is a second layer 208 of boxes remaining. Since there are boxes remaining the method 300 proceeds to compute 306 the height of the first conveyor mechanism 104. The first conveyor mechanism 104 is moved to the height for thesecond layer. The method then grasps 310 the boxes in the second layer 208 until the layer all the boxes in the layer have been removed. The control system determines 312 the layer is finished since no boxes remain in the scene 200. Once the second layer 208 is finished, the control system determines that there are no more boxes in the scene. The method ends 316.

[0112] The process of FIG. 8 is for removing (or unpacking) one panel or wall of objects. As mentioned above, in various embodiments, once the front panel or wall of objects is unpacked according to the method of FIG. 8, the robotic system 100 can move forward to unpack the next panel or wall of objects using the method of FIG. 8, and the process can be repeated until all of the objects to be unpacked are unpacked. A similar, but opposite, process can be used to pack / stack the objects.

[0113] FIG. 9 illustrates another example of a control algorithm 900 for controlling the robotic system in an unloading process. The control algorithm 900 can be same as those described above. For example, the control algorithm may comprise first constructing a box library 901. The box library may be constructed to track and store the size, position and orientation for each of the boxes identified in the scene and dynamically updated upon detection of a change in the scene. FIG. 9 shows an example of an algorithm 920 for constructing the box library 901. The method 920 may comprise obtaining scene data such as by an imaging device 921. The scene data may be based on image data captured by the RGB-D cameras (e g., 3D sensing mast of the system). The image data may be received and processed by one or more processors of the robotic system. In some embodiments, Al-based estimation 923 may be employed to estimate the sizes, positions and orientations for the boxes within the view. In some cases, pre-processing of the capture image data may be performed. In an embodiment, the preprocessing algorithm can include image processing algorithms, such as image smoothing, to mitigate the effect of sensor noise, or image histogram equalization to enhance the pixel intensity values. Al-based approaches as described elsewhere herein may be employed to generate a depth map of the scene or environment. In some cases, computer vision (CV) techniques or computer vision systems may be used to process the sensing data to extract high- level understanding of the environment, object detection, object classification, extraction of the scene depth and estimation of relative positions of objects, extraction of objects’ orientation in space as described above. FIG. 10 shows an example of a constructed box library 1022 that is fed to the box selection algorithm as input for determining a target box.

[0114] The algorithm 920 may monitor and detect a change in the scene 925. As an example, the algorithm may cross-compare the boxes from the current view with history information (previous image frames) to decide if any of the boxes has been moved. The updateinformation is merged to the box library 901 or to update 907 the box library. The box library 901 may also update based on the operation of the end effector 927. For example, once the mobile manipulator system completes a pick, the corresponding box is removed from the box library. The algorithm may further perform the scene change detection 925 to verify the box removal.

[0115] Referring back to the control algorithm 900, based on the 3D scene data, the algorithm may compute one or more box walls (e.g., front wall and back wall) 903 within the scene. In some cases, the scene may have one or more box walls. The one or more box walls along with the environment information may be used to determine a parking location for the robotic system. FIG. 12 shows a top view of a scene 1210 having a front wall of boxes 1211 and a back wall of boxes 1213. The parking location of the robotic system may ensure sufficient clearance between the conveyor 1215 and the front wall of boxes 1211 to avoid collision while ensuring the boxes can be moved safely to the conveyor surface.

[0116] The park location of the robotic system (e.g., x, y coordinates or location in the horizontal plane) may be determined based at least in part on the 3D scene data as described above. As described above, the robotic system or the conveyor system may have a dimension smaller than a regular shipping container, or smaller than a box wall. For example, a length of the first conveyor (e.g., 1320 in FIG. 13) may be shorter than the box wall or smaller than a full width of the container (e.g., width between container wall 1330). A location of the conveyor system or robotic system with respect to the front box wall and the container wall in the horizontal plane may be determined to park the robotic system. The control method may employ a park location control algorithm to compute the park location to navigate the robotic system (e.g., actuating the mobile base) to the park location. The park location may be determined to provide improved efficiency such as grasping more boxes at one park location and provide safely such as sufficient clearance when the conveyor moves to different park locations to avoid collisions. FIG. 13 show s an example of a park location control algorithm 1300. The method may comprise classifying / identifying one or more box walls including boxes in the front wall 1305, 1301 and boxes in the back wall 1303 and other objects in the scene such as container walls 1330. The front wall may be defined as front box wall that can be picked at a current robotic system’s park location and the back box wall may be defined as back wall boxes that cannot be picked at the current robotic system’s park location.

[0117] In some embodiments, the algorithm for determining the park location may be an optimization algorithm. In some cases, the optimization algorithm comprises maximizing anumber of the objects within a reachable space of the robotic arm. As an example, the algorithm may search for the park location by: i) maximize the number of boxes falling within the robot arm 1310 reach (this demands the system stay closer to the box wall for efficiency) ii) have enough clearance 1341 between the robotic system (e.g., conveyor 1320) and the container (e.g., container wall 1330) iii) have enough clearance 1342 between the robotic system (e.g., conveyor 1320) and the front box wall (a large gap for safety) iv) minimize the gap 1342 between the conveyor 1320 and the box wall so that the heavy box is not dropping to the gap in between (e.g., demands a smaller gap 1343 for safety when box is heavy) v) successfully pull boxes out from the box wall and flow out (e.g., demand a larger gap for safety 1344 when box is long)

[0118] The clearance for safety in (iii) may be a pre-determined distance that may be automatically generated by the system or configurable by a user. The initial safety clearance may be adjusted based on the size / weight of the boxes detected in the scene. For example, if the box in the current front wall is heavy or small, the clearance may be reduced to ensure the box not dropping to the gap. In some cases, upon determining the park location, the controller of the system may generate a path for navigating the robotic system (e.g., mobile base 1311) to the park location.

[0119] Referring back to FIG. 9, the method may proceed with computing the top layer for front box wall 905. The method may employ a height control algorithm for computing the box layer and / or the optimal conveyor height. The computation of the height of the conveyor can be the same as those described elsewhere herein. Unlike conventional methods that may simply align the conveyor belt to a bottom side of a layer of boxes, the height control algorithm herein may determine an optimal height for the conveyor to achieve both efficiency (e.g., grasping more boxes at one height) and safety (avoid box dropping to conveyor and damage by setting up a heigh threshold). The optimal height for the conveyor system may be determined to prevent excessive box dropping as well as minimize the lifting movement of the conveyor to save the motion time delaying the unloading process.

[0120] FIG. 10 shows an example of a height control algorithm 1010, 1011 for computing the box layer and / or the height of the conveyor. The algorithm can be the same as those described above. For example, the algorithm may comprise searching for layerheight / conveyor height by maximizing the number of boxes that is within a height threshold. Following is an example of algorithm for determining the layer height / conveyor height:

[0121] i) arrange the box heights by descending order as hQ, h1, hm, adding a negative value duncto consider the perception uncertainty' (e.g., perception uncertainty is based on the robotic system perception accuracy such as 2cm). max / V

[0122] ii) get all the boxes with heights are within the height threshold dmax, the height threshold can be a user defined value to indicate the maximal dropping distance (e.g., a configuration file showing 10 cm for normal weight boxes, a smaller threshold value such as 5 cm for heavy boxes) or determined based on real-time sensed box information (e.g., weight) as described above.

[0123] iii) The conveyor height is hN(e.g., box layer height 1221 in FIG. 12).

[0124] iv) pick the boxes whose height is larger than hNaccording to a rule (e.g., right to the left unless the heights are very different which will prefer pick top one first). The rule may determine the order or logic of selecting a target box for picking which can be determined by the task planning algorithm. In some cases, the rule may be pre-determined and / or configurable by a user or the system.

[0125] v) if the picking failed for the i-th highest box (i < N) due to collision between the robotic arm and the conveyor, the algorithm may search a new conveyor height by moving down the conveyor incrementally by 5, as long as the box height and the new conveyor height is less than 6 / max.

[0126] The above height control algorithm may be executed to dynamically control the height of the conveyor in accordance with movement of the robotic arm. As shown in the above (such as operation v), in some cases, the conveyor may be lifted or low ered within one box layer to avoid collision. Alternatively, the conveyor may not move until a layer of boxes is completed.

[0127] Referring back to FIG. 9, the method may proceed with planning for the next box movement 909. The present disclosure may provide a task planning algorithm for picking up the next box. The task planning algorithm may comprise selecting a box. generating a grasp planning and motion planning for moving the end effector.

[0128] FIG. 10 shows an example of a task planning algorithm 1020, in accordance with some embodiments of the present disclosure. As shown in the example, the algorithm may comprise box selection 1021, grasp planning 1023 and motion planning 1025.

[0129] In some cases, box selection algorithm 1021 may comprise ranking the box picking order based on the box’s position. In some cases, the box picking order may be further modified based on the shape, dimension, orientation of the stacked boxes to prevent the boxes from dropping. The box selection algorithm may comprise pre-determined logics, for instance, downstream box should be picked first than upper stream boxes, since the box needs to be moved out from interference faster. If between two boxes, the height differs over a predetermined threshold, the algorithm may pick the higher one first to minimize the likelihood of dropping. The pre-determined threshold to trigger the operation may be configurable, user defined (e.g., 0.5dmax) or automatically determined by the system based on empirical data or a configuration file. Output of the box selection algorithm may be a target box (i.e., next box to pick up). Alternatively, the box selection algorithm may employ artificial intelligence (Al) algorithms that take as input the 3D scene data and output a target box to pick up as described elsewhere herein.

[0130] The grasp planning algorithm 1023 may determine the grasp location on the box while avoid collision betw een the robotic arm and the conveyor. The grasp planning algorithm 1023 may comprise generating the grasps on the target box. The algorithm may comprise searching multiple faces on the target box, and for each face, searching for multiple gripper orientations, and for each gripper orientation, searching for multiple offset positions starts from centered picking. The algorithm may be an optimization algorithm that iterating the searching or exploring various locations until an optimal offset positions for grab is determined.

[0131] The generated grasps may be further checked for collision. In some cases, the grasp planning algorithm may take as input information about the box (e.g., size, position relative to the conveyor belt, w eight, maximum payload of the robotic arm. etc.) and output the grasp location on the box. The collision free grasps are output for the motion planning 1025. As an example, when the box selection output includes a box located below a conveyor height or the front side is not accessible (e.g., boxes at the bottom layer), the grasp planning algorithm 1023 determine the top face of the box to grasp and lift the box to the conveyor surface. As another example, the topmost layer of boxes in the stack may not be accessible from the top side and the grasp planning algorithm may output the front side and location on the front side for grabbing.

[0132] As described above, when lifting an object from the side, the robot payload drops significantly due to a distance of the object located away from the motor resulting in a large torque applied to the motor. The grasp planning algorithm 1023 may generate an optimal picking location for grasping the object (e.g., from the top side as illustrated in 1011) unless in certain locations such as a higher location that the top side cannot be easily accessed (e.g., illustrated in 1010), the robot system may grasp the object from the side and drag the box onto the conveyor. In some cases, the height for the conveyor may be computed in accordance with executing the task planning algorithm such that the conveyor height is optimized in accordance with the robot arm motion.

[0133] In some cases, the task planning system or grasp planning algorithm may further improve the throughput of the system such as by determining a layer of boxes with height above a conveyor height but within a height threshold as described above and may merge a box with a bottom side located beneath a conveyor height. As shown in FIG. 19, a box that is positioned beneath a conveyor height may be grasped from the top side 1910 and may be lifted onto the conveyor along a non-linear path 1920. The task planning system may determine whether to merge a lower box into a current layer of boxes based on factors such as the height difference (e.g., smaller height difference is likely to trigger a merge), weight of the object and payload capacity7of the robotic arm.

[0134] The task planning algorithm can accommodate various complex scenes such as determining grasping location and selection of target box and / or motion of the end effector for boxes not aligned, leveled and leaning on one another.

[0135] The motion planning algorithm 1025 may include running multiple motion planning threads (e.g., 3 threads. 4 threads, 5 threads. 6 threads, etc.) for the grasps of each orientation. The motion planning results on each orientation may be ranked first according to the path length. Then the paths of different orientations may be cross-compared to generate the final result.

[0136] In alternative embodiments, the box selection algorithm, motion planning, or grasp planning for the end effector may employ neural network as described above. For instance, the algorithm may be a model trained using reinforcement learning as described elsew here herein that take as input the 3D scene and output a motion for the end effector.

[0137] Referring back to FIG. 9, next, the method may determine if the scene has changed 925 and if there is no change detected, the method may actuate the robotic arm to pick and place the box 911. If there is a change in the scene, the method may update the box library907 and repeat the operations 903, 905, 909. The method may control the movement of the conveyor (accessory) 927 to transfer out the box that is placed on the conveyor. The operations may be repeated until a layer is finished 914. Upon finishing a box layer, the height of the conveyor may be calculated to move the conveyor to the next box layer 915. The operations may be repeated until the front box wall is completed 916. Once the front wall is completed, the mobile base of the robotic system may be moved to a next park location 917 for operations on the next box wall.

[0138] The above control algorithm improves efficiency and throughput of the robotic system by reducing conveyor movement with optimization, parallelizing robotic arm and conveyor movement. The control algorithm also ensures safety by capable of moving the boxes with the conveyor to avoid dropping boxes to the floor (e.g., handling fragile boxes with high confidence).

[0139] The control system may be implemented with an on-board computer device, e.g., on-board the robotic system, along with a controller. In another embodiment, the control system may be implemented with an off-board device, e.g., a laptop or desktop outside the robot body. The computer device may comprise one or more processors and computer memory. The computer memory may store computer instructions (e.g., software) that are executed by the processor(s). In particular, for example, the software stored by the computer memory can comprise software for controlling the robotic arm 102 and the lifts for the conveyor mechanisms. As such, the processor(s) can execute the software and based thereon, send signals to the controller(s) to control the robotic arm and the lift mechanisms. In some embodiments, the one or more processors may be a programmable processor (e.g, a central processing unit (CPU), a graphic processing unit (GPU), a general-purpose processing unit or a microcontroller), in the form of fine-grained spatial architectures such as a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and / or one or more Advanced RISC Machine (ARM) processors. In some embodiments, the processor may be a processing unit of a computer system.

[0140] The control system’s processor(s) may comprise one or more, preferably multicore, processors. The memon may comprise multiple memory units and store software or instructions that are executed by the processor(s) as described above. The memory units that store the software / instructions that are executed by the processor may comprise primary computer memory, such as random-access memory (RAM) or read-only memory (ROM). The software may also be stored in secondary computer memory, such as hard disk drives and solid- state drives.

[0141] The software code described above may be implemented using any suitable computer programming language, such as SQL, MySQL, HTML, C, C++, or Python, and using conventional, functional, or object-oriented techniques. Programming languages for computer software and other computer-implemented instructions may be translated into machine language by a compiler or an assembler before execution and / or may be translated directly at run time by an interpreter. Examples of assembly languages include ARM, MIPS, and x86; examples of high-level languages include Ada, BASIC, C, C++, C #, COBOL, Fortran, Java, Lisp, Pascal, Object Pascal, Haskell, and ML; and examples of scripting languages include Bourne script, JavaScript. Python, Ruby, Lua, PHP, and Perl.

[0142] The conveyor system and the mobile manipulator system can be standalone systems and may be compatible with any other existing mechanisms and systems. FIG. 15 shows the mobile manipulator system performing palletizing tasks. The mobile manipulator system can be a standalone system that works with any existing flex or telescopic conveyor. Robotics is to build a general-purpose mobile manipulator. The robotic system can perform multiple tasks without the assembly of the conveyor add-on. The robotic arm or mobile manipulator may perform unloading, palletizing, depalletizing, bin-picking, material transfer, with any existing systems without the conveyor system herein.

[0143] While preferred embodiments of the present invention have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A robotic system for transferring objects, the robotic system comprising: a conveyor system comprising a first conveyor mechanism to receive one or more objects and a lifting mechanism; a mobile manipulator system controlled to grasp and move one or more objects to the first conveyor mechanism; and a controller configured to: i) receive sensor data of an environment containing a plurality of objects, the conveyor system and the mobile manipulator system, ii) determine a conveyor height by identifying a number of the objects within a height threshold above the conveyor height based on the sensor data, iii) select a target object from the layer of objects and determine a grasp motion for actuating the mobile manipulator system to grasp and move the target object to the first conveyor mechanism.

2. The robotic system of claim 1, wherein the conveyor system further comprises a second conveyor connecting the first conveyor mechanism with a third conveyor mechanism, and wherein the mobile manipulator system grasps and moves the target object to the first conveyor mechanism, the second conveyor mechanism or the third conveyor mechanism.

3. The robotic system of claim 2, wherein the first conveyor mechanism and the third conveyor mechanism transfer the plurality' of objects in different directions.

4. The robotic system of claim 1, wherein the lifting mechanism is actuated to lift the first conveyor mechanism to the conveyor height determined in ii).

5. The robotic system of claim 1, wherein the controller is further configured to generate a park location for the first conveyor mechanism relative to the plurality of objects and the environment.

6. The robotic system of claim 5, wherein the conveyor system has a dimension smaller than a dimension of the environment, and wherein the park location is determined to be at a clearance distance to the plurality of objects in the horizontal plane.

7. The robotic system of claim 6. wherein the clearance distance is smaller than a threshold to prevent the plurality of objects from dropping into a clearance gap.

8. The robotic system of claim 1, wherein the sensor data is processed to identify a position, orientation and shape in a three-dimensional (3D) scene for the plurality of objects.

9. The robotic system of claim 8, wherein the controller is further configured to detect a change in the 3D scene based on the sensor data.

10. The robotic system of claim 1, wherein identifying the number of the objects within the height threshold above the conveyor height comprises identifying a maximum number of objects with dropping distances relative to the first conveyor mechanism and wherein the dropping distances are within the height threshold.

11. The robotic system of claim 10, wherein the height threshold is based at least in part on a w eight of the plurality of objects.

12. The robotic system of claim 10, wherein the height threshold is configurable by a user or determined by the robotic system based on weight sensed by the robotic system.

13. The robotic system of claim 1. wherein the target object is selected based at least in part on positions of the plurality of objects.

14. The robotic system of claim 1, wherein determining the grasp motion for actuating the mobile manipulator system comprises searching, for the target object, a side from a plurality of sides of the object for grasping.

15. The robotic system of claim 1, wherein the grasp motion comprises generating a pose and a path for moving an end effector of the mobile manipulator system.

16. The robotic system of claim 15, wherein the path is a non-linear path in 3D space for collision avoidance.

17. The robotic system of claim 1, wherein the grasp motion is generated using a model trained using reinforcement learning.

18. The robotic system of claim 1, wherein the plurality of objects have variable dimensions or are not leveled.

19. The robotic system of claim 1, wherein the conveyor system is removably coupled to a platform of the mobile manipulator system.

20. The robotic system of claim 19, wherein the platform is actuated to move to a park location with aid of a plurality of powered wheels.

21. The robotic system of claim 20, wherein each of the plurality of powered w heels is actuated by two motors.

22. A system for transferring objects, the system comprising: a front conveyor mechanism to receive one or more objects and transfer the one or more objects in a first direction; a transitional conveyor mechanism connected to the front conveyor mechanism and a side conveyor mechanism, wherein the side conveyor mechanism transfers the one or more objects in a second direction that is different from the first direction; a lifting mechanism controlled to move the front conveyor mechanism to a height; and a controller configured to: i) determine, based at least in part on a three-dimensional (3D) scene containing the plurality of objects, a top layer of objects from the plurality of objects to be moved to the front conveyor mechanism, the transitional conveyor mechanism or the side conveyor mechanism, ii) determine the height for the front or the transitional conveyor mechanism, wherein the height is determined at a dropping distance from the top layer of objects, wherein the dropping distance is within a height threshold to reduce a frequency of adjusting the height. iii) determine a park location to move a mobile base on the system utilizing an optimization algorithm.

23. The system of claim 22, wherein the park location is determined to be at a clearance distance to the plurality7of objects in the horizontal plane.

24. The system of claim 23, wherein the clearance distance is smaller than a threshold to prevent the plurality of objects from dropping into a clearance gap.

25. The system of claim 22, wherein the one or more objects are moved to the front conveyor mechanism, the second conveyor mechanism or the third conveyor mechanism by a robotic arm.

26. The system of claim 25, wherein the optimization algorithm comprises maximizing a number of the objects within a reachable space of the robotic arm.

27. The system of claim 22, wherein the controller is further configured to determine a target object from the number of objects in the top layer for picking up according to a pre-determined rule.

28. The system of claim 22, wherein the controller is further configured to determine a new height for the front or the transitional conveyor mechanism upon detection of a collision and wherein the new height is within the height threshold.

Images