Autonomous mobile robotic systems for use at commercial product facilities

Abstract

A robotic system includes a mobile robot for a commercial product facility is provided. The robot, equipped with an object interfacing mechanism, travels to vehicles within the facility. The system also features a detection system to gather environmental data and a control circuit connected to both the robot and detection system. The control circuit identifies target vehicles, determines travel paths, directs the robot to the vehicles, and uses detection data to analyze door characteristics. Based on this analysis, the control circuit devises and executes a door operation strategy, enabling the robot to open or close the doors of trailers or containers.

Description

AUTONOMOUS MOBILE ROBOTIC SYSTEMS FOR USE AT COMMERCIAL PRODUCT FACILITIESRelated Application(s)

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 627,305, filed January 31, 2024, which is incorporated by reference in its entirety herein.Technical Field

[0002] This invention relates generally to a robot system, and specifically to a robot system to interface with vehicles at commercial product facilities.Background

[0003] Truck yards are vital to the logistics and supply chain industry, acting central points for the loading and unloading of goods. Cargo trucks and trailers generally have doors that need specific operations to open and close, tasks traditionally carried out by drivers. These operations, including opening, closing, and inspecting the doors, are integral to management of vehicle and cargo traffic at truck yards.Brief Description of the Drawings

[0004] FIG. l is a block diagram of a system in accordance with some embodiments.

[0005] FIG. 2 illustrates an example of a mobile robot in accordance with some embodiments.

[0006] FIGS. 3-4 illustrate object interfacing mechanisms in accordance with some embodiments.

[0007] FIG. 5 is a flowchart of a method in accordance with some embodiments.

[0008] FIG. 6 is a flowchart of a method for identifying characteristics of a door in accordance with some embodiments.

[0009] FIGS.7A-7D depict an example door of a trailer in accordance with some embodiments.Detailed Description

[0010] Generally speaking, pursuant to various embodiments, systems, apparatuses, devices, and methods are provided herein useful to handle and manipulate doors of trailers / container of vehicles (e.g., yard trucks with a trailer / container). In some embodiments, a robotic system for use at a commercial product facility may comprise a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information of the working environment, and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to identify a target vehicle including a trailer or a container in the working environment, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travel path, obtain, via the detection system, detection data for a door of the trailer or the container of the target vehicle, identify, using the detection data, one or more characteristics of the door of the trailer or the container of the target vehicle, determine, based on the one or more characteristics of the door, a door operation strategy, the door operation strategy including using the mobile robot to open and / or close the door of the trailer or the container, and cause the mobile robot to execute the door operation strategy.

[0011] In a non-automated environment, there are many circumstances where vehicle drivers manually open / close the door of trailers / containers. For example, yard truck drivers should move from the driving seat to the door of the trailer / container (usually at backside of the trailer / container) to open / close the trailer / container door. In some circumstances, yard truck drivers open / close swing doors prior to backing a trailer / container to the dock door or pulling a trailer / container. The introduction of autonomous driver-less vehicles (e.g., driver-less yard trucks) may require an automated opening / closing of the door of trailers / containers.

[0012] Robotic systems and methods of using the robotic systems in accordance with some embodiments described herein may provide autonomous manipulation of doors oftrailers / containers of vehicles (e.g., yard trucks). In some embodiments, the robotic system may include a mobile robot that enables navigation of a working environment and traveling to vehicles at different locations in the working environment. In this manner, a single mobile robot may manipulate a plurality of vehicles at different locations within the working environment, and the vehicles with a trailer / container that need to be opened / closed may not need to be parked at a specific / designated area. Furthermore, the robotic systems and methods of using the robotic systems in accordance with some embodiments described herein do not require a 1 : 1 ratio of robotic system to vehicle. Therefore, the robotic systems, in accordance with some embodiments, may provide cost-efficient and space-efficient trailer / container door manipulation. According to some embodiments, the limits of the reach envelope and the dexterity of the object manipulating mechanism (e.g., an end-of-arm tool coupled to the robotic arm) may be reduced by introducing a mobile robot that is not statically located.

[0013] Various embodiments and examples of systems, devices, apparatus, and methods are described herein. FIGS. 1-7D are provided to illustrate various embodiments. It is noted that when describing certain embodiments, certain features may be shown in one or more of FIGS. 1- 7D.

[0014] FIG. 1 is a block diagram of a robotic system 100 in accordance with some embodiments. The robotic system 100 may include a mobile robot 120, a detection system 130, a memory 104, and a control circuit 102 communicatively coupled to the mobile robot 120, the detection system 130, and the memory 104.

[0015] The mobile robot 120 may navigate a working environment of the mobile robot 120 and travel to vehicles in the working environment. In some embodiments, the mobile robot 120 may be an autonomous mobile robot (AMR) capable of autonomously navigating the working environment. In some embodiments, the working environment may comprise an area of commercial product facility that may receive and interact with vehicles containing commercial products. In some embodiments, the working environment may include, but not limited to, a shipping / receiving yard or a parking area of a commercial product facility such as a warehouse, a fulfillment center, a distribution center, a store, and so on.

[0016] The mobile robot 120 may include a robot controller 122 configured to control the operation and / or motion of the mobile robot 120. In some embodiments, the mobile robot 120 may include a robotic mobility platform 124. In some embodiments, the robotic mobility platform 124 may include, but not limited to, one or more of wheels, drivers, tracks, motorized limbs / legs / arms / feet. For example, the robotic mobility platform 124 may include, but not limited to, a bi-pedal robot, a humanoid robot, a quadruped robot, a wheeled mobile robot, a tracked mobile robot, an airborne autonomous mobile robot, a terrain mobility robot, other types of autonomous mobile robot, and / or any combination thereof.

[0017] In some embodiments, the mobile robot 120 may include a mounted robotic mechanism 126. The mounted robotic mechanism 126 may include, but not limited to, an articulating arm (e.g., a 4, 5, and 6-axis robotic arm), a parallel-link industrial robot (e.g., a Spider robot), a SCARA industrial robot, and other robotic mechanisms (e.g., specifically designed robotic mechanisms), and / or any combination thereof. The mounted robotic mechanism 126 may be mounted to the robotic mobility platform 124, for example, the terrain mobility assembly. In some embodiments, the mounted robotic mechanism 126 may be a custom design robotic arm with two or more axes. In some embodiments, the mounted robotic mechanism 126 may be a robotic arm having four or six axes. In some embodiments, the robotic arm includes multiple arm segments that are pivotally, rotatably, and / or statically attached.

[0018] In some embodiments, the mobile robot 120 may include an object interfacing mechanism 128. The object interfacing mechanism 128 may be configured to grip, pick, release, move, change positions of, rotate, and / or twist various types of objects. The object interfacing mechanism 128 may be capable of manipulating doors of trailers / containers. For example, the object interfacing mechanism 128 may be capable of various actions to open / close a door of a trailer / container (e.g., removing a seal of the trailer door, latching / unlatching one or more bar locks, swinging the door to open or close, securing / fixing the door to the side of the trailer to prevent inadvertent movement of the door and unsecuring / detaching the opened door from the side of the trailer to move the door for closing. The object interfacing mechanism 128 may include, but is not limited to, an end effector (e.g., an end of arm tool, gripper, picker, opener, and so on). In some embodiments, the object interfacing mechanism 128 may include one or more of fingergrippers, suction grippers, other types of end effectors, and / or any combination thereof Some embodiments incorporate the some or all of U.S. Patent Application No. 63 / 536,608 entitled DEVICES AND METHODS FOR OBJECT MANIPULATION, and / or some or all of U.S. Patent Application No. 63 / 536,609, entitled SYSTEMS AND METHODS FOR FRIEGHT MANIPULATION, which are each incorporated herein by reference in their entirety.

[0019] In some embodiments, the mobile robot 120 may include a perception system 132 including one or more on-board sensing devices 134. The perception system 132 may navigate and / or find a travel path of the mobile robot 120 using data and information collected via the one or more on-boarding sensing devices 134. In some embodiments, the one or more on-board sensing devices 134 may include, but not limited to, one or more of a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an RGB overlay camera, a thermal sensor, an electromagnetic wave sensor, an optical sensor, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, and a microphone. The electromagnetic wave sensor may include one or more of a UV sensor and an RF sensor (e.g., including an RFID sensor). In some embodiments, the on-board sensing devices 134 may include an IMU, a LiDAR camera, and other types of sensors. In some embodiments, the perception system 132 and / or the on-board sensing devices 134 may be a part of the detection system 130. For example, at least a part or all of the sensing devices of the detection system 130 may be the onboard sensing devices 134. In some embodiments, the on-board sensing devices 134 may be disposed on, mounted on, and / or physically coupled to the mobile robot 120. In some embodiments, the on-board sensing devices 134 may be disposed on the robotic mobility platform 124._In some embodiments, the detection system 130 may be distributed in multiple locations.

[0020] FIG. 2 illustrates an example of mobile robot 120 in accordance with some embodiments. Referring to FIG. 2, the mobile robot 120 may include the robotic mobility platform 124, a multi-axis robotic arm 227, and the object interfacing mechanism 128. In some embodiments, the robotic mobility platform 124 may include a platform body 223 and one or more moving mechanisms. In some embodiments, the moving mechanism may include one or more robotic limb / robotic leg 225 coupled to the platform body 223. In some embodiments, the robotic mobility platform 124 may be a four-legged robot that can move in various directions (e.g., walkor run forward / backward / right / left / diagonally), rotate clockwise / counterclockwise, climb up and down, jump, and / or flip. In some embodiments, the platform body 223 may include a sensor array 229 including the one or more on-board sensing devices 134. In some embodiments, the sensor array 229 may be disposed on the front side of the platform body 223.

[0021] FIGS. 3-4 illustrate an object interfacing mechanism 128 in accordance with some embodiments. In some embodiments, the object interfacing mechanism 128 may be an end of arm tool (EoAT) that includes two or more fingers 342. In some embodiments, the shape of fingers may generally correspond to an object to be manipulated by the object interfacing mechanism 128 For example, the shape of the gripping surface of the fingers may corresponds to the shape of a handle of a trailer / container door, a latch of a trailer / container door, etc. Referring to FIG. 3, the object interfacing mechanism 128 may include rounded fingers 342. In some embodiments, the object interfacing mechanism 128 may include two rounded fingers 342. Referring to FIG. 4, the object interfacing mechanism 128 may include three fingers 342.

[0022] In some embodiments, the object interfacing mechanism 128 may further include a detachable insert 344 configured to accommodate the difference in external geometry of the objects manipulated by the object interfacing mechanism 128 (e.g., difference in external geometry of the door handles, latches of door locks, etc.). In some embodiments, the detachable insert 344 may be disposed on the interfacing surface of the object interfacing mechanism 128. The interfacing surface may be the surface of the object interfacing mechanism 128 that directly contacts / interfaces with the object manipulated by the object interfacing mechanism 128. In some embodiments, the interfacing surface of the object interfacing mechanism 128 may be the inner surface of the fingers 342 of the object interfacing mechanism 128.

[0023] In some embodiments, the detachable insert 344 may be an insulation member. For example, the detachable insert 344 may be an insulation lining disposed on the inner side of the rounded fingers of the EoAT. In some embodiments, the detachable insert 344 may include a polymer lining. In some embodiments, the polymer for the polymer lining may include urethane.

[0024] In some embodiments, the detachable insert 344 may be detachable such that the detachable insert 344 may be changed / r eplaced depending on the types of trailer / container doorsand / or components of trail er / container doors manipulated by the object interfacing mechanism 128. For example, the detachable insert 344 currently attached to the object interfacing mechanism 128 (e.g., gripper finger 342) may be detached from and another type of detachable insert 344 may be attached to the object interfacing mechanism 128 when the detachable insert 344 currently attached to the object interfacing mechanism 128 is not appropriate to manipulate the next object manipulated by the object interfacing mechanism 128.

[0025] In some embodiments, the object interfacing mechanism 128 may be a gripper including two or more fingers 342. Referring to FIG. 4, the object interfacing mechanism 128 may control each individual finger 342 independently. In some embodiments, the fingers 342 may be pneumatically and / or electromechanically actuated. In some embodiments, the object interfacing mechanism 128 may be water resistant and / or dust resistant. For example, the object interfacing mechanism 128 may be IP68 rated (e.g., water resistance in fresh water to a maximum depth of 1.5 meters for up to 30 minutes and complete protection against dust over extended time) for protection against water and / or dust. In some embodiments, the object interfacing mechanism 128 may be adaptive to shape of an object to be held and / or be replaced depending on each activity conducted by the object interfacing mechanism 128.

[0026] Referring back to FIG. 1, the detection system 130 may be configured to collect detection data 106 (i.e., information of the working environment of the mobile robot 120). In some embodiments, the detection system 130 may include, but not limited to, a 2-dimentional (2D) camera, a 3-dimentional (3D) camera, an RGB-D sensor, an LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, a microphone, or any combination thereof. The detection system 130 may scan or capture data from the working environment in real-time or near real-time during the operation of the mobile robot 120. In some embodiments, the detection system 130 may be local with respect to the mobile robot 120 (where, for example, the detection system 130 may be mounted or physically coupled to the mobile robot 120) or may be physically discrete in whole or in part from the mobile robot 120 as desired. In some embodiments, the detection system 130 may be distributed in multiple locations. In some embodiments, the object interfacing mechanism 128 may include one or more sensors 346 (FIG. 3) and the sensors 346 may collect detailed information ofobjects being currently manipulated by the object interfacing mechanism 128. The sensors 346 may improve the precision of operation of the mobile robot 120.

[0027] In some embodiments, the memory 104 may include a volatile and / or non-volatile memory. In some embodiments, the memory 104 may include a random-access memory (RAM). The memory 104 may serve to store computer instructions that, when executed by the control circuit 102, cause the control circuit 102 to behave as described herein. In some embodiments, the memory 104 may serve, for example, to non-transitorily store computer instructions. As used herein, this reference to "non-transitorily" will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence the may include both non-volatile memory (such as read-only memory (ROM) as well as volatile memory (such as an erasable programmable read-only memory (EPROM).

[0028] The memory 104 may provide storage for the detection data 106 and one or more modules 108. In some embodiments, the detection data 106 may include, but not limited to, image data (e.g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the field of view, sound data of the working environment, force data of force applied to the mobile robot (e.g., to the object interfacing mechanism 128), and / or any other types of sensor data collected via the detection system 130 that may be necessary or used to conduct steps, actions, and / or functions described herein. The memory 104 may also store data generated during the use / operation of the robotic system 100. The one or more modules 108 may include codes executable by the control circuit 102 and, when executed by the control circuit 102, cause the control circuit 102 to perform specific steps, actions, and / or functions designed by each module. In some embodiments, the modules 108 may include, but are not limited to, a point cloud accumulation module, a point cloud filtering module, and / or an image segmentation module. In some embodiments, the modules 108 may further include a distance calculation module, an inverse kinematics module, a motion planning module. In some embodiments, the modules 108 may further include modules facilitating other automation system operations including, but not limited to, a system safety module, a fleet management module, and a warehouse / yard management system interfacing module. In some embodiments, the one or more of the modules 108 may employ a machine learning model 107 toimprove capabilities for object identification, object manipulation, inspection processes, navigating, and ancillary tasks such as exceptional handling and so on. In some embodiments, the detection data 106 captured during the operation of the mobile robot 120 may be used to train the machine learning model 107.

[0029] In some embodiments, the control circuit 102 may operably / communicatively couple to the memory 104, mobile robot 120, and the detection system 130. The control circuit 102 may receive information from the detection system 130 and may store the information (e.g., detection data 106) received from the detection system 130 to the memory 104.

[0030] The control circuit 102 may access the memory 104 and execute the codes stored in the memory 104. In some embodiments, the memory 104 may be integral to the control circuit 102 or may be physically discrete in whole or in part from the control circuit 102 as desired. This memory 104 may also be local with respect to the control circuit 102 (where, for example, both share a common circuit board, chassis, power supply, and / or housing) or may be partially or wholly remote with respect to the control circuit 102 (where, for example, the memory 104 is physically located in another housing or remotely location). In some embodiments, the memory 104 may be distributed in multiple locations.

[0031] The control circuit 102 is configured, for example by using corresponding programming and / or using the modules 108 stored in memory 104, to carry out and / or send signals to carry out one or more of the steps, actions, and / or functions described herein. The control circuit 102 may comprise structure that includes at least one (and typically many) electrically-conductive paths (such as paths comprised of a conductive metal such as copper or silver) that convey electricity in an ordered manner, which path(s) will also typically include corresponding electrical components (both passive (such as resistors and capacitors) and active (such as any of a variety of semiconductor-based devices) as appropriate) to effect one or more of the steps, actions, and / or functions described herein. The control circuit 102, for example, may comprise a fixed-purpose hard-wired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit that is customized by design for a particular use, rather than intended for general-purpose use), a field-programmable gate array (FPGA), and thelike) or can comprise a partially or wholly-programmable hardware platform (including but not limited to microcontrollers, microprocessors, and the like).

[0032] FIG. 5 is a flowchart depicting a method 500 for use with a robotic system in accordance with some embodiments. The method 500 may be performed using the robotic system 100 in accordance with the approaches described above. Although the method 500 is mainly illustrated with the robotic system 100, the method 500 may also be performed with a robotic system differently configured.

[0033] In step 502, the control circuit 102 may identify a target vehicle in the working environment. The target vehicle may be a vehicle with a trailer / container door that needs to be manipulated (e.g., opened, closed, unlocked, locked, etc.). In some embodiments, the control circuit 102 may identify the target vehicle using the detection data 106 (information of the working environment collected via the detection system 130). The working environment may include vehicles and / or other objects in the operation environment / area of the mobile robot 120. In some embodiments, the detection data 106 may include information for the entire / overall area of the working environment collected via the detection system 130 to monitor vehicles in the working environment (e.g., movement of vehicles in the working environment, entering / leaving of vehicles in the working environment). Additionally or alternatively, the control circuit 102 may identify the target vehicle based on signals from vehicles or on signals from a user (e.g., a driver) of the vehicles. For example, when the vehicles are capable of directly or indirectly communicating with the control circuit 102, the vehicles may send signals requesting manipulation of the trailer / container door to the control circuit 102, and based on the requesting signals, the control circuit 102 may identify and / or determine the target vehicle. In some embodiments, the user of the vehicles may send door manipulation requesting signals to the control circuit 102, using a user device (e.g., mobile electronic devices). In some embodiments, based on instructions from an ancillary interfacing yard management system, the target vehicle may be identified, and the mobile robot 120 may travel to the target vehicle. In some embodiments, using the interfacing yard management module stored in the memory 104, the control circuit 102 may identify the target vehicle and cause the mobile robot 120 to travel to the identified target vehicle.

[0034] In step 504, the control circuit 102 may determine a travel path of the mobile robot 120 to the target vehicle. To determine the travel path, the control circuit 102 may determine the current location of the mobile robot and the current location of the target vehicle. In some embodiments, the current location of the mobile robot 120 and / or the target vehicle may be determined based on the detection data 106 collected via the detection system 130, and control circuit 102 may determine, using the determined current locations of the mobile robot 120 and the target vehicle, the travel path of the mobile robot 120 to the target vehicle. In some embodiments, the mobile robot 120 may include a built-in location tracking system (e.g., a GPS system) and the current location of the mobile robot may be determined using the built-in location tracking system. Similarly, when the target vehicle includes a built-in location tracking system, the current location of the target vehicle may be determined using the built-in location tracking system. In some embodiments, in determining the travel path, the control circuit may further consider other objects that may affect the travel of the mobile robot 120 such as static obstacles, frequently detected dynamic obstacles, etc. In some embodiments, the travel path may be from the current location of the mobile robot 120 to the door of the trail er / container of the target vehicle. In some embodiments, the mobile robot 120 may receive the current location of the target vehicle or the travel path of the mobile robot 120 from a yard management system and / or a command and control center that has mapped out the entire operating area (e.g., the working environment of the mobile robot 120) and that can command the mobile robot 120 to travel to the specific location.

[0035] In step 506, the mobile robot 120 may move / travel to the target vehicle according to the determined travel path. In some embodiments, the control circuit 102 may cause the mobile robot 120 to move / travel to the target vehicle according to the determined travel path (e.g., by transmitting the signals for the travel path to the mobile robot 120). In some embodiments, the detection system 130 (in some embodiments, including the perception system 132 of the mobile robot) may continue to collect the information of the working environment while the mobile robot moves / travels toward the target vehicle according to the determined travel path, and the control circuit 102 may recognize, based on the information of the working environment collected during the travel, obstacles that have not been detected / predicted when the control circuit 102 initially determine the travel path of the mobile robot 120 in step 504. In some embodiments, when thecontrol circuit 102 recognizes unpredicted obstacles while the mobile robot 120 travels to the target vehicle, the control circuit 102 may update the travel path in response to the recognition of the unpredicted obstacle and may cause the mobile robot 120 to move / travel to the target vehicle according to the updated travel path.

[0036] In step 508, the control circuit 102 may obtain, from the detection system 130, detection data 106 for the door of the trailer / container of the target vehicle (including detection data 106 for a door lock of the trailer / container, if the trailer / container door includes a door lock). The trailer / container of the target vehicle may use / include various types of doors and door locks. For example, the door of the trailer / container of the target vehicle may include, but not limited to, swing door(s) or a roll door. In some embodiments, the detection data 106 for the door of the trailer / container of the target vehicle may include, but not limited to, image data (e.g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the door (including the door locks if any) of the trailer / container of the target vehicle, sound data for the door area of the target vehicle and / or any other types of sensor data for the door of the trailer / container of the target vehicle. In some embodiments, the 3D point cloud of the door of the trailer / container of the target vehicle may be obtained via a LiDAR sensor of the detection system 130.

[0037] In step 510, the control circuit 102 may identify, using the detection data 106 for the door of the trailer / container of the target vehicle, one or more characteristics of the door of the trailer / container of the target vehicle. The characteristics of the door may include, but are not limited to, the type, shape, size, position, geometry, configuration, number, and / or operation mechanism of the door itself and / or the type, shape, size, position, geometry, configuration, number, presence, and / or operation mechanism of the components of the door (e.g., door locks, door handles, etc.). For example, the characteristics of the door may include the type / number of lock bars of the door of the trailer / container, presence and number of door latch(s) of the trailer / container, hook / latch point(s) of the door of the trailer / container, presence or absence of a seal on the trailer, and so on.

[0038] In some embodiments, the control circuit 102 may use 2D and 3D image data (e.g., 2D images, a 3D point cloud, or a combination thereof) and recognize edges, shapes, and near / far distances of objects to identify the objects (e.g., the target vehicles, the doors of thetrailers / containers, the door locks, the door handles, and other components of the door of the trailer / container, obstacles, objects in the trailers / containers, etc.) and / or the characteristics of the objects. In some embodiments, the control circuit 102 may detect / recognize object edges, shapes, and near / far distance of objects in the working environments of the mobile robot 120. In some embodiments, the control circuit 102 may use the 3D point cloud of the objects, detect the edge of each object in the 3D point cloud, and draw boundary lines for the objects based on the detected edges. Further, the control circuit 102 may determine the near / far distance of each object from the reference point using depth information of the 3D point cloud of the objects.

[0039] In some embodiments, objects in the working environments of the mobile robot 120 may be identified using a machine learning algorithm based on computer vision (CV) models. In some embodiments, the machine learning algorithm may use a CV model trained based on previously captured images of objects as input and object identifiers as categorizations. In some embodiments, the CV model may comprise a deep neural network object recognition model. In some embodiments, the CV model may be trained based on images captured during the operation of the robotic system 100.

[0040] FIG. 6 illustrates an example method 600 to identify the characteristics of the door in step 510 in accordance with some embodiments. In some embodiments, the characteristics of the door identified with the method 600 may include the characteristics of the door lock of the trailer / container door in accordance with some embodiments. In some embodiments, the detection data 106 for the door of the trailer / container of the target vehicle may include a 3D point cloud of the door lock. In some embodiments, the detection data 106 for the door of the trailer / container of the target vehicle may include a 3D point cloud for entire area of the door of the trailer / container of the target vehicle including the door lock. FIGS.7A-7D depict an example door of a trailer and a door lock in accordance with some embodiments. FIGS. 7A-7D may be referred to in describing the method 600, but method 600 is not limited by FIGS. 7A-7D.

[0041] In some embodiments, in step 602, the control circuit 102 may, with the point cloud filtering module, crop the 3D point cloud of the door of the trailer / container to remove the portion of the 3D point cloud not corresponding to the door lock (e g., to remove portion of doors and unrelated objects included in the 3D point cloud of the door of the trailer / container). In someembodiments, the point cloud filtering module may crop the 3D point cloud to leave only the portion of the 3D point cloud corresponding to door lock. FIG. 7A depicts an example of a collected image of a door 702 of a trailer including a door lock 704. FIG. 7B is an enlarged image of portion 730 illustrated in FIG. 7A to show the details of the door lock 704. FIGS. 7C-7D are images to illustrate cropping the 3D point cloud. In some embodiments, the 3D point cloud of the door of the trailer obtained in step 508 may be a 3D point cloud corresponding to the image of FIG. 7A. In some embodiments, the control circuit 102 may crop the 3D point cloud along the cropping boundary 706 (shown in FIG. 7C) of the door lock 704. In this case, the 3D point cloud cropped in step 602 may be a 3D point cloud corresponding to FIG. 7D including the door lock 704 encompassed by the cropping boundary 706.

[0042] In step 604, the control circuit 102 may generate a depth map image based on the cropped 3D point cloud generated in step 602. In some embodiments, the depth map image may be generated by taking each specific point of the cropped 3D point cloud and assigning a value (e g., darkness or color) for each point based on its distance from the reference point.

[0043] In step 606, the control circuit 102 may segment, with the image segmentation module, the depth map image generated in step 604 into image segments. The image segmentation module may include a segmentation algorithm. In some embodiments, each segment may include a matrix of true and false values for each pixel in the depth map image. In the matrix of each segment, the true value may indicate that the pixel of the true value is part of a specific segment, and the false value may indicate that the pixel of the false value is not part of the specific segment.

[0044] In step 608, the control circuit 102 may analyze the image segments to determine the one or more characteristics of the door of the trailer / container of the target vehicle. The control circuit 102 may analyze the image segments using a machine learning algorithm including a trained object identification model configured to identify objects and / or the characteristics of the objects by analyzing the image segments of the objects via artificial intelligence (Al). In some embodiments, the machine learning algorithm may involve patter recognition and / or imitation learning. In some embodiments, in step 608, the control circuit 102 may identify the type, shape, size, position, geometry, configuration, opening mechanism of the door 702 of the trailer / container when the door 702 is in a closed configuration, and closing mechanism of the door 702 of thetrailer / container when the door is in a open configuration. In some embodiments, in step 608, the control circuit 102 may identify the type, shape, size, position, geometry, configuration, unlocking mechanism of the door lock 704 when the door lock is in a lock configuration, and locking mechanism of the door lock 704 when the door lock is in a unlock configuration.

[0045] Although steps 602 to 608 are illustrated with an example of identifying a door lock of the trailer / container door, step 602 to 608 may be performed similarly to identify the characteristics of the door of the trailer / container and / or other components of the door of the trailer / container.

[0046] In some embodiments, steps 602 to 608 may be repeated to identify the characteristics for different components / portions / parts of the door of the trailer / container. In some embodiments, after identifying the characteristics of the first portion of the door (e.g., the door lock 704 of the door 702), steps 602 to 608 may be repeated to identify the characteristics of the second portion of the door (e.g., a handle to open / close the door 702). Steps 602 to 608 may be conducted / repeated as many times as necessary to identify the characteristics of the door of the trailers / containers.

[0047] Referring back to FIG. 5, in step 512, the control circuit 102 may determine, based on the identified characteristics of the door 702 of the trailer / container of the target vehicle, a door operation strategy of the mobile robot 120. In some embodiments, when the door lock 704 is in the lock configuration, the door operation strategy may include unlocking, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the door 702 by manipulating the door lock 704 (e.g., changing the configuration of the door lock 704 from the lock configuration to the unlock configuration by moving / manipulating the components of the door lock according to the unlocking mechanism of the door lock identified in step 510). In some embodiments, when the door 702 is in the closed configuration, the door operation strategy may include opening, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the door 702 by changing the configuration of the door 702 from the closed configuration to the open configuration according to the door opening mechanism identified in step 510. In some embodiments, the door 702 of the trailer / container may include a pair of wings (e.g., a swing door) and opening mechanism of the door may include rotating / pivoting one or two wings of the swing door about the hinge of eachwing. In some embodiments, the door of the trail er / container may include a roll door, and the opening mechanism of the door may include rolling the roll door about a roller by lifting up the lower end of the roll door. In some embodiments, the door opening mechanism may further include latching / fixing, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the opened door (e.g., fully rolled-up door or fully pivoted door) to the side of the trailer / container to secure it to the trailer / container body and prevent inadvertent movement of the opened door. In some embodiments, to secure the opened door, the trailer / container may include a door retainer on the side of the trailer / container to hold / fix the opened door. In some embodiments, when the door 702 is in the open configuration, the door operation strategy may include closing, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the door 702 by changing the configuration of the door 702 from the open configuration to the closed configuration according to the door closing mechanism identified in step 510. In some embodiments, when the door 702 of the trailer / container include the pair of wings (e.g., the swing door), the closing mechanism of the door may include rotating / pivoting back, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the one or two wings of the swing door about the hinge of each wing. In some embodiments, when the door of the trailer / container includes the roll door, the closing mechanism of the door may include unrolling the roll door by pulling down the lower end of the roll door. In some embodiments, when the opened door is fixed to the door retainer, the opened door may be unlatched / released, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), from the door retainer before moving the opened door to the closed configuration. In some embodiments, when the door lock 704 is in the unlock configuration, the door operation strategy may include locking the door 702 by manipulating the door lock 704 (e.g., changing the configuration of the door lock 704 from the unlock configuration to the lock configuration by moving / manipulating the components of the door lock according to the unlocking mechanism of the door lock identified in step 510).

[0048] In step 514, the mobile robot 120 may execute the operation strategy determined in step 512. In some embodiments, the mobile robot 120 may execute the door operation strategy by directly manipulating, e.g., using at least the object interfacing mechanism 128 of the mobile robot 120, the door 702 of the trailer / container. For example, the mobile robot 120 may execute the dooroperation strategy by directly manipulating the door 702 of the trailer / container and / or the component(s) of the door of the trailer / container without using an external tool. The external tool may be a tool that is not a part of the mobile robot and / or the target vehicle.

[0049] In some embodiments, while executing the operation strategy, the control circuit 102 may update the door operation strategy based on the detection data 106 continuously obtained during the execution of the operation strategy. In some embodiments, the detection system 130 may collect detection data 106 for an inside of the trailer / container when the door 702 of the trailer / container has been opened or while the mobile robot 120 is opening the door 702 of the trailer / container, and the control circuit 102 may obtain, from the detection system 130, the detection data 106 for the inside of the trailer / container. In some embodiments, while the execution of the operation strategy, the control circuit 102 may analyze, with the detection data 106 for the inside of the trailer / container, a stability of a stack of objects in the trailer or the container. In some embodiments, the control circuit 102 may update the door operation strategy in response to detection of an unstable object in the trailer / container. In some embodiments, the updated door operation strategy may include using the object interfacing mechanism 128 to prevent the unstable object from falling (e.g., adjusting, with the object interfacing mechanism 128, the positions of objects of the stack of objects and / or stopping the movement of mobile robot to open the door 702 and closing, with the object interfacing mechanism 128, the fully / partially opened door 702).

[0050] In some embodiments, the detection data for the inside of the trailer / container may be collected externally (e.g., via the detection system 130 being outside the trailer / container). For example, the detection system 130 may collect the information of the inside of the trailer / container before opening the trailer door. In some embodiments, to capture the information of the inside of the trailer / container of the target vehicle, the detection system 130 may include, but not limited to, an electromagnetic (EM) wave sensor configured to detect the electromagnetic wave emitted from or reflected at the inside of the trailer / container of the target vehicle. In some embodiments, the detection system 130 may include an EM wave generator and an EM wave reader. The EM waver generator may project EM waves, such as radio waves, infrared waves, or microwaves, into the inside of the trailer of the target vehicle. The EM wave reader or EM wave sensor may receive reflections of the EM waves from the inside of the trailer / container of the target vehicle. The EMwave generator and reader may be separate devices, in the same or separate housings, or can be incorporated into one device. In some embodiments, the EM wave generator and the EM wave reader may be mounted to the mobile robot 120. In other words, the on-board sensing devices 134 may include the EM wave generator and the EM wave reader. In some embodiments, using the captured information from the inside of the trailer / container, the control circuit 102 may determine the stability of objects in the trailer / container. For example, the on-board sensing devices 134 may include the electromagnetic wave sensor to see whether objects (e.g., boxes, cases) stacked in the trailer / container are likely to topple when the trailer / container door is opened.

[0051] In some embodiments, the mobile robot 120 is configured to change / replace the object interfacing mechanisms 128 (e.g., EoATs) based on the identified characteristics of the doors. For example, the mounted robotic mechanism 126 may release the object interfacing mechanism 128 currently coupled thereto and pick and couple another object interfacing mechanism 128 to be used. Additionally or alternatively, changing the object interfacing mechanism 128 may be conducted with an additional object interfacing mechanism. In some embodiments, the mobile robot 120 may further include an additional object interfacing mechanism configured to decouple and release the object interfacing mechanism 128 currently coupled to the mounted robotic mechanism 126 and pick and couple the next object interfacing mechanism 128 to the mounted robotic mechanism 126.

[0052] In some embodiments, a robotic system for use at a commercial product facility may comprise a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information of the working environment, and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to identify a target vehicle including a trailer or a container in the working environment, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travelpath, obtain, via the detection system, detection data for a door of the trailer or the container of the target vehicle, identify, using the detection data, one or more characteristics of the door of the trailer or the container of the target vehicle, determine, based on the one or more characteristics of the door, a door operation strategy, the door operation strategy including using the mobile robot to open and / or close the door of the trailer or the container, and cause the mobile robot to execute the door operation strategy.

[0053] In some embodiments, a method for use with a robotic system at a commercial product facility may comprises identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle including a trailer or a container in a working environment, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfdlment center and a distribution center, and a store, determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, moving the mobile robot to the target vehicle according to the determined travel path, obtaining, via the detection system, detection data for a door of the trailer or the container of the target vehicle, identify, using the detection data, one or more characteristics of the door of the trailer or the container of the target vehicle, determining, based on the one or more characteristics of the door, a door operation strategy, the door operation strategy including using the mobile robot to open and / or close the door of the trailer or the container, and executing, with the mobile robot, the door operation strategy.

[0054] Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

CLAIMSWhat is claimed is:

1. A robotic system for use at a commercial product facility, the robotic system comprising: a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; a detection system configured to collect information of the working environment; and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to: identify a target vehicle including a trailer or a container in the working environment; determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; cause the mobile robot to move to the target vehicle according to the determined travel path; obtain, via the detection system, detection data for a door of the trailer or the container of the target vehicle; identify, using the detection data, one or more characteristics of the door of the trailer or the container of the target vehicle; determine, based on the one or more characteristics of the door, a door operation strategy, the door operation strategy including using the mobile robot to open and / or close the door of the trailer or the container; and cause the mobile robot to execute the door operation strategy.

2. The robotic system of claim 1, wherein the mobile robot comprises a mobility platform comprising one or more of wheels, drivers, and / or motorized limbs.

3. The robotic system of claim 1, wherein the door operation strategy includes unlocking and opening the door of the trailer or the container and / or closing and locking the door of the trailer or the container.

4. The robotic system of claim 1, wherein the control circuit is further configured to obtain, via the detection system, detection data for an inside of the trailer or the container and analyze, with the detection data for the inside of the trailer or the container, a stability of a stack of objects in the trailer or the container.

5. The robotic system of claim 4, wherein the control circuit further configured to update the door operation strategy in response to detection of an unstable object in the trailer or the container.

6. The robotic system of claim 5, wherein the updated door operation strategy includes using the object interfacing mechanism to prevent the unstable object from falling.

7. The robotic system of claim 1, wherein the detection data for the door of the trailer or the container of the target vehicle includes a 3-dimensional (3D) point cloud of the door of the trailer or the container of the target vehicle, and wherein identifying the one or more characteristics of the door of the trailer or the container comprises: cropping the 3D point cloud of the door of the trailer or the container of the target vehicle to remove the 3D point cloud not corresponding to a door lock; generating, based on cropped 3D point cloud, a depth map image; segmenting the depth map image into image segments; and analyzing, using a machine learning algorithm, the image segments to determine one or more characteristics of the door lock.

8. The robotic system of claim 7, wherein the mobile robot is configured to change the object interfacing mechanism based on the characteristics of the door and / or the characteristics of the door lock.

9. The robotic system of claim 1, wherein the object interfacing mechanism includes an end effector comprising two or more fingers.

10. The robotic system of claim 1, wherein the control circuit is further configured to recognize, based on the information of the working environment, an unpredicted obstacle while the mobile robot travels to the target vehicle.

11. The robotic system of claim 10, wherein the control circuit is further configured to update the travel path in response to recognition of the unpredicted obstacle.

12. The robotic system of claim 1, wherein the control circuit is further configured to recognize object edges, shapes, and near / far distance of objects.

13. The robotic system of claim 1, wherein the detection system includes a 2- dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, or any combination thereof.

14. The robotic system of claim 1, the door of the trailer or the container may include a swing door or a roll door.

15. The robotic system of claim 1, the mobile robot executes the door operation strategy by directly manipulating, using the object interfacing mechanism of the mobile robot, the door of the trailer or the container.

16. The robotic system of claim 1 , the mobile robot executes the door operation strategy without using an external tool, wherein the external tool is a tool that is not a part of the mobile robot or the target vehicle.

17. A method for use with a robotic system at a commercial product facility, the method comprising: identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle including a trailer or a container in a working environment, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; moving the mobile robot to the target vehicle according to the determined travel path; obtaining, via the detection system, detection data for a door of the trailer or the container of the target vehicle; identify, using the detection data, one or more characteristics of the door of the trailer or the container of the target vehicle; determining, based on the one or more characteristics of the door, a door operation strategy, the door operation strategy including using the mobile robot to open and / or close the door of the trailer or the container; and executing, with the mobile robot, the door operation strategy.

18. The method of claim 17, further comprising obtaining, via the detection system, detection data for an inside of the trailer or the container and analyzing, with the detection data for the inside of the trailer or the container, a stability of a stack of obj ects in the trailer or the container.

19. The method of claim 18, further comprising updating the door operation strategy in response to detection of an unstable object in the trailer or the container.

20. The method of claim 17, wherein the detection data for the door of the trailer or the container of the target vehicle includes a 3-dimensional (3D) point cloud of the door of the trailer or the container of the target vehicle, and wherein identifying the one or more characteristics of the door of the trailer or the container comprises: cropping the 3D point cloud of the door of the trailer or the container of the target vehicle to remove the 3D point cloud not corresponding to a door lock; generating, based on cropped 3D point cloud, a depth map image; segmenting the depth map image into image segments; and analyzing, using a machine learning algorithm, the image segments to determine one or more characteristics of the door lock.

Images