Autonomous mobile robotic systems for use at commercial product facilities

Abstract

An autonomous mobile robotic system for commercial product facilities includes a mobile robot that navigates and interacts with vehicles in areas like warehouses, fulfillment centers, and stores. The system features an object interfacing mechanism, a detection system to gather environmental data, and a control circuit. The control circuit identifies target vehicles, determines travel paths, and directs the robot to service the vehicles. The control circuit further collects and analyzes data on vehicle components and application objects to develop and execute operation strategies.

Description

AUTONOMOUS MOBILE ROBOTIC SYSTEMS FOR USE AT COMMERCIAL PRODUCT FACILITIESRelated Application(s)

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 627,307, filed January 31, 2024, which is incorporated by reference in its entirety herein.Technical Field

[0002] This invention relates generally to a robot system, and specifically to a robot system to interface with vehicles at commercial product facilities.Background

[0003] Retail truck yards are critical hubs in the supply chain where a high volume of vehicles, including yard trucks with trailers, are managed. These yards handle the loading, unloading, inspection, and servicing of vehicles to ensure they are in good working condition. Efficient and accurate vehicle servicing in retail truck yards are crucial due to the high volume of vehicles and the need to maintain safety, reliability, and compliance.Brief Description of the Drawings

[0004] FIG. l is a block diagram of a system in accordance with some embodiments.

[0005] FIG. 2 illustrates an example of a mobile robot in accordance with some embodiments.

[0006] FIGS. 3A-3B include illustrations of an object interfacing mechanism in accordance with some embodiments.

[0007] FIG. 4 is a flow diagram of a method in accordance with some embodiments.

[0008] FIG. 5 is a flow diagram of a method for identifying the characteristics of the target component in accordance with some embodiments.

[0009] FIGS.6A-6D include illustrations of an example target component in accordance with some embodiments.

[0010] FIG. 7 is a flow diagram of a method for identifying characteristics of the application object in accordance with some embodiments.

[0011] FIG. 8A-8C include illustrations of an example application object in accordance with some embodiments.

[0012] FIG. 9 is an illustration of a wheel and wheel restraint in accordance with some embodiments.Detailed Description

[0013] Generally speaking, pursuant to various embodiments, systems, apparatuses, devices, and methods are provided herein useful to handle, manipulate objects applied to vehicles such as utility cables (e.g., shore-power cables, air brake cables) and wheel restraints for vehicles, more specifically, vehicles with an electric stand-by refrigerated trailer / container or yard trucks. In some embodiments, a robotic system for use at a commercial product facility may comprise a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information of the working environment, and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to identify a target vehicle in the working environment, the target vehicle being necessary to be serviced, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travel path, obtain detection data for an expected area of a target component of the target vehicle, identify, using the detection data for the expected area of the target component, one or more characteristics of the target component of the target vehicle, obtain detection data for an expected area of an application object configured to be applied to the target component of the target vehicle, identify, using the detection data for the expected area of the application object, one or more characteristics of the application object, determine, based on the identified characteristicsof the target component and / or the identified characteristics of the application object, an operation strategy of the mobile robot, and cause the mobile robot to execute the operation strategy.

[0014] In a non-automated environment, there are many circumstances where vehicle drivers manually service their vehicles. For example, yard-truck divers manually service trailer utility connections or manually place wheel restraints after parking the yard truck near a dock door, or designated parking area. The introduction of autonomous driver-less vehicles (e.g., driver-less yard trucks) may require an automated solution for the manual services.

[0015] Robotic systems and methods of using the robotic systems in accordance with some embodiments described herein may provide autonomous vehicle services. In some embodiments, the robotic system may include a mobile robot that enables navigation of a working environment and traveling to vehicles at different locations in the working environment. In this manner, a single mobile robot may service a plurality of vehicles at different locations within the working environment, and the vehicles that need to be serviced may not need to be parked at a specific service area. Furthermore, the robotic systems and methods of using the robotic systems in accordance with some embodiments described herein do not require a 1 : 1 ratio of system to vehicle. Therefore, the robotic systems, in accordance with some embodiments, may provide costefficient autonomous vehicle services. According to some embodiments, the limits of the reach envelope and the dexterity of the object manipulating mechanism (e.g., an end-of-arm tool coupled to the robotic arm) may be reduced.

[0016] Various embodiments and examples of systems, devices, apparatus, and methods are described herein. FIGS. 1-9 are provided to illustrate various embodiments. It is noted that when describing certain embodiments, certain features may be shown in one or more of FIGS. 1- 9.

[0017] FIG. 1 is a block diagram of a robotic system 100 in accordance with some embodiments. The robotic system 100 may include a mobile robot 120, a detection system 130, a memory 104, and a control circuit 102 communicatively coupled to the mobile robot 120, the detection system 130, and the memory 104.

[0018] The mobile robot 120 may navigate a working environment of the mobile robot 120 and travel to vehicles in the working environment. In some embodiments, the mobile robot 120 may be an autonomous mobile robot (AMR) capable of autonomously navigating the working environment. In some embodiments, the working environment may comprise an area of commercial product facility that may receive and interact with vehicles containing commercial products. In some embodiments, the working environment may include, but not limited to, a shipping / receiving yard or a parking area of a commercial product facility such as a warehouse, a fulfillment center, a distribution center, a store, and so on.

[0019] The mobile robot 120 may include a robot controller 122 configured to control the operation and / or motion of the mobile robot 120. In some embodiments, the mobile robot 120 may include a robotic mobility platform 124. In some embodiments, the mobility platform may include, but not limited to, one or more of wheels, drivers, tracks, motorized limbs / legs / arms / feet. For example, the robotic mobility platform 124 may include, but not limited to, a bi-pedal robot, a humanoid robot, a quadruped robot, a wheeled mobile robot, a tracked mobile robot, an airborne autonomous mobile robot, a terrain mobility robot, other types of autonomous mobile robot, and / or any combination thereof.

[0020] In some embodiments, the mobile robot 120 may include a mounted robotic mechanism 126. The mounted robotic mechanism 126 may include, but not limited to, an articulating arm (e.g., a 4, 5, and 6-axis robotic arm), a paralleldink industrial robot (e.g., a Spider robot), a SCARA industrial robot, and other robotic mechanisms (e.g., specifically designed robotic mechanisms), and / or any combination thereof. The mounted robotic mechanism 126 may be mounted to the robotic mobility platform 124, for example, the terrain mobility assembly. In some embodiments, the mounted robotic mechanism 126 may be a custom design robotic arm with two or more axes. In some embodiments, the mounted robotic mechanism 126 may be a robotic arm having four or six axes. In some embodiments, the robotic arm includes multiple arm segments that are pivotally, rotatably, and / or statically attached.

[0021] In some embodiments, the mobile robot 120 may include an object interfacing mechanism 128. The object interfacing mechanism 128 may be configured to grip, pick, release, move, change positions of, rotate, and / or twist various types of objects. The object interfacingmechanism 128 may be capable of manipulating components of vehicles (e.g., a charging socket of the vehicle, a charging socket of the refrigerated trailer / container coupled to a truck, wheels, etc.) and objects applied to the vehicles (e.g., utility cables, air brake cables, wheel chocks, etc.). The object interfacing mechanism 128 may include, but is not limited to, an end effector (e.g., an end of arm tool, gripper, picker, opener, and so on). In some embodiments, the object interfacing mechanism 128 may include one or more of finger grippers, suction grippers, other types of end effectors, and / or any combination thereof. Some embodiments incorporate the some or all of U.S. Patent Application No. 63 / 536,608 entitled DEVICES AND METHODS FOR OBJECT MANIPULATION, and / or some or all of U.S. Patent Application No. 63 / 536,609, entitled SYSTEMS AND METHODS FOR FRIEGHT MANIPULATION, which are each incorporated herein by reference in their entirety.

[0022] In some embodiments, the mobile robot 120 may include a perception system 132 including one or more on-board sensing devices 134. The perception system 132 may navigate and / or find a travel path of the mobile robot 120 using data and information collected via the one or more on-boarding sensing devices 134. In some embodiments, the one or more on-board sensing devices 134 may include, but not limited to, one or more of a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an RGB overlay camera, a thermal sensor, an electromagnetic wave sensor, an optical sensor, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, and a microphone. The electromagnetic wave sensor may include one or more of a UV sensor and an RF sensor (e.g., including an RFID sensor). In some embodiments, the on-board sensing devices 134 may include an IMU, a LiDAR camera, and other types of sensors. In some embodiments, the perception system 132 and / or the on-board sensing devices 134 may be a part of the detection system 130. For example, at least a part or all of the sensing devices of the detection system 130 may be the onboard sensing devices 134. In some embodiments, the on-board sensing devices 134 may be disposed on, mounted on, and / or physically coupled to the mobile robot 120. In some embodiments, the on-board sensing devices 134 may be disposed on the robotic mobility platform 124._In some embodiments, the detection system 130 may be distributed in multiple locations.

[0023] FIG. 2 illustrates an example of mobile robot 120 in accordance with some embodiments. Referring to FIG. 2, the mobile robot 120 may include the robotic mobility platform 124, a multi-axis robotic arm 227, and the object interfacing mechanism 128. In some embodiments, the robotic mobility platform 124 may include a platform body 223 and one or more moving mechanisms. In some embodiments, the moving mechanisms may include a robotic limb / robotic leg 225 coupled to the platform body 223. In some embodiments, the robotic mobility platform 124 may be a four-legged robot that can move in various directions (e.g., walk or run forward / backward / right / left / diagonally), rotate clockwise / counterclockwise, climb up and down, jump, and / or flip. In some embodiments, the platform body 223 may include a sensor array 229 including the one or more on-board sensing devices 134. In some embodiments, the sensor array 229 may be disposed on the front side of the platform body 223.

[0024] FIGS. 3A-3B illustrate an object interfacing mechanism 128 in accordance with some embodiments. In some embodiments, the object interfacing mechanism 128 may be an end of arm tool (EoAT) that includes two or more fingers 342. The fingers 342 may be rounded fingers. The shape of fingers may correspond to the object to be applied to the vehicle. In some embodiments, the shape of fingers may correspond to the shape of the terminal of a utility cable. In some embodiments, referring to FIG. 3A, the object interfacing mechanism 128 may include two rounded fingers 342. Referring to FIG. 3B, the object interfacing mechanism 128 may include three fingers 342.

[0025] In some embodiments, the object interfacing mechanism 128 may further include a detachable insert 344 configured to accommodate the difference in external geometry of the objects applied to vehicle (e.g., difference in external geometry of the terminals of utility cables). The detachable insert 344 may be disposed on the interfacing surface of the object interfacing mechanism 128. The interfacing surface may be the surface of the object interfacing mechanism 128 that directly contacts / interfaces with the object applied to the vehicles. In some embodiments, the interfacing surface of the object interfacing mechanism 128 may be the inner surface of the fingers 342 of the object interfacing mechanism 128.

[0026] In some embodiments, the detachable insert 344 may be an insulation member. For example, the detachable insert 344 may be an insulation lining disposed on the inner side of therounded fingers of the EoAT. In some embodiments, the detachable insert 344 may include a polymer lining. In some embodiments, the polymer for the polymer lining may include urethane. By using the insulating material, the detachable insert 344 may reduce the potential risk of manipulating utility cables / cables.

[0027] In some embodiments, the detachable insert 344 may be detachable such that the detachable insert 344 may be changed / replaced depending on the types of objects to be applied to the vehicle. For example, the detachable insert 344 currently attached to the object interfacing mechanism 128 (e.g., gripper finger 342) may be detached from and another type of detachable insert may be attached when the detachable insert 344 currently attached to the object interfacing mechanism 128 is not appropriate the next object applied to the vehicle.

[0028] In some embodiments, the object interfacing mechanism 128 may be a gripper including two or more figures 342. Referring to FIG. 4, the object interfacing mechanism 128 may control each individual finger 342 independently. In some embodiments, the fingers 342 may be pneumatically and / or electromechanically actuated. In some embodiments, the object interfacing mechanism 128 may be water resistant and / or dust resistant. For example, the object interfacing mechanism 128 may be IP68 rated (e.g., water resistance in fresh water to a maximum depth of 1.5 meters for up to 30 minutes and complete protection against dust over extended time) for protection against water and / or dust. In some embodiments, the object interfacing mechanism 128 may be adaptive to shape of an object to be held and / or be replaced depending on each activity conducted by the object interfacing mechanism 128.

[0029] Referring back to FIG. 1, the detection system 130 may be configured to collect detection data 106 (i.e., information of the working environment of the mobile robot 120). In some embodiments, the detection system 130 may include, but not limited to, a 2-dimentional (2D) camera, a 3-dimentional (3D) camera, an RGB-D sensor, an LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, a microphone, or any combination thereof. The detection system 130 may scan or capture data from the working environment in real-time or near real-time during the operation of the mobile robot 120. In some embodiments, the detection system 130 may be local with respect to the mobile robot 120 (where, for example, the detection system 130 may be mounted orphysically coupled to the mobile robot 120) or may be physically discrete in whole or in part from the mobile robot 120 as desired. In some embodiments, the detection system 130 may be distributed in multiple locations. In some embodiments, the object interfacing mechanism 128 may include one or more sensors 346 and the sensors 346 may collect detailed information of objects being currently manipulated by the object interfacing mechanism 128. The sensors 346 may improve the precision of operation of the mobile robot 120.

[0030] In some embodiments, the memory 104 may include a volatile and / or non-volatile memory. In some embodiments, the memory 104 may include a random-access memory (RAM). The memory 104 may serve to store computer instructions that, when executed by the control circuit 102, cause the control circuit 102 to behave as described herein. In some embodiments, the memory 104 may serve, for example, to non-transitorily store computer instructions. As used herein, this reference to "non-transitorily" will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence the may include both non-volatile memory (such as read-only memory (ROM) as well as volatile memory (such as an erasable programmable read-only memory (EPROM).

[0031] The memory 104 may provide storage for the detection data 106 and one or more modules 108. In some embodiments, the detection data 106 may include, but not limited to, image data (e g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the field of view, sound data of the working environment, force data of force applied to the mobile robot (e.g., to the object interfacing mechanism 128), and / or any other types of sensor data collected via the detection system 130 that may be necessary or used to conduct steps, actions, and / or functions described herein. The memory 104 may also store data generated during the use / operation of the robotic system 100. The one or more modules 108 may include codes executable by the control circuit 102 and, when executed by the control circuit 102, cause the control circuit 102 to perform specific steps, actions, and / or functions designed by each module. In some embodiments, the modules 108 may include, but are not limited to, a point cloud accumulation module, a point cloud filtering module, and / or an image segmentation module. In some embodiments, the modules 108 may further include a distance calculation module, an inverse kinematics module, a motionplanning module. In some embodiments, the modules 108 may further include modules facilitating other automation system operations including, but not limited to, a system safety module, a fleet management module, and a warehouse / yard management system interfacing module. In some embodiments, the one or more of the modules 108 may employ a machine learning model 107 to improve capabilities for object identification, object manipulation, inspection processes, navigating, and ancillary tasks such as exceptional handling and so on. In some embodiments, the detection data 106 captured during the operation of the mobile robot 120 may be used to train the machine learning model 107.

[0032] In some embodiments, the control circuit 102 may operably / communicatively couple to the memory 104, mobile robot 120, and the detection system 130. The control circuit 102 may receive information from the detection system 130 and may store the information (e.g., detection data 106) received from the detection system 130 to the memory 104.

[0033] The control circuit 102 may access the memory 104 and execute the codes stored in the memory 104. In some embodiments, the memory 104 may be integral to the control circuit 102 or may be physically discrete in whole or in part from the control circuit 102 as desired. This memory 104 may also be local with respect to the control circuit 102 (where, for example, both share a common circuit board, chassis, power supply, and / or housing) or may be partially or wholly remote with respect to the control circuit 102 (where, for example, the memory 104 is physically located in another housing or remotely location). In some embodiments, the memory 104 may be distributed in multiple locations.

[0034] The control circuit 102 is configured, for example by using corresponding programming and / or using the modules 108 stored in memory 104, to carry out and / or send signals to carry out one or more of the steps, actions, and / or functions described herein. The control circuit 102 may comprise structure that includes at least one (and typically many) electrically-conductive paths (such as paths comprised of a conductive metal such as copper or silver) that convey electricity in an ordered manner, which path(s) will also typically include corresponding electrical components (both passive (such as resistors and capacitors) and active (such as any of a variety of semiconductor-based devices) as appropriate) to effect one or more of the steps, actions, and / or functions described herein. The control circuit 102, for example, may comprise a fixed-purposehard-wired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit that is customized by design for a particular use, rather than intended for general-purpose use), a field-programmable gate array (FPGA), and the like) or can comprise a partially or wholly-programmable hardware platform (including but not limited to microcontrollers, microprocessors, and the like).

[0035] FIG. 4 is a flowchart depicting a method 400 for use with a robotic system in accordance with some embodiments. The method 400 may be performed using the robotic system 100 in accordance with the approaches described above. Although the method 400 is mainly illustrated with the robotic system 100, the method 400 may also be performed with a robotic system differently configured.

[0036] In step 402, the control circuit 102 may identify a target vehicle in the working environment. The target vehicle may be a vehicle that needs to be serviced. In some embodiments, the control circuit 102 may identify the target vehicle using the detection data 106 (information of the working environment collected via the detection system 130). The working environment may include vehicles and / or objects in the operation environment / area of the mobile robot 120. In some embodiments, the detection data 106 may include information for the entire / overall area of the working environment collected via the detection system 130 to monitor vehicles in the working environment (e.g., movement of vehicles in the working environment, entering / leaving of vehicles in the working environment). Additionally or alternatively, the control circuit 102 may identify the target vehicle based on signals from vehicles or on signals from a user of the vehicles. For example, when the vehicles are capable of directly or indirectly communicating with the control circuit 102, the vehicles may send signals requesting services to the control circuit 102, and based on the service requesting signals, the control circuit 102 may identify and / or determine the target vehicle. In some embodiments, the user of the vehicles (e.g., a driver) may send service requesting signals to the control circuit 102, using the user device (e.g., mobile electronic devices).

[0037] In step 404, the control circuit 102 may determine a travel path of the mobile robot 120 to the target vehicle. To determine the travel path, the control circuit 102 may determine the current location of the mobile robot and the current location of the target vehicle. In some embodiments, the mobile robot 120 may include a built-in location tracking system (e.g., a GPSsystem) and the current location of the mobile robot may be determined using the built-in location tracking system. Similarly, when the target vehicle includes a built-in location tracking system, the current location of the target vehicle may be determined using the built-in location tracking system. In some embodiments, the current location of the mobile robot and / or the target vehicle may be determined based on the detection data 106 collected via the detection system 130, and control circuit 102 may determine, using the determined current locations of the mobile robot 120 and the target vehicle, the travel path of the mobile robot 120 to the target vehicle. In some embodiments, in determining the travel path, the control circuit may further consider other objects that may affect the travel of the mobile robot 120 such as static obstacles, frequently detected dynamic obstacles, etc.

[0038] In step 406, the mobile robot 120 may move / travel to the target vehicle according to the determined travel path. In some embodiments, the control circuit 102 may cause the mobile robot 120 to move / travel to the target vehicle according to the determined travel path (e.g., by transmitting the signals for the travel path to the mobile robot 120). In some embodiments, the detection system 130 (in some embodiments, including the perception system 132 of the mobile robot 120) may continue to collect the information of the working environment while the mobile robot moves / travels toward the target vehicle according to the determined travel path, and the control circuit 102 may recognize, based on the information of the working environment collected during the travel, obstacles that have not been detected / predicted when the control circuit 102 initially determine the travel path of the mobile robot 120 in step 404. In some embodiments, when the control circuit 102 recognizes the unpredicted obstacles while the mobile robot 120 travels to the target vehicle, the control circuit 102 may update the travel path in response to the recognition of the unpredicted obstacle and may cause the mobile robot 120 to move / travel to the target vehicle according to the updated travel path.

[0039] In step 408, the control circuit 102 may obtain from the detection system 130, detection data 106 for an expected area of a target component of the target vehicle. The target component may be the component of the target vehicle that needs to be serviced. For example, the target component may include, but is not limited to, a charging socket of the target vehicle, a charging socket of a refrigerated trailer / container, a cover / case of a socket, a pipe, or an openingto accommodate an air brake cable, a cover / case of an opening, and / or a wheel / tire of a vehicle. In some embodiments, the robotic system 100 may manipulate two or more target components of the target vehicle. The expected area of the target component may be a target component placement area (e.g., an area including the target component and other objects adjacent to the target component). In some embodiments, the detection data 106 for the expected area of the target component may include, but not limited to, image data (e.g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the expected area of the target component, sound data for the expected area of the target component and / or any other types of sensor data for the expected area of the target component. In some embodiments, the 3D point cloud of the expected area of the target component may be obtained via a LiDAR sensor of the detection system 130.

[0040] In step 410, the control circuit 102 may identify, using the detection data 106 for the expected area of the target component, one or more characteristics of the target component. The characteristics may include, but are not limited to, the type, shape, size, position, geometry, configuration, and / or operation mechanism of the target component.

[0041] FIG. 5 illustrates an example method 500 to identify the characteristics of the target component in step 410 in accordance with some embodiments. In some embodiments, the detection data 106 for the expected area of the target component may include a 3D point cloud of the expected area of the target component. FIGS.6A-6D depict an example target component in accordance with some embodiments. In FIGS. 6A-6C, the target component includes a lid / cover of the charging socket of a refrigerated trailer. In FIG. 6D, the target component includes an uncovered charging socket of a refrigerated trailer. FIGS. 6A-6D may be referred to in describing the method 500, but method 500 is not limited by FIGS. 6A-6D.

[0042] In some embodiments, Referring to FIG. 5, in step 502, the control circuit 102 may, with the point cloud filtering module, crop the 3D point cloud of the expected area of the target component to remove the portion of the 3D point cloud not corresponding to the target component (e.g., to remove the background of target component and other objects included in the 3D point cloud of the expected area of the target component). In some embodiments, the point cloud filtering module may crop the 3D point cloud to leave only the portion of the 3D point cloud corresponding to the target component. FIG. 6A depicts an example of a collected image of the expected area oftarget component. In some embodiments, the 3D point cloud of the expected area of the target components obtained in step 408 may be a 3D point cloud corresponding to the image of FIG. 6A. FIG. 6B is an image to illustrate cropping of the 3D point cloud of the expected area of the target component. In some embodiments, the control circuit 102 may crop the 3D point cloud along the cropping boundary 604 of the charging socket cover 602. The 3D point cloud cropped in step 502 may be a 3D point cloud corresponding to the charging socket cover 602 encompassed by the cropping boundary 604.

[0043] In step 504, the control circuit 102 may generate a depth map image based on the cropped 3D point cloud 308 generated in step 502. In some embodiments, the depth map image may be generated by taking each specific point of the cropped 3D point cloud and assigning a value (e.g., darkness or color) for each point based on its distance from the reference point.

[0044] In step 506, the control circuit 102 may segment, with the image segmentation module, the depth map image generated in step 504 into image segments. The image segmentation module may include a segmentation algorithm. In some embodiments, each segment may include a matrix of true and false values for each pixel in the depth map image. In the matrix of each segment, the true value may indicate that the pixel of the true value is part of a specific segment, and the false value may indicate that the pixel of the false value is not part of the specific segment.

[0045] In step 508, the control circuit 102 may analyze the image segments to determine the one or more characteristics of the target component. The control circuit 102 may analyze the image segments using a machine learning algorithm including a trained object identification model configured to identify objects and / or the characteristics of the objects by analyzing the image segments of the objects via artificial intelligence (Al). In some embodiments, the machine learning algorithm may involve patter recognition and / or imitation learning. FIG. 6C illustrates characteristics of the target component identified in step 508. From the analysis in step 508, the control circuit 102 may identify uncovering mechanism of the charging socket cover 602. For example, as illustrated in FIG. 6C, the control circuit may identify that the charging socket cover 602 may be uncovered by rotating / twi sting the charging socket cover 602 in a clockwise direction.

[0046] In some embodiments, when the number of the target component(s) is two or more, the steps 502 to 508 may be repeated to identify the characteristics of each target component. In some embodiments, after identifying the characteristics of the first target component, steps 502 to 508 may be repeated to identify the characteristics of the second target component. For example, FIG. 6D illustrates uncovered charging socket 610. Because the characteristics of the uncovered charging socket 610 may not be identified when the control circuit 102 identify the characteristics of the charging socket cover 602, to identify the characteristics of the uncovered charging socket 610, the control circuit 102 may repeat steps 502 to 508 for the charging socket 610 after mobile robot 120 uncovers the charging socket cover 602. Steps 502 to 508 may be conducted / repeated as many times as necessary to identify the characteristics of the multiple target components. Repetition of steps 502 to 508 to identify the characteristics of the second target component may be conducted after determining the operation strategy for the first target component and executing the operation strategy for the first target component.

[0047] In some embodiments, by conducting steps 502 to 508 for the uncovered charging socket 610, the control circuit 102 may identify the position and configuration of a guide ridge 606 of the charging socket 610 that may guide / indicate a position for inserting a terminal of a utility / charging cable to the charging socket 610.

[0048] Referring back to FIG. 4, in step 412, the control circuit 102 may obtain detection data 106 for an expected area of an application object. When the sensor devices of the detection system 130 are mounted to the mobile robot 120 and the application object is not adjacent to the target component (e.g., apart from the target vehicle), the mobile robot 120 may move between the target component and the application object to collection the detection data 106 for the expected area of the application object. The application object may be the object used in servicing the target vehicle. For example, the application objects may include, but are not limited to, utility cables (power cables), air brake cables, other types of cables, terminals of cables for vehicles / trailers / containers, wheel restraints (wheel chock), etc. In some embodiments, the robotic system 100 may manipulate two or more application objects when it is necessary to service the target vehicles. The expected area of the application object may be an application object placement area (e.g., an area including the application object and other objects adjacent to the applicationobject). In some embodiments, the detection data 106 for the expected area of the application object may include, but not limited to, image data (e.g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the expected area of the application object, sound data for the expected area of the application object and / or any other types of sensor data for the expected area of the application object. In some embodiments, the 3D point cloud of the expected area of the application object may be obtained via a LiDAR sensor.

[0049] In step 414, the control circuit 102 may identify, using the detection data 106 for the expected area of the application object, one or more characteristics of the application object. The characteristics may include, but are not limited to, the type, shape, size, geometry, position, configuration, and / or operation mechanism of the application object.

[0050] FIG. 7 illustrates an example method 700 to identify the characteristics of the application object in step 414 in accordance with some embodiments. In some embodiments, the detection data for the expected area of the target component may include a 3D point cloud of the expected area of the application object. FIGS.8A-8C depict example application objects including a utility cable 802 with a terminal 804 and / or a control panel 806 for the utility cable with a power lever 808 in accordance with some embodiments. FIGS. 8A-8C may be referred to in describing the method 700, but method 700 is not limited by FIGS. 8A-8C.

[0051] In some embodiments, in step 702, the control circuit 102 may, with the point cloud filtering module, crop the 3D point cloud of the expected area of the application object to remove the 3D point cloud not corresponding to the application object or the portion of the application object in question (e.g., to remove the background of the application object and / or the portion of the application object not in question). In some embodiments, the point cloud filtering module may leave only the 3D point cloud corresponding to the application object or the portion of the application object in question. FIG. 8 A depicts an example of a collected image of the expected area of the application object, a utility cable 802. In some embodiments, the 3D point cloud of the expected area of the application object obtained in step 412 may be a 3D point cloud corresponding to the image of FIG. 8A. FIG. 8B an image to illustrate cropping of the 3D point cloud of the expected area of the application object. In some embodiments, the control circuit 102 may crop the 3D point cloud along the cropping boundary 810 to remove the objects and portions not inquestion. The 3D point cloud cropped in step 702 may be a 3D point cloud corresponding to the terminal 804 and the portion of the utility cable 802 encompassed by the cropping boundary 810.

[0052] In step 704, the control circuit 102 may generate a depth map image based on the 3D point cloud 308 cropped in step 702. In some embodiments, the depth map image 310 may be generated by taking each specific point of the cropped 3D point cloud and assigning a value (e.g., darkness or color) for each point based on its distance from the reference point.

[0053] In step 706, the control circuit 102 may segment, with the image segmentation module, the depth map image generated / cropped in step 704 into image segments. The image segmentation module may include a segmentation algorithm.

[0054] In step 708, the control circuit 102 may analyze the image segments to determine the one or more characteristics of the application object. The control circuit 102 may analyze the image segments using a machine learning algorithm including a trained object identification model configured to identify objects and / or the characteristics of the objects by analyzing the image segments of the object via Al. FIG. 8C is an image of the terminal 804 of the utility cable 802 to illustrate characteristics of the application object (the terminal 804) that may be identified in step 708. In some embodiments, the control circuit 102 may identify the size and shape of the terminal 804 of the utility cable 802 and the position and configuration of a guide groove / slot 812 on the terminal 804 of the utility cable 802.

[0055] In some embodiments, when the number of the application obj ect(s) is two or more, the steps 702 to 708 may be repeated to identify the characteristics of each application object. For example, after identifying the characteristics of the terminal 804, steps 702 to 708 may be repeated to identify the characteristics of the next application object, the power lever 808. Repetition of steps 702 to 708 to identify the characteristics of the next application object may be conducted after determining the operation strategy for the first application object (the terminal 804) and executing the operation strategy for the first target component.

[0056] Alternatively or additionally, other methods may be used in step 410 and 414. In some embodiments, the control circuit 102 may use 2D and 3D image data (e.g., 2D images, a 3D point cloud, or a combination thereof) and recognize edges, shapes, and near / far distances ofobjects to identify the objects (e.g., target vehicles, target components, application objects, obstacles, etc.) and / or the characteristics of the objects. In some embodiments, the control circuit may detect / recognize object edges, shapes, and near / far distance of objects in the working environments of the mobile robot 120. In some embodiments, the control circuit 102 may use the 3D point cloud of the objects in the placement area (e.g., expected area of the objects), detect the edge of each object in the 3D point cloud, and draw boundary lines for the objects based on the detected edges. Further, the control circuit 102 may determine the near / far distance of each object from the reference point using depth information of the 3D point cloud of the objects.

[0057] In some embodiments, objects in the working environments of the mobile robot 120 may be identified using a machine learning algorithm based on computer vision (CV) models. In some embodiments, the machine learning algorithm may use a CV model trained based on previously captured images of objects as input and object identifiers as categorizations. In some embodiments, the CV model may comprise a deep neural network object recognition model. In some embodiments, the CV model may be trained based on images captured during the operation of the robotic system 100.

[0058] Referring back to FIG. 4, in step 416, the control circuit 102 may determine, based on the identified characteristics of the target component of the target vehicle and / or the characteristics of the application object, an operation strategy of the mobile robot 120. In some embodiments, when the target component is the charging socket cover 602, the operation strategy may include gripping and twisting, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the charging socket cover 602 to uncover / open the charging socket cover 602. The operation strategy for the charging socket cover 602 may further include covering the charging socket 610 with the charging socket cover 602 after completion of charging. In some embodiments, for the uncovered charging socket 610 of the trailer on the target vehicle and the terminal 804 of the utility cable 802, the operation strategy may include, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), gripping the utility cable 802 and connecting the utility cable 802 to the charging socket 610 of the target vehicle. In some embodiments, connecting the utility cable 802 to the charging socket 610 may include, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), plugging the terminal 804 of the utility cable 802 in the chargingsocket 610 of the target vehicle and rotating the terminal 804 of the utility cable 802 to securely couple the utility cable 802 to the charging socket 610.

[0059] In some embodiments, when the charging socket 610 includes the guide ridge 606 and the terminal 804 of the utility cable 802 includes the guide groove 812, plugging the utility cable 802 may include aligning, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the guide groove 812 of the terminal 804 of the utility cable 802 to the guide ridge 606 of the charging socket 610.

[0060] In some embodiments, the operation strategy may further include turning on, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), power for the utility cable 802 after securely connecting the utility cable 802 to the charging socket 610 of the target vehicle. Turning on power of the utility cable 802 may include biasing / pivoting, with the mobile robot 120 (e.g., with the object interfacing mechanism 128), the power lever 808 on the control panel 806. In some embodiments, the characteristics of the power lever 808 (e.g., position, shape, size, operation mechanism of the power lever 808) may be identified by conducting steps 702 to 708. Turning on the power for the utility cable 802 after securing connecting the utility cable 802 to the charging socket 610 may reduce the potential risk caused by high voltage of the power coming through the utility cable 802.

[0061] FIG. 9 is an image illustrating a wheel and wheel restraint in accordance with some embodiments. In some embodiments, the target component of the target vehicle may include a wheel 902 of the target vehicle, and application object may include a wheel restraint 904. In this case, the operation strategy may include picking the wheel restraint 904 and placing the wheel restraint 904 near the wheel of the target vehicle to prevent unintended movement of the target vehicle, using the mobile robot 120.

[0062] In some embodiments, the operation strategy may include moving the mobile robot 120 from the target component to the application object and / or moving the mobile robot from the application object to the target component.

[0063] In step 418, the mobile robot 120 may execute the operation strategy determined in step 416. While executing the operation strategy, the control circuit 102 may update the operationstrategy based on the detection data 106 continuously obtained during the execution of the operation strategy. In some embodiments, the detection system 130 may include a force sensor configured to measure the resistance force of twisting / rotating the terminal 804 of the utility cable 802 while the mobile robot 120 twists / rotates the terminal 804 of the utility cable 802 after plugging the terminal 804 of the utility cable 802 to the charging socket 610. In some embodiments, the rotation of the of the terminal 804 of the utility cable 802 may be stopped when the measured resistance force is greater than a specific value to avoid damages of the charging socket 610 and / or the terminal 804 of the utility cable 802.

[0064] In some embodiments, the mobile robot 120 is configured to change / replace the object interfacing mechanisms 128 (e.g., EoATs) based on the identified application objects. For example, the mounted robotic mechanism 126 may release the object interfacing mechanism currently coupled thereto and pick and couple another object interfacing mechanism to be used. Additionally or alternatively, changing the object interfacing mechanism 128 may be conducted with an additional object interfacing mechanism. In some embodiments, the mobile robot 120 may further include an additional object interfacing mechanism configured to decouple and release the object interfacing mechanism currently coupled to the mounted robotic mechanism 126 and pick and couple the next object interfacing mechanism to the mounted robotic mechanism 126.

[0065] In some embodiments, a robotic system for use at a commercial product facility may include a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information of the working environment, and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to identify a target vehicle in the working environment, the target vehicle being necessary to be serviced, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travel path, obtain detection data for an expected area of a target component ofthe target vehicle, identify, using the detection data for the expected area of the target component, one or more characteristics of the target component of the target vehicle, obtain detection data for an expected area of an application object configured to be applied to the target component of the target vehicle, identify, using the detection data for the expected area of the application object, one or more characteristics of the application object, determine, based on the identified characteristics of the target component and / or the identified characteristics of the application object, an operation strategy of the mobile robot, and cause the mobile robot to execute the operation strategy.

[0066] In some embodiments, a method for use with a robotic system at a commercial product facility may include identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle in a working environment, the target vehicle being necessary to be serviced, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, moving the mobile robot to the target vehicle according to the determined travel path, obtaining, via the detection system, a 3- dimensional (3D) point cloud of an expected area of a target component of the target vehicle, identifying, with the control circuit and using the 3D point cloud of the expected area of the target component, one or more characteristics of the target component of the target vehicle, obtaining via the detection system, a 3D point cloud of an expected area of an application object configured to be applied to the target component of the target vehicle, identifying, with the control circuit and using the 3D point cloud of the expected area of the application object, one or more characteristics of the application object, determining, with the control circuit and based on the identified target component and / or the identified characteristics of the application object, an operation strategy of the mobile robot, and executing, with the mobile robot, the operation strategy.

[0067] Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

CLAIMSWhat is claimed is:

1. A robotic system for use at a commercial product facility, the robotic system comprising: a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, the mobile robot including an object interfacing mechanism, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; a detection system configured to collect information of the working environment; and a control circuit communicatively coupled to the mobile robot and the detection system, the control circuit configured to: identify a target vehicle in the working environment, the target vehicle being necessary to be serviced; determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; cause the mobile robot to move to the target vehicle according to the determined travel path; obtain detection data for an expected area of a target component of the target vehicle; identify, using the detection data for the expected area of the target component, one or more characteristics of the target component of the target vehicle; obtain detection data for an expected area of an application object configured to be applied to the target component of the target vehicle; identify, using the detection data for the expected area of the application object, one or more characteristics of the application object;determine, based on the identified characteristics of the target component and / or the identified characteristics of the application object, an operation strategy of the mobile robot; and cause the mobile robot to execute the operation strategy.

2. The robotic system of claim 1, wherein the mobile robot comprises a mobility platform comprising one or more of wheels, drivers, and / or motorized limbs.

3. The robotic system of claim 1, wherein the target component of the target vehicle includes a charging socket of the target vehicle and a cover for the charging socket, and the application object includes a utility cable.

4. The robotic system of claim 3, wherein the operation strategy includes opening, with the object interfacing mechanism, the cover for the charging socket.

5. The robotic system of claim 3, wherein the object interfacing mechanism includes a gripper, and wherein the operation strategy includes gripping, with the gripper, the utility cable and connecting the utility cable to the charging socket of the target vehicle.

6. The robotic system of claim 5, wherein the operation strategy further includes turning, with the gripper, on power of the utility cable after connecting the utility cable to the charging socket of the target vehicle.

7. The robotic system of claim 5, wherein connecting the utility cable to the charging socket of the target vehicle includes plugging a terminal of the utility cable in the charging socket of the target vehicle and rotating the terminal of the utility cable to securely couple the utility cable to the charging socket.

8. The robotic system of claim 7, wherein the detection system is further configured to measure rotation resistance while the mobile robot rotating the terminal of the utilitycable and the rotation of the terminal of the utility cable is stopped when the rotation resistance is greater than a specific value.

9. The robotic system of claim 1, wherein the target component of the target vehicle includes a wheel of the target vehicle, and the application object includes a wheel restraint.

10. The robotic system of claim 9, wherein the object interfacing mechanism includes a gripper, and wherein the operation strategy includes picking, with the gripper, the wheel restraint and placing the wheel restraint near the wheel of the target vehicle to prevent unintended movement of the target vehicle.

11. The robotic system of claim 1, wherein the detection data for the expected area of the target component includes a 3-dimensional (3D) point cloud of the expected area of the target component, and wherein identifying the one or more characteristics of the target component of the target vehicle comprises: cropping the 3D point cloud of the expected area of the target component of the target vehicle to remove the 3D point cloud not corresponding to the target component; generating, based on cropped 3D point cloud, a depth map image; segmenting the depth map image into image segments; and analyzing, using a machine learning algorithm, the image segments to determine the one or more characteristics of the target component.

12. The robotic system of claim 1, wherein the detection data for the expected area of the application object includes a 3D point cloud of the expected area of the application object, and wherein identifying the one or more characteristics of the application object comprises: cropping the 3D point cloud of the expected area of the application object to remove the 3D point cloud not corresponding to the application object or a portion of the application object in question;generating, based on cropped 3D point cloud, a depth map image; segmenting the depth map image into image segments; and analyzing, using a machine learning algorithm, the image segments to determine the one or more characteristics of the application object.

13. The robotic system of claim 1, wherein the object interfacing mechanism includes an end effector comprising two or more rounded fingers.

14. The robotic system of claim 13, one or more of the rounded fingers includes a detachable insert on an inner side of the rounded fingers, the detachable insert configured to accommodate external geometry of a utility cable.

15. The robotic system of claim 14, wherein the detachable insert includes a polymer lining.

16. The robotic system of claim 1, wherein the mobile robot is configured to change the object interfacing mechanism based on the identified application object.

17. The robotic system of claim 1, wherein the control circuit is further configured to recognize, based on the information of the working environment, an unpredicted obstacle while the mobile robot travels to the target vehicle.

18. The robotic system of claim 17, wherein the control circuit is further configured to update the travel path in response to recognition of the unpredicted obstacle.

19. The robotic system of claim 1, wherein the control circuit is further configured to recognize object edges, shapes, and near / far distance of objects.

20. A method for use with a robotic system at a commercial product facility, the method comprising:identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle in a working environment, the target vehicle being necessary to be serviced, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; moving the mobile robot to the target vehicle according to the determined travel path; obtaining, via the detection system, detection data for an expected area of a target component of the target vehicle; identifying, with the control circuit and using detection data for the expected area of the target component, one or more characteristics of the target component of the target vehicle; obtaining, via the detection system, detection data for an expected area of an application object configured to be applied to the target component of the target vehicle; identifying, with the control circuit and using the detection data for the expected area of the application object, one or more characteristics of the application object; determining, with the control circuit and based on the identified characteristics of the target component and / or the identified characteristics of the application object, an operation strategy of the mobile robot; and executing, with the mobile robot, the operation strategy.

Images