Inspection systems for use at commercial product facilities

Abstract

An inspection system for a commercial product facility is provided. The system includes a detection system with on-board sensing devices attached to one or more mobile robots. The system's control circuit communicates with the robot and sensors to identify a target vehicle with a trailer, determine the robot's travel path to the vehicle, and move the robot accordingly. The robot and sensors follow a vehicle inspection protocol, collecting detection data from the target vehicle. The system then identifies the vehicle's identifier, trailer identifier, and any defects using the collected data.

Description

INSPECTION SYSTEMS FOR USE AT COMMERCIAL PRODUCT FACILITIES Related Application(s)

[0001] This application claims the benefit of U.S. Provisional Application No.63 / 627,303, filed January 31, 2024, which is incorporated by reference in its entirety herein. Technical Field

[0002] This invention relates generally to a robot system, and specifically to a robot system for use at commercial product facilities. Background

[0003] Retail truck yards are critical hubs in the supply chain where a high volume of vehicles, including yard trucks with trailers, are managed. These yards handle the loading, unloading, and inspection of vehicles to ensure they are in good working condition. Efficient and accurate vehicle inspections in retail truck yards are crucial due to the high volume of vehicles and the need to maintain safety, reliability, and compliance. Brief Description of the Drawings

[0004] FIG. 1 is a block diagram of an inspection system in accordance with some embodiments.

[0005] FIGS.2A-2C include illustrations of examples of mobile robots in accordance with some embodiments.

[0006] FIG. 3 include illustrations of an inspection system in accordance with some embodiments.

[0007] FIG.4 includes an illustration of a vehicle in accordance with some embodiments.

[0008] FIG. 5 includes an illustration of a sensor array in accordance with some embodiments.

[0009] FIG.6 includes an illustration of a mobile robot inspecting a wheel of the vehicle in accordance with some embodiments.

[0010] FIG. 7 is a flowchart of an inspection method in accordance with some embodiments. Docket No.8842-158683-WO_8390WO01

[0011] FIG. 8 is a flowchart of a method for generating the computer vision model in accordance with some embodiments. Detailed Description

[0012] Generally speaking, pursuant to various embodiments, systems, apparatuses, devices, and methods are provided herein useful to inspect vehicles at commercial product facilities (e.g., yard trucks with a trailer / container in a shipping / receiving yard). In some embodiments, an inspection system for use at a commercial product facility comprises a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information from a vehicle in the working environment, the detection system comprising one or more on-board sensing devices physically coupled to the mobile robot, a control circuit communicatively coupled to the mobile robot and the one or more on-board sensing devices, the control circuit configured to identify a target vehicle including a trailer in the working environment, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travel path, cause the mobile robot and the one or more on-board sensing devices to follow a vehicle inspection protocol, obtain, via the one or more on-board sensing devices, detection data corresponding to the target vehicle, an identify, using the detection data, one or more of a vehicle identifier of the target vehicle, a trailer identifier of the target vehicle, and a defect of the target vehicle.

[0013] Vehicle inspection is an essential process that ensures the safety, reliability, and compliance of vehicles. With the increasing complexity of modern vehicles, vehicle inspection has become crucial in identifying potential issues.

[0014] In a non-automated environment, there are many circumstances where vehicle drivers or inspection people manually inspect the condition of vehicles. For example, yard truck drivers should get out of the driver’s seat to inspect the exterior of the vehicles. Further, some Docket No.8842-158683-WO_8390WO01vehicle issues may not be easily detectable by human inspectors. Further, the introduction of autonomous driver-less vehicles (e.g., driver-less yard trucks) may require an automated inspection of vehicles.

[0015] Inspection systems and methods of some embodiments may provide automated inspection of vehicles (e.g., yard trucks with a trailer). In some embodiments, the inspection system may include a mobile robot that enables navigation of a working environment and traveling to vehicles at different locations in the working environment. In this manner, a single mobile robot may manipulate a plurality of vehicles at different locations within the working environment, and the vehicles that need to be inspected may not need to be parked at a specific / designated area. Furthermore, the inspection systems and methods, in accordance with some embodiments, may provide efficient vehicle inspection. According to some embodiments, the limits of the reach envelope and the dexterity of the inspection devices (e.g., sensing devices) may be reduced by introducing a mobile robot that is not statically located. Furthermore, the inspection systems and methods in some embodiments may reduce / eliminate the potential human error, ensuring a higher level of accuracy in identifying defects, faults, and issues.

[0016] Various embodiments and examples of systems, devices, apparatus, and methods are described herein. FIGS. 1-8 are provided to illustrate various embodiments. It is noted that when describing certain embodiments, certain features may be shown in one or more of FIGS.1- 8.

[0017] FIG. 1 is a block diagram of an inspection system 100 in accordance with some embodiments. The inspection system 100 may include a mobile robot 120, a detection system 130, a memory 104, and a control circuit 102 communicatively coupled to the mobile robot 120, the detection system 130, and the memory 104.

[0018] The mobile robot 120 may navigate a working environment of the mobile robot 120 and travel to vehicles in the working environment. In some embodiments, the mobile robot 120 may be an autonomous mobile robot (AMR) capable of autonomously navigating the working environment. In some embodiments, the working environment may comprise an area of commercial product facility that may receive and interact with vehicles containing commercial Docket No.8842-158683-WO_8390WO01products. In some embodiments, the working environment may include, but not limited to, a shipping / receiving yard or a parking area of a commercial product facility such as a warehouse, a fulfillment center, a distribution center, a store, and so on.

[0019] The mobile robot 120 may include a robot controller 122 configured to control the operation and / or motion of the mobile robot 120. The mobile robot 120 may include a robotic mobility platform 124. In some embodiments, the robotic mobility platform 124 may include, but not limited to, one or more of wheels, drivers, tracks, motorized limbs / legs / arms / feet. For example, the robotic mobility platform 124 may include, but not limited to, a bi-pedal robot, a humanoid robot, a quadruped robot, a wheeled mobile robot, a tracked mobile robot, an airborne autonomous mobile robot, a terrain mobility robot, other types of autonomous mobile robot, and / or any combination thereof.

[0020] In some embodiments, the mobile robot 120 may include a mounted robotic mechanism 126. The mounted robotic mechanism 126 may include, but not limited to, an articulating arm (e.g., a 4, 5, and 6-axis robotic arm), a parallel-link industrial robot (e.g., a Spider robot), a SCARA industrial robot, and other robotic mechanisms (e.g., specifically designed robotic mechanisms), and / or any combination thereof. The mounted robotic mechanism 126 may be mounted to the robotic mobility platform 124, for example, the terrain mobility assembly. In some embodiments, the mounted robotic mechanism 126 may be a custom design robotic arm with two or more axes. In some embodiments, the mounted robotic mechanism 126 may be a robotic arm having four or six axes. In some embodiments, the robotic arm includes multiple arm segments that are pivotally, rotatably, and / or statically attached.

[0021] In some embodiments, the mobile robot 120 may include one or more on-board sensing devices 132. The on-board sensing devices 132 may be a part of the detection system 130. The detection system 130 may collect detection data 106 (e.g., information collected from the working environment of the mobile robot 120). In some embodiments, the detection system 130 may include, but not limited to, one or more of a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an RGB overlay camera, a thermal sensor, an electromagnetic wave sensor, an optical sensor, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, and a microphone. The Docket No.8842-158683-WO_8390WO01electromagnetic wave sensor may include one or more of a UV sensor and an RF sensor (e.g., including an RFID sensor). At least a part or all of these sensing devices of the detection system 130 may be the on-board sensing devices 132 disposed on, mounted on, and / or physically coupled to the mobile robot 120. In some embodiments, the on-board sensing devices 132 may be disposed on the robotic mobility platform 124. In some embodiments, the on-board sensing devices 132 may be coupled to the mounted robotic mechanism 126. In some embodiments, the on-board sensing devices 132 may be gripped / held by an object interfacing mechanism 128 (e.g., a gripper to grip the on-board sensing devices 132). In some embodiments, the detection system 130 may be distributed in multiple locations.

[0022] The detection system 130 (including the on-board sensing devices 132) may scan or capture data from the working environment including the vehicles in the working environment in real-time or near real-time for the inspection of the vehicles. In some embodiments, the on- board sensing devices 132 may collect detailed information of components / parts of vehicles for inspection.

[0023] In some embodiments, the mobile robot 120 may further include one or more on- board lighting devices 129 configured to facilitate inspection of the vehicles. In some embodiments, when the vehicle inspection is conducted in a place not bright enough or when the vehicle components being inspected is located in a place to which light may not easily reach out (e.g., an undercarriage of the vehicle), the one or more on-board lighting devices 129 may light the vehicle components being inspected.

[0024] In some embodiments, the mobile robot 120 may further include an object interfacing mechanism 128. The object interfacing mechanism 128 may be configured to grip, pick, release, move, change positions of, rotate, and / or twist various types of objects. In some embodiments, the object interfacing mechanism 128 may include one or more of finger grippers, suction grippers, other types of end effectors, and / or any combination thereof. Some embodiments incorporate the some or all of U.S. Patent Application No. 63 / 536,608 entitled DEVICES AND METHODS FOR OBJECT MANIPULATION, and / or some or all of U.S. Patent Application No. 63 / 536,609, entitled SYSTEMS AND METHODS FOR FRIEGHT MANIPULATION, which are each incorporated herein by reference in their entirety. Docket No.8842-158683-WO_8390WO01

[0025] In some embodiments, the object interfacing mechanism 128 may include an end effector configured to facilitate inspection of vehicles. The object interfacing mechanism 128 may include, but is not limited to, an end of arm tool, a gripper, a picker, a scraper, an opener, a finger and so on. For example, the object interfacing mechanism 128 may include an ice scraper end- effector configured to remove ice on a vehicle component to be inspected prior to capturing images for the inspection. Additionally or alternatively, the object interfacing mechanism 128 may include an end-effector configured to uncover a lid or other types of covers prior to capturing images for the covered vehicle component.

[0026] In some embodiments, the object interfacing mechanism 128 may be a gripper including two or more figures. The object interfacing mechanism 128 may control each individual finger independently. In some embodiments, the fingers may be pneumatically and / or electromechanically actuated. In some embodiments, the object interfacing mechanism 128 may be water resistant and / or dust resistant. For example, the object interfacing mechanism 128 may be IP68 rated (e.g., water resistance in fresh water to a maximum depth of 1.5 meters for up to 30 minutes and complete protection against dust over extended time) for protection against water and / or dust. In some embodiments, the object interfacing mechanism 128 may be adaptive to shape of an object to be held and / or be replaced depending on each activity conducted by the object interfacing mechanism 128.

[0027] In some embodiments, the mobile robot 120 may include a perception system 133. The perception system 133 may include one or more perception sensing devices 134. The perception system 133 may navigate and / or find a travel path of the mobile robot 120 using data and information collected via the one or more perception sensing devices 134. In some embodiments, the one or more perception sensing devices 134 may include, but not limited to, one or more of a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an RGB overlay camera, a thermal sensor, an electromagnetic wave sensor, an optical sensor, an IMU (Inertial Measurement Unit) sensor, a gyroscope, a force sensor, and a microphone. The electromagnetic wave sensor may include one or more of a UV sensor and an RF sensor (e.g., including an RFID sensor). In some embodiments, the perception sensing devices 134 may include an IMU, a LiDAR camera, and other types of Docket No.8842-158683-WO_8390WO01sensors. In some embodiments, the perception system 133 and / or the perception sensing devices 134 may be a part of the detection system 130. For example, at least in part or all of the sensing devices of the detection system 130 may be the perception sensing devices 134. In some embodiments, the perception sensing devices 134 of the perception system 133 may be disposed on, mounted on, and / or physically coupled to the mobile robot 120. In some embodiments, the perception sensing devices 134 may be disposed on the robotic mobility platform 124. In some embodiments, the detection system 130 may be distributed in multiple locations. In some embodiments, at least in part, the perception sensing devices 134 and the on-board sensing devices 132 may be the same sensing devices and / or may be separate sensing devices. For example, at least e part of the perception sensing devices 134 and at least a part of the on-board sensing devices 132 could either be integrated as a single unit / device or configured as distinct, separate units / devices.

[0028] FIGS. 2A-2C illustrate simplified views of examples of the mobile robot 120 in accordance with some embodiments. Referring to FIG.2A-2C, the mobile robot 120 may include the robotic mobility platform 124, the mounted robotic mechanism 126, and the on-board sensing device 132.

[0029] In some embodiments, the robotic mobility platform 124 may include a platform body 223 and one or more moving mechanisms 225. In some embodiments, the moving mechanism 225 may include one or more robotic limb / robotic leg coupled to the platform body 223. In some embodiments, the robotic mobility platform 124 may move, using the moving mechanism 225, in various directions (e.g., walk, run, or sliding, forward / backward / right / left / diagonally), rotate clockwise / counterclockwise, climb up and down, jump, and / or flip. In some embodiments, the one or more moving mechanisms 225 may include robotic legs 251, each with three joints (including the joint connecting the robotic leg 251 to the platform body 223) as illustrated in FIG. 2A. In some embodiments, the one or more moving mechanisms 225 may include robotic legs 253, each with two joints (including the joint connecting the robotic leg 253 to the platform body 223) as illustrated in FIG.2B. In some embodiments, the one or more moving mechanisms 225 may include one or more wheels 255 coupled to the platform body as illustrated in FIG.2C. Docket No.8842-158683-WO_8390WO01

[0030] In some embodiments, the mounted robotic mechanism 126 may include a multi- axis robotic arm 261. In some embodiments, the on-board sensing device 132 may be coupled to the end of the multi-axis robotic arm 261 as illustrated in FIGS. 2A, 2B. In some embodiments, the mobile robot 120 may further include an end of arm tool 263 (e.g., a gripper) and the on-board sensing devices 132 may be coupled to / gripped by the arm tool 263 as illustrated in FIG.2C.

[0031] In some embodiments, the mobile robot 120 may follow and perform a vehicle inspection protocol. The vehicle inspection protocol may include moving the mobile robot 120 to proximate to each of a plurality of locations of the vehicle (e.g., around the vehicle) and collecting with the mobile robot 120 and the on-board sensing devices 132, the information of the plurality of locations of the vehicle. The locations may include, but are not limited to, the front of the vehicle, the rear of the vehicle, the side of the vehicle, the undercarriage of the vehicle, and / or the roof of the vehicle.

[0032] Referring to FIG.2B, in some embodiments, the platform body 223 may include a sensor array 229 including the one or more perception sensing devices 134. In some embodiments, the sensor array 229 may be disposed on the front side of the platform body 223.

[0033] FIG.3 is a simplified view of mobile robots 120 and a vehicle in the inspection in accordance with some embodiments. In some embodiment, the inspection system 100 may include multiple (two or more) mobile robot 120 to inspect multiple locations / components of the vehicle simultaneously. In some embodiments, the multiple mobile robot 120 may concurrently perform a part of the vehicle inspection protocol assigned to each.

[0034] FIG. 4 is a simplified view of a vehicle 470 to illustrate locations / components of the vehicle that may be inspected by the inspection system 100 in accordance with some embodiments. In some embodiments, the locations and / or components of the vehicle that may be inspected by the inspection system 100 may include, but are not limited to, wheels, wheel tires 471, an exterior of a vehicle body of the target vehicle including, but not limited to, undercarriage 472, a bumper 476, and an exhaust 479, an air tank 477, a spare tire 478, and a drive shaft 480. In some embodiments, the vehicle 470 may be a truck with a trailer (herein, the term trailer may be correctively used to indicate a trailer or a container) and the locations and / or components of the Docket No.8842-158683-WO_8390WO01vehicle inspected by the inspection system 100 may include a door 473 of the trailer, and side walls of the trailer 474. In some embodiments, the trailer may be a refrigerated trailer and the locations and / or components of the vehicle inspected by the inspection system 100 may further include a refrigeration unit 475. In some embodiments, the inspection system 100 may further inspect the stability of objects in the trailer. For example, the on-board sensing devices 132 may include the electromagnetic wave sensor to see whether objects (e.g., boxes, cases) stacked in the trailer are likely to topple when the trailer door is opened.

[0035] FIG.5 illustrates an on-board sensor array 533 with multiple sensing devices 532 in accordance with some embodiments. FIG.6 illustrates a part of the mobile robot 120 with the on-board sensor array 533 inspecting a wheel of the vehicle in accordance with some embodiments. In some embodiments, the on-board sensor array 533 may be rotatably coupled to the robotic arm 261 mounted on the robotic mobility platform 124, which allows the on-board sensing device 532 to approach the components of the vehicle in various directions and angles.

[0036] Referring back to FIG. 1, in some embodiments, the memory 104 may include a volatile and / or non-volatile memory. In some embodiments, the memory 104 may include a random-access memory (RAM). The memory 104 may serve to store computer instructions that, when executed by the control circuit 102, cause the control circuit 102 to behave as described herein. In some embodiments, the memory 104 may serve, for example, to non-transitorily store computer instructions. As used herein, this reference to "non-transitorily" will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence the memory may include both non-volatile memory (such as read-only memory (ROM) as well as volatile memory (such as an erasable programmable read-only memory (EPROM).

[0037] The memory 104 may provide storage for the detection data 106 and one or more modules 108. In some embodiments, the detection data 106 may include, but not limited to, image data (e.g., 2D images, 3D images, depth information, 3D point cloud, color data, etc.) of the field of view, sound data of the working environment, force data of force applied to the mobile robot (e.g., to the object interfacing mechanism 128), thermal data, electromagnetic wave sensor data, optical sensor data, location data, and / or any other types of sensor data collected via the detection Docket No.8842-158683-WO_8390WO01system 130 that may be necessary or used to conduct steps, actions, and / or functions described herein. The memory 104 may also store data generated during the use / operation of the inspection system 100. The one or more modules 108 may include codes executable by the control circuit 102 and, when executed by the control circuit 102, cause the control circuit 102 to perform specific steps, actions, and / or functions designed by each module. In some embodiments, the modules 108 may include, but are not limited to, a point cloud accumulation module, a point cloud filtering module, and / or an image segmentation module. In some embodiments, the modules 108 may further include a distance calculation module, an inverse kinematics module, a motion planning module. In some embodiments, the modules 108 may further include modules facilitating other automation system operations including, but not limited to, a system safety module, a fleet management module, and a warehouse / yard management system interfacing module. In some embodiments, the one or more of the modules 108 may employ a machine learning model 107 to improve capabilities for object identification, object manipulation, inspection processes, navigating, and ancillary tasks such as exceptional handling and so on. In some embodiments, the machine learning model 107 may include a computer vision (CV) model 107a. In some embodiments, the detection data 106 captured during the operation of the mobile robot 120 may be used to train the machine learning model 107.

[0038] In some embodiments, the control circuit 102 may operably / communicatively couple to the memory 104, mobile robot 120, and the detection system 130 (including the on-board sensing devices 132). The control circuit 102 may receive information from the detection system 130 and may store the information (e.g., detection data 106) received from the detection system 130 to the memory 104.

[0039] The control circuit 102 may access the memory 104 and execute the codes stored in the memory 104. In some embodiments, the memory 104 may be integral to the control circuit 102 or may be physically discrete in whole or in part from the control circuit 102 as desired. This memory 104 may also be local with respect to the control circuit 102 (where, for example, both share a common circuit board, chassis, power supply, and / or housing) or may be partially or wholly remote with respect to the control circuit 102 (where, for example, the memory 104 is physically Docket No.8842-158683-WO_8390WO01located in another housing or remotely location). In some embodiments, the memory 104 may be distributed in multiple locations.

[0040] The control circuit 102 is configured, for example by using corresponding codes / programming and / or using the modules 108 stored in memory 104, to carry out and / or send signals to carry out one or more of the steps, actions, and / or functions described herein. The control circuit 102 may comprise structure that includes at least one (and typically many) electrically- conductive paths (such as paths comprised of a conductive metal such as copper or silver) that convey electricity in an ordered manner, which path(s) will also typically include corresponding electrical components (both passive (such as resistors and capacitors) and active (such as any of a variety of semiconductor-based devices) as appropriate) to effect one or more of the steps, actions, and / or functions described herein. The control circuit 102, for example, may comprise a fixed- purpose hard-wired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit that is customized by design for a particular use, rather than intended for general-purpose use), a field-programmable gate array (FPGA), and the like) or can comprise a partially or wholly-programmable hardware platform (including but not limited to microcontrollers, microprocessors, and the like).

[0041] In some embodiments, the control circuit 102 may use 2D and 3D image data (e.g., 2D images, a 3D point cloud, or a combination thereof) and recognize edges, shapes, and near / far distances of objects to identify the objects. In some embodiments, the control circuit 102 may detect / recognize object edges, shapes, and near / far distance of objects in the working environments of the mobile robot 120 (e.g., vehicles, trailers of the vehicles, and components / parts of the vehicles, obstacles, etc.). In some embodiments, the control circuit 102 may use the 3D point cloud of the objects, detect the edge of each object in the 3D point cloud, and draw boundary lines for the objects based on the detected edges. Further, the control circuit 102 may determine the near / far distance of each object from the reference point using depth information of the 3D point cloud of the objects.

[0042] FIG.7 is a flowchart depicting an example method 700 for use with an inspection system in accordance with some embodiments. The method 700 may be performed using the inspection system 100 in accordance with the approaches described above. Although the method Docket No.8842-158683-WO_8390WO01700 is mainly illustrated with the inspection system 100, the method 700 may also be performed with a robotic system differently configured.

[0043] In step 702, the control circuit 102 may identify a target vehicle in the working environment. The target vehicle may be a vehicle with a trailer / container that needs to be inspected. In some embodiments, the control circuit 102 may identify the target vehicle using the detection data 106 (information of the working environment collected via the detection system 130). The working environment may include vehicles and / or other objects in the operation environment / area of the mobile robot 120. In some embodiments, the detection data 106 may include information for the entire / overall area of the working environment collected via the detection system 130 to monitor vehicles in the working environment (e.g., movement of vehicles in the working environment, entering / leaving of vehicles in the working environment). Additionally or alternatively, the control circuit 102 may identify the target vehicle based on signals from vehicles or on signals from a user (e.g., a driver) of the vehicles. For example, when the vehicles are capable of directly or indirectly communicating with the control circuit 102, the vehicles may send signals requesting vehicle inspection to the control circuit 102, and based on the requesting signals, the control circuit 102 may identify and / or determine the target vehicle. In some embodiments, the user of the vehicles may send vehicle inspection requesting signals to the control circuit 102, using a user device (e.g., mobile electronic devices).

[0044] In step 704, the control circuit 102 may determine a travel path of the mobile robot 120 to the target vehicle. To determine the travel path, the control circuit 102 may determine the current location of the mobile robot and the current location of the target vehicle. In some embodiments, the current location of the mobile robot 120 and / or the target vehicle may be determined based on the detection data 106 collected via the detection system 130, and control circuit 102 may determine, using the determined current locations of the mobile robot 120 and the target vehicle, the travel path of the mobile robot 120 to the target vehicle. In some embodiments, the mobile robot 120 may include a built-in location tracking system (e.g., a GPS system) and the current location of the mobile robot may be determined using the built-in location tracking system. Similarly, the target vehicle mya include a built-in location system, and when the target vehicle includes a built-in location tracking system, the current location of the target vehicle may be Docket No.8842-158683-WO_8390WO01determined using the built-in location tracking system. In some embodiments, in determining the travel path, the control circuit may further consider other objects that may affect the travel of the mobile robot 120 such as static obstacles, frequently detected dynamic obstacles, etc.

[0045] In step 706, the mobile robot 120 may move / travel to the target vehicle according to the determined travel path. In some embodiments, the control circuit 102 may cause the mobile robot 120 to move / travel to the target vehicle according to the determined travel path (e.g., by transmitting the signals for the travel path to the mobile robot 120). In some embodiments, the detection system 130 (including the perception system 133 of the mobile robot 120) may continue to collect the information of the working environment while the mobile robot 120 moves / travels toward the target vehicle according to the determined travel path, and the control circuit 102 may recognize, based on the information of the working environment collected during the travel, obstacles that have not been detected / predicted when the control circuit 102 initially determine the travel path of the mobile robot 120 in step 704. In some embodiments, when the control circuit 102 recognizes unpredicted obstacles while the mobile robot 120 travels to the target vehicle, the control circuit 102 may update the travel path in response to the recognition of the unpredicted obstacle and may cause the mobile robot 120 to move / travel to the target vehicle according to the updated travel path.

[0046] In step 708, the control circuit 102 may cause the mobile robot 120 and the on- board sensing devices 132 to follow a vehicle inspection protocol. In some embodiments, the mobility platform 124 automatically move the mobile robot 120 proximate to each of a plurality of locations of the target vehicle to perform the vehicle inspection protocol, and the plurality of locations may include a front of the target vehicle, a rear of the target vehicle, a side of the target vehicle, an undercarriage of the target vehicle, and / or a roof of the target vehicle. In some embodiments, the vehicle inspection protocol may include collecting, with the mobile robot 120 and the one or more on-board sensing devices 132, information of the multiple components / parts of the target vehicle at multiple locations around / near the target vehicle. In some embodiments, the vehicle inspection protocol may include collecting, with the on-board sensing devices 132, detailed images corresponding to the multiple components / parts of the target vehicle. Docket No.8842-158683-WO_8390WO01

[0047] For example, the vehicle inspection protocol may include sequentially moving the mobile robot 120 proximate to each of one or more of the wheel tires 471 of the target vehicle 470, the undercarriage 472 of the target vehicle 470, the door 473 of the trailer of the target vehicle 470, the side walls of the trailer 474 of the target vehicle 470, the refrigeration unit 475 of the target vehicle 470, the bumper 476 of the target vehicle 470, the air tank 477 of the target vehicle 470, the spare tire 478 of the target vehicle 470, the exhaust 479 of the target vehicle 470, and the drive shaft 480 of the target vehicle 470 in a proper moving sequence, and capturing, with the on-board sensing devices 132 on the mobile robot 120 when the mobile robot 120 is at the corresponding location, the information from each of one or more of the wheel tires 471 of the target vehicle 470, the undercarriage 472 of the target vehicle 470, the door 473 of the trailer of the target vehicle 470, the side walls of the trailer 474 of the target vehicle 470, the refrigeration unit 475 of the target vehicle 470, the bumper 476 of the target vehicle 470, the air tank 477 of the target vehicle 470, the spare tire 478 of the target vehicle 470, the exhaust 479 of the target vehicle 470, and the drive shaft 480 of the target vehicle 470.

[0048] In some embodiments, the inspection protocol may include scanning, with the detection system 130, an overall structure of the vehicles and the control circuit 102 may determine the proper moving sequence of the mobile robot 120. In some embodiments, the proper moving sequence of the mobile robot 120 may be predetermined.

[0049] In some embodiments, the vehicle inspection protocol may further include capturing, with the on-board sensing devices 132 located outside of the trailer of the target vehicle, the information of the inside of the trailer of the target vehicle before opening the trailer door. In some embodiments, to capture the information of the inside of the trailer of the target vehicle, the detection system may include an electromagnetic (EM) wave sensor configured to detect the electromagnetic wave emitted from or reflected at the inside of the trailer of the target vehicle. In some embodiments, the detection system may include an EM wave generator and an EM wave reader. The EM waver generator may project EM waves, such as radio waves, infrared waves, or microwaves, into the inside of the trailer of the target vehicle. The EM wave reader or EM wave sensor may receive reflections of the EM waves from the inside of the trailer of the target vehicle. The EM wave generator and reader may be separate devices, in the same or separate housings, or Docket No.8842-158683-WO_8390WO01can be incorporated into one device. In some embodiments, the EM wave generator and the EM wave reader may be mounted to the mobile robot 120. In other words, the on-board sensing devices 132 may include the EM wave generator and the EM wave reader.

[0050] In some embodiments, the control circuit 102 may communicate with the target vehicle, a user of the target vehicle, and / or a person in the working environment (e.g., an inspection assistant) to inspect the target vehicle while performing the vehicle inspection protocol when the communication is necessary to complete the vehicle inspection protocol and / or the communication facilitates performing the vehicle inspection protocol. For example, to inspect the target vehicle, the control circuit may send signals requesting to use the wiper blades, turn on / off the left / right signals, flash on / off the hazard light, put down / up the windows, honk the horn, etc. to inspect each of these components.

[0051] In step 710, the control circuit 102 may obtain, from the detection system 130, detection data 106 corresponding to the target vehicles. The detection data may include the data collected with the one or more on-board sensing device 132 while following / performing the vehicle inspection protocol in step 708.

[0052] In step 712, the control circuit 102 may identify, using and / or analyzing the detection data, one or more of a vehicle identifier of the target vehicle, a trailer identifier of the target vehicle, and a defect of the target vehicle. The vehicle identifier may be identified from the detection data corresponding to the vehicle registration plate or other indication assigned to the target vehicle. The trailer identifier of the target vehicle may be printed on the outer surface of the trailer or on the trailer door and may be identified using the detection data therefrom.

[0053] In some embodiments, the defect of the target vehicle may include a defect of one or more of a wheel tire of the target vehicle, an exterior of a vehicle body (including the undercarriage) of the target vehicle, an exterior of the trailer of the target vehicle, a stack of objects in the trailer of the target vehicle, a door of the trailer of the target vehicle, a refrigeration unit of the target vehicle, a bumper of the target vehicle, an air tank of the target vehicle, a spare tire of the target vehicle, an exhaust of the target vehicle, and a drive shaft of the target vehicle. In some embodiments, the defect of the target vehicle includes one or more of a structural damage on a Docket No.8842-158683-WO_8390WO01component / part of the target vehicle, a crack on the component / part of target vehicle, a dent of the component / part of the target vehicle, wear of the component / part of the target vehicle, rust on the component / part of the target vehicle, instability of objects in the trailer of the target vehicle, and a seal defect (e.g., trailer door seal defect) of the target vehicle. For example, the defect of the target vehicle may include wear of tires, a dent of the trailer, etc.

[0054] In some embodiment, the control circuit may identify the defects of the target vehicle using a machine learning algorithm based on the computer vision (CV) model 107a. In some embodiments, the machine learning algorithm may use the CV model 107a trained based on previously captured images for components / parts of the vehicle as input and defect identifiers as categorizations. In some embodiments, the CV model may be trained based on images captured during the previous operation of the inspection system 100.

[0055] FIG.8 is a flowchart depicting an example method 800 to generate the computer vision model 107a in accordance with some embodiments.

[0056] In step 802, previously captured images of vehicles with and / or without defects may be labeled, with each image accompanied by a caption containing information about the specific defect present in the vehicle. In some embodiments, the previously captured images of vehicles may be 2D color images. The caption may include the presence or absence of a defect on the vehicle indicated on the images, and when the captioned images include an indication of one or more defects, the caption may further include the location of the defects of the vehicle indicated on the images. In some embodiments, the location of the defects may be an indication of the part / component of the vehicle (e.g., a tire of the vehicle, a signal of the vehicle, etc.) where the defect exists. In some embodiments, the location of the defects may be more specific, for example, a specific position within the part / component of the vehicle. In step 804, the images may be resized to a consistent resolution. In step 806, the images may be converted to grayscale images. In step 808, the images may be segmented into relevant depth maps. The segmented depth maps may be easily compared to new images captured during the inspection. In step 810, the segmented depth maps may be normalized to increase the accuracy of the CV model by ensuring uniformity of the training data. In step 812, the computer vision may be trained for defect identification using the normalized depth maps to generate the trained CV model 107a. Docket No.8842-158683-WO_8390WO01

[0057] In some embodiments, to increase the accuracy of defect identification using the trained CV model 107a, the control circuit 102 may conduct steps 804 to 810 with the images captured for inspection of the target vehicle. For example, the control circuit 102 may resize the image captured for the inspection to the resolution that is consistent with the resolution of the images of the CV model training dataset, convert the resized images to grayscale, segment the grayscale images into depth maps, and normalize the segmented depth maps.

[0058] Referring back to FIG.7, in some embodiments, in step 712, the machine learning algorithm using the trained CV model 107a may compare knowledges from the images of the training dataset with the images captured for target vehicle inspection and may determine the condition of the target vehicle is good or bad. In some embodiments, in step 712, the machine learning algorithm using the trained CV model 107a may analyze the model’s output to identify the presence and location of the defects on the target vehicle.

[0059] In some embodiments, the trained CV model 107a may be updated. In some embodiments, using the detection data 106 collected during performing the vehicle inspection protocol for the target vehicle and information of the defect of the target vehicle identified in step 712, the steps 802 to 812 may be repeated to update the trained CV model 107a.

[0060] In step 714, the control circuit 102 may analyze the identified defect of the target vehicle to determine whether a service to the target vehicle (e.g., repair, replacement, maintenance of the defected components / parts of the vehicles) is necessary. When the control circuit 102 determines that the service to the target vehicle is necessary, the control circuit 102 may transmit the result of the determination regarding the necessity of the service to a central management system.

[0061] In some embodiments, steps 702 to 714 may be repeated for the next target vehicle. Steps 702 to 714 may be conducted / repeated as many times as necessary to identify the defects of the target vehicles in the working environment.

[0062] In some embodiments, an inspection system for use at a commercial product facility may comprise a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, wherein the working environment comprises an area of the Docket No.8842-158683-WO_8390WO01commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, a detection system configured to collect information from a vehicle in the working environment, the detection system comprising one or more on-board sensing devices physically coupled to the mobile robot, a control circuit communicatively coupled to the mobile robot and the one or more on-board sensing devices, the control circuit configured to identify a target vehicle including a trailer in the working environment, determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, cause the mobile robot to move to the target vehicle according to the determined travel path, cause the mobile robot and the one or more on-board sensing devices to follow a vehicle inspection protocol, obtain, via the one or more on-board sensing devices, detection data corresponding to the target vehicle, an identify, using the detection data, one or more of a vehicle identifier of the target vehicle, a trailer identifier of the target vehicle, and a defect of the target vehicle.

[0063] In some embodiments, a method for use with an inspection system at a commercial product facility may comprise identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle including a trailer in a working environment, the detection system comprising one or more on-board sensing devices physically coupled to the mobile robot, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store, determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle, moving the mobile robot to the target vehicle according to the determined travel path, obtaining, via the one or more on-board sensing devices, detection data corresponding to the target vehicle, and identifying, using the detection data, one or more of a vehicle identifier of the target vehicle, a trailer identifier of the target vehicle, and a defect of the target vehicle.

[0064] Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above described embodiments Docket No.8842-158683-WO_8390WO01without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept. Docket No.8842-158683-WO_8390WO01

Claims

CLAIMS What is claimed is:

1. An inspection system for use at a commercial product facility, the inspection system comprising: a mobile robot configured to navigate a working environment and travel to a vehicle in the working environment, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; a detection system configured to collect information from a vehicle in the working environment, the detection system comprising one or more on-board sensing devices physically coupled to the mobile robot; a control circuit communicatively coupled to the mobile robot and the one or more on-board sensing devices, the control circuit configured to: identify a target vehicle including a trailer in the working environment; determine a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; cause the mobile robot to move to the target vehicle according to the determined travel path; cause the mobile robot and the one or more on-board sensing devices to follow a vehicle inspection protocol; obtain, via the one or more on-board sensing devices, detection data corresponding to the target vehicle; and identify, using the detection data, one or more of: a vehicle identifier of the target vehicle; a trailer identifier of the target vehicle; and a defect of the target vehicle. Docket No.8842-158683-WO_8390WO012. The inspection system of claim 1, wherein the defect of the target vehicle includes one or more of: a structural damage of the target vehicle; a crack of the target vehicle; a dent of the target vehicle; wear of the target vehicle; rust of the target vehicle; instability of objects in the trailer of the target vehicle; and a seal defect of the target vehicle.

3. The inspection system of claim 1, wherein the defect of the target vehicle includes a defect of one or more of: a wheel tire of the target vehicle; an exterior of a vehicle body of the target vehicle; an exterior of the trailer of the target vehicle; a stack of objects in the trailer of the target vehicle; a door of the trailer of the target vehicle; a refrigeration unit of the target vehicle; a bumper of the target vehicle; an air tank of the target vehicle; a spare tire of the target vehicle; an exhaust of the target vehicle; and a drive shaft of the target vehicle.

4. The inspection system of claim 1, wherein the mobile robot comprises a mobility platform comprising one or more of wheels, drivers, and / or motorized limbs.

5. The inspection system of claim 1, wherein the mobile robot comprises a mobility platform configured to automatically move the mobile robot proximate to each of a plurality of locations of the target vehicle to perform the vehicle inspection protocol, the Docket No.8842-158683-WO_8390WO01plurality of locations including a front of the target vehicle, a rear of the target vehicle, a side of the target vehicle, an undercarriage of the target vehicle, and / or a roof of the target vehicle.

6. The inspection system of claim 1, wherein the control circuit is further configured to analyze the identified defect of the target vehicle to determine a necessity of a service to the target vehicle and transmitting a result of determination regarding the necessity of the service to the target vehicle to a central management system.

7. The inspection system of claim 1, wherein the control circuit is configured to identify the defect of the target vehicle using a computer vision model.

8. The inspection system of claim 7, wherein the control circuit is further configured to use the detection data of the target vehicle and the identified defect of target vehicle to update the computer vision model.

9. The inspection system of claim 7, wherein the computer vision model is generated at least by: labeling, with a caption, images of vehicles with and / or without a defect, the caption comprising information on the defect of the vehicle on each of the images; resizing the images to a consistent resolution; converting the images to grayscale; segmenting the images into depth maps; normalizing the depth maps; and training, with the normalized depth maps, a computer vision.

10. The inspection system of claim 9, wherein the caption comprises one or more of: a presence of the defect of the vehicle indicated on the images; and a location of the defect of the vehicle indicated on the images. Docket No.8842-158683-WO_8390WO0111. The inspection system of claim 1, wherein the mobile robot comprises an articulating arm and the one or more on-board sensing devices coupled to an end of the articulating arm.

12. The inspection system of claim 1, wherein the vehicle inspection protocol comprising collecting, with the mobile robot and the one or more on-board sensing devices, information of the target vehicle at multiple locations around the target vehicle.

13. The inspection system of claim 1, wherein the control circuit is further configured to recognize object edges, shapes, and near / far distance of objects.

14. The inspection system of claim 1, wherein the detection system includes one or more of a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, an RGB overlay camera, a thermal sensor, an electromagnetic wave sensor, an optical sensor, an IMU (Inertial Measurement Unit) sensor, a gyroscope, and a force sensor.

15. The inspection system of claim 1, wherein the mobile robot includes an end effector configured to facilitate inspection of the target vehicle.

16. The inspection system of claim 1, wherein the mobile robot includes an on-board lighting device configured to facilitate inspection of the target vehicle.

17. The inspection system of claim 1, wherein the control circuit is further configured to communicate with the target vehicle, a user of the target vehicle, and / or a person in the working environment to inspect the target vehicle while performing the vehicle inspection protocol.

18. A method for use with an inspection system at a commercial product facility, the method comprising: Docket No.8842-158683-WO_8390WO01identifying, with a control circuit coupled to a mobile robot and a detection system, a target vehicle including a trailer in a working environment, the detection system comprising one or more on-board sensing devices physically coupled to the mobile robot, wherein the working environment comprises an area of the commercial product facility that receives and interacts with vehicles containing commercial products, the commercial product facility comprising one or more of a warehouse, a fulfillment center and a distribution center, and a store; determining, with the control circuit, a travel path of the mobile robot from a current location of the mobile robot to the target vehicle; moving the mobile robot to the target vehicle according to the determined travel path; obtaining, via the one or more on-board sensing devices, detection data corresponding to the target vehicle; and identifying, using the detection data, one or more of: a vehicle identifier of the target vehicle; a trailer identifier of the target vehicle; and a defect of the target vehicle.

19. The method of claim 18, further comprising analyzing the identified defect of the target vehicle to determine a necessity of a service to the target vehicle and transmitting a result of determination regarding the necessity of the service to the target vehicle to a central management system.

20. The method of claim 18, wherein the identifying the defect of the target vehicle comprises using a computer vision model. Docket No.8842-158683-WO_8390WO01

Images