Method and architecture for mobile robotic lidar-based mis-localization detection

Abstract

A mobile robot includes one or more sensors configured to generate LiDAR data, one or more motors, and a controller. The controller is programmed to generate a plurality of LiDAR scans of different angular rotations based on the LiDAR data and pose data of the mobile robot, generate transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area, compute a matching score for each of the transformation candidates, select a transformation candidate with a highest matching score among the transformation candidates, determine whether the highest matching score is greater than a threshold, and output a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

Description

METHOD AND ARCHITECTURE FOR MOBILE ROBOTIC LIDAR-BASED MISLOCALIZATION DETECTIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 553,714 filed on February 15, 2024, the entire contents of which are herein incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure generally relates to systems and methods for detecting mislocalization of mobile robots based on LiDAR data and robot pose data.BACKGROUND

[0003] While a mobile robot is moving in an indoor environment, due to sensor system noise, sensor failure, or challenging environment with no features or highly symmetric environment, the mobile robot may have mislocalization issues. The mislocalization may cause the failure of the mission of the mobile robot and require extra human intervention to relocalize the mobile robot. Thus, the mobile robot needs to correct its pose in real time to prevent mislocalization when the mobile robot is moving in the indoor environment.

[0004] Mobile robots rely on a mapped representation of their operating area and utilize either an internal localization system or an external positioning system to determine their precise locations. Inaccurate localization poses risks, leading to task failures and potential hazards such as collisions. Conventional system may exhibit occasional false positives (indicating mislocalization when the mobile robot is correctly localized) or false negatives (indicating correct localization when the mobile robot is mislocalized).

[0005] Accordingly, a need exists for correctly detecting mislocalization of robots without disrupting the robot’s workflow.SUMMARY

[0006] In one aspect, a mobile robot includes one or more sensors configured to generate LiDAR data, one or more motors, and a controller. The controller is programmed to generate a plurality of LiDAR scans of different angular rotations based on the LiDAR data and pose data of the mobile robot, generate transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area, compute a matchingscore for each of the transformation candidates, select a transformation candidate with a highest matching score among the transformation candidates, determine whether the highest matching score is greater than a threshold, and output a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

[0007] In another aspect, a method for detecting mislocalization of a mobile robot is provided. The method includes generating a plurality of LiDAR scans of different angular rotations based on LiDAR data and pose data of the mobile robot, generating transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area, computing a matching score for each of the transformation candidates, selecting a transformation candidate with a highest matching score among the transformation candidates, determining whether the highest matching score is greater than a threshold, and outputting a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

[0008] These and additional features provided by the embodiments described herein will be more fully understood in view of the following detailed description, in conjunction with the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The embodiments set forth in the drawings are illustrative and exemplary in nature and not intended to limit the subject matter defined by the claims. The following detailed description of the illustrative embodiments can be understood when read in conjunction with the following drawings, where like structure is indicated with like reference numerals and in which:

[0010] FIG. 1 depicts an overall system including a mobile robot communicating with a cloud server via a communication network, according to one or more embodiments described and shown herein;

[0011] FIG. 2 depicts the block diagram of the overall system, according to one or more embodiments described and shown herein;

[0012] FIG. 3 depicts a schematic view of the mobile robot, according to the one or more embodiments show and described herein;

[0013] FIG. 4 depicts a flowchart of a method for detecting mislocalization of a mobile robot, according to one or more embodiments shown and described herein;

[0014] FIG. 5 depicts an example of the stack of maps with different resolutions;

[0015] FIG. 6 illustrates a user-defined search window shape with respect to a map and an illustrative representation of the decision-making process for computing transformation candidates;

[0016] FIG. 7 depicts a flowchart for detecting mislocalization when a user manually localizes a mobile robot, according to one or more embodiments shown and described herein;

[0017] FIG. 8 depicts a flowchart for detecting mislocalization when a robot fails a navigation mission, according to one or more embodiments shown and described herein;

[0018] FIG. 9 depicts a flowchart for detecting mislocalization when a user boots up a mobile robot, according to one or more embodiments shown and described herein; and

[0019] FIG. 10 depicts a flowchart for detecting mislocalization when a mobile robot is in an idle state, according to one or more embodiments shown and described herein.DETAILED DESCRIPTION

[0020] The embodiments described herein are directed to systems and methods for detecting mislocalization of mobile robots based on LiDAR data and robot pose data. The method includes generating a plurality of LiDAR scans of different angular rotations based on LiDAR data and pose data of a mobile robot; generating transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area; computing a matching score for each of the transformation candidates; selecting a transformation candidate with a highest matching score among the transformation candidates; determining whether the highest matching score is greater than a threshold; and outputting a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

[0021] The present mislocalization detection system for mobile robots offers substantial benefits across multiple dimensions. First, the present system minimizes operational risks. The present system addresses the common challenge of mislocalization in mobile robotics by utilizing the proposed mislocalization detection method as a risk mitigation tool. The present system reduces the likelihood of errors or accidents by maintaining precise self-awareness of the robot's location. Second, the present system increases autonomy and reliability. The present system enhances the autonomy of mobile robots, minimizes the need for constant human intervention and elevates overall system reliability, rendering them versatile for diverse applications. Third, the present system provides a competitive edge to entities operating in the robotics sector. Fourth, the presentsystem contributes significantly to advancing robotic technologies. By addressing the critical issue of mislocalization, it not only enhances current robotic capabilities but also lays the groundwork for future innovations and developments in the field.

[0022] FIG. 1 depicts an overall system including a mobile robot communicating with a cloud server via a communication network, according to one or more embodiments described and shown herein. In embodiments, the system includes a mobile robot 100, a communication network 120, a cloud server 140, and a robot control system 200. The robot control system 200 may be a local computing device or an edge device, e.g., a computer located at a store, that communicates with the mobile robot 100 and controls movement of the mobile robot 100 locally.

[0023] The main body of the mobile robot 100 may include a front- view camera 116, a FiDAR sensor 118, and a driving unit 230. The front- view camera 116 may be a RGBD camera. The RGBD camera is a type of depth camera that provides both depth (D) and color (RGB) data as the output in real-time. The LiDAR sensor 118 scans the environment of the mobile robot 100 and generates LiDAR data such as a point cloud. The driving unit 230 may move the mobile robot 100 around in the environment.

[0024] The mobile robot 100 may perform predetermined functions or assigned tasks (e.g., serving food and retrieving containers) through communication with the robot control system 200, and may include a support configured to support at least one object. The mobile robot 100 may include at least one of a module (e.g., a grab or a robotic arm module) for loading and unloading an object (e.g., a food tray), an imaging module (e.g., a visible light camera or an infrared camera) for acquiring images of surroundings, a scanner module (e.g., a LIDAR sensor) for acquiring information on obstacles, a sound acquisition module (e.g., a microphone) for acquiring sounds of surroundings, an illuminance acquisition module (e.g., an illuminance sensor) for sensing brightness of surroundings, a speaker module for providing sound information, a display module (e.g., LCD) for providing visual information such as text information, a light emitting module (e.g., LED) for providing visual information such as color information, and a drive module (e.g., a motor) for moving the mobile robot 100.

[0025] For example, the mobile robot 100 may have characteristics or functions similar to those of at least one of a serving robot, a guide robot, a transport robot, a cleaning robot, a medical robot, an entertainment robot, a pet robot, and an unmanned flying robot. Meanwhile, supporting of an object herein should be interpreted as encompassing supporting of a container for containing an object such as food, a means where the container may be placed (e.g., a tray), or the like.

[0026] Meanwhile, according to one embodiment of the present disclosure, the mobile robot 100 may include an application (not shown) for controlling the mobile robot 100. The application may be downloaded from the robot control system 200 or an external application distribution server, such as the cloud server 140. The application may be stored in the memory of the mobile robot 100 such as the one or more memory modules 204 in FIG. 2. Here, at least a part of the application may be replaced with a hardware device or a firmware device that may perform a substantially equal or equivalent function, as necessary.

[0027] The mobile robot 100 may navigate around in an indoor environment, collect FiDAR data, calculate its pose, and detects mislocalization of the mobile robot 100 based on LiDAR scans and global map data. The details of detecting the mislocalization of the mobile robot 100 will be described below with reference to FIG. 4-6.

[0028] Referring now to FIG. 2, various internal components of the mobile robot 100 and the cloud server 104 are illustrated. The mobile robot 100 may include a controller 210 that includes one or more processors 202 and one or more memory modules 204, a satellite antenna 220, a driving unit 230, network interface hardware 240, a screen 110, a microphone 112, a speaker 114, a front-view camera 116, the LiDAR sensor 118, and an odometry sensor 122. In some embodiments, the one or more processors 202, and the one or more memory modules 204 may be provided in a single integrated circuit (e.g., a system on a chip). In some embodiments, the one or more processors 202, and the one or more memory modules 204 may be provided as separate integrated circuits.

[0029] Each of the one or more processors 202 is configured to communicate with electrically coupled components, and may be configured as any commercially available or customized processor suitable for the particular applications that the mobile robot 100 is designed to operate. Each of the one or more processors 202 may be any device capable of executing machine readable instructions. Accordingly, each of the one or more processors 202 may be a controller, an integrated circuit, a microchip, a computer, or any other computing device. The one or more processors 202 are coupled to a communication path 206 that provides signal interconnectivity between various modules of the mobile robot 100. The communication path 206 may communicatively couple any number of processors with one another, and allow the modules coupled to the communication path 206 to operate in a distributed computing environment. Specifically, each of the modules may operate as a node that may send and / or receive data. As used herein, the term “communicatively coupled” means that coupled components are capable of exchanging datasignals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and the like.

[0030] Accordingly, the communication path 206 may be formed from any medium that is capable of transmitting a signal such as, for example, conductive wires, conductive traces, optical waveguides, or the like. Moreover, the communication path 206 may be formed from a combination of mediums capable of transmitting signals. In one embodiment, the communication path 206 comprises a combination of conductive traces, conductive wires, connectors, and buses that cooperate to permit the transmission of electrical data signals to components such as processors, memories, sensors, input devices, output devices, and communication devices. Additionally, it is noted that the term "signal" means a waveform (e.g., electrical, optical, magnetic, mechanical or electromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave, square-wave, vibration, and the like, capable of traveling through a medium.

[0031] The one or more memory modules 204 may be coupled to the communication path 206. The one or more memory modules 204 may include a volatile and / or nonvolatile computer- readable storage medium, such as RAM, ROM, flash memories, hard drives, or any medium capable of storing machine readable instructions such that the machine readable instructions can be accessed by the one or more processors 202. The machine readable instructions may comprise logic or algorithm(s) written in any programming language of any generation (e.g., 1GT, 2GT, 3GT, 4GT, or 5 GT) such as, for example, machine language that may be directly executed by the processor, or assembly language, user-oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into machine readable instructions and stored on the one or more memory modules 204. Alternatively, the machine readable instructions may be written in a hardware description language (HDT), such as logic implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), or their equivalents. Accordingly, the methods described herein may be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components.

[0032] The one or more memory modules 204 may be configured to store one or more modules, each of which includes the set of instructions that, when executed by the one or more processors 202, cause the mobile robot 100 to carry out the functionality of the module described herein. For example, the one or more memory modules 204 may be configured to store a robotoperating module, including, but not limited to, the set of instructions that, when executed by the one or more processors 202, cause the mobile robot 100 to carry out general robot operations.

[0033] The mobile robot 100 may include the satellite antenna 220 coupled to the communication path 206 such that the communication path 206 communicatively couples the satellite antenna 220 to other modules of the mobile robot 100. The satellite antenna 220 is configured to receive signals from global positioning system satellites. Specifically, in one embodiment, the satellite antenna 220 includes one or more conductive elements that interact with electromagnetic signals transmitted by global positioning system satellites. The received signal is transformed into a data signal indicative of the location (e.g., latitude and longitude) of the satellite antenna 220 or a user positioned near the satellite antenna 220, by the one or more processors 202. In some embodiments, the mobile robot 100 may not include the satellite antenna 220.

[0034] The driving unit 230 may comprise actuators, associated drive electronics to control the actuators, and any other external components that may be present in the mobile robot 100. The driving unit 230 may include one or more motors. The driving unit 230 may be configured to receive control signals from the one or more processors 202 and to operate the mobile robot 100 accordingly. The operating parameters and / or gains for the driving unit 230 may be stored in the one or more memory modules 204.

[0035] The mobile robot 100 includes the network interface hardware 240 for communicatively coupling the mobile robot 100 with the cloud server 140 or the robot control system 200. The network interface hardware 240 may be coupled to the communication path 206 and may be configured as a wireless communications circuit such that the mobile robot 100 may communicate with external systems and devices. The network interface hardware 240 may include a communication transceiver for sending and / or receiving data according to any wireless communication standard. For example, the network interface hardware 240 may include a chipset (e.g., antenna, processors, machine readable instructions, etc.) to communicate over wireless computer networks such as, for example, wireless fidelity (Wi-Fi), WiMax, Bluetooth, IrDA, Wireless USB, Z-Wave, ZigBee, or the like. In some embodiments, the network interface hardware 240 includes a Bluetooth transceiver that enables the mobile robot 100 to exchange information with the cloud server 140 or the robot control system 200.

[0036] The mobile robot 100 may include the screen 110 coupled to the communication path 206 such that the communication path 206 communicatively couples the screen 110 to othermodules of the mobile robot 100. The screen 110 may display information about a task currently implemented by the mobile robot 100, for example, delivering items, picking up items, and the like.

[0037] The mobile robot 100 includes the microphone 112 coupled to the communication path 206 such that the communication path 206 communicatively couples the microphone 112 to other modules of the mobile robot 100. The microphone 112 may be configured for receiving user voice commands and / or other inputs to the mobile robot 100. The microphone 112 transforms acoustic vibrations received by the microphone 112 into a speech input signal.

[0038] The mobile robot 100 includes the speaker 114 coupled to the communication path 206 such that the communication path 206 communicatively couples the speaker 114 to other modules of the mobile robot 100. The speaker 114 transforms data signals into audible mechanical vibrations. The speaker 114 outputs audible sound such that a user proximate to the mobile robot 100 may interact with the mobile robot 100.

[0039] The mobile robot 100 includes a front- view camera 116. The front- view camera 116 may include, but not limited to, RGBD sensor, or depth sensors configured to collect depth information of a target area. The front- view camera 116 may have any suitable resolution and may be configured to detect radiation in any desirable wavelength band, such as an ultraviolet wavelength band, a near-ultraviolet wavelength band, a visible light wavelength band, a near infrared wavelength band, or an infrared wavelength band.

[0040] The mobile robot 100 includes the TiDAR sensor 118. The TiDAR sensor 118 emits a laser beam which bounces off a target such as walls, doors, and people and returns to the TiDAR sensor 118. The TiDAR sensor 118 calculates the time it takes for the beam to return and determines the distance to the target. The TiDAR sensor 118 creates a 3D map of its surroundings using point clouds. The TiDAR sensor 118 helps the mobile robot 100 to avoid collisions with objects in the moving path of the mobile robot 100.

[0041] The mobile robot 100 includes the odometry sensor 122. The odometry sensor 122 is a motion sensor that provides data including the position, velocity, and attitude of the mobile robot 100. The odometry sensor 122 is positioned in the wheels of the mobile robot 100 and measures how far the wheels have turned.

[0042] The cloud server 140 includes a controller 260 that includes one or more processors 262 and one or more memory modules 264, and network interface hardware 268. The one or more processors 262, one or more memory modules 264, and the network interface hardware 268 may becomponents similar to the one or more processors 202, one or more memory modules 204, and the network interface hardware 240, as described above.

[0043] FIG. 3 depicts a schematic view of the mobile robot, according to the one or more embodiments show and described herein.

[0044] The mobile robot 100 may include a top tray 310, a touch screen 110, a middle tray 330, a front-view camera 116, a bussing bucket 340, a LiDAR sensor 118, a base 350 including a control system and an inertial measurement unit, and driving wheels 360 with wheel odometry sensors. FIG. 3 illustrates an example structure of the mobile robot 100, and the mobile robot 100 may have a different design with additional elements or less elements than the mobile robot 100 depicted in FIG. 3.

[0045] FIG. 4 depicts a flowchart of a method for detecting mislocalization of a mobile robot, according to one or more embodiments shown and described herein.

[0046] In step 410, the mobile robot 100 may preprocess the LiDAR data 404 to remove small clusters. The LiDAR data 404 may be LiDAR data 404 captured by the LiDAR sensor 118 of the mobile robot 100. The controller of the mobile robot 100 may remove small clusters from the LiDAR data to enhance robustness in dynamic environments. Small clusters may represent unmapped obstacles such as movable obstacles including chairs, tables, people, or any other objects that are commonly encountered in busy restaurant settings during rush hours, in contrast with fixed objects such as floor, walls, ceiling of a building. Preprocessing the LiDAR data by removing small clusters can make the detection method more robust to small noises in the environment.

[0047] In step 420, the mobile robot may preprocess map data to generate a stack of maps from a map with the highest resolution to a map with the lowest resolution. In embodiments, the mobile robot 100 may receive the map data from the cloud server 140 and preprocess the received map data to generate the stack of maps. The map data may be a map generated based on LiDAR data from mobile robots navigating in the environment. In some embodiments, the mobile robot 100 may receive the map data from an edge device or another mobile robot within a communication range. Generating the stack of maps is described with reference to FIG. 5.

[0048] FIG. 5 depicts an example of the stack of maps with different resolutions. The map 510 represents an original map and holds the highest resolution. The mobile robot 100 may receive the original map from the cloud server 140. The map 520 is a map with a lower resolution than the map 510. The map 530 is a map with the lowest resolution among the three maps.

[0049] The controller of the mobile robot 100 may generate the map 520 out of the map 510.For example, a window of nine pixels (3 x 3 square) is passed over every second pixel, in every other row of the map 510. Values of the pixels of the map 520 are determined based on the corresponding window in the map 510. For example, the value of the pixel 522 is determined based on the values of the nine pixels in the window 512 in the map 510. Different methods of decimations of a higher resolution map into a lower resolution map may be used. Among the varied resolutions, users have the flexibility to configure the lowest resolution to employ for mislocalization detection. Lower resolution enhances computation speed, however requires an increase in memory usage.

[0050] Referring back to FIG. 4, in step 430, the mobile robot generates a plurality of LiDAR scans of different angular rotations based on the LiDAR data and pose data of the mobile robot. The LiDAR data may be LiDAR data preprocessed in step 410. The pose data may be obtained by the mobile robot 100. For example, an image-based localization model or an image localization model may be used by the mobile robot 100 to navigate an area or environment, such as a service location. An example image-based localization model may take as input an image (e.g., an image capture in a particular environment or location) and output a set of potential poses or pose estimates. The pose data may include a candidate location (e.g., a grid cell of a configurable size in a grid corresponding to a map of a navigable area). The pose data may include a set of (x, y, yaw) values, where x and y are map or world coordinates and yaw is a yaw angle corresponding to orientation information.

[0051] In embodiments, the controller of the mobile robot 100 may generate a plurality of LiDAR scans of different angular rotations within a user-defined angular search window based on the LiDAR data and pose data of the mobile robot. A LiDAR scan of an angular rotation refers to the data collected by a LiDAR sensor as it rotates, capturing 3D points at various angles around its field of view, creating a complete picture of the surrounding environment by measuring distances to objects at different angular positions throughout a full rotation.

[0052] The angular step size of the LiDAR scan may be customized by a user. If the angular step size is not specified, the angular step size may be determined based on the resolution of the map being used and a maximum LiDAR scan range ensuring coverage of the highest resolution.

[0053] In step 440, the mobile robot generates transformation candidates for the plurality of LiDAR scans and computes a matching score for each of the transformation candidates. In embodiments, the controller of the mobile robot 100 generates transformation candidates for theplurality of LiDAR scans within a linear search window based on a scan matching between the plurality of LiDAR scans and a first map of an area. The first map of the area is one of the stack of maps computed in step 420. The linear step size of the linear search window may be determined by the resolution of the first map. The linear step size determines the resolution and accuracy of the sensor's measurements, allowing for precise mapping and object detection.

[0054] In embodiments, the controller of the mobile robot 100 may compute a matching score for each of the transformation candidates. The matching score refers to a numerical value that indicates how well a current LiDAR scan aligns with a reference scan, e.g., the first map, essentially representing the confidence level of the match between the two scans. Higher matching scores signify a better match and a more accurate pose estimation in the robot's current position. The matching score may be calculated by a scan matching algorithm used in robotics applications. For example, different scan matching algorithms, such as iterative closest point (ICP) or Normal Distributions Transform (NDT), use different cost functions to calculate the matching score, often based on the distance between corresponding points in the two scans after aligning them with corresponding transformation candidate. A high matching score indicates a good match between the scans, meaning the calculated pose is likely accurate, while a low matching score suggests a poor match and potential errors in pose estimation.

[0055] In embodiments, the controller of the mobile robot 100 may selectively compute matching scores for some of the transformation candidates instead of computing matching scores for all of the transformation candidates considering a user-defined search window shape. FIG. 6 illustrates a user-defined search window shape with respect to a map and an illustrative representation of the decision-making process for computing transformation candidates. In FIG. 6, the broken line circle denotes a user-defined linear search window 610, and the square 620 is used for identifying potential candidates, designed to over-approximate the user-defined linear search window. Here, the user-defined linear search window 610 is non-square. The controller of the mobile robot 100 may determine whether there is an overlap between a LiDAR map grid cell corresponding to a certain LiDAR scan and the user-defined linear search window. A LiDAR map grid cell is a cell in a raster or gridded data format that represents an area on the ground. LiDAR data may be stored in this format, which is made up of a regular grid of cells that are all the same size.

[0056] If there is no overlap between the LiDAR map grid cell corresponding to the certainLiDAR scan and the user-defined linear search window, the controller of the mobile robot 100 skipscalculating the matching score for the transformation candidate related to the certain LiDAR scan. If there is an overlap between the LiDAR map grid cell corresponding to the certain LiDAR scan and the user-defined linear search window, the controller of the mobile robot 100 calculates the matching score for the transformation candidate related to the certain LiDAR scan. For example, by referring to FIG. 6, the LiDAR map grid cell 622 overlaps with the user-defined linear search window 610. Thus, the controller of the mobile robot 100 calculates a matching score for the transformation candidate related to the LiDAR map grid cell 622. Similarly, the controller of the mobile robot 100 calculates a matching score for the transformation candidate related to the LiDAR map grid cell 624 because there is an overlap between the LiDAR map grid cell 624 and the user- defined linear search window 610. In contrast, there is no overlap between the LiDAR map grid cell 626 and the user-defined linear search window 610. Thus, the controller of the mobile robot 100 skips calculating the matching score for the transformation candidate related to the LiDAR map grid cell 626.

[0057] Referring back to FIG. 4, in step 450, the mobile robot sorts all generated transformation candidates based on their matching scores in descending order.

[0058] In step 460, the mobile robot determines whether the resolution of the map used for generating transformation candidates in step 440 is the highest among the stack of maps generated in step 420. If the resolution of the map used for generating transformation candidates in step 440 is not the highest among the stack of maps, the process goes to step 470. For example, the map used for generating transformation candidates in step 440 may be the map 530 in FIG. 5, the resolution of which is the lowest among the stack of maps 510, 520, 530. Then, the process goes to step 470.

[0059] If the map used for generating transformation candidates in step 440 has the highest resolution among the stack of maps, the process goes to step 480. For example, the map used for generating transformation candidates in step 440 may be the map 510 in FIG. 5, the resolution of which is the highest among the stack of maps 510, 520, 530. Then, the process goes to step 480.

[0060] In step 470, the mobile robot obtains a higher resolution map and generates additional transformation candidates based on the higher resolution map. In embodiments, the mobile robot 100 obtains LiDAR scans by having the linear step size of the LiDAR sensors, and generates additional transformation candidates for the LiDAR scans based on a scan matching between the LiDAR scans and a second map of the area with a resolution higher than the first map.

[0061] For example, the first map may be the map 530 in FIG. 5 and the second map may be the map 520 in FIG. 5. The mobile robot calculates the matching score for each of the additionaltransformation candidate with respect to the map 520. Then, the process returns to step 450 and then proceeds to step 460. In step 460, because the resolution of the map 520 is still not the highest among the stack of maps 510, 520, 530, the mobile robot 100 obtains LiDAR scans by further having the linear step size of the LiDAR sensors, and generates additional transformation candidates for the LiDAR scans based on a scan matching between the LiDAR scans and the map 510, the resolution of which is the highest among the stack of maps 510, 520, 530. Then, the process returns to step 450, and proceeds to steps 460 and 480.

[0062] In step 480, the mobile robot selects the transformation candidate with the highest matching score against the highest resolution map. For example, the mobile robot 100 selects the transformation candidate with the highest matching score against the map 510 which has the highest resolution among the stack of maps 510, 520, 530. This represents the optimal transformation within the search window, both angularly and linearly, ensuring the best alignment between the LiDAR and map.

[0063] In step 490, the mobile robot determines whether the highest matching score of the selected transformation candidate is greater than a threshold. If the mobile robot 100 determines that the highest matching score of the selected transformation candidate is greater than the threshold, then the process proceeds to step 492, and the mobile robot 100 determines that the mobile robot 100 is localized. If the mobile robot 100 determines that the highest matching score of the selected transformation candidate is not greater than the threshold, then the process proceeds to step 494, and the mobile robot 100 determines that the mobile robot 100 is mislocalized.

[0064] In embodiments, the threshold may be adjusted based on environment. The mobile robot 100 may obtain information on a current environment in which the mobile robot operates, and adjust the threshold based on the information on the current environment. For example, the threshold may be adjusted based on specific scenarios. Specifically, the threshold may be adjusted higher when the mobile robot 100 is in a cautious situation, e.g., boot-up or manual localization. The threshold may be adjusted lower when the mobile robot 100 is in a scenario where a less conservative approach is acceptable, e.g., rush hours.

[0065] FIG. 7 depicts a flowchart for detecting mislocalization when a user manually localizes a mobile robot, according to one or more embodiments shown and described herein.

[0066] In step 710, the mobile robot 100 receives a manual localization request from a user. For example, the user may input a manual localization request via the touch screen 110 of the mobile robot 100.

[0067] In step 720, in response to the manual localization request, the mobile robot 100 sets the current position of the mobile robot 100. The mobile robot 100 may scan environment using the LiDAR sensor 118 and calculate the pose of the mobile robot 100.

[0068] In step 730, the mobile robot 100 runs mislocalization detection based on the current robot pose and LiDAR data. The detailed process of mislocalization detection is described above with reference to FIG. 4.

[0069] In step 740, the mobile robot 100 determines whether the mobile robot 100 is localized or not. If it is determined that the mobile robot 100 is localized, the mobile robot 100 notifies the user that the robot is localized successfully in step 750. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is localized successfully so that the user can see the notification. As another example, the mobile robot 100 output audible sounds, via a speaker, that the robot is localized.

[0070] If it is determined that the mobile robot 100 is not localized, the mobile robot 100 notifies the user that the mobile robot 100 is mislocalized in step 760. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is mislocalized so that the user can see the notification. As another example, the mobile robot 100 output audible sounds, via a speaker, that the robot is mislocalized. Then, the user may retry localization of the user.

[0071] FIG. 8 depicts a flowchart for detecting mislocalization when a robot fails a navigation mission, according to one or more embodiments shown and described herein.

[0072] In step 810, the mobile robot 100 fails to navigate to a destination. Then, the mobile robot 100 sets the current position of the mobile robot 100. The mobile robot 100 may scan environment using the LiDAR sensor 118 and calculate the pose of the mobile robot 100.

[0073] In step 820, the mobile robot 100 runs mislocalization detection based on the current robot pose and LiDAR data. The detailed process of mislocalization detection is described above with reference to FIG. 4.

[0074] In step 830, the mobile robot 100 determines whether the mobile robot 100 is localized or not. If it is determined that the mobile robot 100 is localized, the mobile robot 100 notifies the user that the robot is localized successfully. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is localized successfully so that the user can see the notification.

[0075] If it is determined that the mobile robot 100 is not localized, the mobile robot 100 notifies the user that the mobile robot 100 is mislocalized in step 840. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is mislocalized so that the user can see the notification. Then, the user may retry localization of the user.

[0076] FIG. 9 depicts a flowchart for detecting mislocalization when a user boots up a mobile robot, according to one or more embodiments shown and described herein.

[0077] In step 910, the mobile robot 100 finishes hotting up. Then, the mobile robot 100 sets the current position of the mobile robot 100. The mobile robot 100 may scan environment using the FiDAR sensor 118 and calculate the pose of the mobile robot 100.

[0078] In step 920, the mobile robot 100 runs mislocalization detection based on the current robot pose and LiDAR data. The detailed process of mislocalization detection is described above with reference to FIG. 4.

[0079] In step 930, the mobile robot 100 determines whether the mobile robot 100 is localized or not. If it is determined that the mobile robot 100 is localized, the mobile robot 100 notifies the user that the robot is localized successfully. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is localized successfully so that the user can see the notification.

[0080] If it is determined that the mobile robot 100 is not localized, the mobile robot 100 notifies the user that the mobile robot 100 is mislocalized in step 940. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is mislocalized so that the user can see the notification. Then, the user may retry localization of the user.

[0081] FIG. 10 depicts a flowchart for detecting mislocalization when a mobile robot is in an idle state, according to one or more embodiments shown and described herein.

[0082] In step 1010, the mobile robot 100 determines whether it is time to mislocalization detection. When the mobile robot 100 is in an idle state, the mobile robot 100 is not currently on a navigation mission. In this scenario, users have the flexibility to configure the conditions that trigger mislocalization detection. In embodiments, a common configuration may involve two criteria: 1) the mobile robot 100 has been idle for a specified duration (X amount of time); and 2) the mobile robot 100 has traversed a certain distance (Y meters) since the last instance of mislocalization detection being triggered. The different criteria may be sued for triggering mislocalization detection. If it is determined that it is time to detect mislocalization, the mobilerobot 100 may scan environment using the LiDAR sensor 118 and calculate the pose of the mobile robot 100.

[0083] In step 1020, the mobile robot 100 runs mislocalization detection based on the current robot pose and LiDAR data. The detailed process of mislocalization detection is described above with reference to FIG. 4.

[0084] In step 1030, the mobile robot 100 determines whether the mobile robot 100 is localized or not. If it is determined that the mobile robot 100 is localized, the mobile robot 100 notifies the user that the robot is localized successfully. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is localized successfully so that the user can see the notification.

[0085] If it is determined that the mobile robot 100 is not localized, the mobile robot 100 notifies the user that the mobile robot 100 is mislocalized in step 1040. For example, the mobile robot 100 displays, on the screen 110, a notification that the robot is mislocalized so that the user can see the notification. Then, the user may retry localization of the user.

[0086] The system according to the present disclosure provides several technical advantages. First, the present system provides seamless integration with existing systems. The present system can seamlessly integrate with existing robotic setups, and provide immediate response to mislocalization without disrupting the normal operation of the robot. Activation of the mislocalization detection method is confined to instances when the mobile robot is idle or unable to navigate, preventing interference with routine operations and mitigating potential risks.

[0087] Second, the present system provides robustness in challenging environments. With LiDAR data preprocess by removing small clusters, the present system exhibits a high degree of robustness in dynamic environments, effectively filtering out small unmapped or dynamic objects through the LiDAR-based approach.

[0088] Third, the present system provides flexibility in definition mislocalization search window. The present system provides the flexibility for users to define “mislocalization” since the present system works with any shapes of search window.

[0089] The present system provides human-robot interaction enhancements. The present system enhances the interaction between humans and mobile robots by providing direct feedback when the mobile robot deviates from expected performance.

[0090] It is noted that the terms "substantially" and "about" may be utilized herein to represent the inherent degree of uncertainty that may be attributed to any quantitative comparison, value, measurement, or other representation. These terms are also utilized herein to represent the degree by which a quantitative representation may vary from a stated reference without resulting in a change in the basic function of the subject matter at issue.

[0091] It is noted that the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Although the terms first, second, and the like may be used herein to describe various elements, components, steps and / or operations, these terms are only used to distinguish one element, component, step or operation from another element, component, step, or operation.

[0092] The recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise.

[0093] While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be utilized in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.

Claims

CLAIMS1. A mobile robot comprising: one or more sensors configured to generate LiDAR data; one or more motors; a controller programmed to: generate a plurality of LiDAR scans of different angular rotations based on the LiDAR data and pose data of the mobile robot; generate transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area; compute a matching score for each of the transformation candidates; select a transformation candidate with a highest matching score among the transformation candidates; determine whether the highest matching score is greater than a threshold; and output a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

2. The mobile robot of claim 1, wherein the controller is further programmed to: preprocess the LiDAR data by removing clusters for unmapped obstacles from theLiDAR data prior to generating the plurality of LiDAR scans.

3. The mobile robot of claim 2, wherein the unmapped obstacles are movable obstacles including chairs or tables in the area.

4. The mobile robot of claim 1, wherein the controller is further programmed to: generate a plurality of maps of the area with different resolutions.

5. The mobile robot of claim 4, wherein the controller is further programmed to: determine whether a resolution of the first map of the area is highest among the different resolutions; in response to determining that the resolution of the first map of the area is not highest among the different resolutions, generate additional transformation candidates for theplurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a second map of the area with a resolution higher than the first map; select a transformation candidate with another highest matching score among the additional transformation candidates; determine whether the another highest matching score is greater than the threshold; and output a notification that the mobile robot is mislocalized in response to determining that the another highest matching score is not greater than the threshold.

6. The mobile robot of claim 1, wherein generating the transformation candidates for the plurality of LiDAR scans comprises: obtaining a user-defined search window of a non-square shape in the first map; determining whether there is an overlap between a map grid cell of corresponding LiDAR scan and the user-defined search window; computing the transformation candidate for corresponding LiDAR scan in response to determining that there is an overlap between the map grid cell of the corresponding LiDAR scan and the user-defined search window; and skipping computing the transformation candidate for corresponding LiDAR scan in response to determining that there is no overlap between the map grid cell of the corresponding LiDAR scan and the user-defined search window.

7. The mobile robot of claim 1, wherein the controller is further programmed to: obtain information on a current environment in which the mobile robot operates; and adjust the threshold based on the information on the current environment.

8. The mobile robot of claim 1, wherein the controller is further programmed to: determine that the mobile robot is localized in response to determining that the highest matching score is greater than the threshold.

9. The mobile robot of claim 1, wherein the controller is further programmed to: autonomously navigate the mobile robot by controlling the one or more motors based on the generated LiDAR data.

10. The mobile robot of claim 1, wherein the controller is further programmed to: determine whether it is time to trigger a mislocalization detection; and in response to determining that it is time to trigger the mislocation detection, generate the transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area.

11. The mobile robot of claim 10, wherein the controller is further programmed to: determine whether the mobile robot has been idle for a predetermined time; determine whether the mobile robot has traversed a predetermined distance since last instance of mislocalization detection being triggered; and determine that it is time to trigger the mislocalization detection in response to determining that the mobile robot has been idle for a predetermined time and that the mobile robot has traversed a predetermined distance since last instance of mislocalization detection being triggered.

12. A method for detecting mislocalization of a mobile robot, the method comprising: generating a plurality of LiDAR scans of different angular rotations based on LiDAR data and pose data of the mobile robot; generating transformation candidates for the plurality of LiDAR scans based on a scan matching between the plurality of LiDAR scans and a first map of an area; computing a matching score for each of the transformation candidates; selecting a transformation candidate with a highest matching score among the transformation candidates; determining whether the highest matching score is greater than a threshold; and outputting a notification that the mobile robot is mislocalized in response to determining that the highest matching score is not greater than the threshold.

13. The method of claim 12, further comprising: preprocessing the LiDAR data by removing clusters for unmapped obstacles from the LiDAR data prior to generating the plurality of LiDAR scans.

14. The method of claim 12, further comprising: generating a plurality of maps of the area with different resolutions.

15. The method of claim 14, further comprising: determining whether a resolution of the first map of the area is highest among the different resolutions; in response to determining that the resolution of the first map of the area is not highest among the different resolutions, generating additional transformation candidates for the plurality of TiDAR scans based on a scan matching between the plurality of TiDAR scans and a second map of the area with a resolution higher than the first map; selecting a transformation candidate with another highest matching score among the additional transformation candidates; determining whether the another highest matching score is greater than the threshold; and outputting a notification that the mobile robot is mislocalized in response to determining that the another highest matching score is not greater than the threshold.

16. The method of claim 12, wherein generating the transformation candidates for the plurality of TiDAR scans comprises: obtaining a user-defined search window of a non-square shape in the first map; determining whether there is an overlap between a map grid cell of corresponding TiDAR scan and the user-defined search window; computing the transformation candidate for corresponding TiDAR scan in response to determining that there is an overlap between the map grid cell of the corresponding TiDAR scan and the user-defined search window; and skipping computing the transformation candidate for corresponding TiDAR scan in response to determining that there is no overlap between the map grid cell of the corresponding TiDAR scan and the user-defined search window.

17. The method of claim 12, further comprising: obtaining information on a current environment in which the mobile robot operates; and adjusting the threshold based on the information on the current environment.

18. The method of claim 12, further comprising: determining that the mobile robot is localized in response to determining that the highest matching score is greater than the threshold.

19. The method of claim 12, further comprising: autonomously navigating the mobile robot by controlling one or more motors of the mobile robot based on the TiDAR data.

20. The method of claim 12, further comprising: determining whether the mobile robot has been idle for a predetermined time; determining whether the mobile robot has traversed a predetermined distance since last instance of mislocalization detection being triggered; determining that it is time to trigger the mislocalization detection in response to determining that the mobile robot has been idle for a predetermined time and that the mobile robot has traversed a predetermined distance since last instance of mislocalization detection being triggered; and in response to determining that it is time to trigger the mislocation detection, generating the transformation candidates for the plurality of TiDAR scans based on a scan matching between the plurality of TiDAR scans and a first map of an area.

Images