Autonomous mapping system with human feedback for autonomous mobile robots and method employed thereof

Abstract

Exemplary embodiments of the present disclosure are directed towards an autonomous mapping system designed for autonomous mobile robots, integrating human feedback to enhance exploration and mapping capabilities. The system comprises an autonomous mobile robot equipped with cameras and sensors for capturing environmental data, along with a processing device executing SLAM, frontier detection, and auto-exploration algorithms to autonomously map environments. A computing device, linked to both the robot and a cloud server via a network, features an auto mapping monitoring and controlling module. This module utilizes captured images to create a comprehensive top-down view, applying perspective transform and image stitching algorithms. It facilitates real-time user feedback and commands, enabling users to guide the robot, define virtual no-go zones, and adjust exploration velocities remotely. The processing device of the robot integrates these inputs to dynamically adapt its exploration strategies, including halting, recalibration, and adjusting navigational constraints based on real-time user interactions.

Description

“AUTONOMOUS MAPPING SYSTEM WITH HUMAN FEEDBACK FOR AUTONOMOUS MOBIEE ROBOTS AND METHOD EMPEOYED THEREOF”CROSS REFERENCEThis patent application claims priority benefit of Indian Provisional Patent Application No: 202341049350 entitled “AUTONOMOUS MAPPING SYSTEM WITH HUMAN FEEDBACK FOR AUTONOMOUS MOBILE ROBOTS AND METHOD EMPLOYED THEREOF”, filed on 21stJuly 2023. The entire contents of the patent application is hereby incorporated by reference herein in its entirety.TECHNICAL FIELD

[0001] The present disclosure generally relates to the field of autonomous mobile robotics. More particularly, the present disclosure relates to the autonomous mapping system with human feedback for autonomous mobile robots and method employed thereof. Additionally, the present disclosure operates autonomous mobile robots (AMR) autonomously while being monitored by a human operator providing real-time feedback and enabling the operator to guide the robot when necessary.BACKGROUND

[0002] In recent years, the robotics sector has seen remarkable advancements due to the increasing demands of human needs. Mobile robots with autonomous navigation capabilities have emerged as valuable assets, offering reduced dependency on human intervention and enabling them to perform tasks more efficiently. One critical aspect of autonomous navigation is path planning, which involves determining the optimal route for a robot to reach its desired state from the current position.

[0003] Traditionally, mapping environments using autonomous mobile robots (AMRs) have required human intervention, either through manual exploration or specialized equipment. However, this approach proves to be time-consuming, labor-intensive, and impractical for large-scale mapping tasks. Such limitations hinder the speed and accuracy of mapping results, preventing the seamless integration of AMRs in various scenarios. Moreover, the lack of remote monitoring capabilities poses further challenges in controlling and managing autonomous mobile robots. Human operators would need to be physically present forsupervision and constant oversight, restricting the deployment of robots in environments where human access is difficult, hazardous, or inconvenient.

[0004] Generally, the process of mapping environments with AMR involves human operators physically guiding the robot's movement. This could be achieved either by using a controller to steer the robot through desired areas or by manually pushing it while collecting data from various onboard sensors. The human supervision is crucial to ensure comprehensive coverage and accurate mapping of the environment. However, such manual control and supervision processes are not only cumbersome but also limit the potential of AMRs to operate efficiently and independently.

[0005] In the light of the aforementioned discussion, there exists a need for an advanced autonomous mapping technology that leverages human feedback and real-time monitoring to enable seamless mapping without constant human intervention.SUMMARY

[0006] The following presents a simplified summary of the disclosure in order to provide a basic understanding of the reader. This summary is not an extensive overview of the disclosure and it does not identify key / critical elements of the invention or delineate the scope of the invention. Its sole purpose is to present some concepts disclosed herein in a simplified form as a prelude to the more detailed description that is presented later.

[0007] An objective of the present disclosure is directed towards an autonomous mapping system with human feedback for autonomous mobile robots and method employed thereof.

[0008] Another objective of the present disclosure is directed towards providing an autonomous mapping system with human feedback for autonomous mobile robots, which allows robots to independently explore and map environments while being monitored remotely by a user / human operator.

[0009] Another objective of the present disclosure is directed towards a system that employs stitched camera images with transformed perspective from the robot's onboard or attached cameras to provide real-time feedback to a user / human operator, this feedback enables the operator to guide the autonomous mobile robot when necessary and avoid unnecessary regionsand setting virtual no-go zones for the robot.

[0010] Another objective of the present disclosure is directed towards a system that allows mobile robots to autonomously navigate and map environments without requiring constant human intervention, reducing dependency on manual control.

[0011] Another objective of the present disclosure is directed towards a system that provides advanced mapping algorithms, sensors, and onboard processing capabilities to ensure accurate and real-time mapping results, saving time and resources.

[0012] Another objective of the present disclosure is directed towards a system that provides remote monitoring it allows for centralized observation and control of robots' activities, status, and performance from a distance.

[0013] Another objective of the present disclosure is directed towards a system that allows the user to provide real-time feedback it enables prompt decision-making and supervision, improving operational control even without the physical presence of the robot.

[0014] Another objective of the present disclosure is directed towards a system that takes the decisions autonomously based on real-time mapping data and remote monitoring feedback allowing robots to adapt, avoid obstacles, and optimize navigation paths, enhancing autonomy and performance.

[0015] Another objective of the present disclosure is directed towards a system that enables user feedback with autonomous mapping and exploration, creating an efficient and adaptable system for mobile robots to perform mapping tasks accurately and independently.

[0016] Another objective of the present disclosure is directed towards a system that uses advanced greedy auto-mapping or greedy auto-SLAM technology with a user feedback system for efficient mapping of environments.

[0017] Another objective of the present disclosure is directed towards a system that allows the user to provide feedback and guide the robot when necessary, enhancing control and adaptability.

[0018] Another objective of the present disclosure is directed towards a system that allows the user to set real-time virtual no-go zones to prevent the robot from exploring specific areas.

[0019] Another objective of the present disclosure is directed towards a system that enables the robot to explore and map environments autonomously while allowing user feedback for improved mapping accuracy and efficiency.

[0020] According to an exemplary aspect of the present disclosure, an autonomous mobile robot comprises one or more cameras configured to capture environmental images, one or more sensors configured to collect environmental data, a processing device configured to process the captured images and collected data from the one or more cameras and sensors.

[0021] According to another exemplary aspect of the present disclosure, the processing device is configured to process the environmental data and execute a SLAM algorithm, a frontier detection algorithm, and a greedy auto-exploration algorithm to generate a map of the environment for allowing the autonomous mobile robot to explore independently.

[0022] According to another exemplary aspect of the present disclosure, a computing device communicatively connected to the autonomous mobile robot and a cloud server over a network, whereby the computing device comprises an automatic mobile robot auto mapping monitoring and controlling module configured to capture environmental images using the cameras on the autonomous mobile robot, wherein the automatic mobile robot auto mapping monitoring and controlling module configured to apply perspective transform algorithms and then stitch the captured images to generate a comprehensive 360-degree top-down view for a user.

[0023] According to another exemplary aspect of the present disclosure, the automatic mobile robot auto mapping monitoring and controlling module is configured to enable the user to provide feedback (real-time guidance) and set virtual no-go zones for the autonomous mobile robot, and transfer velocity commands issued by the user to the processing device of the autonomous mobile robot over the network, the processing device.

[0024] According to another exemplary aspect of the present disclosure, the processing device is configured to autonomously halt the autonomous mobile robot, cancel current explorationgoals, introduce delays for subsequent exploration goals based on user commands, and update and adjust virtual no-go zones based on the user's real-time feedback.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In the following, numerous specific details are set forth to provide a thorough description of various embodiments. Certain embodiments may be practiced without these specific details or with some variations in detail. In some instances, certain features are described in less detail so as not to obscure other aspects. The level of detail associated with each of the elements or features should not be construed to qualify the novelty or importance of one feature over the others.

[0026] FIG. 1 is an example diagram depicting an autonomous mapping system with human feedback for autonomous mobile robots, in accordance with one or more exemplary embodiments.

[0027] FIG. 2 is a block diagram depicting an embodiment of the autonomous mapping system with human feedback for autonomous mobile robots, in accordance with one or more exemplary embodiments.

[0028] FIG. 3 is a flowchart depicting an exemplary method for autonomous mapping for autonomous mobile robots, in accordance with one or more exemplary embodiments.

[0029] FIG. 4 is a flowchart depicting an exemplary method for selecting exploration goals from the free spaces and unknown areas using a unique greedy auto-exploration algorithm, in accordance with one or more exemplary embodiments.

[0030] FIG. 5 is a block diagram illustrating the details of a digital processing system in which various aspects of the present disclosure are operative by the execution of appropriate software instructions.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0031] It is to be understood that the present disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The present disclosure is capable of other embodiments and ofbeing practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

[0032] The use of “including”, “comprising” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms “a” and “an” herein do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced item. Further, the use of terms “first”, “second”, and “third”, and the like, herein do not denote any order, quantity, or importance, but rather are used to distinguish one element from another.

[0033] Referring to FIG. 1 is an example diagram 100 depicting an autonomous mapping system with human feedback for autonomous mobile robots, in accordance with one or more exemplary embodiments. The system 100 includes an autonomous mobile robot 102, a network 104, a cloud server 106, a computing device 108, cameras 110, a processing device 112, a switch 114, one or more Lidars 116, 3D cameras 118, and an automatic mobile robot auto mapping monitoring and controlling module 120.

[0034] The autonomous mobile robot 102 may be configured to independently explore and map diverse environments while enabling real-time monitoring and guidance from a human operator. The autonomous mobile robot 102 may include the switch 114. The switch 114 may include but is not limited to a manual switch and the like. The autonomous mobile robot 102 may include one or more cameras 110, the one or more Lidars 116, and the one or more 3D cameras 118. The switch 114, the one or more cameras 110, the one or more Lidars 116, and the one or more 3D cameras 118 may be electrically coupled to the processing device 112. The processing device 112 may include but not limited to, a microcontroller (for example ARM 7 or ARM 11), a Raspberry Pi, a microprocessor, a digital signal processor, a microcomputer, a field programmable gate array, a mini PC, a programmable logic device, a state machine or logic circuitry, and the like.

[0035] The computing device 108 may include a user device. The user may include but not limited to trained operators, and the like . The computing device 108 may include but not limited to, a personal digital assistant, smartphones, personal computers, a mobile station, computing tablets, a handheld device, a telephone, a mobile phone, a digital processing system, and soforth. The computing device 108 may include the automatic mobile robot auto mapping monitoring and controlling module 120. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to enable the user to monitor and interact with the autonomous mobile robot 102. The automatic mobile robot auto mapping monitoring and controlling module 120 may be communicatively connected with the processing device 112 and the cloud server 106 through a network 104. The network 104 may include but not limited to, an Internet of Things (loT network devices), an Ethernet, a wireless local area network (WLAN), or a wide area network (WAN), a Bluetooth low energy network, a ZigBee network, a WIFI communication network e.g., the wireless high speed internet, or a combination of networks, a cellular service such as a 4G (e.g., LTE, mobile WiMAX) or 5G cellular data service, a RFID module, an NFC module, wired cables, such as the world-wide-web based Internet, or other types of networks may include Transport Control Protocol / Intemet Protocol (TCP / IP) or device addresses (e.g. network-based MAC addresses, or those provided in a proprietary networking protocol, such as Modbus TCP, or by using appropriate data feeds to obtain data from various web services, including retrieving XML data from an HTTP address, then traversing the XML for a particular node) and so forth without limiting the scope of the present disclosure.

[0036] The automatic mobile robot auto mapping monitoring and controlling module 120 may be any suitable application downloaded from GOOGLE PLAY® (for Google Android devices), Apple Inc.'s APP STORE® (for Apple devices), or any other suitable database. The automatic mobile robot auto mapping monitoring and controlling module 120 may be a desktop application that runs on Windows or Linux or any other operating system and may be downloaded from a webpage or a CD / USB stick etc. In some embodiments, the automatic mobile robot auto mapping monitoring and controlling module 120 may be software, firmware, or hardware that is integrated into the computing device 108, and may present a web page to the user by way of a browser, wherein the webpage comprises a hyperlink may direct the user to uniform resource locator (URL).

[0037] The processing device 112 may be configured to operate the autonomous mobile robot 102 independently explore and map environments using the greedy auto-mapping technology which consists of a SLAM algorithm, frontier detection algorithm, and a unique greedy autoexploration algorithm. The processing device 112 may recognize free spaces, and unknown areas / frontiers using a frontier detection algorithm, which first recognizes the edges of thepresent map in real-time, and then recognizes the number of points that are free and unoccupied from those edges, the algorithm then checks the intersection of those points with no go zones that are set by the user monitoring the robot in real-time and then publishes the points that are not intersecting as free spaces, and unknown areas / frontiers. The processing device 112 may select exploration goals from the free spaces / unknown areas / frontiers using a unique greedy auto-exploration algorithm based on factors such as the distance from the frontier, size of the frontier, the angle between robot’s current heading direction and the direction of the path to reach that frontier, and the number of neighbouring frontiers.

[0038] The one or more cameras 110 may be connected to one or more camera extensions. The one or more cameras 110 may be configured to capture the environment images and the captured environment images may be transferred to the cloud server 106 and the computing device 108 using the network 104. The cloud server may include the automatic mobile robot auto mapping monitoring and controlling module 120. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to stitch the environment images either by using the homography matrix which is computed by feature matching or by using the pre-determined homography matrix, as all the camera locations on the robot are known. The computing device 108 may include the automatic mobile robot auto mapping monitoring and controlling module 120. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to stitch the environment images either by using the homography matrix which is computed by feature matching or by using the pre-determined homography matrix, as all the camera locations on the robot are known to the user. The automatic mobile robot auto mapping monitoring and controlling module 120 may apply the perspective transformation with pre-calibrated ROI and then stitch the captured image frames to provide a good 360-degree top view for the user monitoring the autonomous mobile robot 102. The autonomous mobile robot 102 may include the switch 114. The switch 114 may be configured to activate a cloud server mode and the cloud server mode enable the user to collaborate with the autonomous mobile robot 102 from any location via cloud servers 106. Once the switch 114 is pressed the processing device 112 may start sending and receiving data from the cloud server 106.

[0039] The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to enable the user to issue velocity commands while autonomous mapping. The automatic mobile robot auto mapping monitoring and controlling module 120 may beconfigured to transfer the velocity commands to the processing device 112 over network 104. The processing device 112 may be configured to autonomously halt for a brief delay and cancel the current exploration goal, introducing a delay for the subsequent exploration goal as long as velocity commands are being issued by the user. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to enable the user dynamically set virtual no-go zones in real time to prevent the autonomous mobile robot 102 from entering and exploring those areas. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to send a stop signal to the autonomous mobile robot 102 when the autonomous mobile robot 102 is trying to enter and explore a zone where it is not supposed to map. The autonomous mobile robot 102 then halts until the virtual no-go zone is set up by the user and continues the exploration process taking into account the virtual no-go zone. Markers are also shown on the map in the user interface to represent free spaces and unknown areas which ought to be explored by the autonomous mobile robot 102 this gives a good visualization for the user monitoring the autonomous mobile robot 102 to set up no-go zones quickly. In accordance with one or more exemplary embodiments of the present disclosure, when the user and autonomous mobile robot 102 are connected to the same Wi-Fi the communication between the user and the AMR is enabled using loT protocols.

[0040] Referring to FIG. 2 is a block diagram 200 depicting an embodiment of the autonomous mapping system with human feedback for autonomous mobile robots, in accordance with one or more exemplary embodiments. The processing device 112 may be configured to operate the autonomous mobile robot 102 independently explore and map environments using the greedy auto-mapping technology which consists of a SLAM algorithm, frontier detection algorithm, and a unique greedy auto-exploration algorithm. The processing device 112 may recognize free spaces, and unknown areas / frontiers using a frontier detection algorithm, which first recognizes the edges of the present map in real-time, and then recognizes the number of points that are free and unoccupied from those edges, the algorithm then checks the intersection of those points with no go zones that are set by the user monitoring the robot in real-time and then publishes the points that are not intersecting as free spaces, and unknown areas / frontiers. The processing device 112 may select exploration goals from the free spaces, unknown areas / frontiers using a unique greedy auto-exploration algorithm based on factors such as the distance from the frontier, size of the frontier, the angle between robot’s current heading direction and the direction of the path to reach that frontier, and the number of neighbouring frontiers.

[0041] The one or more cameras 110 may be connected to one or more camera extensions . The one or more cameras 110 may be configured to capture the environment images and the captured environment images may be transferred to the cloud server 106 and the computing device 108 using the network 104. The cloud server may include the automatic mobile robot auto mapping monitoring and controlling module 120. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to stitch the environment images either by using the homography matrix which is computed by feature matching or by using the pre-determined homography matrix, as all the camera locations on the robot are known. The computing device 108 may include the automatic mobile robot auto mapping monitoring and controlling module 120. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to stitch the environment images either by using the homography matrix which is computed by feature matching or by using the pre-determined homography matrix, as all the camera locations on the robot are known. The automatic mobile robot auto mapping monitoring and controlling module 120 may apply the perspective transformation with pre-calibrated ROI and then stitch the captured image frames to provide a good 360-degree top view for the user monitoring the autonomous mobile robot 102. The autonomous mobile robot 102 may include the switch 114. The switch 114 may be configured to activate a cloud server mode and the cloud server mode enable the user to collaborate with the autonomous mobile robot 102 from any location via cloud servers 106. Once the button 114 is pressed the processing device 112 may start sending and receiving data from the cloud server 106.

[0042] The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to enable the user to issue velocity commands. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to transfer the velocity commands to the processing device 112 over network 104. The processing device 112 may be configured to autonomously halt for a brief delay and cancels the current exploration goal, introduces a delay for the subsequent exploration goal as long as velocity commands are being issued by the user. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to enable the user dynamically set virtual no-go zones in real time to prevent the autonomous mobile robot 102 from entering and exploring those areas. The automatic mobile robot auto mapping monitoring and controlling module 120 may be configured to send a stop signal to the autonomous mobile robot 102 when the autonomousmobile robot 102 is trying to enter and explore a zone where it is not supposed to map. The autonomous mobile robot 102 then halts until the virtual no-go zone is set up by the user and continues the exploration process taking into account the virtual no-go zone. Markers are also shown on the map in the user interface to represent free spaces and unknown areas which ought to be explored by the autonomous mobile robot 102 this gives a good visualization for the user monitoring the autonomous mobile robot 102 to set up no-go zones quickly. In accordance with one or more exemplary embodiments of the present disclosure, when the user and autonomous mobile robot 102 are connected to the same Wi-Fi the communication between the user and the AMR is enabled using loT protocols.

[0043] FIG. 3 is a flowchart 300 depicting an exemplary method for autonomous mapping for autonomous mobile robots, in accordance with one or more exemplary embodiments. As an option, the method 300 is carried out in the context of the details of FIG.l, FIG.2. However, the method 300 is carried out in any desired environment. Further, the aforementioned definitions are equally applied to the description below.

[0044] The method commences at step 302, operating the autonomous mobile robot independently to explore and map environments using a greedy auto-mapping technology which consists of a SLAM algorithm, frontier detection algorithm, and a unique greedy autoexploration algorithm. Thereafter, at step 304, recognizing free spaces, and unknown areas / frontier of the present map in real-time by the processing device using a frontier detection algorithm. Thereafter, at step 306, recognizing a plurality of points that are free and unoccupied from the edges and checking the intersection of points with no-go zones that are set by a user by the processing device. Thereafter, at step 308, selecting exploration goals from the free spaces, unknown areas / frontiers by the processing device using a unique greedy autoexploration algorithm based on factors such as the distance from the frontier, size of the frontier, the angle between robot’s current heading direction and the direction of the path to reach that frontier, and the number of neighbouring frontiers. Thereafter, at step 310, capturing environmental images by the processing device using cameras and collecting data from its sensors while the autonomous Mobile Robot during the exploration. Thereafter, at step 312, transferring the captured images and sensor data to the computing device by the processing device. Thereafter, at step 314, applying perspective transformation algorithms and then stitching the received images by the computing device and generating a comprehensive 360- degree top view for the user. Thereafter, at step 316, enabling the user to provide real-timeguidance and set virtual no-go zones for the autonomous mobile robot by the computing device. Thereafter, at step 318, transferring the velocity commands issued by the user to the processing device over the network. Thereafter, at step 320, cancelling current exploration goals, introducing delays, and passing command velocities to the autonomous mobile robot by the processing device. Thereafter, at step 322, updating and adjusting virtual no-go zones based on the user's real-time feedback by the processing device.

[0045] In accordance with one or more exemplary embodiments, a pre -calibrated quadrilateral region of interest (ROI) covering several square feet is defined on the floor for each camera's frame or video stream to transform the perspective. This quadrilateral ROI is then stretched to form a perfect square or rectangular frame, creating a top-down view. To achieve this, a perspective transformation matrix is computed using the coordinates of the pre-calibrated quadrilateral ROI and a corresponding dummy or pseudo square / rectangular ROI. This transformation matrix is then applied using the warp perspective algorithm to stretch the original image to the desired square / rectangular shape, providing a top-down view of each frame or video stream from the camera. Once the frames or video streams have been perspective -transformed, they can be stitched together. This stitching can be done using a feature-matching algorithm such as SIFT or SURF. Alternatively, if the camera locations are known, a pre-determined homography matrix can be used to stitch these images together. The result is a comprehensive 360-degree top-down view for the user monitoring the autonomous mobile robot.

[0046] Referring to FIG. 4 is a flowchart 400 depicting an exemplary method for selecting exploration goals from the free spaces and unknown areas using a unique greedy autoexploration algorithm, in accordance with one or more exemplary embodiments. As an option, the method 400 is carried out in the context of the details of FIG.1, FIG.2, and FIG.3 However, the method 400 is carried out in any desired environment. Further, the aforementioned definitions are equally applied to the description below.

[0047] The method commences at step 402, generating frontiers from free spaces after checking the intersection of the points with no-go zones. Thereafter, at step 404, calculating the cost of each frontier by the processor using the frontier cost formula: FRONTIER COST=(distance scale x distance from that frontier)-(size scalexsize of t hat frontier)+(heading scale x angle between robot’s current heading direction and direction ofthe path to reach that frontier)+(neighbours scalexnumber of neighbouring frontiers).Thereafter, at step 406, selecting the frontier with the least frontier cost value as the exploration goal by the processor.

[0048] Referring to FIG. 5 is a block diagram 500 illustrating the details of a digital processing system 500 in which various aspects of the present disclosure are operative by execution of appropriate software instructions. The Digital processing system 500 may correspond to the computing device (or any other system in which the various features disclosed above can be implemented).

[0049] Digital processing system 500 may contain one or more processors such as a central processing unit (CPU) 510, random access memory (RAM) 520, secondary memory 530, graphics controller 560, display unit 570, network interface 580, and input interface 590. All the components except display unit 570 may communicate with each other over communication path 550, which may contain several buses as is well known in the relevant arts. The components of Figure 5 are described below in further detail.

[0050] CPU 510 may execute instructions stored in RAM 520 to provide several features of the present disclosure. CPU 510 may contain multiple processing units, with each processing unit potentially being designed for a specific task. Alternatively, CPU 510 may contain only a single general-purpose processing unit.

[0051] RAM 520 may receive instructions from secondary memory 530 using communication path 550. RAM 520 is shown currently containing software instructions, such as those used in threads and stacks, constituting shared environment 525 and / or user programs 526. Shared environment 525 includes operating systems, device drivers, virtual machines, etc., which provide a (common) run time environment for the execution of user programs 526.

[0052] Graphics controller 560 generates display signals (e.g., in RGB format) to display unit 570 based on data / instructions received from CPU 510. Display unit 570 contains a display screen to display the images defined by the display signals. Input interface 590 may correspond to a keyboard and a pointing device (e.g., touchpad, mouse) and may be used to provide inputs. Network interface 580 provides connectivity to a network (e.g., using Internet Protocol), and may be used to communicate with other systems (such as those shown in Figure 1) connectedto the network 104.

[0053] Secondary memory 530 may contain a hard drive 535, flash memory 536, and removable storage drive 537. Secondary memory 530 may store the data software instructions (e.g., for performing the actions noted above with respect to the Figures), which enable digital processing system 500 to provide several features in accordance with the present disclosure.

[0054] Some or all of the data and instructions may be provided on removable storage unit 540, and the data and instructions may be read and provided by removable storage drive 537 to CPU 510. Floppy drive, magnetic tape drive, CD-ROM drive, DVD Drive, Flash memory, removable memory chip (PCMCIA Card, EEPROM) are examples of such removable storage drive 537.

[0055] Removable storage unit 540 may be implemented using medium and storage format compatible with removable storage drive 537 such that removable storage drive 537 can read the data and instructions. Thus, removable storage unit 540 includes a computer readable (storage) medium having stored therein computer software and / or data. However, the computer (or machine, in general) readable medium can be in other forms (e.g., non-removable, random access, etc.).

[0056] In this document, the term "computer program product" is used to generally refer to removable storage unit 540 or hard disk installed in hard drive 535. These computer program products are means for providing software to digital processing system 500. CPU 510 may retrieve the software instructions, and execute the instructions to provide various features of the present disclosure described above.

[0057] The term “storage media / medium” as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or solid-state drives, such as storage memory 530. Volatile media includes dynamic memory, such as RAM 520. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0058] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus (communication path) 550. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0059] In accordance with one or more exemplary embodiments of the present disclosure, the processing device is configured to recognize free spaces and unknown areas using the frontier detection algorithm and to select exploration goals from the free spaces and unknown areas using the greedy auto-exploration algorithm.

[0060] In accordance with one or more exemplary embodiments of the present disclosure, the processing device is configured to recognize edges of the present map in real-time, determine free and unoccupied points from those edges, and check intersections of those points with no- go zones set by the user.

[0061] In accordance with one or more exemplary embodiments of the present disclosure, the automatic mobile robot auto mapping monitoring and controlling module is configured to dynamically update the 360-degree top-down view based on continuous data input from the autonomous mobile robot.

[0062] In accordance with one or more exemplary embodiments of the present disclosure, the automatic mobile robot auto mapping monitoring and controlling module is configured to display markers representing free spaces and unknown areas on the map interface, providing visualization for the user to set virtual no-go zones.

[0063] In accordance with one or more exemplary embodiments of the present disclosure, the automatic mobile robot auto mapping monitoring and controlling module is configured to apply a homography matrix computed by feature matching or a pre-determined homography matrix to stitch the captured images.

[0064] In accordance with one or more exemplary embodiments of the present disclosure, theprocessing device is configured to recognize a plurality of points that are free and unoccupied from the edges and publish the points that are not intersecting with no-go zones as free spaces and unknown areas / frontiers.

[0065] In accordance with one or more exemplary embodiments of the present disclosure, the automatic mobile robot auto mapping monitoring and controlling module is configured to enable collaboration from any location via cloud servers when the switch on the autonomous mobile robot is activated.

[0066] In accordance with one or more exemplary embodiments of the present disclosure, the processing device is configured to select exploration goals from the free spaces, unknown areas / frontiers using a unique greedy auto-exploration algorithm based on factors such as the distance from the frontier, size of the frontier, the angle between the robot’s current heading direction and the direction of the path to reach that frontier, and the number of neighboring frontiers.

[0067] In accordance with one or more exemplary embodiments of the present disclosure, capturing environmental images using one or more cameras on the autonomous mobile robot and collecting environmental data using one or more sensors on the autonomous mobile robot.

[0068] In accordance with one or more exemplary embodiments of the present disclosure, processing the captured images and collected data from the one or more cameras and sensors to generate a map of the environment by a processing device.

[0069] In accordance with one or more exemplary embodiments of the present disclosure, operating an autonomous mobile robot to independently explore and map environments using a SLAM algorithm, a frontier detection algorithm, and a greedy auto-exploration algorithm executed by the processing device enabled in the autonomous mobile robot.

[0070] In accordance with one or more exemplary embodiments of the present disclosure, transferring the captured images and collected data to a cloud server and a computing device over a network.

[0071] In accordance with one or more exemplary embodiments of the present disclosure,applying perspective transform algorithms and stitching the captured images to generate a comprehensive 360-degree top-down view on the computing device using an automatic mobile robot auto mapping monitoring and controlling module.

[0072] In accordance with one or more exemplary embodiments of the present disclosure, enabling a user to provide real-time feedback and set virtual no-go zones for the autonomous mobile robot through the automatic mobile robot auto mapping monitoring and controlling module on the computing device.

[0073] In accordance with one or more exemplary embodiments of the present disclosure, transferring velocity commands issued by the user to the processing device of the autonomous mobile robot over the network.

[0074] In accordance with one or more exemplary embodiments of the present disclosure, autonomously halting the autonomous mobile robot, cancelling current exploration goals, and introducing delays for subsequent exploration goals based on the user commands received by the processing device.

[0075] In accordance with one or more exemplary embodiments of the present disclosure, updating and adjusting virtual no-go zones based on the user's real-time feedback.

[0076] In accordance with one or more exemplary embodiments of the present disclosure, generating frontiers from free spaces after checking the intersection of the points with no-go zones.

[0077] In accordance with one or more exemplary embodiments of the present disclosure, selecting the frontier with the least frontier cost value as the exploration goal.

[0078] In accordance with one or more exemplary embodiments of the present disclosure, enabling the user to draw real-time virtual no-go zones on the map interface, the autonomous mobile robot will recognize and avoid during exploration.

[0079] In accordance with one or more exemplary embodiments of the present disclosure, dynamically updating the 360-degree top-down view on the computing device based oncontinuous data input from the autonomous mobile robot.

[0080] Reference throughout this specification to “one embodiment”, “an embodiment”, or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in an embodiment” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0081] Although the present disclosure has been described in terms of certain preferred embodiments and illustrations thereof, other embodiments and modifications to preferred embodiments may be possible that are within the principles of the invention. The above descriptions and figures are therefore to be regarded as illustrative and not restrictive.

[0082] Thus the scope of the present disclosure is defined by the appended claims and includes both combinations and sub-combinations of the various features described hereinabove as well as variations and modifications thereof, which would occur to persons skilled in the art upon reading the foregoing description.

Claims

CLAIMSWe Claim:

1. An autonomous mapping system with human feedback for autonomous mobile robots, comprising: an autonomous mobile robot comprises one or more cameras configured to capture environmental images, one or more sensors configured to collect environmental data, a processing device configured to process the captured images and collected data from the one or more cameras and sensors; the processing device configured to process the environmental data and execute a SLAM algorithm, a frontier detection algorithm, and a greedy autoexploration algorithm to generate a map of the environment for allowing the autonomous mobile robot to explore independently; a computing device communicatively connected to the autonomous mobile robot and a cloud server over a network, whereby the computing device comprises an automatic mobile robot auto mapping monitoring and controlling module configured to capture environmental images using the cameras on the autonomous mobile robot, wherein the automatic mobile robot auto mapping monitoring and controlling module configured to apply perspective transform algorithms and then stitch the captured images to generate a comprehensive 360-degree top-down view for a user; the automatic mobile robot auto mapping monitoring and controlling module configured to enable the user to provide feedback (real-time guidance) and set virtual no-go zones for the autonomous mobile robot and transfer velocity commands issued by the user to the processing device of the autonomous mobile robot over the network, the processing device; and the processing device configured to autonomously halt the autonomous mobile robot, cancel current exploration goals, introduce delays for subsequent exploration goals based on user commands, and update and adjust virtual no-go zones based on the user's real-time feedback.. The system as claimed in claim 1, wherein the processing device is configured to recognize free spaces and unknown areas using the frontier detection algorithm and to select exploration goals from the free spaces and unknown areas using the greedy autoexploration algorithm.

3. The system as claimed in claim 1, wherein the automatic mobile robot auto mapping monitoring and controlling module is configured to issue stop signals to the autonomous mobile robot when it approaches a virtual no-go zone, causing the robot to halt until a new exploration path is determined.

4. The system as claimed in claim 1, wherein the processing device is configured to recognize edges of the present map in real-time, determine free and unoccupied points from those edges, and check intersections of those points with no-go zones set by the user.

5. The system as claimed in claim 1, wherein the automatic mobile robot auto mapping monitoring and controlling module is configured to dynamically update the 360-degree top- down view based on continuous data input from the autonomous mobile robot.

6. The system as claimed in claim 1, wherein the automatic mobile robot auto mapping monitoring and controlling module is configured to display markers representing free spaces and unknown areas on the map interface, providing visualization for the user to set virtual no-go zones.

7. The system as claimed in claim 1, wherein the automatic mobile robot auto mapping monitoring and controlling module is configured to apply a homography matrix computed by feature matching or a pre-determined homography matrix to stitch the captured images.

8. The system as claimed in claim 1, wherein the processing device is configured to recognize a plurality of points that are free and unoccupied from the edges and publish the points that are not intersecting with no-go zones as free spaces and unknown areas / frontiers.

9. The system as claimed in claim 1, wherein the automatic mobile robot auto mapping monitoring and controlling module is configured to enable collaboration from any location via cloud servers when the switch on the autonomous mobile robot is activated.

10. The system as claimed in claim 1, wherein the processing device is configured to select exploration goals from the free spaces, unknown areas / frontiers using a unique greedy autoexploration algorithm based on factors such as the distance from the frontier, size of the frontier, the angle between the robot’s current heading direction and the direction of the path to reach that frontier, and the number of neighboring frontiers.

11. A method for autonomous mapping with human feedback for autonomous mobile robots, comprising: capturing environmental images using one or more cameras on the autonomous mobile robot and collecting environmental data using one or more sensors on the autonomous mobile robot; processing the captured images and collected data from the one or more cameras and sensors to generate a map of the environment by a processing device; operating an autonomous mobile robot to independently explore and map environments using a SLAM algorithm, a frontier detection algorithm, and a greedy auto-exploration algorithm executed by the processing device enabled in the autonomous mobile robot; transferring the captured images and collected data to a cloud server and a computing device over a network; applying perspective transform algorithms and stitching the captured images to generate a comprehensive 360-degree top-down view on the computing device using an automatic mobile robot auto mapping monitoring and controlling module;enabling a user to provide real-time feedback and set virtual no-go zones for the autonomous mobile robot through the automatic mobile robot auto mapping monitoring and controlling module on the computing device; transferring velocity commands issued by the user to the processing device of the autonomous mobile robot over the network; autonomously halting the autonomous mobile robot, canceling current exploration goals, and introducing delays for subsequent exploration goals based on the user commands received by the processing device; and updating and adjusting virtual no-go zones based on the user's real-time feedback.

12. The method of claim 11, further comprising a step of generating frontiers from free spaces after checking the intersection of the points with no-go zones.

13. The method of claim 11 , further comprising a step of selecting the frontier with the least frontier cost value as the exploration goal.

14. The method of claim 11, further comprising a step of enabling the user to draw realtime virtual no-go zones on the map interface, the autonomous mobile robot will recognize and avoid during exploration.

15. The method of claim 11, further comprising a step of dynamically updating the 360- degree top-down view on the computing device based on continuous data input from the autonomous mobile robot.

Images