Lifelong robot learning for mobile robots

Abstract

A method for improving a mobile robot configured to perform a task in an environment using an operational program is disclosed. Data is received, the data being recorded by the mobile robot using one or more sensors when the mobile robot is navigated in the environment to perform the task. A database and / or model associated with the environment is updated to include the recorded data therein. The operational procedure of the mobile robot can be modified based on the database and / or the model to generate a modified operational procedure for performing the task in the environment, the modified operational procedure improving performance of the mobile robot. Furthermore, based on the database and / or the model, suggestions for improving the performance of the mobile robot in performing the task in the environment can be determined and displayed to a user for consideration.

Description

Technical Field

[0001] The devices and methods disclosed in this document relate to mobile service robots, and more particularly to lifelong robot learning for mobile service robots. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be prior art by virtue of being included in this section.

[0003] In current robotic vacuum products, the associated smartphone app can show the cleaned areas within a home or the trajectory of the vacuum robot during its last cleaning task within the home. These robotic vacuums can also use images from an on-board camera to detect different objects. However, these robotic vacuums do not use the trajectory data from previous cleaning tasks to improve the performance or efficiency of future cleaning tasks, and do not associate the camera images with the trajectory or other sensor data.

[0004] In lifelong learning research for robotic vacuums, techniques have been developed to create a consistent semantic map according to different tasks and use the experience from previous tasks to improve navigation performance. In some methodologies, the surface area is semantically divided into cluttered and non-cluttered regions, and the coverage pattern is planned to sequentially cover these regions. However, these techniques improve performance by changing the behavior of the robotic vacuum while keeping the environment the same. Summary of the Invention

[0005] A method for operating a mobile robot system is described. The mobile robot system includes a mobile robot configured to perform tasks in an environment using an operating program. The method includes receiving first data that is recorded by the mobile robot at least in part using at least one sensor while the mobile robot navigates in the environment to perform a task. The method further includes updating at least one of a database and a model associated with the environment to include the first data. The method further includes at least one of the following: (1) modifying the operating program based on at least one of the database and the model to generate a modified operating program for performing a task in the environment, the modified operating program improving the performance of the mobile robot, and (2) determining a recommendation for improving the performance of the mobile robot when performing a task in the environment based on at least one of the database and the model, and causing the recommendation to be displayed to a user.

[0006] Another method for operating a mobile robot system is described. The mobile robot system includes a mobile robot configured to perform tasks in an environment using an operating program. The method includes receiving first data that is recorded by the mobile robot at least in part using at least one sensor while the mobile robot navigates in the environment to perform the task. The method further includes updating at least one of a database and a model associated with the environment to include the first data. The method further includes modifying the operating program based on at least one of the database and the model to generate a modified operating program for performing the task in the environment, the modified operating program improving the performance of the mobile robot. The method further includes providing the modified operating program to the mobile robot, the mobile robot being configured to use the modified operating program to perform the task in the environment again.

[0007] Another method for operating a mobile robot system is described. The mobile robot system includes a mobile robot configured to perform tasks in an environment using an operating program. The method includes receiving first data that is recorded by the mobile robot at least in part using at least one sensor while the mobile robot navigates in the environment to perform the task. The method further includes updating at least one of a database and a model associated with the environment to include the first data. The method further includes determining, based on at least one of the database and the model, a recommendation for improving the performance of the mobile robot when performing the task in the environment and causing the recommendation to be displayed to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing aspects and other features of the method are explained in the following description in conjunction with the accompanying drawings.

[0009] Figure 1 A mobile robot system is shown.

[0010] Figure 2A An exemplary embodiment of the mobile robot of the mobile robot system is shown.

[0011] Figure 2B An exemplary embodiment of the cloud backend of the mobile robot system is shown.

[0012] Figure 2C An exemplary embodiment of the personal electronic device of the mobile robot system is shown.

[0013] Figure 3 A flowchart of a method for improving the performance and efficiency of a mobile robot that navigates in an environment to perform a task is shown.

[0014] Figure 4AA simple histogram model is shown that indicates the relative proportion or amount of time spent by the mobile robot at various locations or regions within the environment 40 while performing a task.

[0015] Figure 4B - 4C A Gaussian mixture model is shown that indicates the relative proportion or amount of time spent by the mobile robot at various locations or regions within the environment while performing a task.

[0016] Figure 4D - 4F A mean shift clustering model is shown that indicates the relative proportion or amount of time spent by the mobile robot at various locations or regions within the environment while performing a task.

[0017] Figure 5 Shown is an exemplary corrected trajectory that is superimposed on an environmental map including "no-go" areas.

[0018] Figure 6 Shown are multiple possible base station localizations superimposed on the environmental map. Detailed Description

[0019] For the purpose of facilitating an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written description. It should be understood that this is not intended to limit the scope of the present disclosure. It should be further understood that the present disclosure includes any changes and modifications to the illustrated embodiments and includes additional applications of the principles of the present disclosure that would typically occur to those skilled in the art to which the present disclosure pertains.

[0020] Overview

[0021] Reference Figure 1 And Figure 2A - 2C , a mobile robot system 10 is described. The mobile robot system 10 includes at least one mobile robot 20 that is configured to perform tasks in an environment 40 using operating programs (e.g., onboard software and algorithms). In the embodiments described in detail herein, in particular, the mobile robot 20 is a robotic vacuum cleaner or a robotic mop that is configured to navigate in the environment 40 to clean the floor surface in the environment 40. However, those skilled in the art should appreciate that the systems and methods described herein can be applied to a variety of mobile robots that autonomously navigate in an environment to perform some tasks.

[0022] When the mobile robot 20 performs tasks in an environment, the mobile robot system 10 advantageously utilizes lifelong learning to improve the performance and efficiency of the mobile robot 20 over time. In particular, over time, the mobile robot 20 can perform tasks multiple times in the same environment 40. Each execution of a task may be referred to herein as a "mission". During each mission, the mobile robot 20 records and accumulates data including trajectory data, image data, and event data (e.g., mission failures or collision events), which are typically in the form of time series data with timestamps attached to the data. Due to different starting localizations, temporary clutter in the environment 40, people and pets moving around the environment 40, doors opening / closing in the environment 40, and sensing noise, even in the same environment, the trajectory data and image data will be different for each mission. By accumulating data from multiple missions, it is possible to identify and ignore outliers and erroneous data when appropriate.

[0023] The mobile robot system 10 extracts useful information from the accumulated data, which can be used to help improve the performance or efficiency of the mobile robot 20 during future missions in the environment 40. In some embodiments, for the purpose of abstracting the data to make it easier for the mobile robot 20 to process or for the user to understand, one or more models are applied to the accumulated data, or one or more models are trained based on the accumulated data. As new data is collected, the models are further refined.

[0024] Using the extracted information, the mobile robot system 10 can automatically modify the operating program of the mobile robot 20 for a specific environment 40, or suggest modifications to the operating program to the user, or suggest changes that the user can make to the environment 40, which will improve the performance or efficiency of the mobile robot 20 for future missions in the environment.

[0025] In some embodiments, based on the extracted information, the mobile robot system 10 suggests setting or automatically sets "no-go" areas within the environment 40 that the mobile robot 20 should avoid during future missions to improve its performance or efficiency. For example, in Figure 1 the illustration, the environment 40 includes various obstacles 42 (such as furniture) that cause certain areas to be inaccessible to the mobile robot or only partially accessible and may trap the mobile robot 20. These areas can be advantageously marked as "no-go" areas and only need to be avoided by the mobile robot 20.

[0026] In some embodiments, based on the extracted information, the mobile robot system 10 recommends adjusting or automatically adjusts the trajectory planning or area prioritization such that certain problematic areas within the environment 40 are visited last during future tasks to improve its performance or efficiency. For example, if the mobile robot 20 is a robotic vacuum cleaner or the like, there may be areas within the environment 40 that are difficult to clean, such as the entrance of the environment 40 with a dirty doormat or frequently cluttered with debris 44 or waste (such as shoes). It may be preferable for the robotic vacuum cleaner to first clean the remainder of the environment 40 before cleaning the entrance area to ensure that the robotic vacuum cleaner has sufficient power to clean the entire environment before spending too much time at the entrance area.

[0027] In some embodiments, based on the extracted information, the mobile robot system 10 identifies problematic objects and recommends modifying or automatically modifies the operating procedure of the mobile robot 20 such that the mobile robot 20 avoids that object or similar objects during future tasks to improve its performance or efficiency. For example, in Figure 1 the illustrated example, the environment 40 includes debris 44 on the floor of the environment. The debris 44 may be small objects such as shoes or wires that are easily pushed around or run over by the mobile robot 20, but which have the potential to cause the mobile robot 20 to become stuck (e.g., because a shoelace or wire becomes entangled with the wheels of the mobile robot 20).

[0028] In some embodiments, based on the extracted information, the mobile robot system 10 identifies objects within the environment 40 and recommends that the user remove the identified objects from the environment 40 before performing future tasks to improve the performance or efficiency of the mobile robot 20. For example, these identified objects may include debris 44 or other problematic objects within the environment 40.

[0029] In some embodiments, based on the extracted information, the mobile robot system 10 identifies a new location for the base station 46 of the mobile robot 20 and recommends that the user move the base station 46 to the new location to improve the performance or efficiency of the mobile robot 20. For example, the new location may reduce the average travel time or distance required for the mobile robot 20 to perform tasks within the environment 40.

[0030] Continuing to refer to Figure 1, in at least some embodiments, the mobile robot system 10 further includes a cloud backend 50. In particular, the cloud backend 50 can be configured to store the accumulated data collected by the mobile robot 20. Similarly, the cloud backend 50 can be configured to extract information from the accumulated data and determine modifications to be made to the operating procedures of the mobile robot 20 or recommendations to be presented to the user, as discussed above. However, it should be appreciated that these functions can also be performed locally using the mobile robot 20 itself.

[0031] In at least some embodiments, the mobile robot system 10 further includes a personal electronic device 70, such as a mobile phone or a tablet computer, through which the user can manage and operate the mobile robot 20. The recommendations discussed above can be presented to the user via an associated application on the personal electronic device. Such an application can also be used to operate and configure the mobile robot 20. However, it should be appreciated that these functions can also be performed locally using the mobile robot 20 itself (such as using a user interface integrated with the mobile robot 20).

[0032] Mobile Robot

[0033] Figure 2A An exemplary embodiment of the mobile robot 20 is shown. In the illustrated embodiment, for example, the mobile robot 20 includes a processor 22, a memory 24, one or more sensors 26, one or more actuators 28, and at least one network communication module 30. It will be appreciated that the illustrated embodiment of the mobile robot 20 is merely an exemplary embodiment, which represents only any one of the various ways or configurations of a mobile robot that autonomously navigates in an environment to perform some tasks.

[0034] The processor 22 is configured to execute instructions to operate the mobile robot 20 to implement the features, functions, characteristics, and / or the like as described herein. To this end, the processor 22 is operably connected to the memory 24, one or more sensors 26, and one or more actuators 28. The processor 22 generally includes one or more processors that can operate in parallel or otherwise cooperate with each other. Those skilled in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Thus, the processor 22 can include a system having a central processing unit, a graphics processing unit, multiple processing units, a dedicated circuit for implementing a function, programmable logic, or other processing systems.

[0035] The memory 24 is configured to store data and program instructions that, when executed by the processor 22, enable the mobile robot 20 to perform the various operations described herein. As will be appreciated by those skilled in the art, the memory 24 can be any type of device capable of storing information accessible by the processor 22, such as a memory card, ROM, RAM, hard disk drive, magnetic disk, flash memory, or any of the various other computer-readable media used as data storage devices. As discussed further below, the processor 22 is configured to execute the program instructions of the operating program 32 stored in the memory 24 to navigate in the environment 40 to perform tasks, such as cleaning the floor surface in the environment 40. In at least one embodiment, the operating program 32 utilizes an environmental map 34 that virtually represents the environment 40 to assist in performing the tasks.

[0036] One or more sensors 26 can include a variety of different sensors. In some embodiments, the sensors 26 include sensors configured to measure one or more accelerations, rotational rates, and / or orientations of the mobile robot 20. In one embodiment, the sensors 26 include one or more accelerometers configured to measure the linear acceleration of the mobile robot 20 along one or more axes (e.g., roll axis, pitch axis, and yaw axis), one or more gyroscopes configured to measure the rotational rate of the mobile robot 20 along one or more axes (e.g., roll axis, pitch axis, and yaw axis), and / or an inertial measurement unit configured to measure all of the above.

[0037] In some embodiments, the sensors 26 include one or more cameras configured to capture a plurality of images of the environment 40 as the mobile robot 20 navigates through the environment 40. The (one or more) cameras generate image frames of the environment 40, each image frame including a two-dimensional pixel array. Each pixel has a corresponding photometric information (intensity, color, and / or luminance). In some embodiments, the (one or more) cameras are configured to generate RGB-D images, where each pixel has corresponding photometric information and geometric information (depth and / or distance). In such an embodiment, for example, the (one or more) cameras can take the form of two RGB cameras configured to capture stereoscopic images (from which depth and / or distance information can be derived), or take the form of an RGB camera with an associated IR camera (configured to provide depth and / or distance information).

[0038] In some embodiments, sensor 26 includes an optical sensor (e.g., lidar or any other time-of-flight or structured-light based sensor) configured to emit measurement light (e.g., laser) and receive the measurement light after the measurement light is reflected throughout environment 40. In time-of-flight based embodiments, processor 22 is configured to calculate the time-of-flight and / or round-trip time of the measurement light. Based on the calculated time-of-flight and / or round-trip time, processor 22 can generate an environmental map 34, for example, in the form of a point cloud or a raster map. In structured-light based embodiments, processor 22 applies algorithms to extract the 3D contour of the surface (onto which the structured light is projected) (e.g., based on the fringe pattern generated on the surface).

[0039] One or more actuators 28 include at least motors of a locomotion system, which, for example, drive a set of wheels to move mobile robot 20 throughout environment 40 to perform tasks. Additionally, in some embodiments, one or more actuators 28 include at least a vacuum suction system configured to vacuum the floor surface as mobile robot 20 navigates through environment 40. Of course, mobile robot 20 performing other tasks in the environment can include different types of actuators 28 suitable for other tasks.

[0040] Network communication module 30 can include one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in a communication module to enable communication with various other devices, which at least include cloud backend 50 and / or personal electronic device 70. In particular, network communication module 30 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and / or a Wi-Fi router (not shown), and one or more cellular modems configured to communicate with a wireless telephone network. Additionally, network communication module 30 can include a module (not shown) configured to enable communication with personal electronic device 70.

[0041] Mobile robot 20 can also include a corresponding battery or other power source (not shown) configured to power various components within mobile robot 20. In one embodiment, the battery of mobile robot 20 is a rechargeable battery configured to be charged when mobile robot 20 is connected to base station 46, which is configured for use with mobile robot 20.

[0042] Cloud Back - end

[0043] Figure 2BAn exemplary embodiment of the cloud backend 50 is shown. In at least some embodiments, the cloud backend 50 achieves improvements in the performance and efficiency of the mobile robot 20 through lifelong learning. The cloud backend 50 includes one or more cloud servers 52 and one or more cloud storage devices 62. The cloud servers 52 may include servers configured to provide various functional services for the cloud storage backend. Depending on the features provided by the cloud backend 50, the servers include web servers or application servers, but at least include one or more database servers configured to manage the task data received from the mobile robot 20 and stored in the cloud storage device 62. For example, each cloud server 52 includes a processor 54, a memory 56, a user interface 58, and a network communication module 60. It will be appreciated that the illustrated embodiment of the cloud server 52 is only one exemplary embodiment of the cloud server 52 and merely represents any one of the various ways or configurations of a personal computer, server, or any other data processing system operating in the manner set forth herein.

[0044] The processor 54 is configured to execute instructions to operate the cloud server 52 to implement the features, functions, characteristics, and / or the like described herein. To this end, the processor 54 is operably connected to the memory 56, the user interface 58, and the network communication module 60. The processor 54 generally includes one or more processors that may operate in parallel or otherwise cooperate with each other. Those skilled in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Thus, the processor 302 may include a system having a central processing unit, a graphics processing unit, multiple processing units, dedicated circuitry for implementing functions, programmable logic, or other processing systems.

[0045] The cloud storage device 62 is configured to store the task data received from the mobile robot 20. The cloud storage device 62 may be any type of long-term non-volatile storage device capable of storing information accessible by the processor 54, such as a hard disk drive, a solid-state drive, or any one of the various other computer-readable storage media recognized by those skilled in the art. Similarly, the memory 56 is configured to store program instructions that, when executed by the processor 54, cause the cloud server 52 to perform the various operations described herein, including managing the task data stored in the cloud storage device 62. The memory 56 may be a combination of any type of device capable of storing information accessible by the processor 302, such as a memory card, ROM, RAM, a hard disk drive, a magnetic disk, a flash memory, or any one of the various other computer-readable media recognized by those skilled in the art.

[0046] The cloud server 52 can be operated locally or remotely by an administrator. For local operation convenience, the cloud server 52 can include a user interface 58. As will be recognized by those skilled in the art, in at least one embodiment, the user interface 58 can suitably include an LCD display or the like, a mouse or other pointing device, a keyboard or other keypad, speakers, and a microphone. Alternatively, in some embodiments, the administrator can operate the cloud server 52 remotely from another computing device that communicates with the cloud server 52 via a network communication module 60 and has a similar user interface.

[0047] The network communication module 60 provides an interface that allows communication with any of a variety of devices, which at least includes the mobile robot 20 and the personal electronic device 70. In particular, the network communication module 60 can include a local area network port that allows communication with any of a variety of local computers housed in the same or nearby facilities. Generally, the cloud server 52 communicates with remote computers via the Internet through an independent modem and / or router of the local area network. Alternatively, the network communication module 60 can further include a wide area network port that allows communication via the Internet. In one embodiment, the network communication module 60 is equipped with a Wi-Fi transceiver or other wireless communication device. Thus, it will be appreciated that communication with the cloud server 52 can occur via wired communication or via wireless communication. Any of a variety of known communication protocols can be used to accomplish the communication.

[0048] The cloud server 52 is configured to store and manage a task database for the mobile robot 20 in a secure manner and is configured to provide access to the task database via the mobile robot 20 and via the personal electronic device 70. The task database is stored on a cloud storage device 62 and can include task data 64 received from the mobile robot 20, a copy of the environmental map 34 used by the mobile robot 20, and one or more models 66 generated at least in part based on the task data 64. In addition, the memory 56 stores program instructions of a lifelong learning program 68 for improving the performance and efficiency of the mobile robot through lifelong learning using the task database stored on the cloud storage device 62.

[0049] Personal Electronic Device

[0050] Figure 2CAn exemplary embodiment of a personal electronic device 70 is shown. The personal electronic device 70 includes a processor 72, a memory 74, a display screen 76, and at least one network communication module 78. The processor 72 is configured to execute instructions to operate the personal electronic device 70 to implement the features, functions, characteristics, and / or the like described herein. To this end, the processor 72 is operably connected to the memory 74, the display screen 76, and the network communication module 78. The processor 72 generally includes one or more processors that can operate in parallel or otherwise cooperate with each other. Those skilled in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Thus, the processor 72 can include a system having a central processing unit, a graphics processing unit, multiple processing units, dedicated circuits for implementing functions, programmable logic, or other processing systems.

[0051] The memory 74 is configured to store data and program instructions that, when executed by the processor 72, enable the personal electronic device 70 to perform the various operations described herein. As those skilled in the art will recognize, the memory 74 can be any type of device capable of storing information accessible by the processor 72, such as a memory card, ROM, RAM, hard disk drive, magnetic disk, flash memory, or any of the various other computer-readable media used as data storage devices. In addition to this, the memory 74 stores a mobile robot application 80. As discussed in further detail below, the processor 72 is configured to execute the program instructions of the mobile robot application 80 to operate and configure the mobile robot 20.

[0052] The display screen 76 can include any of various known types of displays, such as an LCD screen or an OLED screen. In some embodiments, the display screen 76 can include a touch screen configured to receive touch input from a user. Alternatively, or additionally, the personal electronic device 70 can include additional user interfaces, such as buttons, switches, a keyboard or other keypad, a speaker, and a microphone.

[0053] The network communication module 78 can include one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in a communication module to enable communication with various other devices, which at least include the cloud backend 50 and / or the mobile robot 20. In particular, the network communication module 78 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and / or a Wi-Fi router (not shown), and one or more cellular modems configured to communicate with a wireless telephone network. In addition, the network communication module 78 can include a module (not shown) configured to enable communication with the mobile robot 20.

[0054] The personal electronic device 70 may also include a corresponding battery or other power source (not shown) configured to power the various components within the personal electronic device 70. In one embodiment, the battery of the personal electronic device 70 is a rechargeable battery configured to be charged when the personal electronic device 70 is connected to a battery charger, and the battery charger is configured for use with the mobile robot 20.

[0055] Method for Improving the Performance and Efficiency of a Mobile Robot

[0056] Various methods and processes for improving the performance and efficiency of a mobile robot through lifelong learning are described below. In these descriptions, statements that a method, processor, and / or system perform some task or function refer to a controller or processor (e.g., the processor 54 of the cloud server 52) executing programming instructions stored in a non-transitory computer-readable storage medium (e.g., the memory 56 of the cloud server 52) operably connected to the controller or processor to manipulate data or operate one or more components in the cloud server 52 to perform the task or function. Additionally, the steps of the methods may be executed in any feasible chronological order, regardless of the order shown in the figures or described in the steps.

[0057] Figure 3 A flowchart of a method 100 for improving the performance and efficiency of a mobile robot that navigates in an environment to perform tasks is shown. The method 100 advantageously utilizes lifelong learning to continuously adapt to a specific environment 40 (in which the mobile robot 20 is deployed) and improve the performance and efficiency of the mobile robot 20 in performing tasks in the specific environment 40.

[0058] Method 100 begins by recording task data (block 110) using the sensors of the mobile robot as the mobile robot navigates in the environment to perform tasks. In particular, the processor 22 of the mobile robot 20 executes the instructions of the operating program 32 to operate the sensors 26 and the actuators 28 to cause the mobile robot 20 to navigate in the environment 40 and perform tasks. As discussed above, in at least some embodiments, the mobile robot 20 is particularly a robotic vacuum cleaner or a robotic mop, and the task performed in the environment 40 is to clean the floor surface in the environment 40. However, those skilled in the art should appreciate that the methods described herein may be applicable to a wide variety of mobile robots that must autonomously navigate in an environment to perform some task.

[0059] When the mobile robot 20 navigates in the environment 40 to perform a task, the processor 22 receives multiple task data from the sensors 26 and writes the task data into the memory 24. In some embodiments, the processor 22 compresses the task data before writing it into the memory 24. As used herein, the term "task data" refers to any data recorded by the mobile robot 20 during the execution of a task, including at least: (1) raw sensor data from the sensors 26 of the mobile robot 20, (2) any data derived by the processor 22 of the mobile robot 20 from the raw sensor data, and (3) event data recorded by the mobile robot 20, where the event data describes events that occur during the task.

[0060] For example, the raw sensor data may include RGB images, IR images, or RGB-D images captured by the camera of the mobile robot 20. Additionally, for example, the raw sensor data may include accelerations, rotational rates, and / or orientations measured by the accelerometer, gyroscope, or inertial measurement unit of the mobile robot 20. If a certain position sensor is provided in the mobile robot 20, then the raw sensor data may include the position of the mobile robot 20 within the environment 40.

[0061] Conversely, the derived sensor data may also include the position of the mobile robot 20 within the environment 40, which is derived from images, accelerations, rotational rates, and / or orientations (e.g., using visual and / or visual-inertial odometry methods such as Simultaneous Localization and Mapping (SLAM)). Additionally, the derived sensor data may include time-of-flight and / or return-time derived from light measurements captured by the light sensors of the mobile robot 20.

[0062] Finally, the event data includes data such as: collision data (which identifies the time when the mobile robot 20 collides with something), stuck data (which identifies the time when the mobile robot 20 gets stuck), and task status data (which identifies the start time or end time of the task, or whether the task is successful or failed). The event data may include data derived from the raw sensor data (e.g., detecting an event based on the sensor data), and data recorded only by the operating program 32 of the mobile robot 20 (e.g., recording the time or location associated with the detected event).

[0063] Next, in at least some embodiments, the processor 22 operates the network communication module 30 to transmit the recorded task data to the cloud server 52 of the cloud backend 50. The processor 54 of the cloud server 52 receives the recorded task data via the network communication module 60 and writes the recorded task data to the cloud storage device 62. In some embodiments, the processor 54 compresses the task data before writing it to the cloud storage device 62. In at least one embodiment, the mobile robot 20 records task data throughout the task, and after the task is completed or otherwise terminated, the mobile robot 20 transmits the recorded task data to the cloud server 52 (e.g., when connected to the base station 46). However, in other embodiments, the mobile robot 20 may stream the recorded task data to the cloud server 52 in real time during the task or according to some other schedule.

[0064] Method 100 then loads an existing database or model associated with the environment, or establishes a new database or model (block 120). In particular, as will be discussed in more detail below, task data is recorded by the mobile robot 20 within the same environment 40 over multiple tasks and stored in the database, and / or used to generate and refine one or more models. The database and / or the models will be used to improve the performance or efficiency of the mobile robot 20 in performing tasks in the environment 40.

[0065] In the illustrated embodiment, the database and / or the model(s) are stored on the cloud storage device 62 of the cloud backend (i.e., as task data 64 and the model(s) 66, as Figure 2B shown). Throughout the specification, these methods will be generally described with reference to this embodiment. However, it should be appreciated that in some embodiments, the task data 64 and the model(s) 66 are stored locally in the memory 24 of the mobile robot 20, and any processing of the task data 64 and the model(s) 66 is performed locally by the processor 22 of the mobile robot 20.

[0066] After receiving the recorded task data from the mobile robot 20, the processor 54 identifies the task data 64 and / or the model(s) 66 in the cloud storage device 62 associated with the corresponding mobile robot 20 and / or the corresponding environment 40, and prepares to update or revise the task data 64 and / or the model(s) 66 based on the newly recorded task data. In some embodiments, the processor 54 identifies the task data 64 and / or the model(s) 66 associated with the corresponding mobile robot 20 and / or the corresponding environment 40 based on user input (e.g., via the user interface of the personal electronic device 70).

[0067] Alternatively, if there is no task data 64 and / or one or more models 66 associated with the respective mobile robot 20 and / or the respective environment 40 in the cloud storage device 62, the processor 54 generates a new database and / or one or more new models 66 for the respective mobile robot 20. Similarly, if the user selects to clear (e.g., via the user interface of the personal electronic device 70) the existing task data 64 and / or one or more models 66 associated with the respective mobile robot 20 and / or the respective environment 40, the processor 54 clears the existing task data 64 and / or one or more models 66 associated with the respective mobile robot 20 and / or the respective environment 40, and generates a new database and / or one or more new models 66 for the respective mobile robot 20 and / or the respective environment 40.

[0068] In at least some embodiments, the task data 64 captured by the mobile robot 20 is stored in the cloud storage device 62 (or the memory 24 of the mobile robot 20) in the form of a database. The database may store each type of task data in its original form or in a compressed form. For example, image data may be compressed before being stored in the database, while event data may be stored in its original form.

[0069] In some embodiments, at least some of the task data 64 is stored in the form of a model that is configured to abstract the task data such that it is easier for the mobile robot 20 to process or for the user to understand. For this purpose, any machine learning or statistical model may be employed, such as clustering (e.g., mean shift, k-means), function approximation (e.g., Gaussian mixture model, neural network), or simple statistical analysis (e.g., mean and variance, histogram).

[0070] In some embodiments, the model(s) 66 may include a machine learning model, such as a convolutional neural network, a recurrent neural network, or the like. As used herein, the term "machine learning model" refers to a system or collection of program instructions and / or data configured to implement an algorithm, process, or mathematical model (e.g., a neural network) that predicts or otherwise provides a desired output based on a given input. It will be appreciated that, generally, many or most of the parameters of a machine learning model are not explicitly programmed, and in the traditional sense, a machine learning model is not explicitly designed to follow specific programming rules in order to provide a desired output for a given input. Instead, a corpus of training data is provided to the machine learning model, and the machine learning model identifies or "learns" the implicit patterns and statistical relationships in the data from the corpus, and the implicit patterns and statistical relationships are generalized to make predictions or otherwise provide outputs relative to new data inputs. The results of the training process are embodied in a plurality of learned parameters, kernel weights, and / or filter values that are used in various components of the machine learning model to perform various operations or functions.

[0071] To this end, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) trains or generates a model that represents at least some of the task data 64 received from the mobile robot 20. In some embodiments, the task data used to train or generate the model is not permanently stored in the cloud storage device 62 (or the memory 24 of the mobile robot 20), but is only temporarily stored to train or generate the model and is subsequently deleted. In at least some embodiments, the model(s) 66 are stored in the cloud storage device 62 (or the memory 24 of the mobile robot 20) in the form of model parameters (e.g., model coefficients, machine learning model weights, etc.).

[0072] In some embodiments, a model is generated that summarizes one or more types of task data. In particular, in one embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) trains or generates a model that indicates the attributes of the task data. In one example, the model indicates a mathematical function that fits and thus estimates or summarizes one or more types of task data. In another example, the model indicates the frequency of one type of task data relative to a particular classification, organization, or categorization scheme (e.g., in the form of a histogram). In any case, the model not only provides a useful summary of the task data, but also reduces the storage requirements for the task database.

[0073] In one embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) trains or generates a model that is configured to receive multiple locations corresponding to one or more tasks of the mobile robot 20 within the environment 40 and output a relative proportion or amount of time spent performing the tasks at various locations or regions within the environment 40 (i.e., a heat map), and / or output a specific location or region within the environment 40 where the mobile robot spends the most amount of time performing the tasks (i.e., a hot spot). For example, such a model can include a simple histogram model, a mean shift clustering model, a Gaussian mixture model, or a similar model.

[0074] Figure 4A A simple histogram model 200 is shown, which indicates the relative proportion or amount of time spent by the mobile robot 20 at various locations or regions within the environment 40 when performing tasks. In the histogram model 200, darker shaded cells indicate regions within the environment 40 where the mobile robot 20 spends the most time, while lighter shaded cells indicate regions within the environment 40 where the mobile robot 20 spends relatively less time. In the illustration, the histogram model 200 includes a group of dark cells near the top, which corresponds to the area around the base station 46 within the environment 40. In addition, the histogram model 200 includes other smaller groups of dark cells, which generally correspond to the clutter 44 within the environment 40.

[0075] In some embodiments, other types of models can be similarly used to represent location information. Figure 4B A Gaussian mixture model 210 is shown, which similarly indicates the relative proportion or amount of time spent by the mobile robot 20 at various locations or regions within the environment 40 when performing tasks. The Gaussian mixture model 210 is formed based on four Gaussian distributions. Figure 4C A similar Gaussian mixture model 220 formed based on eight Gaussian distributions is shown. Figure 4D A mean shift clustering model 230 is shown, which similarly indicates the relative proportion or amount of time spent by the mobile robot 20 at various locations or regions within the environment 40 when performing tasks. The mean shift clustering model 230 has a bandwidth of 0.1 meter. Figure 4E A similar mean shift clustering model 240 with a bandwidth of 0.2 meter is shown. Figure 4F A similar mean shift clustering model 250 with a bandwidth of 0.4 meter is shown.

[0076] In one embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) trains or generates a model that is configured to receive multiple positions of the mobile robot 20 within the environment 40 corresponding to one or more tasks located from a specific base station, and output a distribution of the task time or the travel distance from the corresponding base station location. A number of such models can be determined with respect to a number of base station locations. In some embodiments, instead of receiving position data from the mobile robot 20, position data of tasks from different base station locations can be simulated. For example, such models can include simple histogram models, mean shift clustering models, Gaussian mixture models, or similar models.

[0077] In some embodiments, a model representing or predicting the relationship between two or more types of task data is generated. In particular, in one embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) trains or generates a model that is configured to: receive a first type of task data (e.g., raw sensor data) recorded by the mobile robot, and output a prediction result regarding a second type of task data (e.g., event data). In this way, the model represents the relationship between the first type of task data and the second type of task data. However, the corresponding raw data of the first type and the second type received from the mobile robot 20 and used to train or generate the model do not need to be permanently stored themselves. In this way, the model not only achieves prediction capabilities, but also reduces the storage requirements for the task database.

[0078] In one embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) trains or generates a model that is configured to predict that the mobile robot 20 fails to complete a task based on specific sensor data (e.g., image data, acceleration, rotation rate, and / or orientation, position data, or time-of-flight data). In a specific example, the model receives an image of the environment 40 captured by the mobile robot 20, and outputs a prediction result regarding whether the mobile robot 20 will fail to complete the task (e.g., an image including certain objects may indicate failure). For example, such models can include neural networks.

[0079] In another embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) trains or generates a model configured to predict that the mobile robot 20 is stuck based on specific sensor data (e.g., image data, acceleration, rotational rate, and / or orientation, position data, or time-of-flight data). In a particular example, the model receives an image of the environment 40 captured by the mobile robot 20 and outputs a prediction result as to whether the mobile robot 20 will be stuck (e.g., an image including certain objects may indicate being stuck). For example, such a model may include a neural network.

[0080] Returning to Figure 3 , method 100 then updates a database or model associated with the environment to include the recorded sensor data (block 130). In particular, as cited above, the task data recorded by the mobile robot 20 within the same environment 40 through multiple tasks is stored in the database and / or used to generate and refine one or more models. Thus, when new task data is recorded and received, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) updates the task data 64 and / or the model(s) 66 to include the newly recorded task data from the mobile robot 20.

[0081] In the case where the task data 64 is stored in the form of a database, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) adds the newly recorded task data from the mobile robot 20 to the existing task data stored in the database, thereby creating combined task data that includes both the newly recorded task data from the mobile robot 20 and the existing previously recorded task data. In some embodiments, at least some of the existing data is compressed, and the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) decompresses the existing data before combining the newly recorded task data with the existing data in the database. Once combined, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) compresses the combined data if appropriate.

[0082] In the case where one or more models 66 represent at least some of the task data received from the mobile robot 20, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the newly recorded task data as new training data or new source data to refine the corresponding one or more models 66. In at least some embodiments, during such a refinement process, based on the newly recorded task data, the model parameters (e.g., model coefficients, machine learning model weights, etc.) of one or more of the one or more models 66 are modified or updated. In this way, the one or more models 66 are corrected to reflect the newly recorded task data from the mobile robot 20 while retaining the previous learning based on the previously recorded task data from the mobile robot 20.

[0083] Method 100 then modifies the operating program of the mobile robot based on the database or model to improve the task performance of the mobile robot (block 140). In particular, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) extracts useful information from the task data 64 and / or one or more models 66, which include the task data equivalent to the value of several tasks collected by the mobile robot 20. For example, the extracted information may include, but is not limited to, task completion time, distribution of robot position data, objects observed during the task, position and speed before failure, and image data before failure.

[0084] Based on the extracted information, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines a modification to the operating program 32 of the mobile robot 20 that will improve the performance of the mobile robot 20 when performing tasks in the environment 40. For example, the improvement in performance may include: reducing the completion time of future tasks (e.g., a robotic vacuum cleaner or a robotic mop takes less time to clean the floor surface of the environment 40); providing a fully automated experience for the user by not getting stuck and not requiring manual intervention; improving the quality metrics of the tasks performed (e.g., a robotic vacuum cleaner or a robotic mop achieves a greater degree of cleanliness after completing the task); or performing the tasks only in a way that is more satisfactory to the user in certain aspects (e.g., less annoyance to the user).

[0085] Next, in some embodiments, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) operates a network communication module 60 (or a network communication module 30) to transmit information to the personal electronic device 70, the information describing a determined modification to the operating procedure 32. The processor 72 receives, via the network communication module 78, the information describing the determined modification to the operating procedure 32, and operates a display 78 to display to the user a recommendation (including the modification to the operating procedure). In some embodiments, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) further estimates and displays to the user the benefits of the recommended modification to the operating procedure 32.

[0086] In response to receiving an input from the user approving the recommendation, the processor 72 operates the network communication module 78 to transmit the approval to the cloud server 52 (or the mobile robot 20). In response to receiving the approval, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) modifies the operating procedure 32 of the mobile robot 20 based on the determined modification to generate a modified operating procedure.

[0087] In an alternative embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) automatically modifies the operating procedure 32 of the mobile robot 20 based on the determined modification to generate a modified operating procedure without approval from the user.

[0088] If the modified operating procedure is determined at the cloud backend 50, then the processor 54 of the cloud server 52 operates the network communication module 60 to transmit the modified operating procedure to the mobile robot 20. The processor 22 of the mobile robot 20 receives the modified operating procedure from the cloud server 52 via the network communication module 30, and stores the modified operating procedure in the memory 24.

[0089] In a first example, based on the task data 64 and / or one or more models 66, a determined modification to the operating procedure 32 sets a "no-go" area within the environment 40 that the mobile robot 20 should avoid during future tasks to improve its performance or efficiency. In particular, if the mobile robot 20 tends to spend a long time at a particular location, it can be inferred that there may be debris around that location. The mobile robot 20 can reduce the task time by automatically setting a "no-go" area or by suggesting to the user to set a "no-go" area.

[0090] To this end, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines, based on the task data 64 and / or one or more models 66, a "no-go" area within the environment 40 that the mobile robot 20 should not enter when performing tasks in the environment (i.e., the "no-go" area).

[0091] In at least one embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the model 66 to determine "no-go" areas, which identifies the relative proportion or amount of time spent performing tasks at various locations or regions within the environment 40 (i.e., heatmap), and / or identifies specific locations or regions within the environment 40 where the mobile robot spends the most amount of time performing tasks (i.e., hotspots). For example, these models can include statistical models (e.g., histogram model 200), function approximation models (e.g., Gaussian mixture models 210, 220), or clustering models (e.g., mean shift clustering models 230, 240, 250).

[0092] In another embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the model 66 to determine "no-go" areas, which identifies locations or regions within the environment 40 associated with task failure events or robot stuck events.

[0093] Alternatively, or additionally, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines a modified trajectory for performing a task in the environment 40 based on the task data 64 and / or the model(s) 66, which will improve the performance of the mobile robot 20 when performing tasks in the environment 40. The modified trajectory is determined based on the "no-go" areas and based on the environmental map 32. In some embodiments, a quick (sub-optimal) trajectory is determined locally by the processor 22 of the mobile robot 20, and then, the true optimal trajectory is determined by the processor 54 of the cloud server 50. Figure 5 Shown is an exemplary modified trajectory 300, which is superimposed on an environmental map 332 including "no-go" areas 302.

[0094] In a second example, based on the task data 64 and / or the model(s) 66, the determined modification to the operating procedure 32 adjusts the trajectory planning or area prioritization such that certain problematic areas within the environment 40 are visited last during future tasks to improve its performance or efficiency. If the mobile robot 20 tends to spend a long time at a specific location, it can be inferred that there may be debris around that location. The mobile robot 20 can reduce the task time by automatically accessing potential clutter areas later in the task or by suggesting that the user should access potential clutter areas later in the task.

[0095] To this end, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines, based on the task data 64 and / or the model(s) 66, an area within the environment 40 that the mobile robot 20 should enter later than other areas within the environment 40 when performing a task in the environment 40.

[0096] In at least one embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the model 66 to determine an area within the environment 40 that the mobile robot 20 should enter later than other areas within the environment 40, where the model 66 identifies the relative proportion or amount of time spent performing a task at various locations or areas within the environment 40 (i.e., a heatmap), and / or identifies specific locations or areas within the environment 40 where the mobile robot spends the most amount of time performing a task (i.e., hotspots). For example, these models can include statistical models (e.g., the histogram model 200), function approximation models (e.g., the Gaussian mixture models 210, 220), or clustering models (e.g., the mean shift clustering models 230, 240, 250).

[0097] Alternatively, or additionally, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines, based on the task data 64 and / or the model(s) 66, a modified trajectory for performing a task in the environment 40 that will improve the performance of the mobile robot 20 when performing a task in the environment 40. The modified trajectory is determined based on the previously determined area within the environment 40 that the mobile robot 20 should enter later than other areas within the environment 40, and based on the environment map 32. In some embodiments, a fast (sub-optimal) trajectory is locally determined by the processor 22 of the mobile robot 20, and then, the true optimal trajectory is determined by the processor 54 of the cloud server 50.

[0098] In a third example, based on the task data 64 and / or the model(s) 66, a determined modification to the operating program 32 identifies a problematic object and modifies the operating program 32 of the mobile robot 20 such that the mobile robot 20 avoids that object or similar objects during future tasks to improve its performance or efficiency. In particular, if failures often occur when an object with a specific appearance is in the image, then the mobile robot 20 can avoid objects with a similar appearance during future tasks. This can be achieved by using self-supervised learning with both image data and event (failure) data, where an inference model is trained to distinguish whether an object should be avoided.

[0099] To this end, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines the objects in the environment 40 that the mobile robot 20 should avoid when performing a task in the environment 40 based on the task data 64 and / or the model(s) 66.

[0100] In at least one embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the model 66 to determine the objects in the environment 40 that should be avoided. The model 66 receives an image (or other sensor data) of the environment 40 captured by the mobile robot 20 and outputs a prediction result regarding whether the mobile robot 20 will fail to complete the task (e.g., an image including certain objects may indicate failure). For example, such a model may include a neural network.

[0101] In another embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses the model 66 to determine the objects in the environment 40 that should be avoided. The model 66 receives an image (or other sensor data) of the environment 40 captured by the mobile robot 20 and outputs a prediction result regarding whether the mobile robot 20 will get stuck (e.g., an image including certain objects may indicate getting stuck). For example, such a model may include a neural network.

[0102] Returning to Figure 3 , the method 100 then displays suggestions (block 150) for improving the task performance of the mobile robot to the user based on a database or a model. In particular, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) extracts useful information from the task data 64 and / or the model(s) 66, which includes task data equivalent to the value of several tasks collected by the mobile robot 20. For example, the extracted information may include, but is not limited to, the task completion time, the distribution of the robot position data, the objects observed during the task, the position and speed before failure, and the image data before failure.

[0103] Based on the extracted information, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines the suggested actions (e.g., with respect to the environment 40) that the user can take, which will improve the performance of the mobile robot 20 when performing tasks in the environment 40.

[0104] In some embodiments, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) operates a network communication module 60 (or a network communication module 30) to transmit information to a personal electronic device 70 that describes a recommended action that a user can take. A processor 72 receives, via a network communication module 78, information that describes the recommended action and operates a display 78 to display a recommendation (including the recommended action) to the user. In some embodiments, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) further estimates and displays to the user the benefits of taking the recommended action.

[0105] In a first example, based on task data 64 and / or one or more models 66, the recommended action identifies an object within the environment 40 and recommends that the user remove the identified object from the environment 40 before performing a future task to improve the performance or efficiency of the mobile robot 20. In particular, if the mobile robot 20 tends to spend a long time at a particular location, it can be inferred that there may be debris around that location. The mobile robot 20 can reduce task time by suggesting that the user remove potential debris.

[0106] To this end, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) determines, based on task data 64 and / or one or more models 66, an object within the environment 40 that should be removed from the environment 40. The processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) operates a network communication module 60 (or a network communication module 30) to transmit information to the personal electronic device 70 that describes the object (e.g., a location within the environment or an image of the object) that should be removed from the environment 40. A processor 72 receives, via a network communication module 78, information that describes the object and operates a display 78 to display a recommendation that includes the information (describing the object that should be removed from the environment 40) to the user.

[0107] In one embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) uses a model 66 to determine the location of an object that should be removed from the environment 40, the model 66 identifying a relative proportion or amount of time spent performing a task at various locations or regions within the environment 40 (i.e., a heat map), and / or identifying a particular location or region within the environment 40 where the mobile robot spends the most amount of time performing a task (i.e., a hot spot). For example, these models can include statistical models (e.g., a histogram model 200), function approximation models (e.g., Gaussian mixture models 210, 220), or clustering models (e.g., mean shift clustering models 230, 240, 250).

[0108] In another embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) uses a model 66 to determine the positioning of an object that should be removed from the environment 40, and the model 66 identifies positions or areas within the environment 40 associated with task failure events or robot stuck events.

[0109] In yet another embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) uses a model 66 to determine an object that should be removed from the environment 40, and the model 66 receives images (or other sensor data) of the environment 40 captured by the mobile robot 20 and outputs a prediction result regarding whether the mobile robot 20 will fail to complete a task (e.g., an image including certain objects may indicate failure). For example, such a model may include a neural network.

[0110] In yet another embodiment, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) uses a model 66 to determine an object that should be removed from the environment 40, and the model 66 receives images (or other sensor data) of the environment 40 captured by the mobile robot 20 and outputs a prediction result regarding whether the mobile robot 20 will get stuck (e.g., an image including certain objects may indicate getting stuck). For example, such a model may include a neural network.

[0111] In a second example, based on the task data 64 and / or one or more models 66, a recommended action identifies a new positioning for the base station 46 of the mobile robot 20, and the user is recommended to move the base station 46 to the new positioning to improve the performance or efficiency of the mobile robot 20. For example, the new positioning may reduce the average travel time or distance required for the mobile robot 20 to perform a task in the environment 40.

[0112] To this end, a processor 54 of the cloud server 52 (or a processor 22 of the mobile robot 20) determines a recommended new positioning of the base station of the mobile robot 20 within the environment 40 based on the task data 64 and / or one or more models 66, and the recommended new positioning will improve the performance of the mobile robot when performing a task in the environment 40. Figure 6 Multiple possible base station positionings 400 are shown superimposed on the environment map 332. In the illustration, the darker shaded possible base station positionings 400 have longer predicted task times, while the lighter shaded possible base station positionings 400 have shorter predicted task times.

[0113] The processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) operates the network communication module 60 (or the network communication module 30) to transmit information to the personal electronic device 70, the information describing a new location for the base station. The processor 72 receives the information describing the new location via the network communication module 78, and operates the display 78 to display a suggestion (including the information describing the new location for the base station) to the user.

[0114] In one embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) determines and identifies a new location for the base station by comparing trajectories from different base (charging) station locations (based on previous or simulated tasks), the new location potentially resulting in a shorter task time or travel distance.

[0115] In another embodiment, the processor 54 of the cloud server 52 (or the processor 22 of the mobile robot 20) uses a model 66 to determine a new location for the base station, the model 66 receiving multiple positions of the mobile robot 20 within the environment 40 corresponding to one or more tasks from a particular base station location, and outputting a distribution of the task time or the travel distance to the corresponding base station location. For example, such a model can include a simple histogram model, a mean shift clustering model, a Gaussian mixture model, or similar models.

[0116] Method 100 then operates the mobile robot to perform a future task (block 160). In particular, as discussed above, if a modified operating program is generated, it is provided to the mobile robot 20 and stored in the memory 24. In a future task, the processor 22 of the mobile robot 20 executes the instructions of the modified operating program 32 to operate the sensors 26 and the actuators 28 to navigate the mobile robot 20 within the environment 40 and perform the task. As in previously performed tasks, when the mobile robot 20 navigates within the environment 40 to perform the task, the processor 22 receives multiple task data from the sensors 26 and writes the task data to the memory 24. Next, in at least some embodiments, the processor 22 operates the network communication module 30 to transmit the recorded task data to the cloud server 52 of the cloud backend 50.

[0117] Embodiments within the scope of the present disclosure may also include non-transitory computer-readable storage media or machine-readable media for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored thereon. Such non-transitory computer-readable storage media or machine-readable media can be any available medium that can be accessed by a general-purpose computer or a special-purpose computer. By way of example and not limitation, such non-transitory computer-readable storage media or machine-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium (which can be used to carry or store desired program code means in the form of computer-executable instructions or data structures). Combinations of the above should also be included within the scope of non-transitory computer-readable storage media or machine-readable media.

[0118] For example, computer-executable instructions include instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a particular function or a group of functions. Computer-executable instructions also include program modules executed by a computer in a stand-alone environment or a network environment. Generally, program modules include routines, programs, objects, components, and data structures, etc., which perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code means for performing the method steps disclosed herein. A particular sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in these steps.

[0119] Although the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, it should be regarded as illustrative and not restrictive. It should be understood that only the preferred embodiments are presented, and all changes, modifications, and additional applications within the spirit of the present disclosure are desired to be protected.

Claims

1. A method for operating a mobile robot system, the mobile robot system including a mobile robot configured to perform tasks in an environment using an operating program, the method comprising: Receiving first data that is recorded by the mobile robot at least in part using at least one sensor while the mobile robot navigates in the environment to perform the task; Updating at least one of a database and a model associated with the environment to include the first data; and At least one of the following: Modifying the operating program based on at least one of the database and the model to generate a modified operating program for performing the task in the environment, the modified operating program improving the performance of the mobile robot; And Determining a recommendation for improving the performance of the mobile robot when performing the task in the environment based on at least one of the database and the model, and causing the recommendation to be displayed to a user.

2. The method according to claim 1, further comprising: Providing the modified operating program to the mobile robot, the mobile robot being configured to use the modified operating program to perform the task in the environment again.

3. The method according to claim 1, wherein the first data includes at least one of the following: (i) position data indicating the position of the mobile robot in the environment when performing the task, (ii) an image of the environment when performing the task, and (iii) event data indicating an event that occurs when performing the task.

4. The method according to claim 1, updating the database further comprising: Adding the first data to the database that stores a plurality of second data that are recorded by the mobile robot during multiple executions of the task in the environment.

5. The method according to claim 4, adding the first data to the database further comprising: Decompressing the plurality of second data; Combining the first data with the decompressed plurality of second data to generate combined data; And Compressing the combined data.

6. The method according to claim 1, updating the model further comprising: Performing at least one of refining and training the model using the first data.

7. The method according to claim 6, wherein the model is configured to receive data recorded by the mobile robot and output a prediction result regarding an event.

8. The method according to claim 7, wherein the model is configured to output a prediction result regarding at least one of the following: (i) the mobile robot fails to complete the task, and (ii) the mobile robot gets stuck.

9. The method according to claim 6, wherein the model is configured to: receive data recorded by the mobile robot and identify a position in the environment where the mobile robot spends the most amount of time when performing the task.

10. The method according to claim 1, further comprising: Based on at least one of the database and the model, determining a modification to the operating procedure, the modification being configured to improve the performance of the mobile robot when performing the task in the environment.

11. The method according to claim 10, wherein modifying the operating procedure further comprises: Automatically modifying the operating procedure to incorporate the modification, thereby generating the modified operating procedure.

12. The method according to claim 10, wherein causing the suggestion to be displayed to the user further comprises: Causing the suggestion including the modification to the operating procedure to be displayed to the user, and in response to receiving an input from the user approving the suggestion, the operating procedure is modified to incorporate the modification.

13. The method according to claim 10, wherein determining the modification further comprises: Based on at least one of the database and the model, determining an area within the environment that the mobile robot should not enter when performing the task in the environment.

14. The method according to claim 10, wherein determining the modification further comprises: Based on at least one of the database and the model, determining an area within the environment that the mobile robot should enter later than other areas within the environment when performing the task in the environment.

15. The method according to claim 10, wherein determining the modification further comprises: Based on at least one of the database and the model, determining an object in the environment that the mobile robot should avoid when performing the task in the environment.

16. The method according to claim 10, wherein determining the modification further comprises: Based on at least one of the database and the model, determining a corrected trajectory for performing the task in the environment, the corrected trajectory being configured to improve the performance of the mobile robot when performing the task in the environment.

17. The method according to claim 1, wherein causing the suggestion to be displayed to the user further comprises: Based on at least one of the database and the model, determining a proposed new location of the base station of the mobile robot within the environment, the proposed new location being configured to improve the performance of the mobile robot when performing the task in the environment; and Causing the suggestion indicating the proposed new location of the base station to be displayed to the user.

18. The method according to claim 1, wherein causing the suggestion to be displayed to the user further comprises: Based on at least one of the database and the model, determining an object in the environment that should be removed from the environment; and Causing the suggestion indicating the object that should be removed from the environment to be displayed to the user.

19. A method for operating a mobile robot system, the mobile robot system including a mobile robot configured to perform a task in an environment using an operating procedure, the method comprising: Receive first data that is recorded by the mobile robot at least in part using at least one sensor as the mobile robot navigates in the environment to perform the task; Update at least one of a database and a model associated with the environment to include the first data; Modify the operating procedure based on the at least one of the database and the model to generate a modified operating procedure for performing the task in the environment, the modified operating procedure improving the performance of the mobile robot; And Provide the modified operating procedure to the mobile robot, the mobile robot being configured to use the modified operating procedure to perform the task in the environment again.

20. A method for operating a mobile robot system, the mobile robot system including a mobile robot configured to perform a task in an environment using an operating procedure, the method comprising: Receive first data that is recorded by the mobile robot at least in part using at least one sensor as the mobile robot navigates in the environment to perform the task; Update at least one of a database and a model associated with the environment to include the first data; and Based on the at least one of the database and the model, determine a recommendation for improving the performance of the mobile robot as it performs the task in the environment and cause the recommendation to be displayed to a user.

Images