Context and User Experience Based Robot Control

Abstract

The machine-readable medium can include instructions for presenting a user interface, which when executed by a processor can cause the processor to cause the user interface to display an image representing a mobile cleaning robot in communication with the processor. The processor can also display lines representing the mobile cleaning robot's environment, and the image and lines can together indicate the status of the mobile cleaning robot.

Description

[Technical Field]

[0001] Priority Application This application is a continuation of U.S. Patent Application No. 16 / 933,401, filed July 20, 2020, the contents of which are incorporated herein by reference in their entirety. [Background technology]

[0002] Autonomous mobile robots can move around in an environment and perform various categories of functions and operations, including, but not limited to, safety operations, infrastructure or maintenance operations, navigation or mapping operations, inventory control operations, and robot / human interaction operations. Some mobile robots, known as cleaning robots, can autonomously perform cleaning tasks in an environment, such as a home. Many types of cleaning robots are autonomous to some degree and in different ways. For example, a cleaning robot can perform cleaning tasks, picking up (e.g., sucking) debris from floor surfaces in its environment as it moves through it. Summary of the Invention [Means for solving the problem]

[0003] An autonomous mobile robot (hereinafter "mobile robot") may be controlled locally (e.g., via controls on the robot) or remotely (e.g., via a remote handheld device) to move around in an environment. A mobile application implemented on a handheld computing device (e.g., a mobile phone) or the like may be configured to display various information organized in a user interface display. A user can use the mobile application to view the robot's status and to select instructions for operating a processor to send commands to the robot, such as to start or end a mission or cleaning routine. In some situations, the current status of the robot may not be apparent based on the information presented by the user interface.

[0004] A device, system, or method of the present application can help address this problem by changing an image representing the robot and its environment based on the robot's status. For example, an isometric view of the robot and its environment (e.g., represented by lines) can indicate that the robot is in a ready status. Also, when the robot's status changes to a mission status (e.g., the robot is performing part of a mission), etc., a processor can instruct a display device to change the user interface to display a second or different image representing the robot or the robot's environment. Such a change in the represented image can help indicate to the user the change in the robot's status. Additionally, a textual status indicator can also be changed based on the robot's status to help indicate the robot's status to the user.

[0005] In one example, a machine-readable medium can include instructions for presenting a user interface, which, when executed by a processor, can cause the processor to display an image in the user interface representing a mobile cleaning robot in communication with the processor. The processor can also display lines representing the mobile cleaning robot's environment, and the image and lines can together indicate the status of the mobile cleaning robot.

[0006] This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application file with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0007] Various embodiments are illustrated by way of example in the accompanying drawing figures. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present subject matter. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a plan view of a mobile cleaning robot in an environment. [Figure 2A] FIG. 2 is a bottom view of the mobile cleaning robot. [Figure 2B] FIG. 1 is an isometric view of a mobile cleaning robot. [Figure 3] 3 is a cross-sectional view of the mobile cleaning robot taken along the indication 3-3 of FIG. 2A. [Figure 4A] 1 is a diagram illustrating a communication network in which a mobile cleaning robot operates and an example of data transmission in the network. [Figure 4B] FIG. 1 illustrates an exemplary process for exchanging information between a mobile robot and other devices in a communication network. [Figure 5] FIG. 1 is a block diagram of a robot scheduling and control system. [Figure 6A] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6B] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6C] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6D] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6E] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6F] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6G] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6H] FIG. 2 illustrates a user interface of a mobile device. [Figure 6I] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 6J] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 7] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 8] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 9] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 10] FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. [Figure 11]FIG. 1 illustrates a user interface of a mobile device displaying an image representing a mobile robot in a user's home. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 shows a plan view of a mobile cleaning robot 100 in an environment 40 according to at least one example of the present disclosure. The environment 40 may be a dwelling, such as a house or apartment building, and may include rooms 42a-42e. Obstacles, such as a bed 44, a table 46, and an island 48, may be located in the rooms 42 of the environment. Each of the rooms 42a-42e may have a floor surface 50a-50e, respectively. Some rooms, such as room 42d, may include a rug, such as rug 52. The floor surface 50 may be of one or more types, such as hardwood, ceramic, low-pile carpet, medium-pile carpet, long (or high) pile carpet, or stone.

[0010] The mobile cleaning robot 100 can be operated, such as by a user 60, to autonomously clean the environment 40 in a room-by-room manner. In some examples, the robot 100 can clean the floor surface 50a in one room, such as room 42a, before moving on to the next room, such as room 42d, to clean surfaces in room 42d. Different rooms can have different types of floor surfaces. For example, room 42e (which can be a kitchen) can have a hard floor surface, such as wood or ceramic tile, while room 42a (which can be a bedroom) can have a carpeted surface, such as medium-pile carpet. Other rooms, such as room 42d (which can be a dining room), can include multiple surfaces, where a rug 52 is positioned within room 42d.

[0011] During cleaning or navigating operations, the robot 100 can use data collected from various sensors (such as optical sensors) and calculations (such as odometry and obstacle detection) to develop a map of the environment 40. Once the map is created, the user 60 can define rooms or zones (such as room 42) within the map. The map can be presentable to the user 60 in a user interface, such as a mobile device, allowing the user 60 to, for example, direct or change cleaning preferences.

[0012] Also, during operation, the robot 100 can detect surface types within each of the rooms 42, which may be stored on the robot or other devices. The robot 100 can update the map (or data associated with the map) to include or describe the surface types of the floor surfaces 50a-50e of each room 42 of the environment. In some examples, the map can be updated to show different surface types, such as within each of the rooms 42.

[0013] In some examples, the user 60 can define a behavioral control zone 54, for example, using the methods and systems described herein. In response to the user 60 defining the behavioral control zone 54, the robot 100 can move toward the behavioral control zone 54 to confirm the selection. After confirmation, autonomous operation of the robot 100 can be initiated. In autonomous operation, the robot 100 can initiate an action in response to being at or near the behavioral control zone 54. For example, the user 60 can define an area of ​​the environment 40 that is prone to soiling to be the behavioral control zone 54. In response, the robot 100 can initiate a focused cleaning action in which the robot 100 performs focused cleaning of a portion of the floor surface 50d in the behavioral control zone 54.

[0014] Robot components FIG. 2A shows a bottom view of the mobile cleaning robot 100. FIG. 2B shows an isometric view of the mobile cleaning robot 100. FIG. 3 shows a cross-sectional view of the mobile cleaning robot 100 along designation 3-3 in FIG. 2A. FIG. 3 also shows the designations for bottom, top, front, and rear orientations. FIGS. 2A-3 are discussed together below.

[0015] The cleaning robot 100 may be an autonomous cleaning robot that autonomously traverses the floor surface 50 while collecting debris 75 from different portions of the floor surface 50. As depicted in FIGS. 2A and 3 , the robot 100 comprises a body 200 that is movable across the floor surface 50. The body 200 may include a plurality of articulated structures on which the movable components of the cleaning robot 100 are mounted. The articulated structures may include, for example, an outer housing for covering the internal components of the cleaning robot 100, a chassis on which the drive wheels 210 a and 210 b and the cleaning rollers 205 a and 205 b (of the cleaning assembly 205) are mounted, a shock absorber mounted to the outer housing, etc.

[0016] 2A , the main body 200 includes a front portion 202a having a substantially rectangular shape and a rear portion 202b having a substantially semicircular shape. The front portion 202a may be one-third to one-half of the front side of the cleaning robot 100, and the rear portion 202b may be one-half to two-thirds of the rear side of the cleaning robot 100. The front portion 202a may include two lateral sides 204a and 204b that are substantially perpendicular to a front side 206 of the front portion 202a.

[0017] 2A , the robot 100 may include a drive system including actuators 208a and 208b, such as motors, operable with drive wheels 210a and 210b. The actuators 208a and 208b may be mounted to the body 200 and operably coupled to the drive wheels 210a and 210b, which are rotatably mounted to the body 200. The drive wheels 210a and 210b support the body 200 above the floor surface 50. When driven, the actuators 208a and 208b can rotate the drive wheels 210a and 210b to move the robot 100 autonomously across the floor surface 50.

[0018] The controller 212 may be located within the housing and may be a programmable controller, such as a single or multiple board computers, a direct digital controller (DDC), or a programmable logic controller (PLC). In other examples, the controller 212 may be any computing device, such as a portable computer, for example, a smartphone, tablet, laptop, desktop computer, or any other computing device that includes a processor, storage, and communications capabilities. The storage device 213 may be one or more types of storage devices, such as volatile or non-volatile storage, read-only storage (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, other storage devices, and other storage media. The storage device 213 may be located within the housing 200 and may be coupled to and accessible by the controller 212.

[0019] The controller 212 can operate the actuators 208a and 208b to move the robot 100 around and autonomously navigate the floor surface 50 during cleaning operations. The actuators 208a and 208b are operable to drive the robot 100 in a forward drive direction (and optionally in a rearward drive direction) and to turn the robot 100. The robot 100 can include casters 211 that support the body 200 above the floor surface 50. The casters 211 can support the rear portion 202b of the body 200 above the floor surface 50, and the drive wheels 210a and 210b can support the front portion 202a of the body 200 above the floor surface 50.

[0020] As shown in FIG. 3 , the suction assembly 118 may be carried within the body 200 of the robot 100, for example, at the rear portion 202b of the body 200. The controller 212 may operate the suction assembly 118 to generate an airflow that flows through a gap near the cleaning roller 205, through the body 200, and out of the body 200. The suction assembly 118 may include, for example, an impeller that generates the airflow when rotated. The airflow and the cleaning roller 205 cooperate to draw debris 75 into the robot 100 when rotated. A cleaning bin 322 mounted on the body 200 may contain debris 75 drawn by the robot 100, and a filter in the body 200 separates the debris 75 from the airflow 120 before it enters the suction assembly 118 and is exhausted out of the body 200. In this regard, debris 75 is captured in both the cleaning bin 322 and the filter before the airflow 120 is exhausted from the body 200 .

[0021] As shown in Figures 2A and 3, the cleaning head 205 and cleaning rollers 205a and 205b may be positioned on the front portion 202a of the body 200 between the lateral sides 204a and 204b. The cleaning rollers 205a and 205b may be operatively coupled to actuators 214a and 214b, e.g., motors, respectively. The cleaning head 205 and cleaning rollers 205a and 205b may be positioned in front of the cleaning container 322, which is positioned in front of the suction assembly 118. In Figures 2A and 2B, the substantially rectangular shape of the front portion 202a of the body 200 may facilitate allowing the cleaning rollers 205a and 205b to be longer than rollers for a cleaning robot with, for example, a circular body.

[0022] The cleaning rollers 205a and 205b may be mounted on the housing 124 of the cleaning head 205, for example, indirectly or directly on the main body 200 of the robot 100. Specifically, the cleaning rollers 205a and 205b are mounted on the underside of the front portion 202a of the main body 200 when the underside faces the floor surface 50, so as to draw in debris 75 on the floor surface 50 during the cleaning operation.

[0023] The housing 124 of the cleaning head 205 may be mounted to the body 200 of the robot 100. In this regard, the cleaning rollers 205a and 205b are also mounted to the body 200 of the robot 100, e.g., indirectly mounted to the body 200 via the housing 124. Alternatively or additionally, the cleaning head 205 is a removable assembly of the robot 100, in which the housing 124 on which the cleaning rollers 205a and 205b are mounted is removably mounted to the body 200 of the robot 100. The housing 124 and cleaning rollers 205a and 205b are removable as a unit from the body 200 such that the cleaning head 205 can be easily replaced with a replacement cleaning head 205.

[0024] The control system may further include a sensor system with one or more electrical sensors, as described herein, that can generate signals indicative of the current location of the robot 100 and that can generate signals indicative of the location of the robot 100 as it moves along the floor surface 50.

[0025] Cliff sensors 134 (shown in FIG. 2A) may be positioned along a bottom portion of housing 200. Each of cliff sensors 134 may be an optical sensor that may be configured to detect the presence or absence of an object below it, such as floor surface 50. The cliff sensors 134 may be coupled to controller 212. Proximity sensors 136a, 136b, and 136c (shown in FIG. 2B) may be positioned near front face 126 of robot body 200. In other examples, proximity sensors 136 may be located on other portions of body 200. Each of proximity sensors 136a, 136b, 136c may comprise an optical sensor facing outward from body 200 and may be configured to generate a signal based on the presence or absence of an object in front of the optical sensor. The proximity sensors 136 may be coupled to controller.

[0026] The bumper 138 can be removably secured to the body 200 and can be movable relative to the body 200 while being mounted to the body 200. In some examples, the bumper 138 forms part of the body 200. For example, the bumper 138 can form the side surfaces 128, 130 and the front surface 126. Collision sensors 139a and 139b (collision sensors 139) can be coupled to the body 200 and can be engaged or configured to interact with the bumper 138. The bump sensor 139 can comprise a photoelectric sensor, a capacitive sensor, a switch, or other sensor capable of detecting contact between the robot 100, i.e., the bumper 138, and an object in the environment 40. The bump sensor 139 can be coupled to the controller 212.

[0027] The image capture device 140 may be a camera coupled to the body 200 and directed toward the upper portion 125. The image capture device 140 generates signals based on images of the environment 40 of the robot 100 as the robot 100 moves about the floor surface 50. The image capture device 140 may send signals to the controller 212 for use in navigation and cleaning routines.

[0028] The obstacle monitoring sensor 141 (shown in FIG. 2B ) can include an optical sensor facing outward from the side 128 of the body 200, configured to detect the presence or absence of an object adjacent the side 128 of the body 200. The obstacle monitoring sensor 141 can emit a light beam horizontally, in a direction perpendicular to the forward drive direction F of the robot 100 and perpendicular to the side 128 of the robot 100. In some examples, at least some of the proximity sensors 136 a, 136 b, 136 c and the obstacle monitoring sensor 141 each include an optical emitter and an optical detector. The optical emitter emits a light beam outward from the robot 100, e.g., horizontally outward, and the optical detector detects reflections of the light beam that reflect off objects near the robot 100. The robot 100 can determine the time of flight of the light beam, using, for example, the controller 212, thereby determining the distance between the optical detector and the object, and therefore the distance between the robot 100 and the object.

[0029] Side brushes 142 can be coupled to the underside of the robot 100 and can be coupled to motors 144 operable to rotate the side brushes 142 relative to the main body 200 of the robot 100. The side brushes 142 can be configured to draw in debris to move it toward the cleaning assembly 205 or to move it away from the edge of the environment 40. The motors 144 configured to drive the side brushes 142 can be in communication with the controller 112. The brushes 142 can rotate about a non-horizontal axis, for example, an axis that forms an angle between 75 and 90 degrees with the floor surface 50. The non-horizontal axis can, for example, form an angle between 75 and 90 degrees with the longitudinal axes 126a and 126b of the rollers 205a and 205b.

[0030] The brush 142 is a lateral brush that is offset laterally from the center of the robot 100 so that it extends beyond the perimeter of the body 200 of the robot 100. Similarly, the brush 142 can be offset forward from the center of the robot 100 so that it extends beyond the front face 126 of the body 200.

[0031] Robot movement In some example operations, the robot 100 may be propelled in a forward drive direction or a reverse drive direction. The robot 100 may also be propelled to turn in place or to turn while moving in a forward drive direction or a reverse drive direction.

[0032] When the controller 212 causes the robot 100 to perform a task, the controller 212 operates the motor 208 to drive the drive wheels 210 and propel the robot 100 along the floor surface 50. The controller 212 can also operate the motor 214 to rotate the rollers 205 a and 205 b, the motor 144 to rotate the brush 142, and the motor of the suction system 118 to generate airflow. The controller 212 can execute software stored in the memory 213 to operate the various motors of the robot 100 and thereby cause the robot 100 to perform various running and cleaning behaviors.

[0033] Various sensors on the robot 100 may be used to aid the robot in navigating and cleaning within the environment 40. For example, the cliff sensor 134 may detect obstacles such as drop-offs and cliffs below the portion of the robot 100 where the cliff sensor 134 is located. The cliff sensor 134 may send a signal to the controller 212 so that the controller 212 can reorient the robot 100 based on the signal from the cliff sensor 134.

[0034] The proximity sensors 136a, 136b, and 136c can generate signals based on the presence or absence of an object in front of the optical sensor. For example, detectable objects include obstacles such as furniture, walls, people, and other objects in the environment 40 of the robot 100. The proximity sensors 136 can send signals to the controller 212 so that the controller 212 can redirect the robot 100 based on the signals from the proximity sensors 136.

[0035] In some examples, a bump sensor 139a can be used to detect movement of the bumper 138 along the front-to-back axis of the robot 100. A bump sensor 139b can also be used to detect movement of the bumper 138 along one or more sides of the robot 100. The bump sensor 139 can send a signal to the controller 212 so that the controller 212 can redirect the robot 100 based on the signal from the bump sensor 139.

[0036] Image capture device 140 may be configured to generate signals based on images of environment 40 of robot 100 as robot 100 moves about floor surface 50. Image capture device 140 may transmit such signals to controller 212. Image capture device 140 may be tilted in an upward direction, for example, tilted between 30 and 80 degrees from floor surface 50 across which robot 100 moves. When tilted upward, image capture device 140 may capture images of wall surfaces of the environment such that features corresponding to objects in the wall surfaces can be used for localization.

[0037] In some examples, the obstacle monitoring sensors 141 can detect detectable objects, including obstacles such as furniture, walls, people, and other objects in the environment of the robot 100. In some implementations, the sensor system may include obstacle monitoring sensors along the side 130, which can detect the presence or absence of objects adjacent to the side 130. One or more of the obstacle monitoring sensors 141 can also serve as obstacle detection sensors, similar to the proximity sensors described herein.

[0038] The robot 100 may also include sensors for tracking the distance traveled by the robot 100. For example, the sensor system may include encoders associated with the motors 208 for the drive wheels 210, which may track the distance traveled by the robot 100. In some implementations, the sensors may include optical sensors that face downward toward the floor surface. The optical sensors may be positioned to direct light through the bottom of the robot 100 toward the floor surface 50. The optical sensors may detect reflections of light and may detect the distance traveled by the robot 100 based on changes in floor features as the robot 100 travels along the floor surface 50.

[0039] The controller 212 can use data collected by sensors in the sensor system to control the driving behavior of the robot 100 during a mission. For example, the controller 212 can use sensor data collected by obstacle detection sensors (cliff sensors 134, proximity sensors 136, and collision sensors 139) of the robot 100 to enable the robot 100 to avoid obstacles in the environment of the robot 100 during a mission.

[0040] The sensor data may also be used by the controller 212 for simultaneous localization and mapping (SLAM) techniques, in which the controller 212 extracts features of the environment represented by the sensor data and builds a map of the floor surface 50 of the environment. The sensor data collected by the image capture device 140 may also be used for techniques such as vision-based SLAM (VSLAM), in which the controller 212 extracts visual features corresponding to objects in the environment 40 and builds a map using these visual features. As the controller 212 directs the robot 100 around the floor surface 50 during a mission, the controller 212 can use SLAM techniques to detect features represented in the collected sensor data and compare them to previously stored features to determine the location of the robot 100 within the map. The map formed from the sensor data may indicate the location of passable and impassable spaces within the environment. For example, the location of obstacles may be indicated on the map as impassable spaces, and the location of open floor spaces may be indicated on the map as passable spaces.

[0041] Sensor data collected by any of the sensors may be stored in storage 213. Other data generated for SLAM techniques, including mapping data forming a map, may also be stored in storage 213. This data generated during a mission may include persistent data generated during a mission that can be used during further missions. In addition to storing software for causing robot 100 to perform its behaviors, storage 213 may store data resulting from processing of sensor data for access by controller 212. For example, a map may be usable and updatable by controller 212 of robot 100 from one mission to another to navigate robot 100 around floor surface 50.

[0042] Persistent data, including a persistent map, helps enable the robot 100 to efficiently clean the floor surface 50. For example, the map can enable the controller 212 to direct the robot 100 to open floor spaces and avoid impassable spaces. The controller 212 can also use the map to optimize the path taken during a mission to help plan the robot 100's travel through the environment 40 for subsequent missions.

[0043] In some implementations, the robot 100 may include a light directing system 137 positioned on the upper portion 125 of the robot 100. The light directing system 137 may include a light source positioned in a lid 147 that covers the waste bin 322 (shown in FIG. 2B ). The light source may be positioned to direct light toward the periphery of the lid 147. The light source is positioned so that every portion of a continuous ring on the upper portion 125 of the robot 100 can be illuminated. The continuous ring is positioned in a recessed portion of the upper portion 125 of the robot 100 so that the light source can illuminate the surface of the robot 100 when activated.

[0044] Network Example 4A is a diagram illustrating, by way of example and non-limiting example, a communications network 400 that enables networking between the mobile robot 100 and one or more other devices, such as a handheld device 404, a cloud computer system 406, or another autonomous robot 408 separate from the mobile robot 100. Using the communications network 410, the robot 100, the handheld device 404, the robot 408, and the cloud computer system 406 can communicate with each other to send and receive data. In some examples, the robot 100, the robot 408, or both the robot 100 and the robot 408 communicate with the handheld device 404 through the cloud computer system 406. Alternatively or additionally, the robot 100, the robot 408, or both the robot 100 and the robot 408 communicate directly with the handheld device 404. Various types and combinations of wireless networks (e.g., Bluetooth, radio frequency, optical-based, etc.) and network architectures (e.g., mesh networks) may be employed by the communications network 410.

[0045] In some examples, the mobile device 404 may be a remote device that can be tethered to the cloud computer system 406 and allow a user to provide input. The mobile device 404 may include user input elements such as, for example, one or more of a touchscreen display, buttons, a microphone, a mouse, a keyboard, or other devices that respond to input provided by a user. The mobile device 404 may include an immersive medium (e.g., virtual reality) with which the user interacts to provide input. In these examples, the mobile device 404 may be a virtual reality headset or a head-mounted display.

[0046] A user can provide inputs corresponding to commands to the mobile robot 100. In such cases, the handheld device 404 can send signals to the cloud computer system 406, causing the cloud computer system 406 to send the command signals to the mobile robot 100. In some implementations, the handheld device 404 can present an augmented reality image. In some implementations, the handheld device 404 can be a smartphone, a laptop computer, a tablet computer device, or other mobile device.

[0047] According to some examples discussed herein, the mobile device 404 may include a user interface configured to display a map of the robot's environment. A robot path, such as that specified by a range planner, may be displayed on the map. The interface can receive user commands to modify the environment map by, among other things, adding, removing, or modifying no-go zones in the environment, adding, removing, or modifying intensive cleaning zones in the environment (such as areas requiring repeated cleaning), restricting the robot's travel direction or pattern in portions of the environment, or adding or modifying a cleaning order.

[0048] In some examples, communication network 410 may include additional nodes. For example, a node of communication network 410 may include an additional robot. A node of communication network 410 may also include network-connected devices capable of generating information about environment 40. Such network-connected devices may include one or more sensors, such as acoustic sensors, image acquisition systems, or other sensors that generate signals, to detect characteristics of environment 40 from which features can be extracted. Network-connected devices may also include home cameras, smart sensors, and the like.

[0049] In the communications network 410, wireless links may utilize various communication schemes, protocols, etc., such as, for example, varieties of Bluetooth®, Wi-Fi®, Bluetooth® Low Energy 802.15.4, also known as BLE, Worldwide Interoperability for Microwave Access (WiMAX®), infrared channels, or satellite bands. In some examples, the wireless links may include any cellular network standard used to communicate between mobile devices, including, but not limited to, standards qualified as 1G, 2G, 3G, 4G, or 5G. When utilized, a network standard may qualify as one or more generations of a mobile telecommunications standard by implementing a specification or standard, such as, for example, a specification maintained by the International Telecommunications Union. For example, the 4G standard may correspond to the International Mobile Telecommunications Advanced (IMT-Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE®, LTE Advanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standards may use a variety of channel access methods, such as FDMA, TDMA, CDMA, or SDMA.

[0050] FIG. 4B illustrates an example process 401 for exchanging information between devices in a communication network 410 including a mobile robot 100, a cloud computer system 406, and a mobile device 404.

[0051] In some example operations, a cleaning mission can be initiated by pressing a button on the mobile robot 100 (or handheld device 404) or can be scheduled for a future time or date. The user can select a set of rooms to be cleaned during the cleaning mission or can command the robot to clean all rooms. The user can also select a set of cleaning parameters to be used in each room during the cleaning mission.

[0052] During a cleaning mission, the mobile robot 100 can track 410 its status, including its location, the operational events occurring during cleaning, and the time spent cleaning. The mobile robot 100 can send 412 the status data (e.g., one or more of location data, operational event data, and time data) to a cloud computer system 406, which can calculate 414 a time estimate for the area to be cleaned, such as by a processor 442. For example, a time estimate can be calculated for cleaning a room by averaging the actual cleaning times for that room collected during one or more previous cleaning missions for that room. The cloud computer system 406 can send 416 the time estimate data along with the robot status data to the mobile device 404. The mobile device 404 can present 418 the robot status data and the time estimate data on a display, such as by a processor 444. The robot status data and the time estimate data can be presented on the display of the mobile device 404 as an editable mission timeline in several graphical views or as a mapping interface.

[0053] The user 402 can view 420 robot status data and time estimate data on the display, and can input 422 new cleaning parameters and manipulate the order or identity of rooms to be cleaned. The user 402 can also remove rooms from the cleaning schedule of the mobile robot 100. In another example, the user 402 can select an edge cleaning mode or a deep cleaning mode for the rooms to be cleaned.

[0054] The display on the portable device 404 may be updated 424 when the user changes the cleaning parameters or cleaning schedule. For example, if the user changes the cleaning parameters from one cleaning to two cleanings, the system updates the estimated time to provide an estimate based on the new parameters. In this example, the estimate is approximately doubled. In another example, if the user removes a room from the cleaning schedule, the overall time estimate is reduced by the approximate time required to clean the removed room. Based on input from the user 402, the cloud computer system 406 can calculate 426 a time estimate for the area to be cleaned, which can then be transmitted 428 (e.g., by wireless transmission, applying a protocol, broadcasting a wireless transmission) back to the portable device 404 for display. Data regarding the calculated time estimate 426 can also be transmitted 446 to the robot's controller 430. Based on input from the user 402 received by the mobile robot's controller 430, the controller 430 can generate 432 command signals. The command signal commands the mobile robot 100 to perform 434 an action, such as a cleaning action. As the cleaning action is performed, the controller 430 can keep track 410 of the status of the mobile robot 100, including the location of the mobile robot 100, the motion events that occur during cleaning, and the time spent cleaning. In some examples, live updates regarding the status of the robot 100 can additionally be provided to the mobile device 404 or a home electronics system (e.g., an interactive speaker system) via push notifications.

[0055] Upon performing the action 434, the controller 430 may determine 436 whether the received command signal includes a command to complete the cleaning mission. If the command signal includes a command to complete the cleaning mission, the robot may be commanded to return to its dock and, upon return, may transmit information such that the cloud computer system 406 may generate 438 a mission summary that may be sent to and displayed 440 by the portable device 404. The mission summary may include a timeline or a map. A timeline may show which rooms were cleaned, the time spent cleaning each room, the motion events tracked in each room, etc. A map may show which rooms were cleaned, the motion events tracked in each room, the type of cleaning performed in each room (e.g., sweeping or mopping), etc.

[0056] In some examples, communication can occur directly between the mobile robot 100 and the portable device 404. For example, the portable device 404 can be used to send one or more commands through a wireless communication method, such as Bluetooth or Wi-Fi, to instruct the mobile robot 100 to perform a cleaning operation (task).

[0057] Operations for process 401, and other processes described herein, such as one or more steps discussed with respect to Figures 6A-6J, may be performed in a distributed manner. For example, the cloud computer system 406, the mobile robot 100, and the handheld device 404 may coordinate one or more of the operations with one another. Operations described as being performed by one of the cloud computer system 406, the mobile robot 100, and the handheld device 404 are, in some implementations, performed at least in part by two or all of the cloud computer system 406, the mobile robot 100, and the handheld device 404.

[0058] 5 is a diagram of a robot scheduling and control system 500 configured to generate and manage mission routines for a mobile robot (e.g., mobile robot 100) and control the mobile robot to perform missions according to the mission routines. Robot scheduling and control system 500, and methods using same, as described herein according to various embodiments, can be used to control one or more mobile robots of various types, such as a mobile cleaning robot, a mobile mopping robot, a lawn mowing robot, or a space monitoring robot.

[0059] The system 500 may include a sensor circuit 510, a user interface 520, a user action detection device 530, a controller circuit 540, and a storage circuit 550. The system 500 may be implemented in one or more of the mobile robot 100, the portable device 404, the autonomous robot 408, or the cloud computer system 406. In an example, part or all of the system 500 may be implemented in the mobile robot 100. Part or all of the system 500 may be implemented in a device separate from the mobile robot 100, such as the portable device 404 (e.g., a smartphone or other mobile computing device) communicatively coupled to the mobile robot 100. For example, the sensor circuit 510 and at least a portion of the user action detection device 530 may be included in the mobile robot 100. The user interface 520, the controller circuit 540, and the storage circuit 550 may be implemented in the portable device 404. The controller circuitry 540 can execute computer-readable instructions (e.g., a mobile application or "app") to perform the tasks of scheduling and generating instructions for controlling the mobile robot 100. The mobile device 404 can be communicatively linked to the mobile robot 100 through an intermediate system, such as a cloud computer system 406, as shown in Figures 4A and 4B. Alternatively, the mobile device 404 can communicate with the mobile robot 100 through a direct communication link without an intermediate system device.

[0060] The sensor circuitry 510 may include one or more sensors, including, for example, optical sensors, cliff sensors, proximity sensors, collision sensors, imaging sensors, or obstacle detection sensors, among others, such as those discussed above with reference to FIGS. 2A-2B and 3. Some of the sensors can sense obstacles (e.g., occupied areas such as walls), paths, and other open spaces in the environment. The sensor circuitry 510 may include an object detection device 512 configured to detect objects in the robot's environment and recognize the objects as, for example, doors, clutter, walls, dividers, furniture (e.g., tables, chairs, sofas, benches, beds, desks, dressers, cupboards, bookshelves, etc.), or furnishing elements (e.g., appliances, rugs, curtains, pictures, drapes, lamps, cookware, built-in ovens, ranges, dishwashers, etc.), among others.

[0061] The sensor circuitry 510 can detect spatial, contextual, or other semantic information about the detected object. Examples of semantic information can include the detected object's identity, location, physical attributes, or state, and spatial relationship to other objects, among other characteristics of the detected object. For example, for a detected table, the sensor circuitry 510 can identify a room or area in the environment (e.g., a kitchen) that houses the table. The spatial, contextual, or other semantic information can be associated with the object to create a semantic object (e.g., a kitchen table), which can be used to create object-based cleaning task routines, as discussed below.

[0062] The user interface 520, which may be implemented on a portable computing device such as the mobile device 404, includes a user input 522 and a display 524. A user may use the user input 522 to create a mission routine 523. The mission routine 523 may include data representing an editable schedule for at least one mobile robot to perform one or more tasks. The editable schedule may include a time or order for performing cleaning tasks. In an example, the editable schedule may be represented by a timeline of tasks. The editable schedule may optionally include a time estimate for completing a mission or a time estimate for completing specific tasks in a mission. The user interface 520 may include user interface controls that allow a user to create or modify the mission routine 523. In some examples, the user input 522 may be configured to receive a user's voice commands to create or modify the mission routine. The portable computing device may include a speech recognition and dictation module for converting the user's voice commands into device-readable commands understood by the controller circuitry 540 to create or modify the mission routine.

[0063] The display device 524 can present, among other information, information about the mission routines 523, the progress of the mission routines being executed, information about the robots in the home and their operational status, and maps with semantically annotated objects. The display device 524 can also display user interface controls that allow a user to manipulate the display of information, schedule and manage mission routines, and control the robots to perform missions. Examples of user interfaces 520 are discussed below, such as with reference to Figures 6A-6J.

[0064] Controller circuit 540, an example of controller 212, can interpret mission routines 523, such as those provided by a user via user interface 520, and can control at least one mobile robot to perform missions in accordance with mission routines 523. Controller circuit 540 can create and maintain maps including semantically annotated objects and can use such maps to schedule missions and navigate the robot around an environment. In an example, controller circuit 540 can be included in a portable computing device, such as handheld device 404. Alternatively, controller circuit 540 can be at least partially included in a mobile robot, such as mobile robot 100. Controller circuit 540 can be implemented as part of a microprocessor circuit, which can be a dedicated processor such as a digital signal processor, an application-specific integrated circuit (ASIC), a microprocessor, or other type of processor for processing information, including physical activity information. Alternatively, the microprocessor circuit can be a processor capable of receiving and executing sets of instructions that implement the functions, methods, or techniques described herein.

[0065] The controller circuit 540 may comprise a circuit set that includes one or more other circuits or subcircuits, such as a mission controller 542, a map management circuit 546, and a cruise controller 548. These circuits or modules may implement the functions, methods, or techniques described herein, alone or in combination. In an example, the hardware of the circuit set may be invariably designed (e.g., hard-wired) to perform specific operations. In an example, the hardware of the circuit set may include variable-connection physical components (e.g., execution units, transistors, simplex circuits, etc.) that include computer-readable media that are physically changed (e.g., magnetically, electrically movable locations of invariable mass particles, etc.) to encode instructions for specific operations. When connected to the physical components, the underlying electrical properties of the hardware components are changed, for example, from insulator to conductor or vice versa. When operating, the instructions can cause the embedded hardware (e.g., execution units or loading mechanisms) to create, via variable connections, elements of the circuitry configured in the hardware to perform a portion of a specific operation. The computer-readable medium may also be communicatively coupled to other components of the circuit set when the device is operating. In some instances, any of the physical components may be used by more than one member of more than one circuit set. For example, under operation, an execution unit may be used by a first circuit in a first circuit set at one time and then, at a different time, by a second circuit in the first circuit set or by a third circuit in the second circuit set.

[0066] The mission controller 542 can receive a mission routine 523 from the user interface 520. As discussed above, the mission routine 523 includes data representing an editable schedule, including at least one of the time and order, for performing one or more tasks. In some examples, the mission routine 523 can represent a personal mode for executing the mission routine. For a mobile cleaning robot, examples of personal cleaning modes may include "standard cleaning," "deep cleaning," "quick cleaning," "spot cleaning," or "edge and corner cleaning." Each of these mission routines defines a respective room or floor surface area to be cleaned and an associated cleaning pattern. For example, a standard cleaning routine may include a larger range of areas in the user's residential environment, such as all rooms, than a quick cleaning routine. A deep cleaning routine may include one or more repeating cleaning zones, including multiple passes over the same area, a longer cleaning time period, or the application of a greater cleaning power. The personal cleaning mode may be manually created or changed by a user, such as via the user interface 520, or may be automatically activated by an event, such as detected by the user action detection device 530. In an example, the controller circuit 540 may communicate with a light source to automatically adjust the lighting in a target room or floor area, and cause the mobile cleaning robot to navigate to clean the target room or floor area with the adjusted lighting. For example, for a "deep cleaning routine," the controller circuit 540 may automatically activate a light switch to increase the lighting in the room or area to be cleaned, and the navigation controller 548 may cause the mobile cleaning robot to navigate to the illuminated room or area to perform cleaning according to the deep cleaning routine.

[0067] A mission routine 523, such as a personal cleaning mode (e.g., a deep cleaning mode), may include one or more tasks characterized by spatial or contextual information for each of the objects in the environment, or one or more tasks characterized by a user's experience, such as user behavior or routine activities associated with use of a room or area in the environment. The mission controller 542 may include a mission interpreter 543 to extract information from the mission routine 523 about the location for the mission (e.g., the room or area to be cleaned relative to the objects detected in the environment), the time and / or order for performing the mission relative to the user's experience, or the manner in which to clean the identified room or area. The mission interpreter 543 may interpret the mission routine using information about the objects and semantics of the objects detected by the object detector 512, user behavior detected by the user behavior detector 530, or a map generated and maintained by the map manager circuit 546.

[0068] Compared to location- and map-based missions, context- and user-experience-based mission routines can be designed to add user-personalized content. Because context- and user-experience-based mission routines are more consistent with the natural language description of the mission, they enable more intuitive communication between the user and the robot, allowing the mobile robot to execute the mission in a manner that is generally understandable between the user and the robot. Furthermore, including contextual information and user experience in the mission description enhances the content of the mission routine, adds more intelligence to the robot's behavior, and enhances the user experience of personalized control of the mobile robot. Examples of context- and user-experience-based mission routines and the interpretation of the mission routines by the mission interpreter 543 are discussed below.

[0069] The mission manager 544 can monitor the progress of a mission. In an example, the mission manager 544 can generate a mission status report indicating completed tasks (e.g., rooms cleaned) and remaining tasks to be performed (e.g., rooms to be cleaned according to a mission routine). In an example, the mission status report can include an estimate of the time to complete the mission, the elapsed time for the mission, the time remaining for the mission, an estimate of the time to complete tasks in the mission, the elapsed time for tasks in the mission, or the remaining time for tasks in the mission. The time estimate to complete the entire mission or tasks in the mission can be based on characteristics of the environment, such as the approximate square footage or area measurement of the space to be cleaned, the number of rooms to be traversed, the dirt status, or the soiling of one or more target areas, such as detected by the sensor circuitry 510. Additionally or alternatively, the time estimate can be based on past mission completion times or a run-through of all rooms for purposes of calculating the time estimate.

[0070] The mission optimizer 545 can pause, interrupt, or modify a mission routine or its tasks, such as in response to user input or a triggering event. Mission changes can occur during the execution of a mission routine. Examples of mission changes can include adding a new task to a mission, removing an existing task from a mission, or prioritizing one or more tasks in a mission (e.g., changing the order of tasks, such as cleaning a particular "hot spot," such as a dirtier area, before other areas in a home). A triggering event that causes the mission optimizer 545 to change the time or order of tasks in a mission can be a particular type of user behavior, such as room occupancy, indicating the presence or absence of a person in a room. Room occupancy can be detected by a behavior detector 530, such as one communicatively linked to a security camera in the room. Alternatively, room occupancy can be detected by an object detector 512 linked to a sensor included in the mobile robot. To perform a mission routine or task based on a user experience, such as "clean the living room when unoccupied," in response to detecting an occupied room, the controller circuit 540 may pause the mission or modify the mission routine, such as by rescheduling or postponing a task that was scheduled to be performed in the occupied room until it is no longer occupied by the user or when commanded by the user.

[0071] Other examples of user behavior include user engagement in a sound detection event, such as answering a phone call, watching television, listening to music, or having a conversation. The sound detection event can be detected by the behavior detection device 530, such as by being communicatively coupled to the sound sensor. To execute a task routine or task based on the user experience, such as "don't vacuum while I'm watching TV," the controller circuitry 540 can pause the task or modify the task routine, such as by rescheduling or postponing a task that interferes with the sound detection event, until the sound detection event is over or when instructed by the user.

[0072] In some examples, the mission optimizer 545 can receive a time allocation for completing the mission and can prioritize one or more tasks in the mission routine based on the time allocation. To execute a mission routine or task based on a user experience, such as "clean as many rooms as possible in the next hour," the mission manager 544 can estimate the time to complete each task in the mission (e.g., the time required to clean each room) based on the size of the room, the dirtiness of the room, or the completion time of past missions or tasks. The mission optimizer 545 can modify the mission routine by identifying and prioritizing those tasks that can be completed within the allotted time.

[0073] The map management circuit 546 can generate and maintain a map of the environment, or portions of that map. In an example, the map management circuit 546 can generate semantically annotated objects by associating objects, such as those detected by the object detection unit 512, with semantic information, such as spatial or contextual information. Examples of semantic information can include the location, identity, or status of the objects in the environment, or constraints on the spatial relationships between the objects, among other object or inter-object characteristics. The semantically annotated objects can be graphically displayed on a map to create a semantic map. The semantic map can be used for mission control by the mission controller 542 or for robotic navigation control by the navigation controller 548. The semantic map can be stored in the storage circuit 550.

[0074] Semantic annotations may be added to objects algorithmically. In an example, the map management circuit 546 may employ SLAM techniques to use sensor data (e.g., image data or infrared sensor data, etc.) to detect, classify, or identify objects and determine their state or other characteristics. Other techniques for feature extraction and object identification, such as shape algorithms, heuristics, or off-board learning algorithms, may be used to infer meaning from the sensor data. For example, the map management circuit 546 may apply image detection or image classification algorithms to recognize specific types of objects or to analyze images of objects to determine their state (e.g., whether a door is open, closed, or locked). Alternatively or additionally, semantic annotations may be added by a user via the user interface 520. Identifications, attributes, and states, among other characteristics and constraints, may be manually added to the semantic map and associated with objects by the user.

[0075] The cruise control device 548 can navigate the mobile robot to perform missions according to a mission routine. In an example, the mission routine can include a series of rooms or floor surface areas to be cleaned by the mobile cleaning robot. The mobile cleaning robot can have a suction assembly (such as suction assembly 118) and can use suction to capture debris as the mobile cleaning robot (such as robot 100) passes over a floor surface (such as surface 50). In another example, the mission routine can include a series of rooms or floor surface areas to be cleaned by a mobile mopping robot. The mobile mopping robot can have a cleaning pad for wiping or scrubbing the floor surface. In some examples, the mission routine can include tasks scheduled to be performed by two mobile robots linked together sequentially, in parallel, or in some other specified order or pattern. For example, the cruise control device 548 can navigate the mobile cleaning robot to vacuum a room and then navigate the mobile mopping robot to mop the vacuumed room.

[0076] In an example, a mission routine may include one or more cleaning tasks characterized by or referenced to spatial or contextual information of objects in the environment, such as objects detected by object detection device 512. In contrast to room-based cleaning missions that specify specific rooms or areas to be cleaned by the mobile cleaning robot (e.g., rooms or areas as shown on a map), object-based missions may include tasks that associate an area to be cleaned with objects in that area, such as "clean under the dining table," "clean along the risers in the kitchen," "clean near the range," "clean under the couch in the living room," or "clean the cupboard area of ​​the sink." As discussed above with reference to FIG. 5 , sensor circuit 510 can detect objects in the environment and spatial and contextual information associated with the objects. Controller circuit 540 can create semantically annotated objects by establishing associations between detected objects and spatial or contextual information, such as using maps created and stored in storage circuit 550. The mission interpreter 543 can interpret the mission routine to determine a target cleaning area relative to the detected objects and navigate the mobile cleaning robot to perform the cleaning mission.

[0077] In some examples, the object referenced in the task routine may include the dirt status of a room or area. An example task routine may include "clean dirty area." The object detection device 512 may detect the dirt status or dirtiness. The controller circuit 540 may prioritize cleaning of one or more rooms or floor surface areas according to their respective dirtiness. For example, a task routine may include a first area that is dirtier than a second area, which has a higher dirtiness than a third area. The controller circuit 540 may prioritize the task tasks such that the first area is cleaned first, followed by the second area, then the third area, and so on. The controller circuit 540 may additionally or alternatively prioritize cleaning of one or more rooms or floor surface areas according to the distribution of dirt in those rooms or areas. A room with more widely spread dirt (i.e., higher spatial distribution) will have a lower priority in the task routine and will be cleaned later than a room with more spatially concentrated dirt.

[0078] In an example, a task routine may be characterized by or refer to a user's experience using a room or objects in the room. A user experience represents a personal manner of interacting with a room or objects in the room. Examples of user experiences may include a user's time, pattern, frequency, or preference for using a room or area in an environment, or a user's behavior or daily routine associated with the use, lack of use, or manner of use of a room or area in an environment. In an example, an experience-based task routine may include an "after-dinner cleaning routine" that defines cleaning tasks for areas that may be affected by preparing and serving dinner, such as the kitchen floor and the floor area around the dining table. In another example, an experience-based task routine may include an "after-shower cleaning routine" that defines cleaning tasks for areas that may be affected by a user taking a shower, such as the bathroom floor. In some examples, a user's experience-based task routine may be defined with respect to the user's activities or daily routines. For example, an experience-based task routine may include "clean all rooms after I leave home" or "clean the living room before I arrive home."

[0079] The execution of task routines based on a user's experience may be manually activated or modified by a user, such as through the user interface 520. For example, a user may manually initiate an "After-Dinner Cleaning Routine" after dinner or an "After-Shower Cleaning Routine" after a shower. Examples of the user interface 520 and user interface controls for creating, activating, monitoring, or modifying task routines are discussed below, such as with respect to FIGS. 6A-6J. Task routines based on a user's experience may also be automatically activated in response to detection of a user activity, such as by a user activity detection device 530. As shown in FIG. 5, the user activity detection device 530 may be configured to detect user activity associated with use, lack of use, or a mode of use of a room or area in the environment. In an example, the user activity detection device 530 may be communicatively coupled to one or more sensors, including, for example, external sensors (e.g., sensors included in a mobile robot, such as a camera) or stationary sensors located in a room or appliance, such as in a smart home ecosystem. For example, the controller circuit 540 can activate an "after dinner cleaning routine" in response to detecting that the dishwasher is turned on, such as via a sensor on the dishwasher. In another example, the controller circuit 540 can activate a "clean all rooms after I leave the house" in response to detecting that the front door is locked or the garage door is closed, such as via a smart door lock sensor. In another example, the user behavior detection device 530 can request and establish communication with the user's digital calendar (such as one stored on the user's mobile phone) and retrieve the user's daily schedule therefrom. The controller circuit 540 can activate or change tasks based on the schedule of the calendar event. For example, a calendar event for a doctor's appointment during a specific time period may indicate that the user will not be at the residence, and the controller circuit 540 can activate the task of "clean all rooms after I leave the house" during the time of that calendar event.

[0080] User Interface Example 6A-6J show an example of a user interface, such as a screen outline of a user interface 520 of system 500, for creating or implementing cleaning mission routines and controlling a mobile robot to perform cleaning missions in an environment. The user interface may be part of a portable computing device, such as a smartphone, mobile phone, personal digital assistant, laptop computer, tablet, smartwatch, or other portable computing device, that can send and receive signals related to the robot's cleaning missions. In an example, the portable computing device is mobile device 404.

[0081] The user interface may be configured to present on a display device (such as display device 524) information about one or more robots in the user's residence, their respective operational status, one or more editable mission routines (e.g., cleaning missions), and the progress of missions being performed. In some examples, a map of the environment or a portion of the map may be displayed along with objects in the environment. The user interface may also receive user commands, such as via user input 522, to create or modify mission routines, manage the map, and control robot navigation and mission execution.

[0082] 6A shows an example of a device 600A configured to display a diagram of a mission routine created for one or more of a user's mobile robots. The diagram can disclose relevant information to the user based on the user's past interactions with the robot, previous use of the robot, or based on mission circumstances, such as the robots active in the mission, the nature of the mission, and the progress of the mission.

[0083] The device 600A may include a display device 602 configured to display a user interface 604A. Through the single user interface 604 (as shown in FIGS. 6A-6J), a user can adjust the robot's behavior. The user interface 604A may include one or more user interface controls (e.g., buttons, selectable instructions, check boxes, drop-down lists, list boxes, sliders, links, tab strips, text boxes, charts, windows, among others) that allow a user to select various functions of mission scheduling and robot control. The user interface 604A may also include one or more shelves, which may be collections or groups of instructions, buttons, lists, text boxes, etc.

[0084] The user interface 604A may include (or be configured to display) an image 606A of a mobile cleaning robot (robot image 606A). The robot image 606A may represent a mobile cleaning robot (such as robot 100) in an environment, such as environment 40 discussed above, where the robot is in communication with a processor of a control device, such as handheld device 404. The image 606A may be an isometric or perspective view and may represent or indicate that the robot is in a ready status. The ready status may be a status in which the robot is able to perform a routine, task, or mission. That is, the robot has no errors that need to be addressed by the user and has enough battery power to at least begin a mission.

[0085] 6A also shows lines 608A and 610A, which may be images of lines displayed on user interface 604A. Lines 608A and 610A may represent the mobile cleaning robot's environment (such as environment 40 of robot 100) (a floor or surface). Lines 608A and 610A may be isometric or perspective lines and may represent or indicate that the robot is in a ready status. In some examples, lines 608A and 610A, along with image 606A, may indicate that the robot is in a ready status.

[0086] The user interface 604A may also be configured to display a text status indicator 612A that may indicate the status of the robot. For example, as shown in FIG. 6A , the text status indicator 612A may present "Ready to Suck," which may indicate to the user that the robot is in a ready status. In some examples, the text status indicator 612A, along with the lines 608A and 610A and the image 606A, may indicate that the robot is in a ready status.

[0087] User interface 604A can be configured to display additional status indicators, such as output indicator 614 and name indicator 616, where output indicator 614 can indicate the robot's power or battery level, and name indicator 616 can indicate the name of the robot represented by image 606A (shown in FIG. 6A as "Alfred") for which a selectable control is presented in user interface 604A.

[0088] The user interface 604A may also be configured to display routine instructions 618. The routine instructions 618 may be located near the image 606A or near the lines 606A and 608B. The routine instructions 618 may be selectable by a user to instruct the processing circuitry to present a new screen for creating a new routine for the robot to perform. For example, the new routine may be for cleaning one or more rooms or zones of an environment. Following creation of the new routine, the processor or processing circuitry may communicate the newly created routine to the mobile cleaning robot.

[0089] The user interface 604A may also be configured or instructed to display instructions 620A. The instructions 620A may display the text "Empty Bin" and may be selectable instructions to instruct the mobile cleaning robot to empty its bin, such as at a discharge station. The instructions 620A may, in other examples, be selectable to instruct the robot to perform other tasks or routines.

[0090] The user interface 604A can also be configured to display, among other shelves, a favorites shelf 622 and a schedule shelf 624. The favorites shelf 622 is discussed in more detail below.

[0091] 6B shows a user interface 604B of a portable device 600B displaying an image representing a mobile robot in a user's home. The portable device 600 and display device 602 can be similar to those discussed above, and the user interface 604B can be modified relative to the user interface 604A shown in FIG.

[0092] As shown in FIG. 6B, the image 606B of the mobile cleaning robot (robot image 606B) can differ from image 606A of FIG. 6A and can represent a change in the status of the mobile robot represented by robot image 606B (or image 606). For example, robot image 606B can be a top or plan view depicting the robot. Displaying robot image 606B as a top view can indicate that the robot is in a mission status. That is, the robot may be past the start of a cleaning routine or mission. Or, the robot may be in the middle of performing a portion of a cleaning routine or mission.

[0093] Additionally, lines 608B and 610B can be substantially horizontal and vertical lines, respectively, that can represent a top view or plan view of the environment, and a top view of lines 608B or 610B can indicate that the mobile cleaning robot has a mission status, i.e., the robot may be past the start of a cleaning routine or mission.

[0094] The change in image from robot image 606A (first robot image) to robot image 606B (second robot image) in each of user interfaces 604A and 604B can indicate to the user that the robot's status has changed. In the example of FIGS. 6A and 6B, the change in image from 606A to 606B can indicate that the robot has changed from a ready status to a mission status (the robot is performing a portion of a mission or has passed the start of a mission). Such a change in image can be implemented by the processor after a command is sent to the robot to begin or start a routine (and, optionally, after an acknowledgment from the robot or a routine start confirmation).

[0095] Similarly, the change from lines 608A and 610A to lines 606B and 610B can indicate that the robot has changed from a ready status to a mission status (the robot is performing a portion of the mission or has passed the start of the mission). Also, the change from lines 608A and 610A to lines 606B and 610B (from perspective or isometric lines to top or plan views) and the change in the robot image from image 606A to 606B (from perspective or isometric lines to top or plan views) can both indicate a change in the robot's status. Changing both lines 608 and 610 and image 606 can help increase the user's awareness of the change in the robot's status.

[0096] 6B also shows that the text status indicator 612B can be modified to display "Inhaling: Kitchen," which can indicate to the user that the robot is in a status of inhaling in the kitchen. The processor or processing circuitry can modify the text status indicator 612 to reflect the mission status after the processor sends commands to the robot to begin the mission. Additional status indicators are discussed below.

[0097] Additionally, the processor or processing circuitry can modify the user interface 604 to change the instruction 620 to a "pause" instruction 620B after the processor has sent a command to the robot to begin the mission. Selection of the pause instruction 620B by the user causes the processor to send a command to the robot to stop or pause the mission or routine.

[0098] 6A and 6B show that the line 608 and image 606 can change to indicate a change in status. In other examples, either the lines 608 and 610 or the robot image 606 can be changed to indicate a change in status.

[0099] Figure 6C illustrates a user interface 604C of the portable device 600. The portable device 600C and display device 602 can be similar to those discussed above with respect to Figures 6A and 6B, and the user interface 604C can be modified relative to the user interfaces 604A and 604B.

[0100] For example, user interface 604C may display image 606C representing the robot, which is a top view of the robot, but with the robot rotated or oriented at an angle relative to the screen. That is, compared to image 606B of FIG. 6B, image 606C is rotated around the center of the image. Image 606C may represent or indicate to the user that there is an error or problem associated with the robot, such as a charging error. Additionally, text status indicator 612C may output or display text summarizing the error. As shown in FIG. 6C, text status indicator 612C may display "Charging Error," which may indicate to the user that there is an error charging the robot.

[0101] The user interface may also display error indicators 626, which may include rings or circles 628 and 630. Rings 628 and 630 may be concentric rings or circles that surround (or at least partially surround) robot image 606C. Error indicator 626 may indicate to a user that there is an error associated with the mobile cleaning robot, such as a charging error. Error indicator 626, robot image 606C, and text status indicator 612C may together indicate to a user that there is an error associated with the mobile cleaning robot. Displaying multiple indications of an error in user interface 604C may help increase a user's awareness of changes in the robot's status.

[0102] 6C also shows an error information indicator 632, which may be text output on the user interface 604C that can explain additional information about the charging error. For example, the information indicator 632 instructs the user to "remove the mobile cleaning robot from the charging station." The user interface 604C may also include more detailed instructions 634, where selection of the instruction by the user causes instructions sent by the processor to display a new or different user interface 604 that includes text output on the display device 602 to detail the error, such as the charging error.

[0103] Figure 6D illustrates user interface 604D of portable device 600. Portable device 600D and display device 602 can be similar to those discussed above with respect to Figures 6A-6C, and user interface 604D can be modified relative to user interfaces 604A-604C.

[0104] For example, user interface 604D may display a robot list 636, which may be a selectable text output. A user may select each robot name in the robot list 636 to change between user interfaces. The mobile robots included in the robot list 636 may be "Rosie," "Mappy," "Gilfoil," and "Gopher," as shown in FIG. 6D. The robots in list 636 may be of different types, such as a cleaning (e.g., vacuum or sweeping) robot, a mopping robot, or a mowing robot. Two or more robots of the same type (e.g., a cleaning robot) may be listed in the robot list 636.

[0105] User interface 604D also shows lines 608D, 610D, which may represent the environment of a selected robot, such as Rosie, which may be a mopping robot. Lines 608D and 610D may be isometric or perspective views indicating that the mopping robot is in a ready status. Similarly, robot image 606D may be an isometric or perspective view depicting the mopping robot, which view may indicate that the mopping robot is in a ready status.

[0106] Further, the text status indicator 612D may state, "Mopping will begin after suction." This text status indicator 612D may indicate that the mopping robot is ready to start and that the mopping robot is scheduled to start after the cleaning robot (e.g., a suction or sweeping robot, such as the robots in robot images 604A-604C) has completed or finished its task or cleaning routine.

[0107] The processor can be configured to output various messages via the text status indicator 612D. For example, when the processor determines that a mopping pad is not installed on the mopping robot or needs to be replaced, the processor can output text to the text status indicator 612D stating "Please replace the mopping pad." When the processor determines that the mopping robot is low on cleaning fluid or cleaning solution, the processor can output text to the text status indicator 612D stating "Please fill the cleaning solution."

[0108] The processor can then communicate with the mopping robot to determine when an action has been performed. For example, the mopping robot can detect the fluid height of the cleaning solution and send that height to the processor, which can determine whether the user has performed an action. As discussed in more detail below, an alert can be generated by the processor when an action has not been performed, and an alert can be displayed by the user interface 604D when an action has not been performed. When it is determined that an action has been performed, the text status indicator 612 (and other indicators) can be changed to indicate that the robot, such as the mopping robot, is ready to perform its scheduled routine or task.

[0109] Figure 6E illustrates a user interface 604E of the portable device 600. The portable device 600E and display device 602 can be similar to those discussed above with respect to Figures 6A-6D, and the user interface 604E can be modified relative to the user interfaces 604A-604D.

[0110] The user interface 604E may be configured to display lines 608E and 610E, which may be perspective or isometric lines representing the robot's environment. The user interface 604E may also display an image 606E, which may be a perspective or isometric view of the robot (e.g., a mopping robot). Together or separately, the lines 608E and 610E and the robot image 606E may represent that the robot is in a ready status.

[0111] The user interface 604E may display a text status indicator 612E stating, "Ready for wet mopping. Reusable pad installed." The processor may instruct the display device 602 to display such a text status in the text status indicator 612E when the mopping robot is ready to blow dry and when a reusable pad is installed on the robot or when a reusable pad was recently installed on the robot. For example, the text status indicator may state, "Replace mopping pad." Thus, a user can replace the pad on the mobile cleaning robot, and the mobile cleaning robot can detect such an action. Thus, the mobile cleaning robot can transmit to the device 600 that the pad has been replaced, and the processor may output to the display device 602 to change the user interface 604E to state, "Reusable pad installed," which may guide the user that the action of replacing the pad has been recognized by the robot.

[0112] User interface 604E may also present a robot name indicator 638 that may display the name or model of the robot. For example, user interface 604E may display the text "BRAAVA JET 240" in the area of ​​robot name indicator 638, such as to indicate to the user the model or name of the robot displayed in user interface 604E.

[0113] Figure 6F shows a user interface 604F of the portable device 600. The portable device 600F and display 602 can be similar to those discussed above with respect to Figures 6A-6E, and the user interface 604F can be modified relative to the user interfaces 604A-604E. Figure 6G shows a user interface 604G of the portable device 600. The portable device 600G and display 602 can be similar to those discussed above with respect to Figures 6A-6F, and the user interface 604G can be modified relative to the user interfaces 604A-604F. Figures 6F and 6G are discussed together below.

[0114] The user interface 604F may be configured to display lines 608F and 610F, which may be top view lines or plan view lines representing the robot's environment. The user interface 604F may also display an image 606F, which may be a top view or plan view of the robot (e.g., a mopping robot). Together or separately, the lines 608F and 610F and the robot image 606F may represent the robot's mission status, such as the robot performing a mission or step in a cleaning routine. The text status indicator 612F may display "Wet Mopping," which may indicate to the user that the mop is performing a wet mopping routine or mission.

[0115] Thus, when the robot or mop encounters an error, the image 606 can change and an error indication can be displayed. For example, as shown in FIG. 6G, the robot image can be pivoted or oriented at an angle relative to the display device 602. That is, compared to image 606F in FIG. 6F, image 606G is rotated around the center of image 606. Image 606G can represent or indicate to the user that there is an error or problem associated with the robot, such as a stuck or cliff error.

[0116] Additionally, the text status indicator 612G may output or display text summarizing or explaining the error. In some examples, the text status indicator 612G may be positioned near or adjacent to the lines 608 and 610 or the robot image 606. As shown in FIG. 6G, the text status indicator 612G may display "Stuck," which may indicate to a user that the robot has become stuck during a cleaning mission or routine.

[0117] The user interface 604G may also include more detailed instructions 634, where selection of the instruction by the user may cause instructions sent by the processor to display a new or different user interface 604G including text to be output on the display device 602 to detail an error, such as a stuck error. The user interface 604G may further include a virtual wall mode instruction 640, which may indicate that a virtual wall is being activated during a routine or task being performed by the robot.

[0118] The user interface may further display error indicator 626, which may include rings or circles 628 and 630. Rings 628 and 630 may be concentric rings or circles surrounding robot image 606G. Error indicator 626 may indicate to a user that there is an error associated with the mobile cleaning robot, such as a stuck error. Error indicator 626, robot image 606G, and text status indicator 612G may together indicate to a user that there is an error associated with the mobile cleaning robot. Displaying multiple indications of an error in user interface 604G may help increase a user's awareness of changes in the robot's status.

[0119] The user interface 604G may also be configured to display an error information indicator 632, which may be text output to the user interface 604G that may explain additional information about the stuck error. For example, the error information indicator 632 may state, "Mappy has stopped due to a cliff. Moving to a new location." Such a message may indicate to the user what is causing the stuck error (in this example, detecting a cliff) and may provide instructions to the user on how to correct or fix the cause of the error (in this example, moving the robot away from the cliff).

[0120] In some examples, the processor can determine whether the required action indicated by the error indication 632 has been performed with respect to the mobile cleaning robot. For example, the processor can receive a communication from the robot when it has been moved from its stuck location. When the processor determines that the robot has been moved (e.g., the robot continues to communicate to the processor that it is stuck), the processor can display a text alert on the user interface 604G based on the displayed text of the error indication 632, based on the status of the mobile cleaning robot, or based on a determination of whether the required action has been performed. Such an alert can be a text alert 641 or a graphic alert including an error such as "stuck." Such an alert can be displayed outside of an application, such as on a home screen or lock screen of the device 600G.

[0121] The alarm 641 may include a snooze instruction or element 645 that may be selectable to cause the processor to snooze (or clear or remove) the alarm so that the alarm reappears at a later time, such as 10 minutes, 1 hour, or other time frame. As discussed above with respect to the text status indicator 612D, the alarm may also be configured to output "Replace Mopping Pad" when it is determined that the mopping robot does not have a blow-cleaning pad installed or that the robot's mopping pad needs to be replaced. The alarm 641 may also state "Fill Cleaning Container" when the cleaning solution (or fluid or liquid) is low or empty, or is not enough to start or complete a task. The alarm 641 may also state "Empty Container" when the robot's container needs to be emptied to complete or start a task. The alarm 641 may also state "Charge Robot" when the robot has insufficient charge to start or complete a task or routine. For any of these alerts, the robot can communicate to the processor that when the action is completed, the alert 641 should be removed and the mission should commence. Similarly, the robot can communicate to the processor that when the action is not completed, the alert 641 should continue to be displayed or the alert 641 should be displayed again.

[0122] In some examples, the processor may be configured to determine when the user is typically present at the residence based on the connection of the device 600G to the robot, the operation of the robot using the device, the locating, monitoring, or tracking of the device 600G, the schedule of the device, etc. The processor may thereby be configured to display the text alert 641 only when the user is typically present at the residence to help increase the likelihood that the user will remember to take steps required to correct an error in the robot.

[0123] In some examples, the processor may actively sense when the user is in the home and may display the text alert 641 only when the user is typically in the home to help increase the likelihood that the user will remember to take steps required to correct an error in the robot. The processor may actively sense when the user is in the home in a variety of ways, such as through connectivity to the robot or through interaction between the user and an Internet of Things (IoT) device, such as a smart thermostat, smart lights, or smart networked device.

[0124] In some examples, the alert 641 can be associated with a cleaning schedule or a scheduled cleaning event, and the alert 641 can be displayed before the cleaning event is scheduled to begin, which can allow the user to take necessary action before the scheduled cleaning mission or routine. Pre-mission alerts can be sent only when the user is likely to be home, as determined using one or more of the methods discussed above, such as by using a schedule or active sensing.

[0125] Figure 6H shows user interface 604H of portable device 600. Portable device 600H and display device 602 can be similar to those discussed above with respect to Figures 6A-6G, and user interface 604H can be modified relative to user interfaces 604A-604G.

[0126] The user interface 604H can be a menu for creating a new favorite routine or task, and the menu can include a name indicator 639 that can be configured to accept text input from a user (e.g., via a keyboard) to create a name for the new favorite routine. The user interface 604H can be configured to display a color indicator 642 and multiple selectable colors 644a, 644b, 644c, 644d, and 644n, where color 644a has been selected as indicated by an enlarged check mark. That is, when any of the selectable color indicators 644a-644n is selected (e.g., by tapping or pressing), the processor can be configured to change the user interface 604H to display the selected color larger and to include a check mark (or other indicator) within the color indicator 644a-644n. The selected color can be associated with the new favorite routine, as described in more detail below. Although only five color indicators are shown, fewer or more color indicators may be shown or provided in the user interface 604H.

[0127] The user interface 604H may be configured to display a name indicator 646 (e.g., "WILFRED") and a subsequent routine indicator or mission indicator 648. The subsequent routine indicator 468 may be selectable to select an optional subsequent routine. That is, when the subsequent routine indicator 468 is selected, the processor may generate a list or menu of routines for selection by the user. The user may select a subsequent routine that can be scheduled to be performed following the first one of the favorite routines.

[0128] The user interface 604H may also be configured to display an area selection portion 650, in which a user may select (e.g., by tapping or clicking) a checkbox to select an area or room to be cleaned during the new favorite routine. The user interface 604H may also be configured to display a completion indicator 652, which may instruct the processor to complete the creation of the new favorite routine. The completion indicator 652 may include a text indication such as "Create Favorite" to indicate that a new routine is created when the completion indicator 652 is selected.

[0129] FIG. 6I illustrates a user interface 604I for a portable device 600. The portable device 600I and display device 602 can be similar to those previously discussed with respect to FIGS. 6A-6H, and the user interface 604I can be modified relative to the user interfaces 604A-604H. The user interface 604I illustrates how a favorites indicator may appear, and its operation will be described. Any of the user interfaces previously discussed can be modified to include such a favorites indicator.

[0130] The user interface 604I of FIG. 6I shows a play favorites indicator 654I and an add favorites indicator 656. The add favorites indicator 656 may be selectable (e.g., by tapping) by a user to create a new favorite routine or task, such as by creating the user interface 604H of FIG. 6H. The add favorites indicator 656 may include a cross indicator 664 to indicate that a favorite is being added. The add favorites indicator 656 may be located below or in the favorites shelf 622 and may be located next to or near the play favorites indicator 654I.

[0131] The play favorite indicator 654I may be selectable to play, start, or initiate a favorite routine as previously selected by the user. When the play favorite indicator 654I is selected, the processor may send an instruction to the mobile cleaning robot to play the favorite cleaning routine associated with the play favorite indicator 654I. The play favorite indicator 654I may include a play indicator 658I, a description indicator 660, and a duration indicator 662.

[0132] The explanation indicator 660 may indicate to the user the area or room that the robot will clean during the task associated with the play favorite indicator 6541. The duration indicator 662 may indicate to the user the approximate length of time that the robot will need to perform the task associated with the play favorite indicator 6541.

[0133] The play indicator 658I can be a play symbol, such as a triangle, and can be filled or outlined with a color, such as blue. The color, such as blue, can be a color selected by the user during creation of a favorite routine, as discussed with respect to FIG. 6H. In some examples, multiple play favorite indicators 654 can be presented in the user interface 604I, and the user can select which favorite routine indicator is desired to be implemented. When multiple play favorite indicators 654 are presented, the play indicators 658 of each play favorite indicator 658 can have a different color (or various colors) associated with different favorite routines, which can help the user distinguish between the favorite routines.

[0134] Figure 6J illustrates user interface 604J of portable device 600. Portable device 600J and display device 602 can be similar to those discussed above with respect to Figures 6A-6I, and user interface 604J can be modified relative to user interfaces 604A-604I.

[0135] 6J shows interface 604J, which may be similar to interface 604I in which play favorites indicator 654J may be for a "Kitchen" cleaning routine. Play indicator 658J for play favorites indicator 654J may be a different color than the color for play favorites indicator 654I. For example, play favorites indicator 654J may be green. Such a change in color may help a user recognize (or remember) the distinction between the favorites.

[0136] Figure 7 shows a user interface of a mobile device displaying an image representing a mobile robot in a user's home. Figure 8 shows a user interface of a mobile device displaying an image representing a mobile robot in a user's home. Figure 9 shows a user interface of a mobile device displaying an image representing a mobile robot in a user's home. Figure 10 shows a user interface of a mobile device displaying an image representing a mobile robot in a user's home. Figure 11 shows a user interface of a mobile device displaying an image representing a mobile robot in a user's home.

[0137] Notes and Examples The following non-limiting examples detail particular aspects of the present subject matter that, among other things, solve the problems and provide the benefits discussed herein.

[0138] Example 1 is a machine-readable medium including instructions for presenting a user interface, the instructions being configured, when executed by a processor, to cause the processor to display in the user interface an image representing a mobile cleaning robot in communication with the processor and a line representing the mobile cleaning robot's environment, the image and the line together indicating the status of the mobile cleaning robot.

[0139] In Example 2, the subject matter of Example 1 optionally includes that the image is an isometric or perspective view depicting the mobile cleaning robot, the isometric or perspective view indicating that the mobile cleaning robot is in a ready status.

[0140] In Example 3, the subject matter of Example 2 optionally includes that the displayed lines are isometric or oblique lines of the represented environment, and the isometric or oblique lines, together with an image representing the mobile cleaning robot, indicate that the mobile cleaning robot is in a ready status.

[0141] In Example 4, the subject matter of Example 3 optionally includes: the image is a top view representing a mobile cleaning robot, the top view indicating that the mobile cleaning robot is in a mission status; and the displayed lines are a top view of lines of a represented environment, the lines of the top view indicating that the mobile cleaning robot is in a mission status.

[0142] In Example 5, the subject matter of any one or more of Examples 1-4 optionally includes the processor being further configured to display an error indicator at least partially surrounding the image of the mobile cleaning robot, the error indicator indicating an error in the mobile cleaning robot.

[0143] In Example 6, the subject matter of Example 5 optionally includes wherein the error indicator is an image surrounding the image representing the mobile cleaning robot.

[0144] In Example 7, the subject matter of Example 6 optionally includes, wherein the error indicator is an image of concentric circles around the image representing the mobile cleaning robot.

[0145] In Example 8, the subject matter of any one or more of Examples 1-7 optionally includes wherein the processor is further configured to display selectable routine instructions for creating a new routine and, following creation of the new routine, communicate the new routine to the mobile cleaning robot.

[0146] In Example 9, the subject matter of Example 8 optionally includes, wherein the routine instructions are positioned on the user interface near an image representing a mobile cleaning robot.

[0147] In Example 10, the subject matter of any one or more of Examples 1-9 optionally includes: the processor is further configured to display a text display area adjacent to the image representing the mobile cleaning robot, the text display area is configured to display text based on a status of the mobile cleaning robot, and the processor is further configured to display text in the text display area including user instructions for performing a required action with respect to the mobile cleaning robot based on the status of the mobile cleaning robot.

[0148] In Example 11, the subject matter of Example 10 optionally includes wherein the processor is further configured to determine whether a required action has been performed with respect to the mobile cleaning robot, and to display a text alert in a user interface based on the displayed text, based on a status of the mobile cleaning robot, and based on a determination of whether the required action has been performed.

[0149] In Example 12, the subject matter of Example 11 optionally includes, wherein the alert includes a snooze component selectable to cause the processor to display the alert at a later time.

[0150] In Example 13, the subject matter of Example 12 optionally includes wherein the processor is further configured to actively sense when the user is in the residence by interacting with Internet of Things devices and to display the text alert only when the user is typically in the residence.

[0151] In Example 14, the subject matter of any one or more of Examples 12-13 optionally includes wherein the processor is further configured to determine when the user is typically at home and to display the text alert only when the user is typically at home.

[0152] Example 15 is a mobile cleaning robot system including a mobile cleaning robot, the mobile cleaning robot system comprising: a display device configured to present a user interface; and processing circuitry in communication with the mobile cleaning robot and the display device, the processing circuitry configured to: display a plurality of cleaning routine instructions, each of different selectable colors, to create a cleaning routine; display selectable start routine instructions, including a play instruction having a play color that matches the color of a selected cleaning routine; and, when a start routine instruction is selected, to instruct the mobile cleaning robot to perform the selected cleaning routine.

[0153] In Example 16, the subject matter of Example 15 optionally includes that the selectable start routine instruction is located on a favorites shelf on the user interface.

[0154] In Example 17, the subject matter of Example 16 optionally includes wherein the processing circuitry is configured to display a second selectable start routine indication in a favorites shelf.

[0155] In Example 18, the subject matter of any one or more of Examples 15-17 optionally includes that the second selectable start routine instruction includes a second playback instruction having a second playback color different from the playback color.

[0156] In Example 19, the subject matter of any one or more of Examples 15-18 optionally includes wherein the processing circuitry is configured to display, on a user interface, an image representing the mobile cleaning robot in communication with the processing circuitry, indicating a status of the mobile cleaning robot.

[0157] In Example 20, the subject matter of Example 19 optionally includes wherein the processing circuitry is configured to display a line representing an environment of the mobile cleaning robot, the image and the line together indicating a status of the mobile cleaning robot.

[0158] In Example 21, the subject matter of Example 20 optionally includes, wherein the image is an isometric or perspective view depicting the mobile cleaning robot, the isometric or perspective view indicating that the mobile cleaning robot is in a ready status.

[0159] In Example 22, the subject matter of any one or more of Examples 20-21 optionally includes that the displayed lines are isometric or oblique lines of the represented environment, and the isometric or oblique lines, together with an image of the mobile cleaning robot, indicate that the mobile cleaning robot is in a ready status.

[0160] Example 23 is a method for operating a mobile cleaning robot system including a mobile cleaning robot and a display device, the method including the steps of: displaying a first image on a user interface of the display device, the first image indicating a readiness status of the mobile cleaning robot to communicate with the display device; sending a command to the mobile cleaning robot to begin a cleaning mission; and after sending the command, displaying a second image on the user interface, the second image indicating a mission status of the mobile cleaning robot.

[0161] In Example 24, the subject matter of Example 23 optionally includes the step of displaying a line representing an environment of the mobile cleaning robot, wherein the first image and the line together indicate a readiness status of the mobile cleaning robot.

[0162] In Example 25, the subject matter of Example 24 optionally includes a step of displaying a second set of lines after sending the command, wherein the second set of lines and the second image together indicate a mission status of the mobile cleaning robot.

[0163] In Example 26, the subject matter of any one or more of Examples 23-25 ​​optionally includes a step of displaying an error indicator at least partially surrounding the first image or the second image representing the mobile cleaning robot, the error indicator indicating an error in the mobile cleaning robot.

[0164] In Example 27, the subject matter of Example 26 optionally includes, wherein the error indicator is an image of concentric circles around the first image or the second image.

[0165] In Example 28, the apparatus, system, or method of any one or any combination of Examples 1 through 27 may be optionally configured such that all proposed elements or options are available for use or selection.

[0166] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of example, specific embodiments in which the invention may be practiced. These embodiments are referred to herein as "examples." These examples may include elements in addition to those shown or described. However, the inventors also contemplate examples in which only the elements shown or described are provided. Furthermore, the inventors also contemplate examples using any combination or permutation of the elements shown or described (or one or more aspects thereof), either with respect to the specific example (or one or more aspects thereof) or with respect to any other example (or one or more aspects thereof) shown or described herein.

[0167] In the event of a conflicting usage between this document and any document incorporated by reference, the usage in this document controls. The terms "including" and "in which" are used herein as the plain English equivalents of the terms "comprising" and "wherein," respectively. Also, in the appended claims, the terms "including" and "comprising" are intended to be open-ended, meaning that systems, devices, articles, compositions, formulas, or processes that include elements in addition to the elements listed after such term in a claim are still deemed to be within the scope of that claim.

[0168] The foregoing description is intended to be illustrative, not limiting. For example, the examples in the foregoing description (or one or more aspects thereof) can be used in combination with each other. Other embodiments may be utilized by those skilled in the art, etc., upon review of the foregoing description. The Abstract is provided to comply with 37 CFR §1.72(b) to allow the reader to quickly ascertain the nature of the disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the foregoing Detailed Description, various features may be grouped together to streamline the disclosure. This should not be construed as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description as an example or embodiment, with each claim standing on its own as a separate embodiment, and such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full range of equivalents to which such claims are entitled. [Explanation of symbols]

[0169] 40 Environment Rooms 42a, 42b, 42c, 42d, and 42e 44 beds 46 tables 48 Island 50, 50a, 50b, 50c, 50d, 50e floor surface 52 Rugs 54 Behavioral Control Zone 60 users 75 Garbage 100 Mobile Cleaning Robot 112 Control device 118 Suction assembly, suction system 120 airflow 124 Case 125 Upper part 126 Front side 126a, 126b longitudinal axis 128, 130 Side 134 Cliff Sensor 136, 136a, 136b, 136c proximity sensors 137 Light indicator system 138 Buffer 139, 139a, 139b collision sensors 140 Image acquisition device 141 Fault monitoring sensor 142 Side Brush 144 Motor 147 Lid 200 units 202a front part 202b rear part 204a, 204b Side parts 205 Cleaning assembly, cleaning roller, cleaning head 205a, 205b cleaning roller 206 Front side 208 Motor 208a, 208b actuators 210, 210a, 210b driving wheels 211 Caster 212 Control device 213 Storage device 214 Motor 322 Cleaning containers, garbage containers 400 Communication Network 401 Information Exchange Process 402 users 404 Mobile Devices 406 Cloud Computer System 408 other autonomous robots 430 Control device 442 processor 500 Robot Scheduled Control System 510 Sensor Circuit 512 Object detection device 520 User Interface 522 User Input Section 523 Mission Routine 524 Display device 530 User behavior detection device 540 Control Device Circuit 542 Mission Control Device 543 Mission Interpreter 544 Mission Management Device 545 Mission Optimizer 546 Map Management Circuit 548 Navigation control device 550 Storage circuit 600, 600A, 600B, 600C, 600D, 600E, 600F, 600G, 600H, 600I Portable Devices 602 Display device 604, 604A, 604B, 604C, 604D, 604E, 604F, 604G, 604H, 600I, 604J User Interface 606, 606A, 606B, 606C, 606D, 606E, 606F Mobile cleaning robot images, robot images 608, 608A, 608B, 608D, 608E, 608F, 610, 610A, 610B, 610D, 610E, 610F line 612, 612A, 612B, 612C, 612D, 612E, 612F, 612G Text status indicator 614 Output instruction section 616 Name indicator 618 Routine Instructions 620, 620A, 620B instructions 622 Favorites Shelf 624 Schedule Shelf 626 Error indicator 628, 630 ring, circle 632 Error Information Indicator 634 Further Instructions 636 Robot List 638 Robot name indicator 639 Name indicator 641 Alarm 642 Color indicator 644a, 644b, 644c, 644d, 644n Color indicator 646 Name indicator 648 Subsequent Routine Instruction Department, Mission Instruction Department 650 Area Selection Section 652 Completion instruction section 654, 654I, 654J Playback Favorites Indicator 656 Add Favorites Instructions 658, 658I, 658J Playback instruction section 660 Explanation and Instructions 662 Duration indicator 664 Cross indicator F forward drive direction

Claims

1. 1. A machine-readable medium containing instructions for presenting a user interface, the instructions, when executed by a processor, causing the processor to: displaying on the user interface an image representing a mobile cleaning robot in communication with the processor; displaying a line representing the environment of the mobile cleaning robot; the image and the line together indicate a status of the mobile cleaning robot; the image indicates that the mobile cleaning robot is in a ready status; the displayed line, together with the image representing the mobile cleaning robot, indicates that the mobile cleaning robot is in a ready status; The processor: displaying a text display area adjacent to the image representing the mobile cleaning robot, the text display area configured to display text based on the status of the mobile cleaning robot; displaying text in the text display area including user instructions for performing necessary actions regarding the mobile cleaning robot based on the status of the mobile cleaning robot; further configured to: The processor: determining whether the required action has been performed with respect to the mobile cleaning robot; Displaying a text alert in the user interface based on the displayed text, based on the status of the mobile cleaning robot, and based on the determination of whether the required action has been performed. further configured as follows: The processor: determining when a user is typically at home; Displaying the text alert only when the user is typically at home The machine-readable medium further configured to:

2. The machine-readable medium of claim 1 , wherein the image is an isometric or perspective view depicting the mobile cleaning robot.

3. The machine-readable medium of claim 2 , wherein the displayed lines are isometric or oblique lines of the represented environment.

4. 4. The machine-readable medium of claim 3, wherein the image is a top view of the mobile cleaning robot, the top view indicating that the mobile cleaning robot is in a mission status, and the displayed line is a top view of a line of the represented environment, the line of the top view indicating that the mobile cleaning robot is in a mission status.

5. The processor:

5. The machine-readable medium of claim 1, further configured to display an error indicator at least partially surrounding the image of the mobile cleaning robot, the error indicator indicating an error in the mobile cleaning robot.

6. The machine-readable medium of claim 5 , wherein the error indicator is an image surrounding the image representing the mobile cleaning robot.

7. The machine-readable medium of claim 6 , wherein the error indicator is an image of concentric circles around the image representing the mobile cleaning robot.

8. The processor: Displays selectable routine instructions for creating a new routine; Following creation of the new routine, communicating the new routine to the mobile cleaning robot.

8. The machine-readable medium of claim 1, further configured to:

9. The machine-readable medium of claim 8 , wherein the routine instructions are positioned on the user interface near the image representing the mobile cleaning robot.

10. 1. A machine-readable medium containing instructions for presenting a user interface, the instructions, when executed by a processor, causing the processor to: displaying on the user interface an image representing a mobile cleaning robot in communication with the processor; displaying a line representing the environment of the mobile cleaning robot; the image and the line together indicate a status of the mobile cleaning robot; The processor: displaying a text display area adjacent to the image representing the mobile cleaning robot, the text display area configured to display text based on the status of the mobile cleaning robot; displaying text in the text display area including user instructions for performing necessary actions regarding the mobile cleaning robot based on the status of the mobile cleaning robot; further configured to: The processor: determining whether the required action has been performed with respect to the mobile cleaning robot; Displaying a text alert in the user interface based on the displayed text, based on the status of the mobile cleaning robot, and based on the determination of whether the required action has been performed. further configured as follows: the text alert includes a re-notification component selectable to cause the processor to display the text alert at a later time; The processor: Interacting with Internet of Things devices to actively sense when users are in their homes Displaying the text alert only when the user is typically at home The machine-readable medium further configured to:

11. A machine-readable medium containing instructions for presenting a user interface, the instructions, when executed by a processor, causing the processor to: displaying on the user interface an image representing a mobile cleaning robot in communication with the processor; displaying a line representing the environment of the mobile cleaning robot; the image and the line together indicate a status of the mobile cleaning robot; The processor: displaying a text display area adjacent to the image representing the mobile cleaning robot, the text display area configured to display text based on the status of the mobile cleaning robot; displaying text in the text display area including user instructions for performing necessary actions regarding the mobile cleaning robot based on the status of the mobile cleaning robot; further configured to: The processor: determining whether the required action has been performed with respect to the mobile cleaning robot; Displaying a text alert in the user interface based on the displayed text, based on the status of the mobile cleaning robot, and based on the determination of whether the required action has been performed. further configured as follows: the text alert includes a re-notification component selectable to cause the processor to display the text alert at a later time; The processor: determining when a user is typically at home; Displaying the text alert only when the user is typically at home The machine-readable medium further configured to:

Images