User feedback on potential obstacles and error conditions detected by autonomous mobile robots

Abstract

To provide an autonomous mobile robot capable of detecting potential obstacles while operating in an environment (e.g., while performing a cleaning mission).SOLUTION: A mobile computing device includes a user input device and a controller. The user input device includes a display, and the controller is operably connected to the user input device and configured to execute instructions to perform operations. The operations include presenting on the display, information about one or more areas that were not cleaned by an autonomous cleaning robot during a first mission. The operations further include transmitting data corresponding to a user-selected subset of the one or more areas to cause the robot to clean the user-selected subset during a second mission.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] This specification relates to obtaining and using user feedback regarding obstacles detected by an autonomous mobile robot, and related systems and methods. [Background technology]

[0002] Autonomous mobile robots include autonomous cleaning robots that autonomously perform cleaning tasks within an environment, such as a home. A wide variety of cleaning robots are autonomous to some degree and in various ways. The cleaning robot may include a controller configured to autonomously navigate the robot around an environment, allowing the robot to pick up debris as it moves. The cleaning robot may include sensors to avoid obstacles within the environment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent Application No. 15 / 863,591 [Patent Document 2] U.S. Patent Application No. 16 / 588,295 Summary of the Invention [Problem to be solved by the invention]

[0004] An autonomous mobile robot can detect potential obstacles and / or error conditions while operating within an environment (e.g., while performing a cleaning mission). The autonomous mobile robot can provide information about these detected potential obstacles and / or error conditions to a user and receive user feedback about these detected potential obstacles and / or error conditions. This disclosure describes various ways in which user feedback about detected potential obstacles and / or error conditions can be obtained and used in the operation of the robot. When the robot detects a potential obstacle and / or error condition, it can transmit information about the potential obstacle and / or error condition to the user's mobile computing device. For example, this information can include data representing a classification of the detected potential obstacle or type of error condition, the location of the robot at the time the potential obstacle and / or error condition was detected, and an image captured by the robot at the time the potential obstacle and / or error condition was detected.

[0005] The mobile computing device can receive information transmitted by the robot and display at least a portion of this information on a display. In some examples, the mobile computing device can request feedback from a user. For example, the mobile computing device can display on a display user-selectable options to provide feedback for controlling the autonomous mobile robot's response to a potential obstacle, an area where a potential obstacle is located, or an area where an error condition was detected. For example, a user can select an option to instruct the autonomous mobile robot to clean the area (e.g., because the obstacle or source of the error condition has been addressed by the user) or to avoid the area (e.g., because the obstacle or source of the error condition has not or cannot be addressed by the user). These user-selectable options can also be used to provide feedback to the robot regarding whether a detected potential obstacle and / or error condition is a legitimate obstacle and / or error condition. For example, in some examples, obstacles and / or error conditions detected by the robot become potential obstacles and / or potential error conditions that the user can review to identify whether they are legitimate obstacles and / or legitimate error conditions. If these potential obstacles and / or potential error conditions are not legitimate obstacles and / or error conditions, the user may provide feedback to indicate that the obstacles and / or error conditions do not exist.

[0006] In some examples, the autonomous mobile robot may detect one or more obstacles during a first cleaning mission, resulting in one or more areas not being cleaned by the autonomous cleaning robot during the initial cleaning mission. The robot may then perform a second cleaning mission after receiving user feedback regarding the one or more obstacles or areas. In some examples, the second cleaning mission may be a tidy mission in which the autonomous cleaning robot navigates against the one or more obstacles or areas based on the user feedback. For example, the user feedback regarding the one or more obstacles or areas may include a user-selected subset of one or more areas that should be cleaned by the robot in the tidy mission (e.g., because the obstacles were removed or because the detected potential obstacles were not legitimate obstacles).

[0007] The foregoing advantages may include, but are not limited to, those described below and elsewhere herein. [Means for solving the problem]

[0008] Implementations described herein can improve a user's experience interacting with an autonomous mobile robot. Images captured by the autonomous mobile robot when obstacles and / or error conditions are detected can provide the user with information about the robot's environment. For example, the images can help the user understand why one or more areas of the environment were not cleaned by the robot during a cleaning mission. In some implementations, multiple user-selectable options can be displayed on the display, providing the user with an intuitive way to interact with and provide feedback to the robot, particularly regarding potential obstacles and error conditions detected during a cleaning mission.

[0009] Implementations described herein may also improve the robot's coverage within an environment and its performance in further cleaning missions. For example, multiple user-selectable options enable a user to provide useful user feedback to the robot. In some examples, the user feedback can provide information not captured by one or more sensors of the robot. The user feedback can also assist the robot in processing and interpreting data captured by one or more sensors of the robot. In some examples, the user feedback can notify the robot that an obstacle has been removed, allowing the robot to clean areas during a tidy mission that were not cleaned during a first cleaning mission. In some examples, the user feedback can be used to create no-go zones, which can reduce the likelihood that the robot will encounter an error condition in a later cleaning mission. In some examples, the user feedback can inform the robot whether a detected potential obstacle is a legitimate obstacle, which can improve the robot's detection of obstacles in later cleaning missions. In some examples, users may choose to provide data regarding potential obstacles, error conditions, and associated user feedback to the database, which data may be used to improve the performance of various autonomous mobile robots owned by various users.

[0010] Implementations described herein can also improve the efficiency of cleaning missions performed by a robot. For example, tidy missions performed based on user feedback regarding potential obstacles and / or error conditions can enable a robot to efficiently clean areas not cleaned during a first cleaning mission while minimizing re-cleaning of areas already cleaned during the first cleaning mission. In some examples, areas not cleaned during a first cleaning mission may have been left uncleaned for other reasons and may not be cleaned in subsequent missions. The tidy mission can provide a mechanism for ensuring cleaning of such areas.

[0011] In one aspect, a mobile computing device includes a user input device and a controller operably connected to the user input device. The user input device includes a display, and the controller is configured to execute instructions to perform operations. The operations include displaying information on the display regarding one or more areas not cleaned by the autonomous cleaning robot during a first mission. Further, the operations include transmitting data corresponding to a user-selected subset of the one or more areas to cause the autonomous cleaning robot to clean the user-selected subset during a second mission.

[0012] Implementations may include one or more of the features described below or elsewhere herein. Implementations may include combinations of the following features.

[0013] In some implementations, the information about the one or more areas may include a location of each of the one or more areas. In some implementations, the information about the one or more areas may include data representing a potential obstacle detected in each of the one or more areas. In some implementations, the data representing the potential obstacle detected in each of the one or more areas may include one or more images of the potential obstacle, and in some implementations, the operations may further include displaying a representation of the one or more images of the potential obstacle on the display. In some implementations, the data representing the potential obstacle detected in each of the one or more areas may include a label indicating a classification of the potential obstacle. In some implementations, the operations may further include displaying an affordance corresponding to a plurality of user-selectable options for each of the one or more areas on the display. In some implementations, the operations may further include transmitting data corresponding to a user selection of one of the plurality of user-selectable options for each of the one or more areas to the autonomous cleaning robot to navigate the autonomous cleaning robot through the one or more areas during the second mission. In some implementations, the plurality of user-selectable options may include a first option to navigate the autonomous cleaning robot to clean each area in the one or more areas and a second option to navigate the autonomous cleaning robot to avoid each area in the one or more areas. In some implementations, the plurality of user-selectable options may include an option to indicate an absence of obstacles in each area in the one or more areas. In some implementations, the plurality of user-selectable options may include an option to navigate the autonomous cleaning robot to avoid each area in the one or more areas during a second mission and not avoid the each area during one or more missions after the second mission.In some implementations, the plurality of user-selectable options may include an option to navigate the autonomous cleaning robot to avoid each area in the one or more areas during the second mission and one or more missions after the second mission. In some implementations, affordances corresponding to the plurality of user-selectable options may be displayed on the display after completion of the first mission. In some implementations, the second mission may be initiated within 12 hours after completion of the first mission. In some implementations, the operations may further include displaying an affordance corresponding to a provision option on the display, the provision option enabling the mobile computing device to transmit information about the one or more areas and data corresponding to the user-selected subset to a database that stores data from a plurality of users. In some implementations, the operations may further include displaying a map of the environment including the one or more areas on the display. In some implementations, the operations may further include displaying information about the user-selected subset on the display after completion of the second mission. In some implementations, the first mission and the second mission may be consecutive missions. In some implementations, the second mission may include: i) the autonomous cleaning robot starting to move from the dock to a first area of ​​the user-selected subset, ii) the autonomous cleaning robot starting to move to all remaining areas of the user-selected subset, and iii) the autonomous cleaning robot starting to move from the final area of ​​the user-selected subset to the dock. In some implementations, the autonomous cleaning robot may clean the user-selected subset during the second mission without cleaning all of the areas that the autonomous cleaning robot cleaned during the first mission.

[0014] In another aspect, an autonomous cleaning robot includes a drive system, an obstacle detection sensor, and a controller operatively connected to the drive system and the obstacle detection sensor. The drive system is operable to support the autonomous cleaning robot above a floor surface and navigate the autonomous cleaning robot across the floor surface. The obstacle detection sensor is operable to detect potential obstacles as the autonomous cleaning robot navigates across the floor surface, and the controller is configured to execute instructions to perform operations. The operations include performing a first mission and detecting one or more potential obstacles located in one or more areas on the floor surface during the first mission. Further, the operations include transmitting data corresponding to the detected potential obstacles and the one or more areas to a mobile computing device. Further, the operations include receiving data corresponding to a user-selected subset of the one or more areas from the mobile computing device and performing a second mission to clean the user-selected subset of the one or more areas.

[0015] Implementations may include one or more of the features described below or elsewhere herein. Implementations may include combinations of the following features.

[0016] In some implementations, the obstacle detection sensor may comprise an image capture device positioned on the autonomous cleaning robot to capture an image of a portion of a floor surface in front of the autonomous cleaning robot. In some implementations, the data corresponding to the detected potential obstacles and the one or more areas may include data representing an image of each obstacle among the one or more detected potential obstacles. In some implementations, the image may include a single image, and in some implementations, the image may include a series of images. In some implementations, the data corresponding to the detected potential obstacles and the one or more areas may include a location of each area among the one or more areas. In some implementations, the data corresponding to the detected potential obstacles and the one or more areas may include a label indicating a classification of each potential obstacle among the detected potential obstacles. In some implementations, the second mission may be initiated within 12 hours after completion of the first mission. In some implementations, these operations may further include transmitting data corresponding to an updated state of a user-selected subset of the one or more areas during the second mission to the mobile computing device. In some implementations, the data corresponding to the updated state of the user-selected subset may include an indication of a portion of the user-selected subset that was cleaned during the second mission. In some implementations, the data corresponding to the user-selected subset of the one or more areas may include, for each area of ​​the one or more areas, data corresponding to a user selection of one of a plurality of user-selectable options. In some implementations, the plurality of user-selectable options may include a first option that causes the autonomous cleaning robot to clean each area of ​​the one or more areas during the second mission and a second option that causes the autonomous cleaning robot to avoid each area of ​​the one or more areas during the second mission.In some implementations, the plurality of user-selectable options may include an option to cause the autonomous cleaning robot to avoid each area in the one or more areas during the second mission and not avoid each area during one or more missions after the second mission. In some implementations, the plurality of user-selectable options may include an option to cause the autonomous cleaning robot to avoid each area in the one or more areas during the second mission and one or more missions after the second mission. In some implementations, the plurality of user-selectable options may include an option to indicate the absence of obstacles located in each area in the one or more areas. In some implementations, these operations may further include updating the obstacle detection module based on the user selection. In some implementations, the first mission and the second mission may be consecutive missions. In some implementations, performing the second mission may include i) initiating movement from the dock to a first area in the user-selected subset, ii) initiating movement to all remaining areas of the user-selected subset, and iii) initiating movement from a final area of ​​the user-selected subset to the dock. In some implementations, the second mission may include cleaning a user-selected subset of one or more areas without cleaning all of the areas cleaned by the autonomous cleaning robot during the first mission.

[0017] In another aspect, a mobile computing device includes a user input device and a controller operably connected to the user input device. The user input device includes a display, and the controller is configured to execute instructions to perform operations. The operations include receiving, from the autonomous cleaning robot, data corresponding to an error condition detected by the autonomous cleaning robot and a portion of an image captured by the autonomous cleaning robot, the portion of the image being associated with the detected error condition. Further, the operations include displaying, in response to receiving the data corresponding to the detected error condition, a representation of the portion of the image on the display and an indicator of the detected error condition.

[0018] Implementations may include one or more of the features described below or elsewhere herein. Implementations may include combinations of the following features.

[0019] In some implementations, the portion of images associated with the error condition may be captured near the location of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition. In some implementations, the portion of images may include images captured before the autonomous cleaning robot detected the error condition, and in some implementations, the portion of images may include images captured after the autonomous cleaning robot detected the error condition. In some implementations, the portion of images associated with the error condition may include a single image, and in some implementations, the portion of images associated with the error condition may include a series of images. In some implementations, the data corresponding to the detected error condition may include at least one of a location of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition, a time when the autonomous cleaning robot detected the error condition, or a type of error condition. In some implementations, the type of error condition may be related to a component of the autonomous cleaning robot, and the component may include at least one of a drive system, a cleaning assembly, or a brush. In some implementations, the component may be identified for replacement. In some implementations, the type of error condition can be related to limitations on the mobility of the autonomous cleaning robot, and in some implementations, limitations on the mobility of the autonomous cleaning robot can include an inability to complete a mission or an inability to navigate to a dock. In some implementations, the portion of the image can include a captured image of a portion of the environment in front of the autonomous cleaning robot. In some implementations, the portion of the environment in front of the autonomous cleaning robot can include a portion of a floor surface. In some implementations, the indicator of the detected error condition can include a label of the type of error condition, and in some implementations, the indicator of the detected error condition can include a representation of the position of the autonomous cleaning robot at the time the autonomous cleaning robot detected the error condition.

[0020] In another aspect, an autonomous cleaning robot includes a drive system for supporting the autonomous cleaning robot above a floor surface, the drive system operable to navigate the autonomous cleaning robot across the floor surface. The autonomous cleaning robot further includes one or more sensors configured to capture sensor data corresponding to an error condition of the autonomous cleaning robot and an image capture device for capturing an image associated with the error condition. The image can be an image of a portion of an environment in front of the autonomous cleaning robot. The autonomous cleaning robot further includes one or more controllers operably connected to the drive system, the image capture device, and the one or more sensors. The one or more controllers are configured to execute instructions to perform operations. The operations include detecting an error condition of the autonomous cleaning robot based on the sensor data as the autonomous cleaning robot is navigated across the floor surface. The operations further include (i) transmitting information about the detected error condition to a mobile computing device to cause the mobile computing device to display an indicator of the error condition, and (ii) transmitting data representing a portion of the captured image to cause the mobile computing device to display a representation of the portion of the captured image.

[0021] Implementations may include one or more of the features described below or elsewhere herein. Implementations may include combinations of the following features.

[0022] In some implementations, the portion of the environment in front of the autonomous cleaning robot may include a portion of a floor surface. In some implementations, the portion of the captured images may be captured at a position of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition. In some implementations, the portion of the captured images may include a single image, and in some implementations, the portion of the captured images may include a series of images. In some implementations, the information about the error condition may include at least one of a position of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition, a time when the autonomous cleaning robot detected the error condition, or a type of error condition. In some implementations, the type of error condition may be related to a component of the autonomous cleaning robot, which may include at least one of a drive system, a cleaning assembly, or a brush. In some implementations, the component may be identified for replacement. In some implementations, the type of error condition may be related to a limitation in the mobility of the autonomous cleaning robot, which may include an inability to complete a mission or an inability to navigate to a dock.

[0023] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other possible features, aspects, and advantages will become apparent from the following description, drawings, and claims. [Brief explanation of the drawings]

[0024] [Figure 1A] FIG. 1 is a plan view of an example of an environment including an autonomous mobile robot. [Figure 1B] FIG. 1 is a plan view of an example of an environment including an autonomous mobile robot. [Figure 2] FIG. 1 is a perspective view of an autonomous cleaning robot in an environment. [Figure 3] FIG. 1 is a schematic side view of an autonomous cleaning robot in an environment. [Figure 4A] FIG. 2 is a bottom view of the autonomous cleaning robot. [Figure 4B] FIG. 4B is a top perspective view of the robot of FIG. 4A. [Figure 5] 1 is a flow diagram of a process for providing user feedback regarding obstacles detected by an autonomous mobile robot. [Figure 6] 1 is a flow diagram of a process for planning a decluttering mission. [Figure 7] FIG. 1 is a front view of a mobile computing device displaying a notification for planning a tidying mission. [Figure 8A] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing obstacles detected by an autonomous cleaning robot. [Figure 8B] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing obstacles detected by an autonomous cleaning robot. [Figure 8C] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing obstacles detected by an autonomous cleaning robot. [Figure 8D] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing obstacles detected by an autonomous cleaning robot. [Figure 9A] FIG. 1 is a front view of a mobile computing device displaying a user interface for providing user feedback regarding potential obstacles detected by an autonomous cleaning robot. [Figure 9B] FIG. 1 is a front view of a mobile computing device displaying a user interface for providing user feedback regarding potential obstacles detected by an autonomous cleaning robot. [Figure 9C]FIG. 1 is a front view of a mobile computing device displaying a user interface for providing user feedback regarding potential obstacles detected by an autonomous cleaning robot. [Figure 9D] FIG. 1 is a front view of a mobile computing device displaying a user interface for providing user feedback regarding potential obstacles detected by an autonomous cleaning robot. [Figure 10] FIG. 1 is a front view of a mobile computing device displaying a user interface for initiating a tidying mission. [Figure 11A] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing completed tidying missions. [Figure 11B] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing completed tidying missions. [Figure 12] 1 is a flow diagram of a process for displaying information about an error condition detected by an autonomous cleaning robot. [Figure 13A] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing an error condition detected by an autonomous cleaning robot. [Figure 13B] FIG. 1 is a front view of a mobile computing device displaying a user interface for reviewing an error condition detected by an autonomous cleaning robot. [Figure 14] FIG. 1 illustrates an example of a computing device and a mobile computing device. DETAILED DESCRIPTION OF THE INVENTION

[0025] Like reference numbers and symbols in the various drawings refer to like elements.

[0026] Referring to FIG. 1A , an autonomous mobile robot 100 moves across a floor surface 10 within an environment 20. For example, the environment 20 can be a home having a kitchen, a dining room, a bathroom, a bedroom, and a living room. The robot 100 is a cleaning robot, such as a robot vacuum cleaner, a robot mop, or other cleaning robot, that can clean the floor surface 10 as the robot 100 moves across the floor surface 10. The robot 100 can perform various missions, which can include training missions and cleaning missions. During a training mission, the robot moves across the floor surface 10 without cleaning it. During a cleaning mission, the robot 100 moves across the floor surface 10 to clean one or more areas of the environment 20. These areas can be automatically determined by the robot 100 or can be pre-designated by a user. At the start of a cleaning mission, the robot 100 can be stored in a docking station 50 (sometimes called a dock) or can be placed elsewhere within the environment 20. A cleaning mission may be considered complete when the user manually terminates the cleaning mission after the robot 100 has completed navigating or attempted to navigate to various areas within the environment 20, or when the robot 100 encounters an error condition. In some implementations, a cleaning mission is not considered complete until the robot 100 has navigated or attempted to navigate to these various areas a threshold number of times (e.g., two, three, etc.). In some implementations, a cleaning mission is not considered complete until the robot detects a debris load below a threshold amount in at least some of the various areas. A cleaning mission may also include the robot 100 returning to the docking station 50 midway through or at the end of the cleaning mission. In some implementations, a cleaning mission may be considered complete when the robot 100 determines that it is unable to navigate to the docking station 50. The docking station 50 may charge the robot 100.In the example where the robot 100 is a robotic vacuum cleaner that collects debris as it moves around an environment, in some implementations the docking station 50 ejects debris from the robot 100 so that the robot 100 can collect more debris. Once a cleaning mission is complete, the robot 100 stops moving and cleaning around the floor surface 10 until the next cleaning mission begins.

[0027] As described in this disclosure, during a first cleaning mission, the robot 100 may detect potential obstacles in one or more areas of the environment 20. For example, the robot 100 may detect a sock 35 as a potential obstacle in a first area 25A of the floor surface 10, a cord 45 as a potential obstacle in a second area 25B of the floor surface 10, and a rug 55 as a potential obstacle in a third area 25C of the floor surface 10. In some implementations, potential obstacles are detected by processing images captured by an image capture device (e.g., a forward-facing camera) of the robot 100. For example, images captured by the image capture device of the robot 100 may be input to an object detection module 190 (shown in FIG. 4A ) of the robot 100 to detect obstacles in the images (e.g., using image analysis techniques, one or more convolutional neural networks, etc.). Examples of obstacle detection techniques that may be implemented by or in conjunction with robot 100 are described in U.S. patent application Ser. No. 15 / 863,591, filed Jan. 5, 2018, the entire contents of which are incorporated herein by reference.

[0028] In some implementations, to avoid getting stuck and encountering error conditions, the robot 100 may avoid cleaning a first area 25A, a second area 25B, and a third area 25C (collectively referred to as areas 25) during a first cleaning mission. In some implementations, in response to detecting a potential obstacle, the robot may modify its behavior (e.g., pause cleaning, slow down, turn on lights to indicate the detection of an obstacle, etc.). The robot 100 may transmit information about the area 25, potential obstacles associated with the area 25, etc. to a user's mobile computing device. This information may include the location of the area 25, images of associated potential obstacles, and classifications of the potential obstacles. In some examples, the robot may transmit information about only some types of potential obstacles (e.g., small obstacles, obstacles that may cause error conditions, obstacles that can be easily removed from the floor surface 10 by the user, etc.). For example, a dining table 60 may prevent the robot 100 from cleaning a portion of the floor surface 10, but the robot 100 may transmit information about the socks 35 but not about the dining table 60.

[0029] The mobile computing device may display affordances on the display corresponding to user-selectable options for each area 25. These user-selectable options (further described in connection with FIGS. 9A-9D ) allow the user to provide feedback to the robot 100 regarding the area 25 and / or potential obstacles detected within the area 25. For example, if the user picks up the sock 35, the user may provide feedback to have the robot clean area 25A during a second cleaning mission. Alternatively, if the user is unable or unwilling to pick up the sock 35 before the start of the second cleaning mission, the user may provide feedback to have the robot 100 avoid area 25A during the second cleaning mission. In the case of more permanent obstacles, the user may provide feedback to the robot 100 to establish no-go zones. For example, if the user does not intend to move the cord 45 for a significant period of time, the user can provide feedback that establishes a no-go zone around area 25B, causing the robot 100 to avoid area 25B not only during the second cleaning mission but also during cleaning missions after the second cleaning mission. Additionally, the user can provide feedback indicating that no obstacles exist in a particular area. For example, if the robot 100 detects the cord 55 as a potential obstacle by incorrectly identifying the pattern of the cord 55 as a cord, the user can provide feedback to indicate that no obstacles actually exist in area 25C. This feedback can cause the robot 100 to clean area 25C during the second cleaning mission and can also update the object detection module 190 (shown in FIG. 4A ) of the robot 100 (e.g., to reduce false detections of potential obstacles in further cleaning missions).In some examples, a user may choose to provide data regarding each area 25, detected potential obstacles, and / or user feedback to a database that stores data from multiple users (described in further detail herein).

[0030] Referring to FIG. 1B , the robot 100 may perform a second cleaning mission after receiving feedback from the user. In some examples, the second cleaning mission can be a tidy mission. During the tidy mission, the robot 100 may clean a subset of areas 25 that were not cleaned during the first cleaning mission. For example, based on user feedback, the robot 100 may clean areas 25A and 25C while avoiding area 25B. In this example, the user provides feedback to the robot 100 indicating (i) the removal of the sock 35 from area 25A, (ii) the designation of area 25B as a no-go zone, and (iii) the absence of obstacles within area 25C. In response to receiving this user feedback, the robot 100 travels across the floor surface 10 according to the trajectory 70 during the tidy mission. The robot 100 begins the tidy mission by initiating movement from the docking station 50 toward area 25A. Sock 35 is no longer present in area 25A because the user moved it so that robot 100 may clean area 25A. After cleaning area 25A, robot 100 begins moving to area 25C. Because the user provided feedback indicating the absence of obstacles in area 25C, the robot cleans area 25C, even though it had previously identified a potential obstacle in area 25C. After cleaning area 25C, robot 100 begins moving from area 25C to docking station 50, thereby completing the tidy mission. Robot 100 does not attempt to travel to area 25B because the user provided feedback designating area 25B as a no-go zone. Furthermore, during the tidy mission, the robot cleans areas 25A and 25C rather than cleaning the entire area that robot 100 already cleaned during the first cleaning mission. As a result, the tidy mission can save time and energy when compared to a second cleaning mission in which the robot 100 attempts to clean the entire floor surface 10 .

[0031] Example of an autonomous mobile robot An example of an autonomous mobile robot is U.S. Patent Application No. 16 / 588,295, filed September 30, 2019. No. 6,299,699, the entire contents of which are incorporated herein by reference.

[0032] 2 , an autonomous mobile robot 100, such as an autonomous cleaning robot, positioned on a floor surface 10 in an environment 20 includes an image capture device 101 configured to capture an image of the environment 20. Specifically, the image capture device 101 is positioned on a forward portion of the robot 100. A field of view 103 of the image capture device 101 covers at least a portion of the floor surface 10 in front of the robot 100. The image capture device 101 can capture an image of an object located on that portion of the floor surface 10. For example, as shown in FIG. 2 , the image capture device 101 can capture an image representing at least a portion of a rug 30 located on the floor surface 10. This image can be used by the robot 100 to navigate around the environment 20, and in particular, to navigate relative to the rug 30 so as to avoid error conditions that may potentially occur as the robot 100 moves over the rug 30. For example, this image may be used as input to an object detection module 190 (shown in FIG. 4A ) that processes the image (e.g., performs image processing operations on data corresponding to the image) to identify potential obstacles located on the floor surface 10, such as clothing, electrical cords, backpacks, etc.

[0033] 3 and 4A-4B illustrate an example of a robot 100. Referring to FIG. 3, the robot 100 collects debris 105 from the floor surface 10 as it traverses the floor surface 10. The robot 100 can be used to perform one or more cleaning missions within an environment 20 (shown in FIGS. 1A-1B) to clean the floor surface 10. A user can command the robot 100 to initiate a cleaning mission. For example, a user can issue a start command that causes the robot 100 to initiate a cleaning mission upon receipt of the start command. In another example, a user can provide a schedule that causes the robot 100 to initiate a cleaning mission at the scheduled time.

[0034] 4A, the robot 100 includes a housing structure 108. The housing structure 108 may define the structural periphery of the robot 100. In some examples, the housing structure 108 includes a chassis, a cover, a bottom plate, and a bumper assembly.

[0035] The robot 100 includes a drive system 110 that includes one or more drive wheels. The drive system 110 further includes one or more electric motors that include an electric drive portion that forms part of an electrical circuit 106. A housing structure 108 supports the electrical circuit 106, including at least one controller 109, within the robot 100.

[0036] The drive system 110 is operable to propel the robot 100 across the floor surface 10. The robot 100 may be propelled in a forward drive direction F or a backward drive direction R. Additionally, the robot 100 may be propelled so that the robot 100 turns in place or turns while moving in the forward drive direction F or the backward drive direction R. In the example shown in FIG. 4A , the robot 100 includes drive wheels 112 that extend through a bottom portion 113 of the housing structure 108. The drive wheels 112 are rotated by a motor 114 to cause the robot 100 to move along the floor surface 10. The robot 100 further includes a passive swivel 115 that extends through the bottom portion 113 of the housing structure 108. The swivel 115 is not powered. The drive wheels 112 and the swivel 115 cooperate together to support the housing structure 108 above the floor surface 10. For example, swivel 115 is disposed along the rear portion of housing structure 108 and drive wheel 112 is disposed forward of swivel 115 .

[0037] 3, 4A, and 4B, robot 100 is an autonomous mobile floor-cleaning robot that includes a cleaning assembly 116 (shown in FIG. 4A) that is operable to clean floor surface 10. For example, robot 100 is a vacuum cleaning robot in which cleaning assembly 116 is operable to clean floor surface 10 by collecting debris 105 (shown in FIG. 3) from floor surface 10. Cleaning assembly 116 includes a cleaning inlet 117 through which debris collected by robot 100 passes. Cleaning assembly 116 includes one or more rotatable members driven by a drive system (e.g., rotatable member 118 driven by motor 120), and cleaning inlet 117 is positioned between the rotatable members.

[0038] The rotatable member 118 is located on a bottom portion of the robot 100 and is configured to rotate to direct debris within the robot 100, such as into the trash can 124 (shown in FIG. 3). As shown in FIG. 3, the rotatable member 118 is a counter-rotating roller. For example, the rotatable member 118 can be rotatable about parallel horizontal axes 146, 148 (shown in FIG. 4A) to agitate debris 105 on the floor surface 10 and direct the debris 105 toward and into the cleaning inlet 117 and into the suction path 145 (shown in FIG. 3) of the robot 100.

[0039] The robot 100 further includes a vacuum system 119 operable to generate an airflow through cleaning intakes 117 located between the rotatable members 118 and into the trash can 124. The vacuum system 119 includes an impeller and a motor for rotating the impeller to generate the airflow. The vacuum system 119 cooperates with the cleaning assembly 116 to draw debris 105 from the floor surface 10 and into the trash can 124. In some examples, the airflow generated by the vacuum system 119 generates sufficient force to draw debris 105 located on the floor surface 10 upward and through the gaps between the rotatable members 118 and into the trash can 124. In some examples, the rotatable members 118 contact the floor surface 10, thereby agitating the debris 105 located on the floor surface 10, thereby making the debris 105 more easily absorbed by the airflow generated by the vacuum system 119.

[0040] The robot 100 further includes a brush 126 that rotates about a non-horizontal axis, such as an axis that forms an angle between 75 and 90 degrees with the floor surface 10. The robot 100 includes a motor 128 operably coupled to the brush 126 for rotating the brush 126. The brush 126 is rotatable about the non-horizontal axis so as to brush debris located on the floor surface 10 into the cleaning path of the cleaning assembly 116 as the robot 100 moves. The brush 126 is a side brush that is offset laterally from the fore-aft axis FA of the robot 100 and forwardly from the lateral axis LA of the robot 100 such that the brush 126 extends beyond the periphery of the housing structure 108 of the robot 100. Thus, the brush 126 may be able to engage debris located on portions of the floor surface 10 that the rotatable member 118 would not typically reach, such as portions of the floor surface 10 that are located outside the portion of the floor surface 10 directly below the robot 100.

[0041] In addition to the controller 109, the electrical circuitry 106 includes, for example, a memory storage element 144 and a sensor system having one or more electrical sensors. As described herein, the sensor system is capable of generating signals corresponding to the current position of the robot 100 and may generate signals corresponding to the position of the robot 100 as the robot 100 moves along the floor surface 10. The controller 109 is configured to execute instructions to perform one or more operations as described herein. The memory storage element 144 is accessible by the controller 109 and is disposed within the housing structure 108.

[0042] The one or more electrical sensors may be configured to detect features located within the environment 20 of the robot 100. For example, with reference to FIG. 4A , the sensor system includes a cliff sensor 134 that can detect an obstacle, such as a drop-off or cliff, located below the portion of the robot 100 where the cliff sensor 134 is disposed and redirect the robot accordingly. With reference to FIG. 4B , the sensor system further includes one or more proximal sensors (e.g., proximal sensors 136 a, 136 b) that can detect the presence or absence of an object along the floor surface 10 located near the robot 100. The sensor system further includes a bumper system that includes a bumper 138 and one or more bump sensors (e.g., bump sensors 139 a, 139 b) that detect contact between the bumper 138 and an obstacle located within the environment 20. The sensor system further includes one or more obstacle-following sensors (e.g., obstacle-following sensor 141) that can detect the presence or absence of objects located adjacent to the side surfaces of the housing structure 108. For example, detectable objects can include obstacles such as furniture, walls, people, and other objects located within the environment 20 of the robot 100.

[0043] The sensor system may further include an image capture device 101 (shown in FIG. 4B ). The image capture device 101 is positioned on a front portion of the robot 100 and oriented to capture an image of at least a portion of the floor surface 10 in front of the robot 100. In particular, the image capture device 101 may be oriented toward a forward direction F (shown in FIG. 4A ) of the robot 100. The image capture device 101 may be, for example, a camera or an optical sensor. With reference to FIG. 2 , the field of view 103 of the image capture device 101 extends laterally and vertically. The image may represent a portion of the floor surface 10 and other portions of the environment 20 above the floor surface 10. For example, the image may represent portions of wall surfaces and obstacles located in the environment 20 above the floor surface 10. As described herein, the image capture device 101 may generate images for generating a representation of a map of the environment 20 and for controlling navigation of the robot 100 around obstacles.

[0044] The image capture device 101 may also be an obstacle detection sensor, in which case the image captured by the image capture device 101 is used as input to an object detection module 190 (shown in FIG. 4A ) to detect obstacles (e.g., using image analysis techniques) including clothing, electrical cords, backpacks, etc. In some implementations, the object detection module 190 may further receive as input data from other sensors of the robot 100, such as the cliff sensor 134, the proximity sensors 136 a, 136 b, the bump sensors 139 a, 139 b, the obstacle-following sensor 141, etc., to detect obstacles in the environment 20.

[0045] The sensor system may further include additional obstacle detection sensors. For example, active detection techniques (e.g., LIDAR, RADAR, ultrasonic, etc.), passive detection techniques, etc. may be used in place of or in combination with the image capture device 101 to detect potential obstacles in the environment 20 and determine the distance of the potential obstacles from the robot 100. In some implementations, the robot 100 may include lighting that is used in conjunction with the image capture device 101 to determine the distance of obstacles from the robot 100 (e.g., based on shading, reflectivity, etc.).

[0046] The sensor system may further include one or more sensors for detecting error conditions in the robot. For example, one or more current sensors may generate sensor data regarding the current supplied to various components of the robot 100, including the drive wheels 112, the cleaning assembly 116, or the brushes 126. This sensor data may be used to detect when the robot 100 is stuck or when the cleaning assembly 116 is clogged. In some examples, the sensor data may be used to identify a component of the robot 100 (e.g., the drive wheels 112 or the brushes 126) to replace.

[0047] The sensor system may further include sensors (e.g., motor encoders, optical sensors, etc.) for tracking the distance traveled by the robot 100. The controller 109 uses the data collected by the sensors in the sensor system to control the traveling behavior of the robot 100 during a mission. For example, the sensor data may be used by the controller 109 for simultaneous localization and mapping (SLAM) techniques, in which the controller 109 extracts features of the environment 20 indicated by the sensor data and builds a map of the floor surface 10 of the environment 20. The map formed from the sensor data may indicate the location of accessible and unaccessible spaces within the environment 20. For example, the location of obstacles may be indicated on the map as unaccessible spaces, and the location of open floor spaces may be indicated on the map as accessible spaces.

[0048] Sensor data collected by any of these sensors may be stored in the memory storage element 144. Additionally, other data generated for SLAM techniques, including mapping data, may be stored in the memory storage element 144. This data generated during a mission may include persistent data generated during a mission and available for use during additional missions. For example, the mission may be a first mission, and the additional mission may be a second mission performed after the first mission. In addition to storing software for causing the robot 100 to perform behaviors, the memory storage element 144 also stores sensor data for access by the controller 109 from one mission to another, or data resulting from processing the sensor data. For example, a map may be a persistent map available and updatable by the controller 109 of the robot 100 from one mission to another to navigate the robot 100 across the floor surface 10.

[0049] The persistent data, including the persistent map, enables the robot 100 to efficiently clean the floor surface 10. For example, the persistent map enables the controller 109 to direct the robot 100 toward open floor spaces and avoid impassable spaces. Additionally, for subsequent missions, the controller 109 can utilize the persistent map to plan the robot 100's travel through the environment 20 to optimize the path traversed during the mission.

[0050] The robot 100 may further include a wireless transceiver 149 (shown in FIG. 4A ), which allows the robot 100 to wirelessly communicate data using a communication network. The robot 100 may use the wireless transceiver 149 to send and receive data, such as receive data shown on a map and transmit data corresponding to mapping data collected by the robot 100.

[0051] When the controller 109 causes the robot 100 to perform a mission, the controller 109 operates the motors 114 to drive the drive wheels 112 and propel the robot 100 along the floor surface 10. Additionally, the controller 109 operates the motors 120 to rotate the rotatable member 118, the motors 128 to rotate the brushes 126, and the motors of the vacuum system 119 to generate airflow. The controller 109 executes software stored on the memory storage element 144 to cause the robot 100 to perform various driving and cleaning behaviors. Execution of this software operates the various motors of the robot 100 to cause the robot 100 to perform these behaviors.

[0052] Process Example 5 shows a flow diagram of an example process 500 for providing user feedback regarding obstacles detected by the autonomous mobile robot 100. The process 500 includes operations 502, 504, 506, 508, 510, 512, 514, 516, and 518.

[0053] Operations 502 and 504 involve the actions of the robot 100 as it navigates through an environment (e.g., environment 20 shown in FIGS. 1A-1B). At operation 502, the robot 100 begins navigating the environment. For example, the robot 100 may begin navigating the environment in response to receiving a command to begin a cleaning mission. At operation 504, the robot 100 collects sensor data. This sensor data may be collected by the sensor systems of the robot 100 and may be used for various purposes, including generating mapping data of the environment and detecting potential obstacles as the robot 100 navigates through the environment. As the robot 100 navigates through the environment, it may clean floor surfaces if operations 502 and 504 are performed as part of a cleaning mission. In some examples, the robot 100 may navigate through floor surfaces without cleaning.

[0054] Acts 506, 508, and 510 involve displaying to user 80 multiple user-selectable options related to a potential obstacle or the location of a potential obstacle in the environment. In some implementations, each user-selectable option may be displayed as an affordance (e.g., a button, an interactive visual element, etc.) corresponding to the option. Acts 506 and 508 are performed by computing system 90, which may be a controller located on robot 100, a controller located on mobile computing device 85, a remote computing system, a distributed computing system including processors located on multiple devices (e.g., robot 100, mobile device 85, or remote computing system), a processor on an autonomous mobile robot added to robot 100, or a combination of these computing devices. In some examples, one or more of the acts performed by computing system 90 (e.g., acts 506, 508, and 516) may each be performed by a different actor. For example, detecting a potential obstacle using sensor data (operation 506) can be performed on a remote server, while planning the robot's behavior relative to the potential obstacle (operation 516) can be performed on the robot 100.

[0055] At operation 506, the sensor data is utilized to detect potential obstacles located within the environment. For example, images captured by the image capture device 101 of the robot 100 may be analyzed using the object detection module 190 (e.g., using executable instructions of an object detection algorithm) to detect potential obstacles located in the vicinity of the robot 100. At operation 508, the computing system 90 generates data representative of potential obstacles detected in an area within the environment. For example, this data may include a portion of the image captured by the image capture device 101 (e.g., a single image, a series of images, etc., acquired at the robot 100's location when the robot 100 detects a potential obstacle), the location of the obstacle or area, and / or a label indicating the classification of the potential obstacle (e.g., a sock, a cord, a backpack, etc.). At operation 510, the mobile computing device 85 displays affordances to the user 80 corresponding to multiple user-selectable options associated with the potential obstacle or area.

[0056] These user-selectable options allow the user 80 to provide user feedback to the robot 100 regarding potential obstacles and allow the robot 100 to navigate to potential obstacles or areas in a particular manner. The user-selectable options may modify various cleaning parameters of the robot 100. For example, some user-selectable options may cause the robot 100 to perform a particular action, such as cleaning an area, avoiding an area, adjusting the robot's cleaning power in an area, etc. Some user-selectable options may modify the mission (e.g., the next mission, all missions after, missions after a specified number of times, all missions after within a day, a week, a month, etc.) in which the robot 100 performs a particular action. Some user-selectable options allow the robot 100 to include or exclude certain areas in the tidy mission, some options allow modifying the scheduling of the tidy mission (e.g., now, 30 minutes from now, 1 hour from now, 6 hours from now, 12 hours from now, etc.), and some options allow modifying the length of the tidy mission (e.g., 5 minute maximum length, 15 minute maximum length, 30 minute maximum length, 1 hour maximum length, etc.).

[0057] In some implementations, the user-selectable options may include a first option to navigate the robot 100 to clean the area and a second option to navigate the robot 100 to avoid the area. In some examples, multiple options may be displayed for having the robot clean or avoid the area. For example, a first option may cause the robot to avoid the area only during the second cleaning mission, while a second option may cause the robot to avoid the area during the second cleaning mission and in one or more subsequent cleaning missions. The multiple user-selectable options may also further include an option to identify an absence of obstacles in the area (e.g., when a potential obstacle detected by the robot 100 is not actually an obstacle or does not exist). In some examples, the option to identify an absence of obstacles in the area may cause the robot 100 to clean the area. In some examples, the option to identify the absence of obstacles in this area may cause the robot 100 to update the object detection module 190 (e.g., a module utilizing a convolutional neural network) implemented by the computing system 90 to reduce the likelihood of false detection of potential obstacles in subsequent cleaning missions.

[0058] At operation 512, the user 80 makes a user selection of one of the plurality of user-selectable options. For example, based on reviewing information about the potential obstacle (e.g., one or more images of the potential obstacle, the location of the potential obstacle, a label indicating the classification of the potential obstacle, etc.), the user 80 may select one of the user-selectable options to provide feedback to the robot 100 about the potential obstacle (e.g., by interacting with a corresponding affordance). In some examples, the information about the potential obstacle is displayed on the mobile device 85 along with the user-selectable options. At operation 514, the mobile device 85 may display one or more visual elements on the display corresponding to the user selection. For example, the selected option may be shaded, outlined, resized, etc. to indicate that it has been selected.

[0059] At operation 516, the computing system 90 plans a behavior for the robot 100 relative to the potential obstacle or area based on the user selection. As previously described, this planned behavior may include clearing the area or avoiding the area. Planning the behavior of the robot 100 relative to the potential obstacle may also include planning a path or trajectory for the robot to and from the area. In examples where multiple potential obstacles are detected, the planned behavior of the robot 100 may include planning a path or trajectory for the robot 100 between multiple areas relative to multiple potential obstacles. At operation 518, the robot 100 navigates relative to the obstacle or area (e.g., according to the planned robot behavior).

[0060] In some implementations, multiple potential obstacles may be detected in the environment, and process 500 may be performed multiple times (e.g., once for each potential obstacle). In some examples, a user may provide feedback for each potential obstacle simultaneously as it is detected during a mission, providing real-time feedback. In other examples, a user may review multiple potential obstacles at once (e.g., after completing a first mission), and user selection for each potential obstacle may be used to plan the robot's behavior relative to the potential obstacle or area during a second mission (e.g., a tidy mission).

[0061] FIG. 6 shows a flow diagram of a process 600 for planning a tidying mission. Operations 602 and 604 involve the robot 100 as it navigates through an environment (e.g., environment 20 shown in FIGS. 1A-1B). At operation 602, the robot 100 performs a first mission (e.g., a cleaning mission or a training mission). At operation 604, the robot 100 collects sensor data during the first mission. This sensor data may be collected by the sensor systems of the robot 100 and may be used for various purposes, including generating mapping data of the environment and detecting potential obstacles as the robot 100 navigates through the environment. The robot 100 may clean floor surfaces as it navigates through the environment. In some examples, the robot 100 may navigate through floor surfaces without cleaning.

[0062] Acts 606, 608, and 610 involve generating information about one or more areas in the environment where potential obstacles have been detected and displaying this information to user 80. Acts 606 and 608 are performed by computing system 90, which may be a controller located on robot 100, a controller located on mobile computing device 85, a remote computing system, a distributed computing system including processors located on multiple devices (e.g., robot 100, mobile device 85, or remote computing system), a processor on an autonomous mobile robot added to robot 100, or a combination of these computing devices. In some examples, one or more of the acts performed by computing system 90 (e.g., acts 606, 608, and 616) may be performed by different actors. For example, detecting a potential obstacle (act 606) may be performed on a remote server, while planning robot behavior relative to the potential obstacle (act 616) may be performed on robot 100.

[0063] At operation 606, the sensor data is used to detect one or more potential obstacles located within one or more areas on a floor surface (e.g., floor surface 10 shown in FIGS. 1A-1B) during the first mission. For example, images captured by the image capture device 101 of the robot 100 may be analyzed utilizing the object detection module 190 (e.g., using executable instructions associated with an object detection algorithm) to detect potential obstacles located in the vicinity of the robot 100. At operation 608, the computing system 90 generates data corresponding to the detected potential obstacles and one or more areas. For example, this data may include a portion of the image captured by the image capture device 101 (e.g., a single image, a series of images, a video clip, etc., acquired at the location of the robot 100 when the robot 100 detects the potential obstacle), the location of the obstacle or area, and / or a label indicating the classification of the potential obstacle (e.g., a sock, a cord, a backpack, etc.). At operation 610, the mobile computing device 85 displays information regarding the one or more areas to the user 80 on a display. In some implementations, the displayed information may include an image of the potential obstacle, a location of the potential obstacle or area, and / or a label indicating a classification of the potential obstacle. In some implementations, affordances (such as those described in connection with FIG. 5) corresponding to multiple user-selectable options related to the potential obstacle may further be displayed to the user 80.

[0064] At operation 612, the user 80 provides input of a user-selected subset of one or more areas. For example, the subset of one or more areas may include areas that the user 80 intends the robot 100 to clean. As also described in connection with FIG. 5 , in some implementations, the user selects the subset of one or more areas by interacting with an interface displayed by the mobile device 85 (e.g., options are graphically presented for selection on the display). For example, in some implementations, any areas among the one or more areas for which the user selects an option for the robot 100 to clean may be included within this subset.

[0065] At operation 614, the mobile device 85, after receiving input from the user 80, displays one or more visual elements corresponding to the user-selected subset of one or more areas. For example, the one or more visual elements may be displayed in a user interface for initiating a tidying mission (an example of which is shown in FIG. 10 ). The one or more visual elements may include a list of the areas included in the user-selected subset, images associated with the areas included in the user-selected subset, a map depicting the locations of the areas included in the user-selected subset, etc.

[0066] At operation 616, computing system 90 plans a second mission to clean a user-selected subset of one or more areas. Planning this second mission may include planning a path or trajectory for robot 100 within the environment so that the robot travels to each of the areas included in the user-selected subset. At operation 618, the robot performs the second mission.

[0067] In some implementations, the first mission and the second mission are consecutive missions. In some implementations, the second mission may be initiated a period of time (e.g., 1 hour, 6 hours, 12 hours, 24 hours, 48 ​​hours, etc.) after the completion of the first mission. In some implementations, during or after the completion of the second mission, the robot 100 may transmit data corresponding to an updated status of a user-selected subset of one or more areas to the mobile device 85. In some implementations, the mobile computing device 85 may display at least a portion of this data on a display (described in further detail, e.g., in connection with FIGS. 11A-11B).

[0068] 7-10 are front views of a mobile computing device 85 illustrating an example user interface (IU) for providing user feedback regarding obstacles detected by the autonomous mobile robot 100 (as described in connection with FIG. 5) and planning tidying missions (as described in connection with FIG. 6). Referring to FIG. 7, the mobile computing device 85 (sometimes simply referred to as a "mobile device") includes a display 700. After the robot 100 completes a cleaning mission, a push notification 710 may be sent to the mobile device 85 and displayed on the display 700. This push notification 710 may include a message stating that the cleaning mission by the robot 100 (which in this example has the name "Rosie") has been completed, but that some areas were not cleaned during the cleaning mission. The push notification 710 may prompt the user to review these areas for future cleaning. In some implementations, reviewing these areas includes providing user feedback regarding these areas and potential obstacles detected by the robot 100 within these areas. In some implementations, reviewing these areas can include planning a tidying mission.

[0069] 8A , after receiving a push notification 710, a user 80 can interact with a screen displayed on the display 700 (e.g., by interacting with the push notification, by opening a mobile application, etc.) to view obstacles detected by the robot 100. The display 700 may include a timestamp 822 containing the date and time the cleaning mission began or the date and time the cleaning mission was completed. The display 700 may also include a map 802 of the environment in which the robot 100 performed the cleaning mission. In this example, the environment is a home (not shown) that includes a study, dining room, kitchen, living room, and bedroom. The display 700 also includes control options for interacting with the map 802, including a zoom in option 806A, a zoom out option 806B, and a rotate map option 808 to rotate the map (e.g., 90 degrees clockwise). Indicators 804A, 804B, 804C, and 804D (collectively referred to as indicators 804) are displayed on the map 802 to indicate the location of potential obstacles detected by the robot 100 during a cleaning mission. In FIG. 8A , indicator 804A is selected as indicated by a change in color and an increase in size. While indicator 804 is shown as an exclamation point, this is not intended to be limiting and other symbols can be used. For example, in some implementations, indicator 804 may include a symbol corresponding to the type of potential obstacle (e.g., a sock, a shoe, a cord, etc.).

[0070] The display 700 also includes a cleaning report 810 that shows additional information about one or more potential obstacles detected by the robot 100. The cleaning report 810 may include a summary statement 812 that provides information about the total number of potential obstacles detected in the environment during the cleaning mission (e.g., five obstacles in this example). For each potential obstacle, the cleaning report 810 may include an image of the potential obstacle (e.g., images 814A, 814B), a label indicating the classification of the potential obstacle (e.g., labels 816A, 816B), and the area in which the potential obstacle was detected (e.g., areas 818A, 818B). Referring now to FIG. 8D , in some implementations, the cleaning report 810 can be a text report. The cleaning report 810 may include a mission title 830, which may specify the environment 20 in which the robot 100 is located (e.g., "Main Floor") or the type of mission performed by the robot 100 (e.g., "Cleaning Mission," "Training Mission," etc.). In some examples, the title 830 may be edited and customized by the user 80. The cleaning report 810 may also include various entries (e.g., entry 832, entry 840) listed in chronological order to generate a mission timeline. Entry 832 is an example of an entry for a particular area within an environment (e.g., environment 20) and includes a status indicator 834, an area name 836, and time spent in the area 838. In this example, entry 832 indicates that the kitchen was successfully cleaned in 24 minutes. Entry 840 is an example of an entry for a potential obstacle detected within the environment and includes a status indicator 842 (which may be different from the status indicator 834), a message 844, and the time the potential obstacle was encountered 846. In some implementations, message 844 communicates that a potential obstacle has been detected and, in some implementations, may include information about the obstacle, such as the classification and location of the potential obstacle (e.g., "socks in the dining room").In this example, entry 840 indicates that a sock was detected in the dining room at 9:42 a.m. In some implementations, entry 840 can include a user-selectable affordance 890 that, when selected, causes robot 100 to clean an area associated with the location of the potential obstacle during a later mission (e.g., a tidying mission). In some implementations, an image associated with the potential obstacle can be included in cleaning report 810 (e.g., as part of entry 840, next to entry 840, at the bottom of cleaning report 810, etc.). In some implementations, entry 840 can be selected by user 80 (e.g., by touching entry 840, by touching and holding entry 840, by hovering a cursor over entry 840, etc.) to view an image associated with the potential obstacle and / or further information associated with the potential obstacle, including a map showing the location of the potential obstacle in the environment. In some implementations, the cleaning report 810 can further include a summary entry 895, which may include, for example, the date and time of the mission, the duration of the mission, and / or the time it took to complete the mission, etc. In some implementations, only a portion of the descriptive information may be displayed in the cleaning report 810, and in some implementations, more information may be displayed.

[0071] In some implementations, the cleaning report 810 may be displayed before the user 80 reviews further information about each potential obstacle (as described in further detail in connection with FIGS. 9A-9D ). In one example of such an implementation, when the user selects affordance 890 from the cleaning report 810, further review of the potential obstacles may not be necessary to begin a later mission (e.g., a tidy mission). In another example, selecting affordance 890 may immediately return the robot 100 to the area associated with the potential obstacle without requiring feedback about other potential obstacles. In some implementations, the cleaning report 810 is displayed in real time as the mission is being performed, and selecting affordance 890 may return the robot 100 to the area associated with the potential obstacle during the same mission.

[0072] In some implementations, the cleaning report 810 may be displayed after the user 80 has already reviewed further information about each potential obstacle and provided feedback related to the potential obstacle. In such an example, a user selection of an affordance 890 from the cleaning report 810 may adjust or override previously provided feedback about the potential obstacle. For example, after the user has reviewed the potential obstacles, the cleaning report 810 may function as a user interface for planning a tidy mission. An example user interface for planning a tidy mission is described in further detail herein with respect to FIG. 10 .

[0073] 8A , selecting indicator 804A causes information about the potential obstacle associated with indicator 804A to be displayed in a main location within cleaning report 810. In this example, the potential obstacle associated with indicator 804A is a cord located in the study. In some implementations, multiple potential obstacles are displayed simultaneously within cleaning report 810. For example, the potential obstacles may be displayed in an order corresponding to the chronological order in which they were detected by robot 100. In this example, the next potential obstacle detected by robot 100 was a group of socks located in the dining room, as indicated by image 814B, label 816B, and area 818B.

[0074] To review information about the remaining detected potential obstacles, the user 80 can select one of the indicators 804 to view the potential obstacle corresponding to that indicator. Alternatively, the user 80 can scroll through each potential obstacle detected by the robot 100 (e.g., by using a left-to-right or right-to-left gesture on the display 700). For example, referring to FIG. 8B , when the user selects indicator 804B, in response, information about the potential obstacle associated with indicator 804B (e.g., a sock located in the dining room) is displayed in a main location within the cleaning report 810. In this example, the next potential obstacle detected after the sock located in the dining room was clothing located in the bedroom, as indicated by image 814C, label 816C, and area 818C. In this example, the map 802 is partially obscured by the cleaning report 810, so the bedroom is not visible to the user 80. In some implementations, the user 80 can select a minimize option 824 to minimize the cleaning report 810 to view a larger portion of the map 802.

[0075] 8C , the user can scroll through cleaning report 810 to review the remaining potential obstacles detected by robot 100 during the cleaning mission. In this example, the user is scrolling through these potential obstacles, with image 814D, label 816D, and area 818D (corresponding to the backpack located in the living room) positioned in the main location of cleaning report 810. In response to the user scrolling, indicator 804C corresponding to the backpack located in the living room is displayed by changing color and increasing in size, as if it had been directly selected by user 80. In this example, the next potential obstacle detected after the backpack located in the living room was a cord located in the living room, as indicated by image 814E, label 816E, and area 818E.

[0076] At any time while reviewing a potential obstacle detected by the robot 100, the user 80 can select option 820 to begin providing user feedback regarding the potential obstacle. After selecting option 820, a series of images for providing feedback regarding the potential obstacle is displayed to the user 80 on the display 700. FIGS. 9A-9D show examples of such images. Referring to FIG. 9A, a user interface for providing user feedback regarding the potential obstacle includes information about the potential obstacle, including an image 902A of the potential obstacle, a label 908A indicating the classification (e.g., code 904) of the potential obstacle, and an area 910A (e.g., study) of the potential obstacle. In some implementations, image 902A may be a single image captured at the location of the robot 100 (e.g., using image capture device 101 shown in FIG. 2) when the robot detects the potential obstacle. In some implementations, a series of images may be displayed, the series of images being captured near the location of the potential obstacle. In some implementations, a bounding box 906A may be displayed along with the image 902A to highlight particular portions of the image 902A where potential obstacles have been detected (e.g., by an object detection module 190 utilizing a convolutional neural network). In some implementations, areas of the image 902A that are not included within the bounding box 906A may be dimmed to further highlight particular portions of the image 902A where potential obstacles have been detected. In some implementations, the user interface for providing user feedback regarding potential obstacles may include further information, such as a confidence score or accuracy metric associated with the label 908A.

[0077] In addition to information about the potential obstacle (e.g., code 904), display 700 may include multiple user-selectable affordances 912A, 912B, 912C, 912D (collectively referred to as user-selectable affordances 912) associated with the potential obstacle or an area in the environment in which the potential obstacle was detected. In some implementations, each affordance 912 may correspond to one user-selectable option. For example, affordance 912A, when selected by user 80, may cause robot 100 to clean the area during an immediately following mission (e.g., a tidy mission) and / or during other further missions (e.g., future scheduled missions) (because the obstacle was removed by user 80). Affordance 912B may correspond to the absence of an obstacle in the area (e.g., because user 80 determined that the potential obstacle detected by robot 100 was not an obstacle and / or that no obstacles exist in the area). Selecting affordance 912B may allow robot 100 to clean the area during the immediately following mission and / or during other additional missions, and may also allow robot 100 to update object detection module 190 to reduce the likelihood of false detection of potential obstacles in later cleaning missions. Affordance 912C may cause robot 100 to avoid the area during the immediately following cleaning mission (e.g., during a tidy mission) but not during other cleaning missions. Affordance 912D may cause robot 100 to avoid the area during the immediately following cleaning mission (e.g., during a tidy mission) and other cleaning missions. For example, selecting affordance 912D may establish a no-go zone around the area, preventing robot 100 from cleaning the area until the no-go zone is lifted or edited by user 80.When a user selects an affordance from among the plurality of user-selectable affordances 912, a visual element corresponding to the user selection may be displayed on the display 700. For example, the selected affordance may be displayed with a hatched background, a colored background, bold text, a bold outline (as shown by affordance 912B in FIG. 9D), a colored outline, etc. Although the affordances 912 are illustrated with particular icons, this is not intended to be limiting and other icons and / or symbols may be used.

[0078] 9A , the display 700 may further include an affordance 930 corresponding to a user-selectable provision option. When the user selects this affordance 930 (as indicated by the display of a checkmark icon), the mobile device 85 can transmit information about the potential obstacle (e.g., data corresponding to the image 902A, the label 908A, and / or the area 910A) and information about the user selection associated with the potential obstacle (e.g., one of the user-selectable affordances 912) to a database that stores data from multiple users. In some implementations, this data can be used to improve the object detection module 190 of various autonomous mobile robots owned by different users. In some implementations, this data can be used for the development of new autonomous mobile robots. In some examples, the default setting for the affordance 930 is to not transmit any information to the database. In some implementations, the user can control what information is transmitted to the database (e.g., transmitting information corresponding to the label 908A and the user selection, but not transmitting information corresponding to the image 902A). In some implementations, images containing people or human faces may be automatically filtered and prevented from being sent to the database. In some implementations, a user may crop image 902A (e.g., by reshaping bounding box 906A) to remove sensitive data before sending the information to the database. In some implementations, display 700 may include user-selectable options 914 that, when selected, may present user 80 with further information regarding the data to be sent to the database and the intended use of the data.

[0079] The display 700 may further include one or more visual elements corresponding to the number of potential obstacles detected during the cleaning mission and the potential obstacle currently being reviewed. For example, in FIG. 9A , visual elements 916, 990 indicate that five potential obstacles are available for review by the user 80 and that the user 80 is currently reviewing the first of the five potential obstacles. In visual element 916, this is indicated by the first of the five circles being filled in. In visual element 990, this is indicated by displaying a number corresponding to the potential obstacle currently being reviewed (in this example, “1”) and a number corresponding to the total number of potential obstacles available for review (in this example, “5”). In some implementations, the user 80 can scroll through these potential obstacles for review by utilizing a left-to-right or right-to-left swipe motion on the display 700.

[0080] In FIG. 9A , cord 904 has been detected by robot 100 as a potential obstacle. If the cord is unlikely to be removed before the start of the next cleaning mission, user 80 can cause the robot to avoid cord 904 by selecting affordance 912C or affordance 912D. When affordance 912C is selected, robot 100 avoids cord 904 during the immediately following cleaning mission (e.g., the tidy mission), but not during any further cleaning missions thereafter. However, when affordance 912D is selected, robot 100 avoids cord 904 during both the immediately following cleaning mission (e.g., the tidy mission) and any further cleaning missions thereafter. Thus, if user 80 does not anticipate removing cord 904 for a significant period of time, user 80 may choose to select affordance 912D rather than affordance 912C. After selecting an affordance from among multiple user-selectable affordances 912, user 80 may provide feedback regarding other potential obstacles detected by robot 100.

[0081] 9B , a group of socks 918 has been detected by the robot 100 as a potential obstacle. The display 700 in FIG. 9B is similar to the display 700 in FIG. 9A , but displays a different image 902B, a different bounding box 906B, a different label 908B, and a different area 910B associated with the potential obstacle. In this example, the bounding box 906B is displayed along with the image 902B to highlight the portion of the image 902B in which the group of socks 918 was detected. If the user is able to remove the group of socks 918 before the start of the next cleaning mission, the user can select affordance 912A to have the robot 100 clean the area associated with the socks 918 during a later cleaning mission (e.g., during a tidy mission). Alternatively, if the user is unable or unwilling to remove the group of socks 918 before the start of the next cleaning mission, the user can select affordance 912C to have the robot avoid the area associated with the socks 918 in the immediately following cleaning mission (e.g., a tidy mission). In this example, if the user 80 intends to eventually pick up the socks 918, the user 80 may decide not to select affordance 912D because these socks 918 are only temporary obstacles located in the environment. After selecting an affordance from among the user-selectable affordances 912, the user 80 can provide feedback regarding other potential obstacles detected by the robot 100.

[0082] 9C , a shadow 920 on a floor surface 950 is detected by the robot 100 as a potential obstacle. The display 700 in FIG. 9C is similar to the display 700 in FIGS. 9A and 9B , but displays a different image 902C, a different bounding box 906C, a different label 908C, and a different area 910C associated with the potential obstacle. In this example, the bounding box 906C is displayed along with the image 902C to highlight the portion of the image 902C where the potential obstacle was detected. In this example, the robot 100 incorrectly recognizes the shadow 920 located on the floor surface 950 as a code (as indicated by the label 908C). After reviewing the information displayed on the display 700, the user 80 can specify the absence of an obstacle in this area by selecting affordance 912B. Selecting affordance 912B may cause the robot 100 to update the object detection module 190 to reduce false detections of subsequent potential obstacles and clean this area during a subsequent cleaning mission (e.g., a tidy mission). After selecting an affordance from among the user-selectable affordances 912, the user 80 may provide feedback regarding other potential obstacles detected by the robot 100. Referring to FIG. 9D , if the shadow 920 is the last potential obstacle requiring user feedback, affordance 924 may be displayed on the display to allow the user 80 to complete review of the potential obstacles detected by the robot 100. If the user 80 has not completed reviewing the potential obstacles detected by the robot 100, the user 80 may scroll through the potential obstacles and edit their previous user selection.

[0083] Upon selecting affordance 924, a screen may be displayed on display 700 to present a user interface for planning a tidy mission. Referring to FIG. 10 , the user interface for planning a tidy mission may include a “Re-Clean” portion 1000A that includes visual elements corresponding to a user-selected subset of areas to be cleaned during the tidy mission. For example, these visual elements may include images of areas, such as image 1002A, image 1002B, and image 1002C (collectively referred to as images 1002). The user-selected subset of areas may correspond to areas for which the user 80 previously selected an affordance to have the robot 100 clean while reviewing information about corresponding potential obstacles. The images 1002 may each include a selection indicator, such as a check mark, to allow the user 80 to confirm whether the area associated with each image should be cleaned during the tidy mission. For example, image 1002A may include a selection indicator 1004A to confirm that user 80 wants a sock-related area located in the dining room to be cleaned during the tidy mission. Image 1002B may include a selection indicator 1004B to confirm that user 80 wants a clothing-related area located in the bedroom to be cleaned during the tidy mission. Image 1002C may include a selection indicator 1004C to confirm that user 80 wants an incorrectly recognized code-related area located in the living room to be cleaned during the tidy mission. If user 80 does not want to include a re-clean area during the tidy mission, user 80 can touch the corresponding image in "Re-Clean" section 1000A to toggle the selection indicator and deselect the image.

[0084] The user interface for planning a tidying mission may further include an “Avoid This Time” portion 1000B and / or an “Avoid Always” portion (not shown) on the display 700. Each portion may include visual elements corresponding to a subset of areas based on a previous user selection for each area from the plurality of user-selectable affordances 912 (shown in FIGS. 9A-9D ). The user 80 can scroll through the “Clean Again” portion 1000A, the “Avoid This Time” portion 1000B, and the “Always Avoid” portion (not shown) by performing a swipe motion from top to bottom or bottom to top on the display 700 to review the contents of each portion. In some implementations, the user 80 can drag and drop visual elements (e.g., image 1002A) from one portion on the display 700 to another portion on the display 700 (e.g., from the “Clean Again” portion 1000A to the “Avoid This Time” portion 1000B). In some implementations, the user interface for planning the tidy mission may further allow the user 80 to provide cleaning instructions for areas to be included in the re-cleaning in the tidy mission.

[0085] The user interface for planning the tidy mission may further include a message 1010 and corresponding affordances 1006, 1008 to prompt the user 80 to set a start time for the tidy mission. If the user selects affordance 1008, the robot 100 immediately begins the tidy mission. Alternatively, if the user 80 selects affordance 1006, a user interface for scheduling a later tidy mission is displayed to the user 80.

[0086] The robot 100 plans and executes the tidy mission based on user input. Once the tidy mission is completed, the mobile computing device 85 may display a user interface on the display 700 for reviewing the completed tidy mission. Referring to FIG. 11A , the user interface for reviewing the completed tidy mission may include a timestamp 1122 on the display 700, which includes the date and time the tidy mission began or the date and time the tidy mission was completed. The display 700 may also include a map 1102 of the environment in which the robot 100 executed the tidy mission. In this example, the environment is a home (not shown) that includes a study, dining room, kitchen, living room, and bedroom. The display 700 also includes control options for interacting with the map 1102, including a zoom in option 1106A, a zoom out option 1106B, and a rotate map option 1108 for rotating the map (e.g., 90 degrees clockwise). Indicators 1104A, 1104B, 1104C, 1104D, and 1104E (collectively referred to as indicators 1104) are displayed on the map 1102. Indicator 1104A is a shaded area indicating that the user 80 has instructed the robot 100 to avoid the corresponding area of ​​the environment. For example, the user may have assigned a no-go zone around a previously detected code located in a study. Indicators 1104B, 1104C, and 1104D are check marks indicating areas previously avoided by the robot 100 during a first cleaning mission but successfully cleaned by the robot 100 during a tidying mission. For example, these areas may include areas where the user 80 removed a potential obstacle (e.g., a sock in the dining room) or areas where the user 80 provided feedback specifying the absence of an obstacle (e.g., the absence of a falsely detected code in the living room). During the tidy mission, the robot 100 may encounter one or more new potential obstacles that were not detected during the first cleaning mission.For example, indicator 1104E is an exclamation mark indicating a newly detected potential obstacle (e.g., a cord located in the kitchen). In FIG. 11A , selection of indicator 1104E is indicated by a change in color and an increase in size of indicator 1104E. Display 700 may further include on map 1102 a path 1128 for robot 100 to traverse during the tidy mission. In some implementations, path 1128 for robot 100 to traverse during the tidy mission may be planned to avoid robot 100 covering all areas included in the tidy mission in a time- and energy-efficient manner. In some implementations, user 80 may provide robot 100 with a cleaning order for the areas included in the tidy mission, which may assist in planning path 1128.

[0087] The user interface for reviewing completed tidy missions further includes a cleaning report 1110 on the display 700, which displays additional information about any new potential obstacles detected by the robot 100. Similar to the cleaning report 810 (shown in FIG. 8A ), the cleaning report 1110 can include a summary statement 1112 that provides information about the total number of new potential obstacles detected in the environment during the tidy mission (e.g., 1 new obstacle in this example). For each potential obstacle, the cleaning report 1110 may include an image of the potential obstacle (e.g., image 1114), a label indicating the classification of the potential obstacle (e.g., label 1116), and the area in which the potential obstacle was detected (e.g., area 1118). In some implementations, a minimize option 1124 can be selected by the user 80 to minimize the cleaning report 1110 to view a larger portion of the map 1102.

[0088] 11B , in some implementations, the cleaning report 1110 can be a text report. The cleaning report 1110 can include a portion 1130 corresponding to a first mission performed by the robot 100. For example, this portion 1130 can be substantially similar to the text cleaning report 810 described in connection with FIG. 8D . The cleaning report 1110 can further include a portion 1142 corresponding to completed tidy missions. In this example, the portion 1142 includes entries 1132, 1134 listed in chronological order to create a mission timeline. Using the entries 1134 as an example, each entry in the portion 1142 can include a status indicator 1136, an area 1138, and time spent 1140 by the robot 100 within the area. In this example, the robot 100 completes the tidy mission by re-cleaning the living room (i.e., where the code was detected during the first mission) for five minutes and the dining room (i.e., where the sock was detected during the first mission) for seven minutes. In some implementations, the time spent by the robot 100 in an area during the tidy mission can be less than the elapsed time in the area during the first mission because the tidy mission only cleans a portion of the area (e.g., only the portion of the living room where the code was last detected, only the portion of the dining room where the sock was last detected, etc.). In some implementations, each entry in the cleaning report 1110 (e.g., entry 1134) can be selected by the user 80 to see more information about the entry (e.g., a user-selection affordance that causes the robot 100 to perform a cleaning task, a map displaying the location of the area, etc.). In some implementations, only a portion of the information described herein may be displayed in the cleaning report 1110, and in some implementations, more information may be displayed. For example, the cleaning report 1110 may further include information about no-go zones in addition to information about areas corresponding to previously detected potential obstacles.

[0089] The user interface for reviewing completed tidy missions further includes affordance 1120 on display 700 for completing the tidy mission (shown in both FIGS. 11A and 11B ). In some implementations, after completing a first tidy mission, user 80 may not be able to provide feedback regarding newly detected potential obstacles and / or schedule a second, consecutive tidy mission. Once review of the tidy missions is complete, user 80 may exit the user interface for reviewing completed tidy missions by selecting affordance 1120.

[0090] While examples have been provided regarding the robot 100 detecting potential obstacles, in some implementations, the robot 100 may detect an error condition during a cleaning mission. Various types of error conditions are detectable. For example, an error condition may be related to components of the autonomous cleaning robot 100, including the drive system 110 of the robot 100, the cleaning assembly 116 of the robot 100, or the brushes 126 of the robot 100. For example, one or more of these components may break or become stuck. In some implementations, detecting an error condition may include identifying a component of the robot 100 that should be replaced. In some implementations, a type of error condition may be related to limited mobility of the robot 100. For example, the robot 100 may become stuck under furniture, such as a sofa. In other examples, the robot 100 may be able to navigate around a floor surface but may be unable to find a path out of a particular portion of the environment. In other examples, the robot 100 may be unable to navigate to the docking station 50 or may be unable to complete the cleaning mission.

[0091] 12 illustrates a process 1200 for displaying information about an error condition detected by the robot 100. The process 1200 includes operations 1202, 1204, 1206, 1208, 1210, 1212, and 1214.

[0092] Operations 1202 and 1204 involve actions of the robot 100 as it navigates through an environment (e.g., environment 20 shown in FIGS. 1A-1B). At operation 1202, the robot 100 begins navigating the environment. For example, the robot 100 may begin navigating the environment in response to receiving a command to begin a cleaning mission. At operation 1204, the robot 100 collects sensor data. The sensor data is collected by the sensor systems of the robot 100 and may be used for various purposes, including generating mapping data of the environment and detecting error conditions as the robot 100 navigates through the environment. As the robot 100 navigates through the environment, it may clean a floor surface if operations 1202 and 1204 are performed as part of a cleaning mission. In some examples, the robot 100 may navigate a floor surface without cleaning.

[0093] Acts 1206, 1208, and 1210 involve detecting an error condition, generating data corresponding to the detected error condition, and capturing a portion of an image associated with the error condition. Acts 1206, 1208, and 1210 are performed by computing system 90, which may be a controller located on robot 100, a controller located on mobile computing device 85, a remote computing system, a distributed computing system including processors located on multiple devices (e.g., robot 100, mobile device 85, or remote computing system), a processor on an autonomous mobile robot added to robot 100, or a combination of these computing devices. In some examples, one or more of the acts performed by computing system 90 (e.g., acts 1206, 1208, and 1210) may be performed by different actors. For example, detecting the error condition (act 1206) may be performed on robot 100, while capturing a portion of an image associated with the error condition (act 1210) may be performed on a remote server. At operation 1206, sensors are used to detect error conditions in the robot 100. For example, data from current sensors may be used to detect when the drive system 110, cleaning assembly 116, or brushes 126 of the robot 100 are stuck. In some examples, sensor data used for SLAM techniques may be used to detect when the robot 100 is unable to avoid a particular portion of the environment or navigate to the docking station 50. At operation 1208, the computing system 90 generates data corresponding to the detected error condition. For example, this data may include the location of the robot 100 when the error condition was detected and / or a label indicating the type of error condition. At operation 1210, the computing system 90 acquires a portion of the image associated with the error condition.For example, computing system 90 can obtain this portion of the image from an image captured by image capture device 101. This portion of the image associated with the error condition can be a single image, a series of images, or a video clip. In some implementations, this portion of the image can be captured near the position of robot 100 when robot 100 detects the error condition. In some implementations, this portion of the image can be captured before robot 100 detects the error condition. In some implementations, this portion of the image can be captured after robot 100 detects the error condition.

[0094] At operation 1212, the mobile device 85 displays a representation of this portion of the image and an indicator of the detected error condition. For example, as shown in Figures 13A-13B (described in further detail herein), the mobile device 85 may display an image associated with the error condition (e.g., images 1305, 1315), a label indicating the type of error condition (e.g., labels 1312, 1320), and / or the location of the error condition (e.g., indicators 1304, 1318).

[0095] At operation 1214, user 80 reviews the representation of this portion of the image and the indicator of the detected error condition. In some implementations, the user may provide feedback regarding the detected error condition. For example, user 80 may provide a user selection from mobile computing device 85 to stop robot 100 and / or terminate the cleaning mission. In some implementations, user 80 is not given the option to provide feedback regarding the detected error condition. In such examples, user 80 can still understand the context of the error condition detected by robot 100 by reviewing the representation of this portion of the image and the indicator of the detected error condition. In some examples, user 80 may choose to provide data regarding the error condition to a database that stores data from multiple users. This data may be used to reduce the number of error conditions faced by existing autonomous mobile robots and to develop new autonomous mobile robots.

[0096] 13A , an example of a user interface for reviewing an error condition detected by the robot 100 is displayed on the display 700 of the mobile computing device 85. The display 700 may include a timestamp 1322 containing the date and time the error condition was detected. The display 700 may also include a map 1302 of the environment in which the robot 100 is located. In this example, the environment is a home (not shown) that includes a study, dining room, kitchen, living room, and bedroom. The display 700 also includes control options for interacting with the map 1302, including a zoom in option 1306A, a zoom out option 1306B, and a rotate map option 1308 for rotating the map (e.g., 90 degrees clockwise). An indicator 1304 is displayed on the map 1302 to indicate the location of the robot 100 when the error condition was detected during the cleaning mission. The display 700 further includes an error report 1310. The error report 1310 may include a label 1312 indicating the type of error condition. In the example of FIG. 13A , label 1312 indicates that robot 100 (named "Rosie") is stuck. The error report may also include image 1305, which is a portion of an image captured by robot 100. For example, image 1305 may have been captured at or near the location of robot 100 when it detected the error condition. In this example, image 1305 shows cord 1314 on the floor surface. User 80 can review the items displayed on display 700 to gain context regarding the error condition detected by robot 100. For example, user 80 may infer that robot 100 is stuck in the living room of their home because it has gotten caught on electrical cord 1314.

[0097] In some implementations, image 1305 may be captured by robot 100 during a period of time (e.g., 1 second, 5 seconds, 10 seconds, etc.) before the detection of the error condition or during a period of time (e.g., 1 second, 5 seconds, 10 seconds, etc.) after the detection of the error condition. In some implementations, user 80 may review multiple images taken at various times before and / or after the detection of the error condition. This may be advantageous if an initial image (e.g., image 1305) does not provide sufficient context for user 80 regarding the error condition. For example, if robot 100 becomes stuck under a couch and detects an error condition, subsequently captured images may be too dark or may not include features that would allow user 80 to interpret the context of the error condition. However, images captured before the detection of the error condition may allow user 80 to see that robot 100 was moving toward the space under the couch before becoming stuck. In another example, if the robot 100 detects that it cannot navigate to the docking station 50, the initial image 1305 may not contain enough features for the user 80 to determine the current location of the robot 100. However, an image captured after the error condition is detected (e.g., 5 seconds after detecting the error condition) may contain features (e.g., a television, a rug, a potted plant, etc.) that reveal to the user 80 that the robot 100 is currently located in the living room.

[0098] In some implementations, user 80 can select affordance 1316 corresponding to a provision option that provides information about the error condition to a database that stores data from multiple users. For example, when affordance 1316 is selected, mobile computing device 85 can send information about the error condition (e.g., the location of the error condition, sensor data corresponding to the error condition, a portion of an image captured by robot 100, a label indicating the type of error condition, etc.) to the database. This data can be used to reduce the number of error conditions encountered by robot 100 in subsequent cleaning missions or to develop new autonomous mobile robots.

[0099] 13B , another example of a user interface for reviewing an error condition detected by the robot 100 is displayed on the display 700 of the mobile computing device 85. This example is similar to the example described in connection with FIG. 13A , and like elements are given the same reference numerals. An indicator 1318 is displayed on the map 1302 to indicate the location of the robot 100 when an error condition is detected during a cleaning mission. In this example, the error report 1310 includes a label 1320 indicating the type of error condition. In the example of FIG. 13B , the label 1320 indicates that the cleaning assembly 116 of the robot 100 is jammed. The error report also includes an image 1315, which is a portion of an image captured by the robot 100. In this example, the image 1315 shows a pile of clothes 1324 on the floor surface. The user 80 can review the items displayed on the display 700 to gain context for the error condition detected by the robot 100. For example, the user 80 may infer that the cleaning assembly 116 of the robot 100 became stuck while attempting to suck up an article of clothing (e.g., a sock) from a pile of clothes 1324 located in a study. The user may again select affordance 1316, which corresponds to a provision option that provides information about the error condition to a database that stores data from multiple users. For example, when affordance 1316 is selected, the mobile computing device 85 may transmit information about the error condition (e.g., the location of the error condition, sensor data corresponding to the error condition, a portion of an image captured by the robot 100, a label indicating the type of error condition, etc.) to the database. This data may be used to reduce the number of error conditions encountered by the robot 100 in subsequent cleaning missions or to develop new autonomous mobile robots.

[0100] Further alternative implementations While some implementations are described in detail above, other implementations are also possible. For example, in some implementations, there may be a time limit for providing user feedback regarding detected obstacles and / or error conditions (e.g., 12 hours from the completion of the first mission, 18 hours from the completion of the first mission, 24 hours from the completion of the first mission, etc.). In some implementations, if the user 80 does not provide feedback regarding the detected obstacles and / or error conditions with this time limit, information regarding the detected obstacles and / or error conditions may be erased. In some implementations, once the time limit expires, the user 80 may be unable to plan a tidy mission or cause the robot 100 to initiate a tidy mission. In some implementations, a notification (e.g., a push notification) may be displayed on the display 700 of the mobile computing device 85 prior to the expiration of the time limit to remind the user to provide feedback regarding potential obstacles and / or error conditions detected by the robot 100.

[0101] In some implementations, the first mission performed by the robot 100 may be a training mission, in which the robot navigates around the environment 20 without cleaning the floor surface 10. During this training mission, the robot 100 may further detect potential obstacles and / or error conditions (e.g., in a manner substantially similar to the examples described above). Based on user feedback regarding the potential obstacles and / or error conditions, the robot 100 may then perform a second mission. In some examples, the second mission may be a cleaning mission, such as a tidying mission.

[0102] In some implementations, information about one or more potential obstacles and / or error conditions detected by the robot 100 may be shared among multiple autonomous mobile robots (e.g., a fleet of autonomous mobile robots) owned by the user 80. In some implementations, if one robot detects an error condition and is unable to complete a cleaning mission, a second robot may complete the cleaning mission. In some implementations, a first robot may perform a first cleaning mission, and a second robot may perform a tidying mission after receiving feedback from the user 80 about one or more potential obstacles.

[0103] In some implementations, before the user 80 provides feedback regarding potential obstacles and / or error conditions detected by the robot 100, the mobile computing device 85 may cause a user interface for a tutorial to be displayed on the display 700. The tutorial may instruct the user 80 on how to provide feedback regarding potential obstacles and / or error conditions detected by the robot 100. The tutorial may also instruct the user 80 on how to plan a tidying mission. The tutorial may provide information about each of a plurality of user-selectable affordances 912 (shown in FIGS. 9A-9D ) and details about affordances 930, 1316 corresponding to the provision options (shown in FIGS. 9A-9D , 13A-13B ). In some implementations, the tutorial is first shown before the user 80's first attempt to provide feedback regarding potential obstacles and / or error conditions detected by the robot 100. However, in some implementations, the tutorial is accessible at any time, allowing the user 80 to review the tutorial as needed.

[0104] FIG. 14 illustrates an example of a computing device 1400 and a mobile computing device 1450 that can be used to implement the techniques described herein. For example, computing device 1400 and mobile computing device 1450 can represent example elements of mobile device 85 and computing system 90. Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Mobile computing device 1450 is intended to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, and other similar computing devices. Additionally, computing device 1400 or 1450 may include a universal serial bus (USB) flash drive. These USB flash drives may store an operating system and other applications. USB flash drives may include input / output components, such as a wireless transceiver or a USB connector that can be inserted into a USB port of another computing device. The components, their connections and relationships, and their functions illustrated herein are intended solely as examples and not as limitations.

[0105] Computing device 1400 includes processor 1402, memory 1404, storage device 1406, high-speed interface 1408 connected to memory 1404 and multiple high-speed expansion ports 1410, and low-speed interface 1412 connected to low-speed expansion port 1414 and storage device 1406. Processor 1402, memory 1404, storage device 1406, high-speed interface 1408, high-speed expansion port 1410, and low-speed interface 1412 are each interconnected using various buses, which may be mounted on a shared motherboard or otherwise as appropriate. Processor 1402 may process instructions for execution within computing device 1400, including instructions stored in memory 1404 or on storage device 1406 to display graphical information of a GUI on an external input / output device, such as a display 1416 coupled to high-speed interface 1408. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Additionally, multiple computing devices may be connected together, with each device performing a portion of the required operations (eg, as a server bank, as a cluster of blade servers, or as a multiprocessor system).

[0106] The memory 1404 stores information within the computing device 1400. In some implementations, the memory 1404 is a volatile memory unit. In some implementations, the memory 1404 is a non-volatile memory unit. The memory 1404 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0107] The storage device 1406 can provide mass storage for the computing device 1400. In some implementations, the storage device 1406 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. Instructions can be stored on an information carrier. These instructions, when executed by one or more processing devices (e.g., the processor 1402), perform one or more methods, such as those described above. These instructions can also be stored by one or more storage devices (e.g., the memory 1404, the storage device 1406, or memory on the processor 1402), such as a computer-readable or machine-readable medium.

[0108] The high-speed interface 1408 manages bandwidth-intensive operations of the computing device 1400, while the low-speed interface 1412 manages less bandwidth-intensive operations. Such an allocation of functionality is merely an example. In some implementations, the high-speed interface 1408 is coupled to the memory 1404, the display 1416 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1410 that may accept various expansion cards. In this implementation, the low-speed interface 1412 is coupled to the storage device 1406 and the low-speed expansion port 1414. The low-speed expansion port 1414, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices. Such input / output devices may include a scanner 1430, a printing device 1434, or a keyboard or mouse 1436. The input / output devices may also be coupled to the low-speed expansion port 1414 via a network adapter. Such network input / output devices may include, for example, a switch or router 1432 .

[0109] Computing device 1400 may be implemented in a number of different forms, as shown in FIG. 14 . For example, computing device 1400 may be implemented as a standard server 1420 or multiplexed in a cluster of such servers. Furthermore, computing device 1400 may be implemented in a personal computer, such as, for example, a laptop computer 1422. Furthermore, computing device 1400 may be implemented as part of a rack server system 1424. Alternatively, components of computing device 1400 may be combined with other components of a mobile device, such as, for example, mobile computing device 1450. Each such device may include one or more of computing device 1400 and mobile computing device 1450, and the entire system may be comprised of multiple computing devices communicating with each other.

[0110] The mobile computing device 1450 includes, among other components, a processor 1452, memory 1464, an input / output device such as a display 1454, a communication interface 1466, and a transceiver 1468. The mobile computing device 1450 may also include a storage device, such as a microdrive or other device, to provide additional storage. The processor 1452, memory 1464, display 1454, communication interface 1466, and transceiver 1468 are each interconnected using various buses, and multiple of these components may be mounted on a shared motherboard or otherwise, as appropriate.

[0111] The processor 1452 may execute instructions within the mobile computing device 1450, including instructions stored in the memory 1464. The processor 1452 may be implemented as a chipset of chips including individual and multiple analog and digital processors. For example, the processor 1452 may be a complex instruction set computer (CISC) processor, a reduced instruction set computer (RISC) processor, or a minimal instruction set computer (MISC) processor. The processor 1452 may facilitate coordination among other components of the mobile computing device 1450, such as controlling the user interface, applications executed by the mobile computing device 1450, and wireless communications by the mobile computing device 1450.

[0112] The processor 1452 may communicate with a user via a control interface 1458 and a display interface 1456 coupled to a display 1454. The display 1454 may be, for example, a thin film transistor liquid crystal display (TFT-LCD) or an organic light emitting diode (OLED) display, or other suitable display technology. The display interface 1456 may comprise appropriate circuitry for driving the display 1454 to display graphical and other information to the user. The control interface 1458 may receive instructions from the user and translate these instructions for submission to the processor 1452. Additionally, the external interface 1462 may enable near-area communication of the mobile computing device 1450 with other devices by enabling communication with the processor 1452. The external interface 1462 may enable wired communication, for example, in some embodiments, or wireless communication in other embodiments. Multiple interfaces may also be used.

[0113] Memory 1464 stores information within mobile computing device 1450. Memory 1464 may be implemented as one or more of a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Additional memory 1474 may also be provided and connected to mobile computing device 1450 via expansion interface 1472. Expansion interface 1472 may include, for example, a single integrated memory module (SIMM) card interface. Additional memory 1474 may provide additional storage space for mobile computing device 1450 or may store applications or other information for mobile computing device 1450. Specifically, additional memory 1474 may include instructions for implementing or supplementing the processes described above and may include secure information. Thus, for example, additional memory 1474 may be provided as a security module for mobile computing device 1450 and may be programmed with instructions that enable secure use of mobile computing device 1450. Additionally, secure applications may be provided via the SIMM card with additional information, for example by placing identifying information on the SIMM card in an unhackable manner.

[0114] The memory may include, for example, flash memory and / or non-volatile random access memory (NVRAM), as discussed below. In some implementations, the instructions are stored on an information carrier. When executed by one or more processing devices (e.g., processor 1452), the instructions perform one or more methods, such as those described above. The instructions may also be stored by one or more storage devices, such as, for example, one or more computer-readable or machine-readable media (e.g., memory 1464, additional memory 1474, or memory on processor 1452). In some implementations, the instructions may be received as a propagated signal, such as via transceiver 1468 or external interface 1462.

[0115] The mobile computing device 1450 may communicate wirelessly via a communication interface 1466, which may include digital signal processing circuitry, if necessary. The communication interface 1466 may enable communication under various modes or protocols, such as Global System for Mobile Communications (GSM) voice calls, Short Message Service (SMS), Enhanced Message Service (EMS), or Multimedia Messaging Service (MMS) messaging, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Personal Digital Cellular (PDC), Wideband Code Division Multiple Access (WCDMA), CDMA2000, or General Packet Radio Service (GPRS), among others. Such communication may occur via a transceiver 1468, for example, using radio frequencies. Additionally, short-range communication may occur, for example, using Bluetooth, Wi-Fi, or other such transceivers. Additionally, a global positioning system (GPS) receiver module 1470 may provide additional travel-related and location-related wireless data to the mobile computing device 1450, which may be utilized as appropriate by applications running on the mobile computing device 1450. In some implementations, the wireless transceiver 149 of the robot 100 may use any of the wireless transmission technologies enabled by the communication interface 1466 (e.g., to communicate with the mobile device 85).

[0116] Mobile computing device 1450 may also communicate audibly using audio codec 1460, which may receive speech information from a user and convert it into usable digital information. Similarly, audio codec 1460 may generate audio sounds for a user, such as through a speaker, in a handset of mobile computing device 1450. Such sounds may include sounds from a voice call, recorded sounds (e.g., voice messages, music files, etc.), and even sounds generated by applications running on mobile computing device 1450.

[0117] The mobile computing device 1450 may be implemented in a number of different forms as shown in the figure. For example, the mobile computing device 1450 may be implemented as a mobile phone 1480. The mobile computing device 1450 may also be implemented as part of a smartphone, a personal digital assistant 1482, or other similar mobile device.

[0118] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0119] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor. In some implementations, modules (e.g., object detection module), functions (e.g., displaying information on a display), and processes performed by the robot 100, computing system 90, and mobile device 85 (described in connection with FIG. 5) can execute instructions associated with the computer programs described above.

[0120] To enable interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (such as a cathode ray tube (CRT) monitor or a liquid crystal display (LCD) monitor) for displaying information to the user, and a keyboard and pointing device (such as a mouse or trackball) for enabling the user to provide input to the computer. Other types of devices may also be used to enable interaction with the user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including audio input, speech input, or tactile input.

[0121] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., as an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser that allows a user to interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically communicate through a communication network. The relationship of client and server is established by virtue of computer programs running on the respective computers and having a client-server relationship to each other. [Explanation of symbols]

[0123] 10 Floor surface 20 Environment 25 Area 25A First Area 25B Second Area 25C Third Area 30 rugs 35 socks 45 Code 50 Docking Station 55 Rug 60 Dining Table 70 orbit 80 users 85 Mobile Computing Devices 90 Computing Systems 100 robots 101 Image Capture Device 103 Field of view 105 Garbage 106 Electrical Circuits 108 Housing structure 109 Controller 110 Drive System 112 Drive wheels 113 Bottom part 114 Motor 115 Passive swivel wheel 116 Cleaning Assembly 117 Cleaning intake 118 Rotatable member 119 Vacuum System 120 motor 124 Trash Can 126 Brushes 128 motor 134 Cliff Sensor 136a Proximal Sensor 136b Proximal Sensor 138 Bumper 139a Bump sensor 139b Bump sensor 141 Obstacle Following Sensor 144 Memory Storage Elements 145 Inhalation route 146 parallel horizontal axis 148 parallel horizontal axis 149 Radio Transceiver 190 Object Detection Module 190 Object Detection Module 700 display 710 Push Notifications 802 Map 804 indicator 804A Indicator 804B Indicator 804C indicator 806A Zoom-in Option 806B Zoom Out Option 808 Rotation Map Options 810 Cleaning Report 812 Summary Statement 814A Images 814B Images 814C Images 814D Images 814E Images 816A Label 816B Label 816C Label 816D Label 816E Label 818A Area 818B Area 818C Area 818D Area 818E Area 820 Options 822 timestamp 824 Minimize Options 830 titles 832 entries 834 Status Indicator 836 Area Name 838 elapsed time 840 entries 842 Status Indicator 844 Messages 846 hours 890 User-Selectable Affordances 895 Summary Entries 902A Images 902B Images 902C Images 904 Code 906A Bounding Box 906B Bounding Box 906C Bounding Box 908A Label 908B Label 908C Label 910A Area 910B Area 910C Area 912 User-Selectable Affordances 912A User-Selectable Affordances 912B User Selectable Affordances 912C User Selectable Affordances 912D User-Selectable Affordances 914 User Selectable Options 916 Visual Elements 918 socks 918 Socks 920 Shadow 924 Affordance 930 Affordance 950 Floor surface 990 Visual Elements 1000A "Re-cleaning" section, "Re-cleaning" section 1000B "Avoid this time" part 1002 images 1002A Image 1002B Image 1002C Images 1004A Selection Indicator 1004B Selection Indicator 1004C Selection Indicator 1006 Affordance 1008 Affordance 1010 Messages 1102 Map 1104 indicator 1104A Indicator 1104B Indicator 1104C Indicator 1104D Indicator 1104E indicator 1106A Zoom-in option 1106B Zoom Out Option 1110 Cleaning Report 1112 Summary Statement 1114 images 1116 Label 1118 Area 1120 Affordance 1122 timestamp 1124 Minimize Options 1128 Route 1130 parts 1132 entries 1134 entries 1136 Status Indicator 1138 Area 1140 elapsed time 1142 parts 1302 Map 1304 indicator 1305 images 1306A Zoom-in option 1306B Zoom Out Option 1308 Rotation Map Options 1310 Error Report 1312 Label 1314 Cord, Electrical Cord 1315 images 1316 Affordance 1318 indicator 1320 Label 1322 timestamp 1324 Pile of clothes 1400 computing devices 1402 processor 1404 memory 1406 Storage Devices 1408 high-speed interface 1410 High-Speed ​​Expansion Port 1412 Low-speed interface 1414 Low-Speed ​​Expansion Port 1416 Display 1420 Standard Server 1422 laptop computer 1424 Rack Server System 1430 scanner 1432 Router 1434 Print Device 1436 Mouse 1450 Mobile Computing Devices 1452 processor 1454 Display 1456 Display Interface 1458 Control Interface 1460 Audio Codec 1462 External Interface 1464 memory 1466 Communication Interface 1468 Transceiver 1470 Global Positioning System GPS Receiver Module 1472 Expansion Interface 1474 additional memory 1480 mobile phone 1482 Personal Digital Assistant

Claims

1. a user input device having a display; a controller operably connected to the user input device; wherein the controller displaying on the display information regarding one or more areas not cleaned by the autonomous cleaning robot during the first mission; and transmitting data corresponding to a user-selected subset of the one or more areas to cause the autonomous cleaning robot to clean the user-selected subset during a second mission.

10. A mobile computing device configured to execute instructions to perform operations including:

2. The mobile computing device of claim 1 , wherein the information about the one or more areas includes a location of each of the one or more areas.

3. The mobile computing device of claim 1 , wherein the information about the one or more areas includes data representing potential obstacles detected in each of the one or more areas.

4. The mobile computing device of claim 3 , wherein the data representing the potential obstacles detected in each of the one or more areas includes one or more images of the potential obstacles.

5. The mobile computing device of claim 4 , wherein the actions further include displaying the one or more graphical representations of the potential obstacles on the display.

6. The mobile computing device of claim 3 , wherein the data representing the potential obstacles detected in each of the one or more areas includes a label indicating a classification of the potential obstacle.

7. 10. The mobile computing device of claim 1, wherein the actions further include displaying, for each area of ​​the one or more areas, affordances on the display corresponding to a plurality of user-selectable options.

8. 8. The mobile computing device of claim 7, wherein the operations further include transmitting data representing a user selection of one of the plurality of user-selectable options for each of the one or more areas to the autonomous cleaning robot for navigating the one or more areas during the second mission.

9. 8. The mobile computing device of claim 7, wherein the plurality of user-selectable options includes a first option to cause the autonomous cleaning robot to navigate to clean each area in the one or more areas and a second option to cause the autonomous cleaning robot to navigate to avoid each area in the one or more areas.

10. The mobile computing device of claim 7 , wherein the plurality of user-selectable options includes an option for indicating an absence of obstacles within the each of the one or more areas.

11. 8. The mobile computing device of claim 7, wherein the plurality of user-selectable options includes an option to cause the autonomous cleaning robot to avoid a particular area during the second mission and not avoid the particular area during one or more missions after the second mission.

12. 8. The mobile computing device of claim 7, wherein the plurality of user-selectable options includes an option to cause the autonomous cleaning robot to navigate to avoid specific areas during the second mission and during one or more missions after the second mission.

13. The mobile computing device of claim 7 , wherein an indicator of the plurality of user-selectable options is displayed on the display after completion of the first mission.

14. The mobile computing device of claim 1 , wherein the second mission is initiated within 12 hours after completion of the first mission.

15. 10. The mobile computing device of claim 1, wherein the operations further include displaying an affordance on the display corresponding to a provision option, the provision option enabling the mobile computing device to transmit the information about the one or more areas and the data corresponding to the user-selected subset to a database that stores data from multiple users.

16. The mobile computing device of claim 1 , wherein the actions further include displaying on the display a map of an environment including the one or more areas.

17. The mobile computing device of claim 1 , wherein the operations further include displaying information about the user-selected subset on the display after completion of the second mission.

18. The mobile computing device of claim 1 , wherein the first mission and the second mission are consecutive missions.

19. The second mission is: the autonomous cleaning robot commencing movement from a dock to a first area of ​​the user-selected subset; causing the autonomous cleaning robot to begin moving to all remaining areas of the user-selected subset; and the autonomous cleaning robot begins moving from the final area of ​​the user-selected subset to the dock; and 10. The mobile computing device of claim 1, comprising:

20. 10. The mobile computing device of claim 1, wherein during the second mission, the autonomous cleaning robot cleans the user-selected subset without cleaning all of the areas that the autonomous cleaning robot cleaned during the first mission.

21. An autonomous cleaning robot, a drive system for supporting the autonomous cleaning robot above a floor surface, the drive system operable to propel the autonomous cleaning robot across the floor surface; an obstacle detection sensor for detecting potential obstacles as the autonomous cleaning robot moves about the floor surface; a controller operatively connected to the drive system and the obstacle detection sensor; wherein the controller To carry out the first mission, detecting one or more potential obstacles located in one or more areas on the floor surface during the first mission; transmitting data corresponding to the detected potential obstacles and the one or more areas to a mobile computing device; receiving data from the mobile computing device corresponding to a user-selected subset of the one or more areas; and conducting a second mission to clean the user-selected subset of the one or more areas; 1. An autonomous cleaning robot configured to execute instructions to perform an operation, comprising:

22. 22. The autonomous cleaning robot of claim 21, wherein the obstacle detection sensor comprises an image capture device positioned on the autonomous cleaning robot to capture an image of a portion of the floor surface in front of the autonomous cleaning robot.

23. 22. The autonomous cleaning robot of claim 21, wherein the data corresponding to the detected potential obstacles and the one or more areas includes data representing an image of each obstacle among the one or more detected potential obstacles.

24. 24. The autonomous cleaning robot of claim 23, wherein the image comprises a single image.

25. 24. The autonomous cleaning robot of claim 23, wherein the image comprises a series of images.

26. 22. The autonomous cleaning robot of claim 21, wherein the data corresponding to the detected potential obstacles and the one or more areas includes a location of each area within the one or more areas.

27. 22. The autonomous cleaning robot of claim 21, wherein the data corresponding to the detected potential obstacles and the one or more areas includes a label indicating a classification of each potential obstacle among the detected potential obstacles.

28. 22. The autonomous cleaning robot of claim 21, wherein the second mission is initiated within 12 hours after completion of the first mission.

29. 22. The autonomous cleaning robot of claim 21, wherein the operations further include transmitting data to the mobile computing device corresponding to an updated state of the user-selected subset of the one or more areas during the second mission.

30. 30. The autonomous cleaning robot of claim 29, wherein the data corresponding to the updated state of the user-selected subset includes an indication of a portion of the user-selected subset that was cleaned during the second mission.

31. 22. The autonomous cleaning robot of claim 21, wherein the data corresponding to the user-selected subset of the one or more areas includes data corresponding to a user selection of one of a plurality of user-selectable options for each area of ​​the one or more areas.

32. 32. The autonomous cleaning robot of claim 31 , wherein the plurality of user-selectable options include a first option that causes the autonomous cleaning robot to clean each of the areas in the one or more areas during the second mission, and a second option that causes the autonomous cleaning robot to avoid each of the areas in the one or more areas during the second mission.

33. 32. The autonomous cleaning robot of claim 31 , wherein the plurality of user-selectable options includes an option to cause the autonomous cleaning robot to avoid a particular area during the second mission and not avoid the particular area during one or more missions after the second mission.

34. 32. The autonomous cleaning robot of claim 31 , wherein the plurality of user-selectable options includes an option to cause the autonomous cleaning robot to avoid a particular area during the second mission and during one or more missions after the second mission.

35. 32. The autonomous cleaning robot of claim 31, wherein the plurality of user-selectable options includes an option indicating an absence of obstacles located within the each area in the one or more areas.

36. 32. The autonomous cleaning robot of claim 31, wherein the operation further comprises updating an obstacle detection module based on the user selection.

37. 22. The autonomous cleaning robot of claim 21, wherein the first mission and the second mission are consecutive missions.

38. conducting the second mission; Initiating movement from a dock to a first area in the user-selected subset; initiating movement to all remaining areas of said user-selected subset; Initiating movement from the final area of ​​the user-selected subset to the dock; 22. The autonomous cleaning robot of claim 21, comprising:

39. 22. The autonomous cleaning robot of claim 21, wherein the second mission includes cleaning the user-selected subset of the one or more areas without cleaning all of the areas cleaned by the autonomous cleaning robot during the first mission.

40. a user input device having a display; a controller operably connected to the user input device; wherein the controller From autonomous cleaning robots, data corresponding to an error condition detected by the autonomous cleaning robot; and a portion of an image captured by the autonomous cleaning robot, the portion of the image relating to the detected error condition; receiving the displaying a representation of the portion of the image on the display and an indicator of the detected error condition in response to receiving the data corresponding to the detected error condition; and 10. A mobile computing device configured to execute instructions to perform operations including:

41. 41. The mobile computing device of claim 40, wherein the portion of the image associated with the error condition is captured near a location of the autonomous cleaning robot when the autonomous cleaning robot detects the error condition.

42. 41. The mobile computing device of claim 40, wherein the portion of the image includes an image captured before the autonomous cleaning robot detected the error condition.

43. 41. The mobile computing device of claim 40, wherein the portion of the image includes an image captured after the autonomous cleaning robot detects the error condition.

44. 41. The mobile computing device of claim 40, wherein the portion of the image associated with the error condition comprises a single image.

45. The mobile computing device of claim 40 , wherein the portion of the image associated with the error condition comprises a series of images.

46. 41. The mobile computing device of claim 40, wherein the data corresponding to the detected error condition includes at least one of a location of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition, a time when the autonomous cleaning robot detected the error condition, or a type of error condition.

47. 47. The mobile computing device of claim 46, wherein the type of the error condition relates to a component of the autonomous cleaning robot, the component including at least one of a drive system, a cleaning assembly, or a brush.

48. 48. The mobile computing device of claim 47, wherein the component is identified to perform the exchange.

49. 47. The mobile computing device of claim 46, wherein the type of error condition relates to a limitation of mobility of the autonomous cleaning robot.

50. 50. The mobile computing device of claim 49, wherein the limitations on the mobility of the autonomous cleaning robot include an inability to complete a mission or an inability to drive to a dock.

51. 41. The mobile computing device of claim 40, wherein the portion of the image includes a captured image of a portion of an environment in front of the autonomous cleaning robot.

52. 52. The mobile computing device of claim 51, wherein the portion of the environment in front of the autonomous cleaning robot includes a portion of a floor surface.

53. 41. The mobile computing device of claim 40, wherein the indicator of the detected error condition includes a label of a type of the error condition.

54. 41. The mobile computing device of claim 40, wherein the indicator of the detected error condition includes an image of the position of the autonomous cleaning robot at the time the autonomous cleaning robot detected the error condition.

55. An autonomous cleaning robot, a drive system for supporting the autonomous cleaning robot above a floor surface, the drive system operable to propel the autonomous cleaning robot across the floor surface; one or more sensors configured to capture sensor data indicative of an error condition of the autonomous cleaning robot; an image capture device for capturing an image of the error condition, the captured image being an image of a portion of an environment in front of the autonomous cleaning robot; and one or more controllers operatively connected to the drive system, the image capture device, and the one or more sensors; wherein the one or more controllers: detecting the error condition of the autonomous cleaning robot based on the sensor data as the autonomous cleaning robot is traversed across the floor surface; and transmitting to the mobile computing device (i) data representative of the error condition to cause the mobile computing device to display an indicator of the error condition, and (ii) data representative of a portion of the captured image to cause the mobile computing device to display a representation of the portion of the captured image.

1. An autonomous cleaning robot configured to execute instructions to perform an operation, comprising:

56. 56. The autonomous cleaning robot of claim 55, wherein the portion of the environment in front of the autonomous cleaning robot includes a portion of the floor surface.

57. 56. The autonomous cleaning robot of claim 55, wherein the portion of the captured image was captured at a position of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition.

58. 56. The autonomous cleaning robot of claim 55, wherein the portion of the captured images comprises a single image.

59. 56. The autonomous cleaning robot of claim 55, wherein the portion of the captured images comprises a series of images.

60. 56. The autonomous cleaning robot of claim 55, wherein the data representative of the error condition includes at least one of a location of the autonomous cleaning robot when the autonomous cleaning robot detected the error condition, a time when the autonomous cleaning robot detected the error condition, or a type of the error condition.

61. 61. The autonomous cleaning robot of claim 60, wherein the type of error condition is associated with a component of the autonomous cleaning robot, the component including at least one of a drive system, a cleaning assembly, or a brush.

62. 62. The autonomous cleaning robot of claim 61, wherein the component is identified for replacement.

63. 61. The autonomous cleaning robot of claim 60, wherein the type of error condition relates to a limitation in mobility of the autonomous cleaning robot.

64. 64. The autonomous cleaning robot of claim 63, wherein the limitations on the mobility of the autonomous cleaning robot include an inability to complete a mission or an inability to navigate to a dock.

Images