System and method for controlling an autonomous mobile robot

Abstract

A system and method for controlling an autonomous mobile robot using an audio media device is provided. A method for controlling one or more actions of a steerable autonomous mobile robot in a home includes establishing wireless communication between the autonomous mobile robot and a remote computing system and initiating one or more actions of the autonomous mobile robot in response to receiving a wireless command signal from the remote computing system. The autonomous mobile robot is remote from an audio media device deployed in the home. The audio media device is capable of receiving and emitting audio. The remote computing system is configured to associate identification data of the autonomous mobile robot with identification data of the audio media device. The wireless command signal corresponds to an audible user command received by the audio media device.

Description

[Technical Field]

[0001] SUMMARY This specification relates to systems and methods for controlling autonomous mobile robots, and in particular for controlling autonomous mobile robots using audio media devices. [Background technology]

[0002] Many home environments include mobile robots that autonomously navigate the home performing several actions to complete tasks that would otherwise require the attention of the home occupants. For example, some mobile robots are cleaning robots that can autonomously perform cleaning actions within a defined area using a program stored in a memory coupled to a controller. The cleaning robot can clean the home without the user having to manually move the cleaning robot around the home. The cleaning robot may include a manual input device, for example, a button that the user presses to initiate autonomous cleaning actions within the home.

[0003] These same home environments often include computing devices such as desktops, tablets, entertainment systems, and portable communication devices. To allow users to control the computing devices, the computing devices may include manual input devices (e.g., touch screens, keyboards, pointing devices) that provide users with manual control over the operation of the computing devices. Some of these computing devices allow users to control mobile robots using application programs stored on the computing devices. Summary of the Invention [Means for solving the problem]

[0004] In one aspect, a method for controlling one or more actions of a steerable autonomous mobile robot in a home includes establishing wireless communication between the autonomous mobile robot and a remote computing system and initiating one or more actions of the autonomous mobile robot in response to receiving a wireless command signal from the remote computing system. The autonomous mobile robot is remote from an audio media device deployed in the home. The audio media device is capable of receiving and emitting audio. The remote computing system is configured to associate identification data of the autonomous mobile robot with identification data of the audio media device. The wireless command signal corresponds to an audible user command received by the audio media device.

[0005] In another aspect, a method for controlling one or more operations of a steerable autonomous mobile robot in a home includes associating identification data of the autonomous mobile robot with identification data of an audio media device and establishing wireless communication with each of the autonomous mobile robot and the audio media device. The method further includes, upon receiving a wireless instruction signal from the audio media device, transmitting a wireless command signal to the autonomous mobile robot to initiate one or more operations of the autonomous mobile robot. The audio media device is capable of receiving and emitting audio. The audio media device is deployed in the home at a location remote from the autonomous mobile robot. The wireless instruction signal corresponds to an audible user command received by the audio media device.

[0006] In another aspect, a method for providing status of a steerable autonomous mobile robot within a home includes establishing wireless communication between an audio media device and a remote computing system, and emitting an audible signal representative of an operational status of the autonomous mobile robot in response to receiving a wireless notification signal from the remote computing system. The audio media device is capable of receiving and emitting audio and is deployed within the home at a location remote from the autonomous mobile robot. The remote computing system is configured to associate identification data of the autonomous mobile robot with identification data of the audio media device. The wireless notification signal corresponds to status data received by the remote computing system as the autonomous mobile robot autonomously navigates within the home relative to the audio media device.

[0007] In another aspect, a method for providing a status of an autonomous mobile robot located in a home includes associating identification data of the autonomous mobile robot with identification data of an audio media device and establishing wireless communication with each of the autonomous mobile robot and the audio media device. Upon receiving data indicating an operational status of the autonomous mobile robot, the method further includes transmitting a wireless notification signal to the audio media device to cause the audio media device to emit an audible signal representing the operational status of the autonomous mobile robot. The audio media device is capable of receiving and emitting audio. The audio media device is deployed in the home at a location remote from the autonomous mobile robot.

[0008] In another aspect, a method for controlling one or more actions of a steerable autonomous mobile robot in a home includes establishing wireless communication between an audio media device and a remote computing system, and transmitting wireless command signals to the remote computing system in response to receiving an audible user command to initiate one or more actions of the autonomous mobile robot. The audio media device is capable of receiving and emitting audio and is deployed in the home at a location remote from the autonomous mobile robot. The remote computing system is configured to associate identification data of the autonomous mobile robot with identification data of the audio media device.

[0009] In another aspect, a method for providing a status of a steerable autonomous mobile robot within a home includes establishing wireless communication between the autonomous mobile robot and a remote computing system, and transmitting status data from the autonomous mobile robot to the remote computing system indicating an operational status of the autonomous mobile robot while autonomously navigating within the home relative to the audio media device to cause the audio media device to emit an audible signal representative of the operational status of the autonomous mobile robot. The autonomous mobile robot is remote from the audio media device. The audio media device is deployed within the home. The audio media device is capable of receiving and emitting audio. The remote computing system is configured to associate identification data of the autonomous mobile robot with identification data of the audio media device.

[0010] In another aspect, an autonomous mobile robot includes a chassis, a sensing system for generating signals for generating a map of a home, a mobility member for supporting the chassis on a floor surface within a home, and a controller operable with a wireless communication system. The mobility member is capable of driving the autonomous mobile robot to autonomously navigate the floor surface while the sensing system generates signals for generating the map of the home. The controller is configured to initiate one or more actions of the autonomous mobile robot in response to receiving wireless command signals from a remote computing system. The wireless command signals correspond to audible user commands received by an audio media device capable of receiving and emitting audio.

[0011] In another aspect, an autonomous mobile robot includes at least one drive wheel operable to move the mobile robot across a floor surface, a sensor system including navigation sensors, a controller in communication with the sensor system, and a wireless communication system in communication with the controller. The controller is configured to control movement of the mobile robot based on sensor signals generated by the navigation sensors. The controller is configured to initiate one or more actions of the mobile robot in response to receiving wireless command signals from a remote computing system. The wireless command signals indicate user-requested actions and are based on audible user commands received by an audio media device capable of receiving and emitting audio.

[0012] Some aspects may include one or more implementations described below and elsewhere herein. In some examples, a controller performs one or more of the methods described above.

[0013] In some examples, the one or more operations include autonomously navigating through the home relative to the audio media device. The autonomous mobile robot may include an autonomous cleaning robot. The one or more operations may include autonomously navigating to a docking station for the autonomous mobile robot. The one or more operations may include autonomously navigating toward the audio media device. One or more of the methods may further include receiving a location signal from a remote computing system and autonomously navigating through the home relative to the audio media device while localizing to the location of the audio media device. The location signal may indicate the location of the audio media device within the home. One or more of the methods may include generating a map of the home including the locations of devices in wireless communication with the remote computing system while autonomously navigating through the home.

[0014] In some examples, the one or more actions include pausing a cleaning action of the autonomous mobile robot.

[0015] In some examples, initiating the one or more actions includes initiating a cleaning action of the autonomous mobile robot.

[0016] In some examples, one or more of the methods further include autonomously navigating within the home while performing cleaning operations. One or more of the methods may further include, in response to receiving a wireless command signal from a remote computing system, initiating a localized cleaning operation within a portion of the home by navigating the autonomous mobile robot in a predetermined pattern within the portion of the home and increasing the amount of power provided to a cleaning device of the autonomous mobile robot.

[0017] In some examples, the one or more actions include emitting an audible signal.

[0018] In some examples, the one or more actions include storing a user-defined schedule for performing a subsequent action at a future time. The one or more actions may include performing the subsequent action at a future time.

[0019] In some examples, the audible user commands include an audible identifier indicating identification data of the autonomous mobile robot and a voice command corresponding to one or more actions of the autonomous mobile robot. One or more of the methods may further include generating a wireless command signal based on the voice command and selecting a destination of the wireless command signal based on the audible identifier.

[0020] In some examples, one or more of the methods further include, upon receiving an initial user command including an audible identifier indicating the robot type, determining that the identification data of the autonomous mobile robot includes a robot type and that other identification data of the other robot associated with the identification data of the audio media device includes a robot type. One or more of the methods may include transmitting a wireless notification signal to cause the audio media device to issue an audible request for the audible identifier indicating the unique identity of the autonomous mobile robot. The audible user command may correspond to a user response to the audible request. The transmitting of the wireless notification signal may occur after determining that the identification data of the autonomous mobile robot includes the robot type and other identification data.

[0021] In some examples, one or more of the methods further include receiving an initial audible user command including an audible identifier indicating the identity of another autonomous mobile robot not associated with the audio media device. One or more of the methods may further include transmitting a wireless notification signal to cause the audio media device to emit an audible notification indicating that the other autonomous mobile robot is not associated with the audio media device.

[0022] In some examples, one or more of the methods may further include receiving an initial audible user command that does not include any of a plurality of available voice commands corresponding to one or more actions of the autonomous mobile robot. One or more of the methods may further include transmitting a wireless notification signal to cause an audio media device to issue an audible notification indicating that the initial audible user command does not include a voice command corresponding to any of the available voice commands.

[0023] In some examples, one or more of the methods further include estimating a user's location based on acoustic characteristics of the audible user command. Transmitting the wireless command signal may include transmitting a wireless command signal to cause the autonomous mobile robot to navigate toward the estimated user's location.

[0024] In some examples, one or more of the methods further include associating identification data of the other autonomous mobile robot with identification data of the audio media device, where the identification data of the autonomous mobile robot may correspond to a first audible identifier and the identification data of the other autonomous mobile robot may correspond to a second audible identifier distinct from the first audible identifier.

[0025] In some examples, the audible user command includes an audible location identifier corresponding to a predetermined location within the home. One or more of the methods may further include associating identification data of the other autonomous mobile robot with identification data of the audio media device. Sending the wireless command signal to the autonomous mobile robot may include sending the wireless command signal to the autonomous mobile robot upon determining that the distance of the autonomous mobile robot to the predetermined location is less than the distance of the other autonomous mobile robot to the predetermined location.

[0026] In some examples, the audible user command includes an audible location identifier corresponding to one predetermined location of a plurality of predetermined locations within the home. Transmitting the wireless command signal to the autonomous mobile robot may include transmitting the wireless command signal to the autonomous mobile robot to cause the autonomous mobile robot to navigate to the one predetermined location.

[0027] In some examples, one or more of the methods further include receiving an initial audible user command including audible scheduling parameters corresponding to a future time for transmitting a wireless command signal to the autonomous robot to perform one or more operations. Upon determining that the future time conflicts with an event in the user event calendar, one or more of the methods may further include transmitting a wireless notification signal to cause an audio media device to emit an audible notification indicating that the future time conflicts with the event.

[0028] In some examples, one or more of the methods further include receiving an initial audible user command to schedule the event in the user event calendar. One or more of the methods may further include, upon determining that the event corresponds to a predetermined event type, transmitting a wireless notification signal to cause the audio media device to issue an audible request. The audible user command may include a confirmation of the audible request. Transmitting the wireless command signal may include transmitting a wireless command signal to the autonomous mobile robot to initiate one or more actions of the autonomous mobile robot within a predetermined period of time prior to the event.

[0029] In some examples, emitting an audible signal includes emitting an audible signal after determining that a user has entered the home.

[0030] In some examples, the operational status of the autonomous mobile robot includes a schedule of previous operations, and issuing an audible signal may include issuing an audible request to modify the stored schedule of operations based on the schedule of previous operations.

[0031] In some examples, the operational status corresponds to a stationary state of the autonomous mobile robot within the home. Emitting an audible signal may include emitting an audible signal to indicate that the autonomous mobile robot is stationary. The audible signal may indicate the location of the autonomous mobile robot within the home.

[0032] In some examples, emitting an audible signal includes emitting an audible signal to indicate a frequency of a previous movement of the autonomous mobile robot.

[0033] In some examples, emitting an audible signal includes emitting an audible signal to indicate a total duration of previous movements of the autonomous mobile robot within a predetermined period of time.

[0034] In some examples, emitting an audible signal includes emitting an audible signal to indicate an estimated remaining service life of a component of the autonomous mobile robot, which may be a cleaning brush, a cleaning pad, a roller, a battery, a debris bin, or a wheel module.

[0035] In some examples, emitting an audible signal includes emitting an audible signal to identify an obstacle detected by the autonomous mobile robot while the autonomous mobile robot navigates through the house.

[0036] In some examples, emitting an audible signal includes emitting an audible signal to identify a connected device located in the home, the connected device may be wirelessly connected to a remote computing system and associated with the audio media device.

[0037] In some examples, issuing an audible signal includes issuing an audible instruction in response to detecting the error to guide a user to address the root of the error associated with the autonomous mobile robot.

[0038] In some examples, emitting the audible signal includes emitting the audible signal after the autonomous mobile robot has autonomously navigated through the home, The audible signal may identify areas of the home that the autonomous mobile robot did not traverse while autonomously navigating.

[0039] In some examples, emitting an audible signal includes emitting an audible signal upon receiving an audible user request about the operational status of the autonomous mobile robot.

[0040] In some examples, transmitting the wireless notification signal includes transmitting the wireless notification signal to cause an audio media device to emit an audible signal upon determining that an estimated remaining service life of a replaceable part of the autonomous mobile robot is less than a predetermined threshold. The audible signal may include a suggestion to purchase a replacement part for the part of the autonomous mobile robot. One or more of the methods may further include ordering the replacement part upon receiving an audible user command including a confirmation to purchase the replacement part.

[0041] In some examples, associating identification data of the autonomous mobile robot with identification data of the audio media device includes transmitting a wireless command signal based on issuing and receiving an audible association request, and associating identification data of the autonomous mobile robot with identification data of the audio media device upon receiving confirmation of receipt of the audible association request. One of the autonomous mobile robot and the audio media device can issue the audible association request, and the other of the autonomous mobile robot and the audio media device can receive the audible association request.

[0042] In some examples, one or more of the methods further include determining a location of the autonomous mobile robot within the home based on an audible status signal emitted by the autonomous mobile robot and received by the audio media device, wherein emitting an audible signal includes emitting an audible signal to indicate the location of the autonomous mobile robot.

[0043] In some examples, the mobile robot is remote from the audio media device and the audio media device is deployed in a home. The controller may be further configured to establish wireless communication with the remote computing system when the remote computing system associates identification data of the mobile robot with identification data of the audio media device.

[0044] In some examples, the autonomous mobile robot further includes an audio media device, which may be mounted on a chassis of the autonomous mobile robot.

[0045] In some examples, the autonomous mobile robot further includes a variable height member on the chassis and a camera supported by the variable height member. The variable height member may be configured to move perpendicularly relative to the chassis to a plurality of camera height positions. The sensing system may include a camera. The camera may be configured to capture images of the house for generating a map of the house. The one or more actions may include autonomously navigating the autonomous mobile robot to a selected location within the house and moving the variable height member to a selected camera height position to observe an object within the house using the camera. The object may be identified in an audible user command received by the audio media device.

[0046] In some examples, the audio media device is in wireless communication with a remote computing system and is deployed in the environment at a location remote from the autonomous mobile robot.

[0047] In some examples, the audio media device is in wireless communication with a remote computing system and is deployed in the environment at a location remote from the autonomous mobile robot.

[0048] In some examples, the controller is configured to generate a map of the environment based on the sensor signals, and in some cases, the navigation sensor includes a camera configured for visual identification of features and landmarks used in calculating the pose of the robot on the generated map.

[0049] In some examples, the mobile robot is configured to perform a plurality of missions and transmit a wireless signal indicative of cumulative data measured by the sensor system for each of the plurality of missions to cause an audio media device to issue a cumulative summary based on the cumulative data.

[0050] In some examples, the sensor system is configured to provide data indicative of the robot status to the controller. The mobile robot may be configured to transmit wireless signals indicative of the robot status to cause an audio media device to issue an audible status update.

[0051] In some cases, the data indicates the total duration the mobile robot has been operating for multiple missions, the number of missions performed by the mobile robot, the total cumulative distance of floor surface traveled, or the distance traveled by the mobile robot.

[0052] In some cases, the mobile robot is a vacuum cleaning robot. The sensor system can be configured to provide data indicative of a total amount of debris collected by the mobile robot over multiple missions.

[0053] In some cases, the mobile robot is a vacuum cleaning robot operable with a docking station that empties debris from the vacuum cleaning robot. The mobile robot may be configured to send data to a remote computing system via a wireless communication system to enable the remote computing system to calculate an average number of emptying operations per week, a number of emptying operations performed in a certain time period, or a number of emptying operations performed for an area covered by the mobile robot in a certain time period.

[0054] In some examples, the mobile robot further includes a serviceable component and one or more sensors configured to generate a signal indicative of an estimated remaining service life of the serviceable component. In some cases, the serviceable component includes a cleaning brush, a cleaning pad, a roller, a blade, a battery, a container, a wheel, a container filter, or a container lid. In some cases, the controller is configured to transmit a wireless signal indicative of an estimated remaining service life of the serviceable component in response to an audible user command received by the audio media device.

[0055] In some examples, the mobile robot further includes a stall sensor unit configured to detect a stall condition of the mobile robot. The controller may be configured to transmit a wireless signal indicating the detected stall condition of the mobile robot to cause the audio media device to issue a status update indicating the mobile robot is in a stall condition. In some cases, the stall sensor unit includes an optical stationary sensor, a motor stall sensor, a mouse sensor, a gyroscope, an accelerometer, or a stereo camera.

[0056] In some examples, the controller is configured, in response to the sensor system detecting an error, to transmit a wireless signal indicating the error to cause the audio media device to issue an initial audible instruction to address the error.

[0057] In some examples, the mobile robot is a vacuum cleaning robot. The controller may be configured to control movement of the vacuum cleaning robot in the environment to a user-specified room in response to receiving the wireless command signal, wherein the audible user command and the wireless command signal indicate the user-specified room.

[0058] In some examples, the mobile robot is a vacuum cleaning robot, and the controller can be configured to adjust the vacuum intensity in response to the wireless command signal.

[0059] In some examples, the controller is configured to adjust the stored schedule of operations in response to a wireless command signal.

[0060] In some examples, the mobile robot is configured to transmit a wireless signal in response to the wireless command signal to cause an audio media device to emit an audible signal indicating the current location of the mobile robot.

[0061] In some examples, the mobile robot is a vacuum cleaning robot including a dust container, a rotatable roller configured to agitate dust from a floor surface, and an air mover configured to move dust from the floor surface toward the dust container. In some cases, the mobile robot further includes a dust container level sensor configured to detect an amount of dust sucked into the dust container. In some cases, the mobile robot further includes one or more dust sensors configured to detect a dust suction rate. In some cases, the mobile robot further includes a dust filter and a filter sensor for detecting whether the dust filter needs cleaning.

[0062] Advantages of the foregoing may include, but are not limited to, those described below and elsewhere herein. The systems and methods described herein reduce the need for a user to manually interact with a mobile robot to initiate or control robotic operation within a home. Specifically, a user may issue voice commands to an audio media device to control the mobile robot, thereby enabling the user to control the mobile robot even when the user's hands are not available to operate a manual input device operable with the mobile robot. A user may also receive notifications regarding the status of the mobile robot without having to be physically close to the mobile robot. The audio media device may be positioned to provide audible notifications to the user, and these notifications may be provided to the user when the mobile robot autonomously navigates within the home, in some cases leaving the user's vicinity. Because the audio media device issues audible notifications, the user may be informed of the status of the mobile robot without the need for visual distraction.

[0063] The systems and methods described herein may further provide the mobile robot and audio media device with additional data regarding the state of the home that may improve the operation of the mobile robot and audio media device. In some examples, data collected by sensors on the audio media device may be uploaded to the audio media device or used, along with data collected by sensors on the mobile robot, to improve the accuracy or precision of a map of the home that the mobile robot constructs during autonomous navigation within the home. The mobile robot may also use data uploaded to or collected by the audio media device to improve the mobile robot's localization within the home, thereby reducing errors in the mobile robot's navigation behavior.

[0064] The systems and methods described herein can also reduce inefficiencies related to user interactions with or in connection with a mobile robot. For example, the systems and methods describe prediction of recommended actions that, if implemented, may prevent errors in the operation of a mobile robot from occurring. An audio media device can notify a user of the recommended actions before the user would normally realize that such actions would be beneficial. In some cases, a remote computing system can implement the recommended actions, thereby reducing the time spent by a user addressing errors or potential errors related to the mobile robot.

[0065] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other possible features, aspects, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0066] [Figure 1] FIG. 1 is a schematic top view of a mobile robot navigating in a home environment. [Figure 2] FIG. 2 is a block diagram of a control system. [Figure 3A] FIG. 1 is a schematic bottom view of an example mobile robot. [Figure 3B] 1 is a schematic diagram of an example of a sensor mounted on a mobile robot. [Figure 4] FIG. 1 is a schematic front view of an audio media device. [Figure 5] FIG. 1 illustrates a user providing audible commands to an audio media device. [Figure 6A] 1 is a flowchart illustrating the association process. [Figure 6B] FIG. 6B shows a screenshot of the example association process of FIG. 6A. [Figure 7] 1 is a flowchart illustrating a process for sending command signals to a mobile robot. [Figure 8] 10 is a flowchart illustrating a process for issuing an audible notification of the status of a mobile robot. [Figure 9] 1 is a flowchart illustrating a process for implementing recommended actions based on the behavior of a mobile robot. [Figure 10] 10 is a flowchart illustrating a process for issuing an audible command to correct an error condition on a mobile robot. [Figure 11A] FIG. 10 shows the mobile robot in the camera storage position. [Figure 11B] FIG. 10 shows the mobile robot in the camera storage position. [Figure 11C] FIG. 10 shows the mobile robot in the camera storage position. [Figure 11D] FIG. 10 shows the mobile robot in the camera storage position. [Figure 11E] FIG. 11B is a diagram showing the mobile robot of FIGS. 11A to 11D at the camera protruding position. [Figure 11F] FIG. 11B is a diagram showing the mobile robot of FIGS. 11A to 11D at the camera protruding position. [Figure 11G] FIG. 11B is a diagram showing the mobile robot of FIGS. 11A to 11D at the camera protruding position. [Figure 12] FIG. 2 is a schematic diagram illustrating an example of a mobile robot monitoring route. DETAILED DESCRIPTION OF THE INVENTION

[0067] Like reference numbers and designations in the various drawings indicate like elements.

[0068] 1, the remote computing system 200 can control the mobile robot 300 according to voice commands that the user 100 directs at the audio media device 400. Similarly, the remote computing system 200 can cause the audio media device 400 to emit audible signals indicative of the operation of the mobile robot 300. Referring to FIG. 2, which shows a diagram of an exemplary communication network 201, the mobile robot 300 and the audio media device 400 are both wirelessly linked to the remote computing system 200 (graphically represented as a server included in the computing system) to enable communication between the remote computing system 200 and the mobile robot 300 and between the remote computing system 200 and the audio media device 400.

[0069] The communications network 201 enables a user's voice commands directed to the audio media device 400 to be used as the basis for command signals to control the movement of the mobile robot 300. The user 100 operates the audio media device 400 to command the mobile robot 300 by uttering audible commands that are received by the audio media device 400. The remote computing system 200 is communicatively linked to the audio media device 400 and can receive signals from the audio media device 400 corresponding to the audible commands and then transmit another signal (or signals) to the mobile robot 300 to cause the mobile robot 300 to perform an action in accordance with the user's audible command.

[0070] In some examples, the audio media device 400 emits an audible signal that includes an audible notification that informs the user 100 of the status of the operation of the mobile robot 300. The remote computing system 200 generates a command signal to be sent to the audio media device 400 by analyzing input signals from one or more of the mobile robot 300, the mobile computing device 202 (e.g., the smartphone shown in FIG. 2 ), or other suitable sources that may provide information regarding the status of the operation of the mobile robot 300. The remote computing system 200 then transmits the command signal to the audio media device 400 to cause the audio media device 400 to emit the audible signal, thereby providing the user 100 with information regarding the operation of the mobile robot.

[0071] In some cases, the information provided to the user 100 by the audible signal includes a request to implement a recommended action that addresses a current or predicted need for the mobile robot 300. The remote computing system 200 may predict that the recommended action may improve the efficiency or effectiveness of the operation of the mobile robot 300. The audio media device 400 issues an audible request, for example, to adjust the operation of the mobile robot 300 or to implement an action that supports the operation of the mobile robot 300. If the user 100 verbally responds to the audio media device 400 agreeing with the audible request, the remote computing system 200 causes the action to be implemented to address the need for the mobile robot 300. For example, in one example, if the recommended action is to adjust a user-defined schedule for the mobile robot 300, the remote computing system 200 transmits the adjusted schedule to the mobile robot 300 and causes the mobile robot 300 to store the adjusted schedule, for example, in the mobile robot 300 or in a memory associated with the remote computing system 200. If the recommended action is to purchase a replacement part for the mobile robot 300, the remote computing system 200 sends a signal to an online marketplace to order the replacement part to be delivered to the user 100.

[0072] In some implementations, the mobile robot 300 and the audio media device 400 communicate acoustic signals directly with each other. For example, the audio media device 400 communicates acoustic signals directly to the mobile robot 300, and the mobile robot 300 communicates acoustic signals directly to the audio media device 400. The acoustic signals include direct commands and responses, for example, where the mobile robot 300 issues acoustic signals that identify the status of the mobile robot 300 and the audio media device 400 receives the acoustic signals directly. Alternatively or additionally, the audio media device issues acoustic commands that the mobile robot 300 receives directly, and the acoustic commands cause the mobile robot 300 to perform actions.

[0073] In some cases where the mobile robot 300 and the audio media device 400 communicate acoustic signals directly with each other, one of the mobile robot 300 and the audio media device 400 does not include a wireless communication system connecting the device to the wireless communication network 201, while the other of the mobile robot 300 and the audio media device 400 includes a wireless communication system for connecting the device to the wireless communication network 201. For example, if the audio media device 400 includes a wireless communication system, the user 100 provides an audible command to the audio media device 400 or remotely accesses the audio media device 400 to provide a command. The audio media device 400 then issues an acoustic command to the mobile robot 300 to initiate an action, e.g., a cleaning action or an observation action, upon receiving the audible user command to initiate an action. Similarly, if the mobile robot 300 includes a wireless communication system, the mobile robot 300 communicates an acoustic command to the audio media device 400, e.g., causing the audio media device 400 to order more reusable pads, in a manner similar to how the user 100 provides such an audible user command. The acoustic commands include, for example, voice commands or simple audible signals, such as tones associated with particular commands.

[0074] In some examples, the audio sensing capabilities of the audio media device 400 improve the localization of the mobile robot 300. For example, the audio media device 400 may be capable of measuring the loudness and / or direction of the acoustic output from the mobile robot 300, and that information may be used to determine the location of the mobile robot 300. In some examples, the mobile robot 300 localizes relative to the audio media device 400 in the environment and / or relative to the location of other devices emitting acoustic signals. In some implementations, the mobile robot 300 measures the loudness and direction of the acoustic signal and localizes that signal coming from the audio media device 400 and / or other devices.

[0075] In some examples, this type of audio sensing can be used to adjust the operation of audio input devices, e.g., microphones, and audio output devices, e.g., speakers, of the home environment audio media device 400. By communicating acoustic signals with the audio media device 400 at various locations within and / or outside the room 20, the mobile robot 300 enables the audio media device 400 to adjust the sensing level of its audio input device or adjust the output level, e.g., loudness, of its audio output device.

[0076] An exemplary environment for a mobile robot The mobile robot 300 and the audio media device 400 may operate within one or more enclosed spaces or within an environment including one or more enclosed spaces. The environment may include, for example, a home environment, a living space, a work environment, or other environment. The enclosed space may correspond, for example, to a room within the environment. In the exemplary environment shown in FIG. 1 , the environment includes a house 10 in which a user 100, a mobile robot 300, and an audio media device 400 are located. The mobile robot 300 is located in a room 20A. The audio media device 400 is deployed in a room 20B in which the user 100 is also located. The room 20A is adjacent to the room 20B and connected by an entrance 22A. The room 20A is a bedroom including a bed 22 and end tables 24, 26.

[0077] In the example shown in FIG. 1 , the mobile robot 300 autonomously navigates through the room 20A to complete a mission. If the mobile robot 300 is a cleaning robot, the mission may correspond to a cleaning mission to clean the floor of the room 20A. In some examples, the mission may correspond to a patrol in which the mobile robot 300 autonomously navigates to a predetermined location within the house 10. The mobile robot 300 navigates around obstacles located within the room 20A (e.g., the bed 22 and end tables 24, 26) while completing its mission. As the mobile robot 300 moves through the house 10 during a mission, the mobile robot 300 uses its sensors to generate a map of the house 10 and to localize the mobile robot 300 within the map. The mobile robot 300 includes other sensors that generate signals indicative of the status of the mobile robot, such as the status of parts of the mobile robot 300 or the status of a mission or operation being performed by the mobile robot 300.

[0078] In some implementations, in addition to the mobile robot 300 and the audio media device 400, the house 10 includes other devices that communicate with the remote computing system 200. In some implementations, the mobile computing device 202 shown in FIG. 2 is linked to the remote computing system 200 and allows the user 100 to provide input on the mobile computing device 202. The mobile computing device 202 may include user input elements such as, for example, one or more of a touchscreen display, buttons, a microphone, a mouse, a keyboard, or other devices that respond to input provided by the user 100. The mobile computing device 202 may alternatively or additionally include immersive media (e.g., virtual reality) for the user 100 to interact with to provide user input. The mobile computing device 202, in these cases, is, for example, a virtual reality headset or a head-mounted display. The user can provide input corresponding to commands for the mobile robot 300. In such a case, the mobile computing device 202 sends a signal to the remote computing system 200 to cause the remote computing system 200 to send a command signal to the mobile robot 300 .

[0079] 2 depicts the mobile computing device 202 as wirelessly linked to the remote computing system 200, in some implementations the communication network 201 of FIG. 2 includes a wireless link between the mobile computing device 202 and the mobile robot 300 to enable the mobile computing device 202 to transmit wireless command signals directly to the mobile robot 300. The user 100 provides user input to the mobile computing device 202 indicating a command signal, and the mobile computing device 202 then transmits a command signal corresponding to the user input. Various types of wireless networks (e.g., Bluetooth, radio frequency, optical-based, etc.) and network architectures (e.g., mesh networks) may be utilized by the communication network 201.

[0080] 1, home 10 includes linked devices 102A, 102B. In some implementations, linked devices 102A, 102B each include sensors suitable for monitoring home 10, monitoring the occupants of home 10, monitoring the operation of mobile robot 300, monitoring the operation of audio media device 400, etc. These sensors may include, for example, imaging sensors, occupancy sensors, environmental sensors, etc.

[0081] Imaging sensors for the linked devices 102A, 102B may include visible light, infrared cameras, sensors utilizing other portions of the electromagnetic spectrum, etc. The linked devices 102A, 102B transmit images generated by these imaging sensors to the remote computing system 200. Occupancy sensors for the linked devices 102A, 102B may include, for example, one or more of: passive or active transmissive or reflective infrared sensors; time-of-flight or triangulation sensors using light, sonar, or radio frequency; microphones for recognizing sound or sound pressure characteristics of occupancy; airflow sensors; cameras; wireless receivers or transceivers for monitoring frequencies for sufficiently strong received signal strength and / or WiFi frequencies; light sensors capable of detecting natural lighting, artificial lighting, and ambient light, including light emitted from a mobile computing device (e.g., the mobile computing device 202); and / or other suitable sensors for detecting the presence of the user 100 or other occupants in the home 10. The occupancy sensors may alternatively or additionally detect the movement of the user 100 or the movement of the autonomous mobile robot 300. If the motion sensors are sufficiently sensitive to the movement of the autonomous mobile robot 300, the motion sensors of the linked devices 102A, 102B generate signals indicative of the movement of the mobile robot 300. Environmental sensors for the linked devices 102A, 102B may include electronic thermometers, barometers, humidity or moisture sensors, gas detectors, airborne particle counters, etc. The linked devices 102A, 102B transmit sensor signals from a combination of imaging sensors, motion sensors, environmental sensors, and other sensors present in the linked devices 102A, 102B to the remote computing system 200. These signals serve as input data for the remote computing system 200 to implement the processes described herein for controlling or monitoring the operation of the mobile robot 300 and the audio media device 400.

[0082] In some examples, the remote computing system 200 is connected to multiple robotic devices, including a mobile robot 300 and a second mobile robot 301, thus allowing the user 100 to interact with the audio media device 400 to control and monitor the multiple robotic devices 300, 301. As shown in Figure 1, the second mobile robot 301 is located in room 20C, which is connected to room 20B by entrance 22B. Similar to the mobile robot 300, the second mobile robot 301 performs a mission, e.g., a cleaning mission, within room 20C.

[0083] The controllers for each of the audio media device 400, the mobile robot 300, the linked devices 102A, 102B, the second mobile robot 301, and the other devices may establish and maintain wireless links for communication with the remote computing system 200. The controllers may also establish and maintain wireless links directly with each other, for example, to establish and maintain a wireless link between the mobile robot 300 and the audio media device 400, or between the audio media device 400 and one of the linked devices 102A, 102B. Wireless links may also be formed with other remote electronic devices, such as a mobile phone, a tablet, a laptop, another mobile computing device, one or more environmental control devices, other types of electronic devices, etc. In some implementations, the wireless link permits communication with one or more devices including, but not limited to, a smart light bulb, a thermostat, a garage door opener, a door lock, a remote control, a television, a security system, a security camera, a smoke detector, a video game console, another robotic system, or other communication-enabled sensing and / or actuation device or appliance.

[0084] In the communication network 201 shown in FIG. 2 , and in other implementations of the communication network 201, the wireless links may use various communication methods and protocols, such as, for example, Bluetooth class, Wi-Fi, Bluetooth-low-energy, also known as BLE (BLE and BT classic are completely different protocols that simply share branding), 802.15.4, Worldwide Interoperability for Microwave Access (WiMAX), infrared channels, or satellite bands. In some cases, the wireless links include any cellular network standard used to communicate between mobile devices, including, but not limited to, standards recognized as 1G, 2G, 3G, or 4G. If used, the network standard may be recognized as one or more generations of a mobile telecommunications standard by meeting specifications or standards, such as, for example, specifications maintained by the International Telecommunications Union. If used, the 3G standard may correspond, for example, to the International Mobile Telecommunications-2000 (IMT-2000) specification, and the 4G standard may correspond to the International Mobile Telecommunications Advanced (IMT-Advanced) specification. Examples of cellular network standards include AMPS, GSM, GPRS, UMTS, LTE, LTE Advanced, Mobile WiMAX, and WiMAX-Advanced. Cellular network standards can use various channel access methods, for example, FDMA, TDMA, CDMA, or SDMA.

[0085] The communications network 201 allows the user 100 to interact with the audio media device 400 using natural speech to control or monitor the mobile robot 300. In one example shown in FIG. 1 , the remote computing system 200 determines that the battery for the mobile robot 300 (“Robot #1”) has a low enough charge capacity that it would be beneficial to order a replacement battery before the charge capacity becomes insufficient for the mobile robot 300 to perform an operation. The remote computing system 200 causes the audio media device 400 to issue an audible request: “Robot #1 will need a battery soon. Shall I order a new battery?” Upon hearing the audible request, the user 100 responds: “Yes, order a new cleaning pad.” The audio media device 400 sends a signal to the remote computing system 200 indicating the user's agreement to the audible request. The remote computing system 200 then sends a signal to an online marketplace to order a replacement battery to be delivered to the user's 100 home 10.

[0086] In some implementations, both the mobile robot 300 and the second mobile robot 301 are linked to the remote computing system 200. Information regarding the operation of both mobile robots 300, 301 can be transmitted to the remote computing system 200 for use in informing the user 100 of their operation. In some cases, in addition to causing the audio media device 400 to issue an audible request, the remote computing system 200 generates a command signal to cause the audio media device to issue an audible notification indicating the operation of the mobile robots 300, 301. In another example shown in FIG. 1 , the user 100 utters an audible request for information about the second mobile robot 301 ("Robot #2") to be issued by the audio media device 400 ("AMD"), i.e., "AMD. Where is Robot #2?" The audio media device 400 receives the audible request and transmits a wireless signal indicating the audible request to the remote computing system 200. The remote computing system 200 generates the command signal based on the location of the second mobile robot 301. The remote computing system 200 sends a command signal to the audio media device 400, which emits an audible notification indicating the location of the second mobile robot 301, i.e., "Robot #2 is in the bathroom."

[0087] Using the audio media device 400, the user 100 can track the operation of the mobile robot 300 and request commands to be sent to the mobile robot 300. If a second mobile robot 301 is also connected to the remote computing system 200, the audio media device 400 acts as a fleet management tool, allowing the user 100 to selectively control or request information regarding the operation of each of the mobile robots 300, 301. The exemplary process described above and elsewhere herein beneficially allows the user 100 to manage the mobile robot 300 in a convenient and efficient manner. The exemplary process also provides a hands-free method for the user 100 to manage the mobile robot 300, such that the user 100 does not have to manually provide input to command the mobile robot 300. The user 100 remains informed of the operation of the mobile robot 300 without having to manually operate an input device. As the mobile robot 300 moves around the house 10, the user 100 easily tracks the mobile robot 300 without having to know the location of the mobile robot 300 or move in close proximity to the location of the mobile robot 300.

[0088] Furthermore, in cases where the user 100 is unable to focus visual attention on notifications intended for the mobile robot 300, the user 100 continues to monitor the operation of the mobile robot 300 by listening to the audible signals emitted by the audio media device 400. For example, when the user 100 is performing a task that requires the user's visual attention, the user 100 can use verbal communication to command the mobile robot 300. The audio media device 400 also allows the user 100 to use natural verbal communication to control the mobile robot 300 without having to translate visual indicators or icons, thereby improving the ease with which, for example, the elderly, children, the visually impaired, and others can control the operation of the mobile robot 300.

[0089] Mobile robot example Mobile robots 300, 301 use a drive system and a series of sensors to autonomously navigate an environment, e.g., a house 10. FIG. 3A shows a schematic bottom view of the example mobile robot 300 of FIG. 1. FIG. 3A is described with respect to the mobile robot 300, which may additionally or alternatively correspond to a second mobile robot 301. The mobile robot 300 includes a drive system including drive wheels 302. In some cases, caster wheels 304 support the mobile robot 300 above a floor surface. The mobile robot 300 further includes a controller 306 operable with one or more motors connected to the drive wheels 302. The controller 306 of the mobile robot 300 selectively activates the motors to drive the drive wheels 302 and navigate the mobile robot 300 across a floor surface.

[0090] The controller 306 is also operable with a detection system 308. The detection system 308 includes sensors usable by the controller 306 to navigate within the house 10. The detection system 308 has sensors, for example, for detecting obstacles within the house 10 and for generating signals for generating a map of the house 10. The detection system 308 may include obstacle detection sensors, such as time-of-flight sensors for detecting distance to obstacles, cliff detection sensors for detecting steps (e.g., stairs), impact sensors associated with shock absorbers mounted on the mobile robot 300, and contact sensors. The controller 306 operates a drive system for the mobile robot 300 to move around the obstacle when the obstacle detection sensors detect an obstacle.

[0091] The controller 306 uses signals from its sensor system to generate a map of the house 10 by tracking and updating the position and orientation of the mobile robot 300 over time. These mapping sensors include, for example, simultaneous localization and mapping (SLAM) sensors, dead reckoning sensors, and obstacle detection and avoidance (ODOA) sensors. The controller 306 builds a two-dimensional map of the house 10 floor, determines the robot's pose on the map, and determines the location of parts of the house 10 that the mobile robot 300 can traverse (e.g., clear, passable floors). Using signals from the dead reckoning sensors and contact and non-contact obstacle detection sensors, the controller 306 indicates floors where the mobile robot 300 cannot traverse due to obstacles above or below the floor surface. In one example, the controller 306 builds a map of walls and obstacles as they pass through, creating an occupancy grid of passable, occupied space. In some implementations, the map uses a Cartesian or polar coordinate system. In some cases, the map is a topological map, a representational map, or a probabilistic map.

[0092] In some examples, using simultaneous localization and mapping (SLAM) techniques, the controller 306 determines the pose of the mobile robot 300 within a two-dimensional map of the house 10. The SLAM sensors include, for example, one or more cameras for visual identification of features and landmarks used in calculating the robot's pose on the map. The mobile robot 300 includes additional sensors that generate signals to enable the controller 306 to estimate the position and / or orientation of the mobile robot 300 as it moves around the house 10. These sensors, alone or in combination with the SLAM sensors, determine the pose of the mobile robot 300 on a robot map constructed by the mobile robot 300 as it passes through. In some implementations, the controller 306 uses signals from the additional sensors to verify or adjust the pose determined by the SLAM sensors. In some implementations, the additional sensors include an odometer, an accelerometer, a gyroscope, an inertial measurement unit, and / or other sensors that generate signals indicative of the distance traveled, amount of rotation, speed, or acceleration of the mobile robot 300. For example, the mobile robot 300 includes an orientation sensor, such as a gyroscope, that generates a signal indicating the amount the mobile robot 300 has rotated from its nose. In some implementations, the detection system 308 includes a dead-reckoning sensor, such as an IR wheel encoder, to generate a signal indicating the rotation of the drive wheels 302, and the controller 306 uses the detected rotation to estimate the distance traveled by the mobile robot 300. In some implementations, the detection system 308 includes, for example, a laser scanner or time-of-flight sensor that generates sensor readings to determine the distance to observed obstacles and objects in the environment. Alternatively or additionally, the detection system 308 includes a floor-facing optical mouse sensor to determine the distance the mobile robot 300 has drifted laterally on the floor relative to its nose.

[0093] In some implementations, the mobile robot 300 utilizes visual simultaneous localization and mapping (VSLAM) to build its map and determine its current pose on the map. The sensing system 308 includes one or more localization sensors, e.g., a camera 310, that generate signals for the controller 306 to determine the location and orientation of the mobile robot relative to features detected in the environment. In some implementations, the mobile robot 300 includes a visible light camera 310 below the top surface of the robot body, angled upward, e.g., between 30 and 80 degrees from the floor surface along which the mobile robot 300 navigates. The camera 310 aims at locations on walls and ceilings with a high concentration of static elements such as window frames, picture frames, doorway frames, and other objects with visually detectable features like lines, corners, and edges. For example, when the camera 310 is angled upward, the center of the camera's 310's view cone is angled upward such that the center of the view cone is aimed at locations on the walls and ceiling. Using images captured by the camera 310, the controller 306 determines the robot's pose on a map that the mobile robot 300 constructs as it navigates through a room or enclosed space, such as a series of adjacent rooms 20A, 20B, 20C, and 20D (collectively referred to as the enclosed space or room 20).

[0094] The location estimation sensors include sensors on the mobile robot 300 that can generate signals in response to detecting walls and objects in the environment that, in some cases, block impassable floor spaces. In addition to VSLAM cameras, these location estimation sensors include contact sensors, such as impact sensors, as well as non-contact time-of-flight sensors, such as lasers, volumetric point cloud sensors, point line sensors (e.g., time-of-flight line sensors, such as those manufactured by PIXART), IR proximity sensors, LIDAR, and acoustic sensors. The location estimation sensors generate signals from which unique signatures, patterns, or features can be extracted that, among other things, distinguish impassable floors from traversable floors or traversable floor spaces that are added to the augmented robot map as the mobile robot 300 traverses them. When the controller 306 determines that these features have been detected, the controller 306 uses the location and orientation of the mobile robot 300 relative to these detected features to determine the pose of the mobile robot 300 on the map of the house 10. The controller 306 localizes the mobile robot 300 within the house 10 by, among other things, determining the current pose of the mobile robot 300 relative to features corresponding to objects within the house 10. The extracted features indicate the room in which the mobile robot 300 is located. Referring again to FIG. 1 , the extracted features form a unique identifier for each of the rooms 20A, 20B, and 20C. In some implementations, the mobile robot 300 uses the extracted features to determine in which of the rooms 20A, 20B, and 20C it is currently located in responsive to detecting a particular feature or features associated with the room identifier. In some implementations, the mobile robot 300 recognizes pre-identified rooms through object recognition. For example, the mobile robot 300 uses its camera 310 to capture images of objects (e.g., stove, dishwasher, and refrigerator) associated with each room 20. The user 100 communicates to the mobile robot 300 the specific room identifier (e.g., kitchen) associated with those recognizable objects. During a mission, the mobile robot 300 recognizes these objects and communicates their location to, for example, the AMD400 or a mobile computing device by triggering the issuance of an audible alert. 202 to generate an audible alert or to issue a visual alert, for example, by displaying a text notification on the mobile computing device 202 indicating the associated stored room identifier.

[0095] In some implementations, the map is persistent and stored in the remote computing system 200 for access by one or more mobile robots 300, 301 and / or AMD 400. On each subsequent run, the mobile robot 300 updates the persistent map according to changing conditions in the home 10, such as moved furniture or the new location of the AMD 400. In some examples, the mobile robot 300 discovers connected devices in the home 10 and localizes them on the map through standard protocols. This includes the locations of connected lights and speakers, vents, door and window sensors, and other connected devices in the home 10. The mobile robot 300 roams the home 10 and uses RF signatures, visual recognition, received signal strength, and other methods to recognize connected devices in the home 10 and automatically place them on the robot map of the home 10. For example, the mobile robot 300 explores the home 10 and recognizes the NEST thermostat on the living room wall, the connected SAMSUNG refrigerator in the kitchen, and the HUE BLOOM lights in both the family room and the bedroom. The mobile robot 300 maps the recognized connected devices, allowing the user 100 and the AMD 400 to take advantage of this spatial knowledge of the connected devices. For example, the user 100 can turn on the bedroom light by speaking the phrase, "Turn on the bedroom light." The interaction between the AMD 400 and the mobile robot 300 described herein enables this simple speech control via devices detected by the mobile robot 300 as it navigates through the home 10.

[0096] The detection system 308 also generates signals indicative of the movement of the mobile robot 300. In some examples, the detection system 308 includes a stall sensor unit integrated with the drive system that generates a signal indicative of a stall condition of the mobile robot 300, in which the mobile robot 300 is unable to move along a floor surface within the house 10. The stall sensor unit generates a signal indicative of a change in current supplied to the motor of the drive system. The change in current may indicate a stationary condition for the mobile robot 300, in which the mobile robot 300 is unable to move substantially from its current posture. The stall sensor unit may alternatively or additionally include an optical sensor that generates a signal indicative of whether a wheel, e.g., one of the caster wheels 304 or the drive wheel 302, is moving when power is supplied to the motor of the drive wheel 302. The stall sensor unit is, in some cases, a mouse sensor for tracking and detecting movement, or lack of movement, by comparing successive images for changes; in other implementations, the mobile robot 300 relies on an accelerometer to generate a signal indicative of the acceleration of the mobile robot 300. When the controller 306 detects a lack of wheel movement, it determines that the mobile robot 300 is in a stall state.

[0097] In some implementations, the mobile robot 300 includes other sensors, as shown in the schematic diagram of FIG. 3B. The sensing system 308, in some examples, includes a microphone 314 that receives audible signals from the mobile robot's 300 environment. In some cases, the sensing system 308 includes environmental sensors, such as a temperature sensor 2070C, an ambient light sensor 2070D, an air moisture content sensor 2070J, a gas composition, air quality sensor 2070I, or other characteristics of the environment. The sensing system 308 also includes status sensors that indicate the status of the mobile robot 300 or components of the mobile robot 300. These sensors include, for example, a battery charge status sensor to detect the amount of charge or capacity for charge in the mobile robot's 300 power source, and component life sensors, such as a wheel tread sensor to detect the operability of a component or the amount of component life remaining, etc.

[0098] The mobile robot 300 further includes an audio emitting system 312 that enables the mobile robot 300 to emit audible signals. The controller 306 causes the emission of the audible signals, for example, to notify the user 100 of the status of the mobile robot 300, such as the status of parts of the mobile robot 300, the status of the operation of the mobile robot 300, or the status of a mission performed by the mobile robot 300. As described in more detail herein, the controller 306 can operate the audio emitting system 312 to communicate audible signals to be received by a microphone unit 402 of the audio media device 400. In some cases, the audio media device 400 uses a speaker unit to communicate audible signals to be received by a microphone 314 of the mobile robot 300.

[0099] The mobile robot 300 further includes a wireless communication system 316 that enables the mobile robot 300 to communicate with the remote computing system 200, as shown in the communication network 201 of FIG. 2. Using the wireless communication system 316, the controller 306 transmits data to the remote computing system 200. In some examples, the data includes signals generated by sensors in the detection system 308. In some implementations of the mobile robot 300 that include an image capture system 310, captured images can be transmitted directly to the remote computing system 200. In some examples, the mobile robot 300 collects information and builds a map of the house 10, and the controller 306 transmits the map to the remote computing system 200. If the controller 306 includes any state sensors, the controller 306 also transmits information indicative of the state of the mobile robot 300 to the remote computing system 200.

[0100] As described with respect to FIG. 1 , during its navigation of the home 10, the mobile robot 300 performs actions and completes missions within the home 10. The actions performed depend on the type of mobile robot 300. In some cases, when the user 100 directs a voice command to the audio media device 400 to instruct the mobile robot 300 to perform an action, the actions available for execution by the mobile robot 300 depend on the type of mobile robot 300. In addition to illustrating basic components that may be present on many types of mobile robots for implementations described herein, FIG. 3A illustrates components specific to a vacuum cleaning robot, which corresponds to one of many types of mobile robots that may benefit from the processes described herein. Other mobile robots 300 may include floor-scrubbing robots, house monitoring robots, robotic lawn mowers, mopping robots, companion robots, sweeping robots, combinations thereof, and other suitable robots. As described in more detail herein, each of these robots may benefit from the processes and systems described herein.

[0101] In some examples, the mobile robot 300 is a vacuum cleaning robot that includes a cleaning system for sucking up debris from a floor surface. The cleaning system includes, for example, rotatable rollers or brushes 317 that agitate debris from the floor surface into a dustbin (not shown) mounted on the mobile robot 300. The cleaning system includes an air mover that, when activated, moves air, thereby moving debris from the floor surface toward the dustbin. As the mobile robot 300 navigates its environment during a cleaning mission, the mobile robot 300 activates its cleaning system to suck up debris, thereby cleaning the floor surface.

[0102] In some cases, if the mobile robot 300 is a vacuum cleaning robot, the detection system 308 includes a dustbin level sensor that detects the amount of dust sucked into a removable dustbin for the vacuum cleaning robot. The detection system 308 includes one or more dust sensors that detect when the vacuum cleaning robot has sucked up dust or that detect the rate of dust suction. In some examples, the mobile robot 300 includes a dust filter, and the detection system 308 also includes a filter sensor to detect whether the filter needs cleaning.

[0103] The controller 306 has access to a memory that stores information collected by the sensors and routines executable by the controller 306 to cause the mobile robot 300 to perform operations within the home 10. As shown in FIG. 3A , in some examples, the mobile robot 300 includes a memory 318. The routines include, for example, navigation routines for navigating the mobile robot 300 in the home 10. The controller 306 initiates operations of the mobile robot 300 in response to, for example, signals from the sensing system 308 or wireless command signals transmitted to the controller 306 through the wireless communication system 316. If the mobile robot 300 includes a user input device, such as a manually operable button, the input device can be operated by the user 100 to cause the controller 306 to initiate one or more operations of the mobile robot 300. The manually operable button corresponds, for example, to a push button or button icon on a touchscreen display. In some cases, the memory 318 also stores deterministic patterns of movement that the controller 306 implements to navigate the mobile robot 300 through the home 10. The patterns may include, for example, a linear motion pattern, a vine pattern, a cornrow pattern, a spiral pattern, a zigzag pattern, or other patterns including combinations of patterns. Memory 318 also stores data collected by sensors of detection system 308, including any of the dead reckoning sensors, position estimation sensors, status sensors, or other sensors of detection system 308. If controller 306 has constructed a map of home 10, controller 306 optionally stores the map in memory 318 for reuse in subsequent cleaning missions.

[0104] The actions available to the mobile robot 300 depend on the type of mobile robot 300. For example, if the mobile robot 300 is a vacuum cleaning robot as described herein, the memory 318 includes routines for performing floor-cleaning operations. When the vacuum cleaning robot receives a command to begin a cleaning mission, the vacuum cleaning robot performs floor-cleaning operations by autonomously navigating its environment and sucking debris from the floor surface. Floor-cleaning operations include room-cleaning operations in which the controller 306 navigates the vacuum cleaning robot in a pattern, such as a cornrow pattern, a spiral pattern, or other suitable motion pattern, to cover the floor surfaces of a room or multiple rooms.

[0105] In some cases, the floor cleaning operation includes a spot cleaning operation in which the vacuum cleaning robot limits its cleaning operation to a localized area when it receives a command to perform the spot cleaning operation. The localized area may include a relatively large amount of detected debris detected by a debris sensor. As part of the spot cleaning operation, the controller 306 additionally or alternatively increases the power supplied to the vacuum cleaning robot's air mover to allow the debris to be more easily sucked up by the vacuum cleaning robot. To perform the spot cleaning operation, the controller 306 controls the drive system to move the vacuum cleaning robot within the localized area in a predetermined pattern, for example, a spiral pattern. Initiation of any of the floor cleaning operations may also occur in response to a sensor signal. If the vacuum cleaning robot includes a debris sensor, the controller 306 can control the vacuum cleaning robot to perform a spot cleaning operation in response to the detection of debris by the debris sensor. A wireless command signal from the remote computing system 200 can also initiate a spot cleaning operation, a room cleaning operation, or other operation of the vacuum cleaning robot.

[0106] In some implementations, the mobile robot 300 communicates or otherwise interacts with other devices in its environment. The mobile robot 300 includes a rechargeable battery that can be recharged, for example, at a station electrically connectable to the battery. In some cases, the battery is a removable battery that is plugged into the station, and in other cases, the mobile robot 300 docks at the station, thereby allowing the station to recharge the battery. As shown in FIG. 1 , a docking station 104 is located within the room 20A. The docking station 104 includes a charger operable to charge the battery of the mobile robot 300 when the mobile robot 300 is docked to, e.g., physically and / or electrically connected to, the docking station 104. If the mobile robot 300 is a vacuum cleaning robot, the docking station 104 additionally or alternatively serves as a discharge station that includes a motorized bin for transferring waste from the vacuum cleaning robot's waste bin.

[0107] In another example shown in FIG. 1 , a transmitting unit 106 positioned proximate entrance 22B transmits an axial suppression beam spanning at least the length of entrance 22B. The transmitting unit 106 is positioned such that the emitted suppression beam separates room 20C from room 20B. In some cases, the sensing system 308 includes an omnidirectional detector that detects the emitted suppression beam. In response to detecting the suppression beam, the controller 306 navigates the mobile robot 300 to avoid crossing the suppression beam, thereby maintaining autonomous navigation of the mobile robot 300 within or outside room 20C. For example, the mobile robot 300 moves away from the suppression beam when the omnidirectional detector detects the suppression beam. Referring to FIG. 1 , if a second mobile robot 301 includes such an omnidirectional detector, the transmitting unit 106 restricts the second mobile robot 301 to autonomously navigate within room 20C without crossing the suppression beam and, therefore, entrance 22B.

[0108] In some cases, one or more transmitting units emit signals into the environment that can be detected by localization sensors on the mobile robot 300. The signals are, for example, optical or acoustic signals that remain stationary within the house 10. For example, if the transmitting unit transmits a narrowly directed acoustic signal into the house 10 while the mobile robot 300 navigates through the house 10, when an acoustic receiver on the mobile robot 300 receives the acoustic signal, the controller 306 focuses on the acoustic signal. The acoustic signal may be directed toward a wall surface so that the acoustic receiver detects a reflected acoustic signal, or the acoustic signal may be directed toward a floor surface so that the acoustic receiver receives a direct emission of the acoustic signal. The transmitting unit transmits one or more of these signals into the house 10, and the mobile robot 300 uses each of these signals as a localization feature. In some cases, the transmitting unit emits a narrowly focused light beam into the house 10 that the sensing system 308 detects and the controller 306 uses to localize the mobile robot 300 within the house 10.

[0109] Audio Media Device Examples 4 shows an example of an audio media device 400 positionable on a surface within home 10, for example, on a table in room 20B, to enable audio interactions with user 100. These audio interactions include the playback of audible signals emitted by audio media device 400 for user 100 to hear and speech uttered by user 100 to be received by audio media device 400. To support these interactions, audio media device 400 includes audio transducers, for example, one or more microphones and one or more speakers, for converting acoustic signals to electrical signals or electrical signals to acoustic signals. Audio media device 400 includes a microphone unit 402 with at least one microphone for receiving audible signals and at least one speaker unit 404 with at least one speaker for emitting audible signals.

[0110] The audio media device 400, in some cases, includes non-audio control input mechanisms, such as one or more input buttons for increasing / decreasing the volume or for turning the audio media device 400 on or off. In some examples, a visual indicator 406 on the audio media device 400 indicates the status of the audio media device 400, such as when the power for the audio media device 400 is on. The visual indicator 406 indicates, for example, the volume or power status for the audio media device 400.

[0111] The primary form of user input for the audio media device 400 corresponds to speech spoken and received by the user 100, and the primary form of user output corresponds to audible utterances to be heard by the user 100. The user 100 interacts and communicates with the audio media device 400 through natural speech. In some implementations, the audio media device 400 is controllable using only audible signals and therefore does not require the user 100 to operate an input device or display. In such cases, the audio media device 400 does not include tactile input devices such as control buttons, keypads, joysticks, keyboards, touchscreens, and other alternative tactile devices. Furthermore, in some implementations, the audio media device 400 provides notifications and information to the user 100 using only audible signals. The audio media device does not include a display for the user 100 to read, for example, text or graphical output. In some implementations, the audio media device 400 includes a visual indicator 406, while in other implementations, the audio media device 400 can provide non-tactile, non-visual user input and output mechanisms to the user 100.

[0112] In addition to the microphone unit 402, the audio media device 400 may include other sensors, such as an image capture sensor, a motion detection sensor, an optical sensor, a global positioning system (GPS) transceiver, a device presence sensor (e.g., for geofencing), and other sensors that can detect the state of the environment of the audio media device 400. In some cases, the audio media device 400 includes a sensor for detecting the amount of light in the environment, and the audio media device 400 activates a lamp to illuminate the environment in low light conditions. In some cases, the audio media device 400 includes a camera for detecting the distance and orientation of the user 100 from the audio media device 400 or for enabling teleconferencing operations using the camera.

[0113] The audio media device 400 further includes a controller 408 for operating the audio media device 400. The controller 408 has access to memory. In some implementations, the audio media device 400 includes memory 410 to which the controller 408 has access. In some cases, the controller 408 alternatively or additionally uses a wireless communication system 412 to access remote memory, for example, remote memory associated with the remote computing system 200. In addition to enabling remote storage of data, the wireless communication system 412 enables the controller 408 to communicate with the remote computing system 200, for example, as shown in FIG. 2 . Assuming the wireless communication system for the mobile robots 300, 301 is also capable of short-range communication, the wireless communication system 412 may be capable of implementing short-range communication with nearby devices, such as the mobile robots 300, 301. The wireless communication system 412 can transmit information collected by the microphone unit 402 and other sensors from the audio media device 400 to the remote computing system 200. The wireless communication system 412 also allows the controller 408 to receive signals from the remote computing system 200 .

[0114] In some examples, the controller 408 has access to a distance estimation routine that estimates the distance between the microphone unit 402 of the audio media device 400 and the origin of the audio received by the microphone unit 402. If the received audio is, for example, an utterance from the user 100, the controller 408 of the audio media device 400 executes or causes the execution of the distance estimation routine to determine the distance between the user and the microphone unit 402. If the microphone unit 402 has multiple microphones, the distance estimation routine includes, for example, calculating the time difference of arrival of acoustic signals at the microphones.

[0115] The microphone unit 402 may also capture audio other than vocalizations originating from the user 100. In some cases, as the user 100 walks around the audio media device 400, the audio media device 400 detects the footsteps of the user 100. Using these sounds, a distance estimation routine determines the distance of the user 100 from the audio media device 400 and, in some examples, the speed of the user 100.

[0116] The microphone unit 402 can detect acoustic signals emitted by the audio media device 400, particularly by detecting reflections of the emitted acoustic signals within the home 10. In some examples, the controller 408 causes the speaker unit 404 of the audio media device 400 to generate acoustic signals, and the microphone unit 402 receives corresponding acoustic reflections after the acoustic signals travel through the environment of the audio media device 400. The controller 408 can have access to an acoustic mapping routine that, when executed, forms an acoustic map of the environment.

[0117] When the controller 408 executes the acoustic mapping routine, the speaker unit 404 may emit multiple acoustic signals simultaneously and / or sequentially, having different directions, frequencies, intensities, and / or other acoustic characteristics. The acoustic characteristics of the emitted and reflected acoustic signals may indicate the characteristics of surfaces and objects in the environment that cause the reflections. Based on the echo characteristics of the environment and the different intensities of acoustic reflections within the environment, the acoustic mapping routine determines the distances of surfaces and objects within the environment of the audio media device 400. In some implementations, the acoustic mapping routine uses echolocation acoustic ranging or the like to determine the distances and spatial characteristics of the environment of the audio media device 400.

[0118] In some cases, the controller 408 filters the received audio to reduce the effect of noise on the estimated distance between objects in the environment. For example, filtering out reverberation in the acoustic signal, resulting in several acoustic signals having different arrival times at the microphones of the microphone unit 402, reduces the likelihood of estimating distance based on reverberation.

[0119] In some examples, the acoustic mapping routine includes forming a baseline acoustic map of the environment when the environment is devoid of moving entities, humans, or while the audio media device 400 is otherwise stationary within the environment. In FIG. 1 , the environment corresponds to the home 10. The constructed acoustic map of the home 10 includes, for example, the locations of surfaces and other objects that cause acoustic reflections received by the microphone unit 402. The acoustic mapping routine provides the location of the audio media device 400 within the baseline acoustic map relative to the surfaces and other objects within the home 10. The acoustic mapping routine further measures echo direction and origin pose when an occupant, such as the user 100, is present in the home 10. In this regard, the acoustic mapping routine can estimate the location of the user 100 within the acoustic map of the home 10.

[0120] In some cases, the controller 408 extracts distinctive features or landmarks using the acoustic signals detected by the microphone unit 402. The controller 408 uses techniques such as, for example, sound-based SLAM, in which the controller 408 extracts acoustic reflection features from the signals detected by the microphone unit 402. For example, the acoustic reflection features may correspond to a particular geometry of objects within the house 10 that causes a pattern of reflected acoustic signals to be detected by the microphone unit 402.

[0121] These features may enable location estimation of the audio media device 400 within the home 10. In some implementations, the baseline acoustic map corresponds to an acoustic map constructed when the audio media device 400 is at an initial location within the environment. The audio media device 400 may periodically emit sounds according to an acoustic map creation routine to construct subsequent acoustic maps. If a subsequent acoustic map is constructed while the audio media device 400 is in operation after the baseline acoustic map is constructed, the subsequent acoustic map may, in some cases, exhibit a discrepancy with the baseline acoustic map. The discrepancy may indicate that the audio media device 400 has moved from its initial location within the home 10. When the audio media device 400 is moved from its initial location, the locations of some features in the subsequent acoustic map may similarly move relative to their locations in the baseline acoustic map. The acoustic map creation routine causes relative shifting of features to determine the new location of the audio media device 400 in the subsequent acoustic map, thereby enabling the controller 408 to localize the audio media device 400 within the home 10.

[0122] In some cases, in addition to or as an alternative to using the detected distance and geometry of an object to extract features for location estimation, the controller 408 uses other characteristics of the received acoustic signal to extract features for location estimation. After creating a map of the environment using an acoustic mapping routine, the controller 408, for example, causes the speaker unit 404 to emit an acoustic signal at a range of frequencies. Objects within the home 10 may have frequency responses that form unique signatures that can serve as features for location estimation of the audio media device 400 within the home 10. For example, some surfaces absorb acoustic signals at a wide range of frequencies and cause reflections at a narrow range of frequencies. Such frequency responses may be unique to particular surfaces or objects within the home 10. If the audio media device 400 is moved to a new location, the controller 408 can use the unique frequency responses to detect the particular surfaces and then determine the change in distance between the audio media device 400 and the particular surfaces. The controller 408 can thus use the frequency responses as features for location estimation.

[0123] The controller 408 can also use other acoustic localization features. In some implementations, reverberation caused by objects in the environment can exhibit unique characteristics that can be discernible in the acoustic signals received by the microphone unit 402. Detection of reverberation can therefore also serve as a feature for localization. In some cases, the home 10 includes one or more audio-emitting units that emit acoustic signals at unique frequencies. The audio-emitting units act as fixed transmitting beacons that the controller 408 uses to triangulate its location within the home 10.

[0124] The controller 408 has access to a sound recognition routine, which can be executed to recognize different sounds in the audible signal. In some examples, the memory 410 includes the sound recognition routine to be executed by the controller 408. In some implementations, the controller 408 sends a signal corresponding to the audible signal to the remote computing system 200 to cause the remote computing system to execute the sound recognition routine. The sound recognition routine includes a speech recognition subroutine for recognizing speech from words spoken by the user 100. In response to the recognition of a particular word, a corresponding signal is generated, including, for example, a command signal, a notification signal, or other signal. The signal may, for example, cause the audio media device 400, the mobile robot 300, or the second mobile robot 301 to perform an action.

[0125] In some examples, the sound recognition routine is executed to recognize common audible signals expected in a home environment or expected from a home environment with a mobile robot. In some implementations, the sound recognition routine identifies relatively low frequency noises from appliances in the home 10. The sound recognition routine identifies noises originating from the mobile robot 300, such as activation of an air mover if the mobile robot 300 is a vacuum cleaner. In some cases, the sound recognition routine identifies sounds caused by the drive mechanism of the mobile robot 300, such as activation of a motor or movement of the drive wheels 302 on different types of floor surfaces. Alternatively or additionally, the sound recognition routine identifies sounds caused by opening and closing doors and windows in the home.

[0126] In some cases, the acoustic signals emitted by the speaker unit 404 are non-audible, ultrasonic signals. Alternatively, or additionally, the acoustic signals include audible tones. While the speaker unit 404 and the microphone unit 402 have been described as being used for acoustic mapping, the acoustic mapping routine may alternatively cause an auxiliary speaker and an auxiliary microphone to emit and receive acoustic signals, respectively. These acoustic signals for the auxiliary speaker and microphone have frequencies outside the frequency response ranges of the speaker unit 404 and the microphone unit 402, for example. In some cases, the auxiliary speaker and microphone emit and receive acoustic signals in the ultrasonic or inaudible range, while the speaker unit 404 and the microphone unit 402 emit and receive acoustic signals in the audible range. In this regard, the auxiliary speaker causes minimal acoustic interference for the microphone unit 402 when the acoustic speaker emits its acoustic signals. Similarly, when the speaker unit 404 emits its acoustic signals, the speaker unit 404 causes minimal acoustic interference with the auxiliary microphone.

[0127] Example Process for a Mobile Robot and Audio Media Device The wireless communication system for the mobile robot 300 and the audio media device 400 enables transmission of data between the mobile robot 300 and the audio media device 400, in some cases, through a remote computing system 200, as shown in communication network 201 of FIG. 2. The remote computing system 200 includes computing resources separate from the environment of the mobile robot 300 and the audio media device 400, for example, separate from the home 10. In some cases, the remote computing system 200 includes one or more servers 204 that establish wireless links with each of the mobile robot 300 and the audio media device 400. The one or more servers 204 are, in some cases, connected to a network of one or more remotely located servers (a "cloud" computing network 205). The remote computing system 200 includes portions of a network-accessible computing platform implemented as a computing infrastructure, such as processors, storage, software, data access, etc., maintained and accessible through a communication network, as described herein. The remote computing system does not require the user 100 to know the physical location and configuration of the system, nor does the remote computing system require the user 100 to know the routines executed by or the services provided by the remote computing system 200.

[0128] 5, audio media device identification database 502 stores identification data for audio media devices wirelessly linked to remote computing system 200. Mobile robot identification database 504 stores identification data for mobile robots wirelessly linked to remote computing system 200. In the exemplary case described with respect to FIGS. 1, 2, and 5, audio media device identification database 502 includes audio media device identification data for audio media device 400. Mobile robot identification database 504 includes robot identification data for mobile robot 300 and any additional robots, such as second mobile robot 301 shown in exemplary house 10 of FIG. 1.

[0129] 6A , which will now be described in more detail, execution of an association process 600 associates identification data for the mobile robot 300 with identification data for the audio media device 400. After the identification data is associated, the remote computing system 200 can facilitate the transmission of information, such as sensor data, commands, notifications, etc., between the mobile robot 300 and the audio media device 400. In some examples, this increases the efficiency of the computing process, as the remote computing system 200 assists in processing routines and operations and storing information communicated by the audio media device 400 and / or the mobile robot 300.

[0130] The association of identification data for the mobile robot 300 with identification data for the audio media device 400 enables the transmission of data that may enhance the navigation operations of the mobile robot 300. For example, sensors on the audio media device 400 generate signals that are used to estimate the position of the mobile robot 300 or to localize the mobile robot 300 within the home 10. As described with respect to FIG. 3A , dead reckoning and / or localization sensors for the mobile robot 300 may include sensors that generate signals from which features can be extracted to localize the mobile robot 300. The dead reckoning and / or localization sensors may alternatively or additionally include sensors remote from the mobile robot 300, for example, sensors on the audio media device 400 or linked devices 102A, 102B.

[0131] The mobile robot 300 may emit signals detectable by the remote dead reckoning sensors such that changes in the relative position and / or orientation of the mobile robot 300 as it navigates through an environment can be estimated using the output from these sensors. In some implementations, acoustic signals received by the audio media device 400 can be used to determine the position and orientation of the mobile robot 300 relative to the position and orientation of the audio media device 400 as it navigates through the house 10. In some implementations, the audio media device 400 receives acoustic signals originating from the mobile robot 300. If the mobile robot 300 is a vacuum cleaning robot, the acoustic signals correspond, for example, to noise from activation of an air mover of the vacuum cleaning robot. In some cases, activation of a drive system to move the mobile robot 300 generates acoustic signals that are received by the microphone 402 of the audio media device 400. If the mobile robot 300 includes an audio emitting system 312, the mobile robot 300 alternatively emits an acoustic signal to be received by the audio media device 400. The acoustic signal may be either an audible acoustic signal (e.g., within the human hearing range) or an inaudible acoustic signal (e.g., within the ultrasonic range).

[0132] In examples where the audio media device 400 remains stationary within the home 10 while the mobile robot 300 moves relative to the audio media device 400, the strength of the acoustic signals received by the audio media device 400 changes as the mobile robot 300 moves through the home 10. If the location of the audio media device 400 within the home 10 is known or determined using techniques described herein, for example, using an acoustic mapping routine, detection of the acoustic signals originating at the mobile robot 300 can be used to localize the mobile robot 300 within the home 10. Once the audio media device 400 receives the acoustic signals, for example, a distance estimation routine is executed by the remote computing system 200 to estimate the distance between the mobile robot 300 and the audio media device 400. In some cases, the remote computing system 200 also determines the direction of the mobile robot 300 relative to the audio media device 400 (e.g., utilizing one or more direction finding techniques). Because the location of the audio media device 400 within the house 10 has been previously determined, the remote computing system 200 can use the received acoustic signals to localize the mobile robot 300 within the house 10. In some implementations, the robot map and the acoustic map are compared to reduce discrepancies between the maps. In particular, the robot map can be constructed using sensors of the mobile robot 300, such as VSLAM, ODOA sensors, and dead reckoning sensors, while the mobile robot 300 navigates in the environment, and the acoustic map can be constructed by running an acoustic map creation routine while the audio media device 400 operates its microphone unit and speaker unit. In one example of analysis to reduce discrepancies, the remote computing system 200 receives both the robot map and the acoustic map and compares extracted features from the robot map with extracted features from the acoustic map. If the robot map is a two-dimensional map of the floor of the house 10 and the acoustic map is a three-dimensional map of the house 10, the acoustic map can be analyzed to determine a two-dimensional map corresponding to the floor of the house 10. The three-dimensional geometric features on the map may be further processed to correspond to two-dimensional features on the robot map, such as obstacles detected by the robot sensors on or above the floor. The extracted features from the robot map may include features extracted from images captured by the mobile robot's image capture system 310 or features extracted using obstacle detection sensors or other sensors of the sensing system 308, with the extracted features having unique geometric properties. Similarly, the acoustic map constructed by the audio media device 400 also includes extracted features with unique geometric properties. In some cases, by comparing these extracted features from the two maps, the remote computing system 200 determines whether the acoustic map and the robot map have any discrepancies regarding the location or orientation of these extracted features.

[0133] In some implementations, the remote computing system 200 adjusts the acoustic map and / or the robot map in response to detecting a discrepancy using statistical or probabilistic modeling. Once the acoustic and robot maps are constructed, confidence ratings are assigned to features within the maps. The remote computing system 200 uses these confidence ratings to determine the relative predictive accuracy of different portions of each map. When the remote computing system 200 adjusts a map, it may adjust portions of one map according to similar portions in the other map. For example, referring to FIG. 1 , because the mobile robot 300 is located in room 20A, a robot map constructed using the mobile robot 300 may have a higher confidence rating for features extracted from room 20A. On the other hand, because the audio media device 400 is located in room 20B, an acoustic map constructed using the audio media device 400 may have a higher confidence rating for features extracted from room 20B. If the remote computing system 200 detects a mismatch in the maps for both room 20A and room 20B, the remote computing system 200 determines, based on the confidence rating, that the portion of the acoustic map corresponding to room 20A should be adjusted to correspond to the portion of the robot map corresponding to room 20A. Similarly, the remote computing system 200 determines, based on the confidence rating, that the portion of the robot map corresponding to room 20B should be adjusted to correspond to the portion of the acoustic map corresponding to room 20B.

[0134] In some cases, the mobile robot 300 receives a signal from the remote computing system 200 indicating the location of the audio media device 400 within the home 10. For example, the signal may correspond to an acoustic map and include the location of the audio media device 400 within the map. If the mobile robot 300 detects the audio media device 400, the mobile robot 300 focuses on the location of the audio media device 400 while the mobile robot 300 navigates through the home 10 relative to the audio media device 400. In some examples, as the mobile robot 300 navigates through the home 10, the image capture system 310 or other sensors on the mobile robot 300 detect the audio media device 400, thereby enabling an estimation of the location of the audio media device 400 within the robot map. The estimated location of the audio media device 400 is compared to the location indicated by the signal received by the remote computing system 200. The robot map is adjusted if any discrepancies are determined to improve the accuracy of the position estimation of the mobile robot 300 within the home 10.

[0135] The linked devices 102A, 102B (e.g., networked connected devices or devices connected to the communication network 201) also, in some implementations, generate information that the remote computing system 200, the mobile robot 300, and / or the AMD access to improve the accuracy of the robot map and / or acoustic map. The linked devices 102A, 102B include sensors that detect features in the home 10, such as acoustic sensors, image capture systems, or other sensors that generate signals from which features can be extracted. In some instances, the linked devices 102A, 102B transmit information from the sensor signals to the remote computing system 200. The remote computing system 200 correlates information about these features with features on the acoustic map and / or robot map.

[0136] In some implementations, the linked devices 102A, 102B generate their own maps of portions of the home 10 that are compared to the robot map and the acoustic map. The linked devices 102A, 102B include, for example, cameras, optical sensors, ranging sensors, acoustic sensors, or other sensors that generate signals to be used to form a map of the environment of the linked devices 102A, 102B. In some examples, the linked devices 102A, 102B cooperate with each other to form the map. The linked devices 102A, 102B, in some cases, include microphone units and / or speaker units, and an acoustic mapping routine, to be performed by the audio media device 400, is implemented to generate the map. The maps of the linked devices 102A, 102B are additionally or alternatively used to correct discrepancies in the robot map and the acoustic map.

[0137] The linked devices 102A, 102B can also emit signals that are received by sensors on the mobile robot 300 or the audio media device 400. The mobile robot 300 and the audio media device 400 use the signals generated by their sensors in response to signals from the linked devices 102A, 102B to triangulate the position of the mobile robot 300 and the position of the audio media device 400. The emitted signals can be, for example, optical signals, acoustic signals, wireless signals, and other detectable signals that change intensity as they propagate through the environment.

[0138] In addition to enabling interactions between various devices in the home 10, including the mobile robot 300 and the audio media device 400, the communications network 201 also facilitates interactions between the user 100 and the audio media device 400 to control or monitor the mobile robot 300. As described with respect to FIG. 1 , interactions between the user 100 and the audio media device 400 include audible signals emitted by the audio media device 400 to the user 100 and audible signals provided by the user 100 to the audio media device 400. These interactions may affect the behavior of the mobile robot 300 or the second mobile robot 301 or can provide the user 100 with information regarding the behavior of the mobile robot 300 or the second mobile robot 301.

[0139] The controller 408 of the audio media device 400 implements the speech recognition routine or can have the speech recognition routine implemented by, for example, the remote computing system 200. In some examples, the sound recognition routine is stored on the memory 410 of the audio media device 400, allowing the controller 408 to implement the sound recognition routine on the audible signal detected by its microphone. In some cases, the sound recognition routine is stored on the remote computing system 200 or on a database associated with the remote computing system 200. If the sound recognition routine is stored on the remote computing system 200, the controller 408 sends a wireless signal indicative of the audible signal captured by the microphone 402 to the remote computing system 200, and the remote computing system 200 executes the speech recognition routine to recognize words in the audible signal. The remote computing system 200 then generates a signal to cause the audio media device 400, the mobile robot 300, or the second mobile robot 301 to perform an action.

[0140] In some examples, the identification data for the user 100 is associated with the identification data for the audio media device 400 and / or the mobile robot 300. The remote computing system 200 includes, for example, a user identification database 506 that stores identification data for users who use the services of the remote computing system 200. The identification data for users includes, for example, account data unique to each user. The user identification database 506, if present, includes the identification data for the user 100. In some implementations, the identification data for the user 100 includes a speech signature unique to the user's 100's speech pattern. The speech signature is authenticated for use with the audio media device 400 so that the user 100 can utter speech signals to control the audio media device 400. The identification data for the user 100 includes the speech signature, and the speech signature is associated with the identification data for the audio media device 400. If the speech signature is associated with the identification data for the audio media device 400, a sound recognition routine is applied to the spoken words of the user 100 and performs an action corresponding to the recognized words.

[0141] Once the associations for the audio media device 400, the mobile robot 300, and / or the user 100 are completed, the user 100 can utter words to command the audio media device 400 and / or the robot 300. The user 100 utters words corresponding to commands in the command database 508. If the user identification data includes an utterance signature, only the uttered utterance having the utterance signature can be used to command the audio media device 400 or to command the mobile robot using the audio media device 400. The command database 508 includes commands for controlling the audio media device 400 and commands for controlling the mobile robot 300. When the remote computing system 200 selects a command from the command database 508, the remote computing system 200 generates a command signal to be sent to an appropriate device, for example, the audio media device 400 or one of the mobile robots 300 and 301. Upon receiving the command signal, the device performs one or more actions.

[0142] In an example where the user 100 utters a command to control the mobile robot 300, the user's 100 utterance 510 includes, for example, an audible identifier 512 corresponding to identification data for the audio media device 400, an audible identifier 514 corresponding to identification data for the mobile robot 300, and an audible command 516. In some examples, the remote computing system 200 executes a sound recognition routine to parse the audible identifier 512, the audible identifier 514, and the audible command 516. The remote computing system 200 checks that the identification data is associated and then queries the command database 508 to find a command corresponding to the audible command. The remote computing system 200 then generates a command signal to be sent to the appropriate device, for example, the audio media device 400 or the mobile robot 300. If the command signal is used to control the mobile robot 300, when the mobile robot 300 receives the command signal, the command signal causes the mobile robot 300 to perform one or more actions. An example command process 700 for commanding a mobile robot 300 using user-provided utterances is described with respect to FIG.

[0143] In some implementations, when a command signal is sent to the audio media device 400, the audio media device 400 emits an audible notification. The association enables the mobile robot 300 and the audio media device 400 to send information about their operations to the remote computing system 200, so that the audible notification from the audio media device 400 can include status updates of either the audio media device 400 or the mobile robot 300. An example notification process 800 for instructing the audio media device 400 to emit an audible notification is described with respect to FIG. 8. Example information provided by such a notification is also described with respect to the notification process 800.

[0144] The association also allows the remote computing system 200 to cause the audio media device 400 to emit an audible signal regarding the mobile robot 300 without receiving a prompt from the user 100, for example, based on data sent to the remote computing system 200 by the mobile robot 300 and, in some cases, information sent to the remote computing system 200 by the user 100. The remote computing system 200 then determines recommended actions that may enhance the behavior of the mobile robot 300. An example recommendation process 900 for the audio media device 400 that implements the recommended actions is described with respect to FIG. 9. Exemplary recommended actions are also described in relation to FIG. 9.

[0145] In some implementations, during operation of the mobile robot 300, a sensing system for the mobile robot 300 detects an error status associated with the mobile robot 300. The remote computing system 200 receives an indication of the error status and can cause the audio media device to issue audible instructions for the user 100 to correct the root of the error status. Alternatively, the user 100 speaks words that cause the audio media device to issue audible instructions corresponding to the error status of the mobile robot 300. An example error correction process 1000 for causing the audio media device 400 to provide guidance to the user 100 to correct the error in the mobile robot 300 is described with respect to FIG. 10 .

[0146] The operations for each of processes 600, 700, 800, 900, and 1000, as well as other processes described herein, may be performed in a distributed manner. For example, the remote computing system 200, the mobile robot 300, and the audio media device 400 may perform one or more of the operations in coordination with each other. An operation described as being performed by one of the remote computing system 200, the mobile robot 300, and the audio media device 400 is, in some implementations, at least partially performed by two or all of the remote computing system 200, the mobile robot 300, and the audio media device 400.

[0147] Exemplary Association Process 2, identification data for the mobile robot 300 and identification data for the audio media device 400 are associated to establish a channel of wireless communication between the mobile robot 300 and the audio media device 400. Association process 600 of FIG. 6A illustrates example actions performed by the user 100 (user actions 600A), the mobile computing device 202 (mobile device actions 600B), the remote computing system 200 (remote computing system actions 600C), the mobile robot 300 (mobile robot actions 600D), and the audio media device 400 (audio media device actions 600E) to associate identification data for the mobile robot 300 with identification data for the audio media device 400.

[0148] In some examples, as part of the association process 600, the user 100 queries the remote computing system 200 to create one or more user accounts associated with the mobile robot 300 and the audio media device 400. For example, in some implementations, if the user 100 creates a single account, both the mobile robot identification data and the audio media device identification data are associated with the user account. In some cases, the user 100 creates a different account for each of the mobile robot identification data and the audio media device identification data.

[0149] 6A , the user 100 requests (602) an association of the mobile robot 300 with the audio media device 400. The user 100 uses an input device on a user interface device, such as the mobile computing device 202, to request the association. The mobile computing device 202 then sends (604) a request to associate the mobile robot 300 with the audio media device 400 to the remote computing system 200. The remote computing system 200 accesses (606) a user account for the mobile robot identification data and the audio media device identification data.

[0150] In some cases, the association process 600 includes a user verification process in which the user 100 confirms identification data for the mobile robot and the audio media device. To implement this user verification process, the remote computing system 200 transmits (608) the mobile robot identification data and the audio media device identification to the mobile computing device 202. The mobile computing device 202 then displays (610) the audio media device identification data and the mobile robot identification data, and the user 100 operates an input device on the mobile computing device 202 (e.g., operates a touchscreen, a virtual button, a pushbutton, etc.) to confirm (612) that the identification data should be associated.

[0151] In some implementations, the association process 600 alternatively or additionally includes device confirmation from the mobile robot 300 and / or the audio media device 400. In the example of FIG. 6A , the remote computing system 200 requests confirmation of the association from the mobile robot 300 and the audio media device 400 (614). The mobile robot 300 confirms the association (616), and the audio media device 618 also confirms the association. Confirmation from the mobile robot 300 and / or the audio media device 400 indicates that the device is an active device. In some cases, confirmation from the mobile robot 300 and / or the audio media device 400 causes robot identification data to be stored in the memory 410 of the audio media device 400, or audio media device identification data to be stored in the memory 318 of the mobile robot 300.

[0152] In cases where either or both of a user verification process and a device verification process are implemented, after the remote computing system 200 receives a corresponding verification signal, the remote computing system 200 associates (620) the mobile robot identification data with the audio media device identification data. Alternatively, if no verification process is present, the remote computing system 200 associates (620) the mobile robot identification data with the audio media device identification data after the remote computing system 200 accesses (606) the user account. In some examples, the remote computing system 200 associates identification data from different types of mobile robots, such as mobile robots that perform wet floor mopping, home monitoring, or vacuuming. As described with respect to the vacuum cleaning robot of FIG. 3A , in some implementations, the type of mobile robot 300 determines the types of available voice command inputs at the audio media device 400. In this regard, in some implementations, the remote computing system 200 associates (622) a command list for the type of mobile robot 300 with the audio media device identification data. The remote computing system 200 responds to wireless signals corresponding to particular voice commands received by the audio media device 400 for controlling a particular type of mobile robot 300 .

[0153] In some implementations, after the association is completed, the remote computing system 200 transmits a successful association notification (624). One or more of the mobile computing device 202, the mobile robot 300, and the audio media device 400 receive the transmitted success notification, indicating that the association was successful. For example, the mobile computing device 202 issues or displays a successful association notification (626), the mobile robot 300 issues or displays a successful association notification (628), and / or the audio media device 400 issues a successful association notification. A displayed successful association notification includes, for example, a banner or notification on a display screen, a series of illuminated lights, or other suitable visual signal. An audible successful association notification includes, for example, an audible alert or sound, spoken words, or other suitable audible signal indicating that the association operation was successful. In some cases, the successful association notification includes a tactile notification, such as a vibration alert.

[0154] In some examples, the audio media device 400 is associated with multiple mobile robots, for example, both mobile robots 300, 301. The identification data for the mobile robot 300 is separate from the second mobile robot 301. The association process 600 associates the identification data for each mobile robot 300, 301 with the identification data for the audio media device 400.

[0155] 6B illustrates an example of using a mobile computing device 202, e.g., a smartphone, to request an association. In this example, the user 100 has a first user account for the mobile robot 300 and a second user account specific to the audio media device 400. As part of the user 100 requesting (602) the association, the user 100 obtains mobile robot identification data. The user 100 uses the mobile computing device 202 to access the mobile robot user account, and the mobile computing device 202 displays the mobile robot identification data associated with the user account for the mobile robot 300, as shown in screen 650. As part of the user 100 requesting (602) the association, the user 100 also obtains audio media device identification data. The user 100 uses the mobile computing device 202 to access the user account for the audio media device 400, and the mobile computing device 202 then displays the audio media device identification data associated with the user account for the audio media device 400, as shown in screen 652. To request the association, the user 100, for example, uses an application on the mobile computing device 202 to input the mobile robot identification data to be associated with the corresponding audio media device identification data. If the association process 600 includes a user verification process, the mobile computing device 202 displays a verification request on the mobile computing device 202, as shown in screen 654. After the user 100 confirms the association, for example, by invoking a "Yes" user interface button, the remote computing system 200 associates the identification data for the mobile robot 300 and the audio media device 400. The mobile computing device 202 optionally displays a successful association notification, as shown in screen 656.

[0156] In some implementations, the mobile robot 300 and the audio media device 400 interact directly to establish an association between the mobile robot identification data and the audio media device identification data. For example, the user 100 requests association (602) by uttering a voice command for the audio media device 400 to associate with the mobile robot 300. The audio media device 400 uses its wireless communication system 412 to search for nearby mobile robots, for example, over a local area network or using short-range communication. Upon locating the mobile robot 300, the audio media device 400 transmits a signal to associate the identification data. In some cases, the mobile robot 300 uses its wireless communication system 316 to implement such an association process.

[0157] In some examples, the user 100 issues a command to one or both of the audio media device 400 and the mobile robot 300 to associate with each other. The command, for example, causes the remote computing system 200 to transmit a command signal that causes one of the devices to issue an audible association request and the other of the devices to receive the audible association request. Upon receiving confirmation that the audible associated request has been received, for example, by the microphone unit 402 of the audio media device 400 or the microphone of the mobile robot 300, the remote computing system 200 associates the identification data of the mobile robot 300 with the identification data of the audio media device 400.

[0158] In some implementations, the user 100 issues an association command to the mobile robot 300 to associate with the audio media device 400. In response to the command, the mobile robot 300 autonomously navigates the house 10 to search for the audio media device 400, for example, using an image capture system 310 or a microphone 314 mounted on the mobile robot 300. As shown in the example of FIG. 5 , as the mobile robot 300 navigates through the house 10, the image capture system 310 of the mobile robot 300 detects the audio media device 400. Upon locating the audio media device 400, the mobile robot 300 emits an audible association request using the audio emission system 312 to be received by the audio media device 400. The audible association request optionally includes robot identification data. When the audio media device 400 receives the signal, the audio media device 400 transmits a wireless signal to the remote computing system 200 to cause the mobile robot identification data to be associated with the audio media device identification data. In some cases, the audio media device 400 responds to the audible association signal by emitting an audible response signal that is received by a microphone of the mobile robot 300 .

[0159] Example Command Process 5, the user 100 utters voice commands, for example, through utterances 510, that may be received by the audio media device 400 to cause the mobile robot 300 to perform a behavior in accordance with the voice command. Command process 700 of FIG. 7 illustrates example spoken commands that cause the mobile robot 300 to perform a behavior in accordance with the spoken voice command. Command process 700 of FIG. 7 illustrates example actions performed by the user 100 (user actions 700A), the audio media device 400 (audio media device actions 700B), the remote computing system 200 (remote computing system actions 700C), and the mobile robot 300 (mobile robot actions 700D) to cause the mobile robot 300 to perform a behavior or action in accordance with the user's voice command.

[0160] The user 100 provides audible user commands, e.g., voice commands, to the audio media device for the remote computing system 200 to determine which mobile robot 300, 301 to command and to determine specific behaviors to be performed by the mobile robot 300, 301. In the example shown in FIG. 7 , the user 100 provides an utterance (702) including a voice command to control the mobile robot 300. The audio media device 400 generates (704) a wireless signal indicative of the utterance and then transmits the wireless signal to the remote computing system 200. As shown in FIG. 5 , in some cases, the utterance 510 also includes an audible identifier 512, and the audio media device 400 recognizes that the user's utterance also corresponds to a command to be executed by the audio media device 400. The memory 410 of the audio media device 400 optionally includes a sound recognition routine that recognizes the audible identifier 512, such that the audio media device 400 only executes the command preceded by the audible identifier 512 corresponding to the audio media device 400. The command executed by the audio media device 400 corresponds, for example, to transmitting a wireless signal to the remote computing system 200 .

[0161] Upon receiving the wireless signal, the remote computing system 200 parses 706 the wireless signal. The remote computing system 200 may, for example, parse 706 the wireless signal and execute a sound recognition routine to determine the content of the utterance provided 702. The parsed 706 wireless signal may then be analyzed, for example, to check for errors and to generate commands to send to the mobile robot 300.

[0162] In some examples, the remote computing system 200 determines, based on the provided utterance (702), identification data corresponding to the audio media device 400 and identification data corresponding to the mobile robot 300. The remote computing system 200 determines (708) whether the mobile robot identification data and the audio media device identification data are associated based on the identification data. If the remote computing system 200 determines that they are not associated, the remote computing system 200 sends a signal to the audio media device 400 to issue (712) an error notification. In some cases, the identification data is not associated because the user's utterance includes an audible identifier indicating another autonomous mobile robot not associated with the identification data of the audio media device 400, i.e., not indicating the mobile robot 300. The user 100 may attempt to provide (702) another utterance in response to the error notification or may attempt to associate the audio media device identification data and the mobile robot identification data using, for example, the association process 600 of FIG. 6 .

[0163] Alternatively or additionally, the remote computing system 200 analyzes the parsed (706) wireless signal to determine (710) whether the provided (702) utterance includes a valid voice command intended for the mobile robot 300. As shown in FIG. 5 , portions of the parsed (706) wireless signal may correspond to the vocal audible command 516 provided by the user 100. A valid voice command may include, for example, any command in the command database 508. In some cases, a valid voice command may include a predetermined command in the command database 508 intended for a predetermined type of mobile robot 300. If the parsed (706) wireless signal does not include a valid voice command, the remote computing system 200 causes the audio media device 400 to issue (712) an error notification. The error notification informs the user 100 that the user did not provide (702) an utterance with a valid voice command or that the voice command does not correspond to any of the voice commands available for a particular type of mobile robot 300.

[0164] If the parsed (706) wireless signal contains a valid voice command, the remote computing system 200 generates (714) a command signal corresponding to the valid voice command. The command database 508 includes, for example, a lookup table that matches each valid voice command with a corresponding command signal.

[0165] The remote computing system 200 then transmits the command signal to the appropriate destination, for example, by selecting a destination for the generated (714) command signal. In the command process 700, the mobile robot 300 is controlled using the command signal, in which the remote computing system 200 transmits the command signal to the mobile robot 300, for example, as indicated by the audible identifier 514 corresponding to the mobile robot 300. If the audio media device 400 is associated with multiple mobile robots, the audible identifier 514 for each mobile robot is distinct from the other audible identifiers. If the audible identifier 514 corresponds to identification data for the mobile robot 300, the command signal is transmitted to the mobile robot 300. Upon receiving the command signal, the mobile robot 300 performs (716) a behavior or action. Specifically, the mobile robot 300 performs (716) a behavior or action corresponding to the voice command included in the utterance provided (702) by the user 100. In some examples, the remote computing system 200 also causes the audio media device 400 to issue a command success notification (718) to indicate to the user 100 that the user's vocalization was successfully used to command the mobile robot 300.

[0166] In some examples, the command signal causes both the mobile robot 300 and the audio media device 400 to receive a corresponding command. For example, the command signal causes the mobile robot 300 to perform a behavior or routine within the house 10. In some examples, the command signal causes the mobile robot 300 to autonomously navigate the house 10 relative to the audio media device 400. If the mobile robot 300 is a cleaning robot, such as a vacuum cleaning robot, and the command is a cleaning command, the audio media device 400 optionally causes the mobile robot 300 to begin a cleaning operation, for example, by activating an air mover while the mobile robot 300 autonomously navigates the house 10. Additionally, the user 100 can provide a voice command to the mobile robot 300 to pause the cleaning operation.

[0167] If the mobile robot 300 is connectable to a docking station, such as the docking station 104 in FIG. 1 , the command signal may correspond to a command signal that causes the mobile robot 300 to autonomously navigate to the docking station 104. In some cases, the command signal causes the mobile robot 300 to navigate to the docking station 104 and, if the mobile robot 300 is a vacuum cleaning robot with a dustbin, causes the docking station 104 to empty the dust from the mobile robot 300's dustbin. For example, the mobile robot 300 operates the docking station 104 to initiate emptying after proper docking and alignment. In some implementations, the remote computing system 200 is also wirelessly linked to the docking station 104 and sends an emptying command signal directly to the docking station 104 to cause the docking station to empty the dustbin when the mobile robot 300 or the docking station 104 sends a signal indicating a ready status for one or both of the mobile robot 300 and the docking station 104.

[0168] In some implementations, the command signal corresponds to a voice command that the user 100 desires to have the mobile robot 300 navigate toward the audio media device 400. The mobile robot 300 thus autonomously navigates toward the audio media device 400, for example, based on a predetermined location of the audio media device 400. The predetermined location corresponds, for example, to a location determined using the acoustic map creation routine described with respect to FIG. 4 or a location determined using images captured by the image capture system 310 of the mobile robot 300 or some other sensor-based detection (e.g., an IR transceiver) that enables the mobile robot 300 to localize the audio media device 400 on a robot map.

[0169] In some examples, if the mobile robot 300 includes a microphone 314, the remote computing system 200 provides command signals to both the audio media device 400 and the mobile robot 300 to cause the mobile robot 300 to navigate toward the audio media device 400. For example, the remote computing system 200 causes the audio media device 400 to emit an audible navigation signal to be received by the microphone 314 of the mobile robot 300. To navigate toward the audio media device 400, the mobile robot 300 navigates in a direction that increases the strength of the received audible navigation signal. In some implementations, the mobile robot 300 detects the signal strength of signals other than audible signals, such as, for example, wireless signals connecting the audio media device 400 to the communications network 201. The mobile robot 300 detects the signal strength of the wireless signals and uses the signal strength to navigate within the house 10. To navigate toward the audio media device 400, the mobile robot 300 navigates in a direction that increases the strength of the received wireless signals.

[0170] The user 100, in some cases, provides voice commands to adjust a user-defined schedule for controlling the mobile robot's operation. The mobile robot 300 may operate according to this user-defined schedule. For example, if the mobile robot 300 is a cleaning robot, the mobile robot 300 begins cleaning operations at the start time indicated in the user-defined schedule. The utterances provided by the user 100 (702) in the command process 700 thus include voice commands that change or establish start times for the mobile robot's operation. When the remote computing system 200 generates (714) a command signal, the remote computing system 200 adjusts the user-defined schedule according to the changes to the start times. In some cases, the user-defined schedule is stored in memory 318 on the mobile robot 300, and the remote computing system 200 transmits schedule change commands to the mobile robot 300. Alternatively or additionally, the user-defined schedule is stored remotely from the mobile robot 300, for example, on a cloud computing network 205, and the remote computing system 200 modifies the user-defined schedule in the cloud computing network 205. In such a case, the user 100 provides (702) an utterance in a voice command at the present time with the intention of controlling the behavior of the mobile robot 300 in the future without the user 100 having to provide the command at the future time. The remote computing system 200 stores the modified user-defined schedule and sends commands at future times according to the modified user-defined schedule to cause the mobile robot 300 to perform (716) a behavior or action corresponding to the voice command.

[0171] In some implementations, the user 100 provides voice commands to cause the mobile robot 300 to autonomously navigate to the user's 100 current location. To generate a command signal to cause the mobile robot 300 to navigate toward the user's 100 current location, for example, a distance estimation routine described with reference to FIG. 4 is implemented to estimate the distance and orientation between the user 100 and the audio media device 400, thereby allowing the remote computing system 200 to determine the user's 100 pose on the robot map and / or the audio media device map. For example, the remote computing system 200 or the audio media device 400 estimates the distance based on a signal indicative of acoustic characteristics, e.g., a time delay between two microphones of the microphone unit 402 that receive the vocalization provided by the user 100 (702). The remote computing system 200 or the audio media device 400 determines the user's 100 location within the robot map and then generates a command signal to cause the mobile robot 300 to navigate to a location within the robot map that corresponds to the user's 100 estimated location. In some cases, the mobile robot 300 navigates to a position within a predetermined distance of the estimated location of the user 100, for example, within 0.1 to 2 meters of the user 100.

[0172] If the mobile robot 300 is a vacuum cleaning robot, the user 100 can provide a voice command to cause the mobile robot 300 to perform an operation, such as a spot cleaning operation for intensive cleaning in a particularly dirty spot. The current location of the mobile robot 300 may correspond to the center of a localized area where the spot cleaning operation is to be performed. In some cases, the user 100 provides (702) an utterance with a voice command to cause the mobile robot 300 to clean at the user's 100's current location. Thus, in addition to causing the mobile robot 300 to move to a position proximate the user's 100's location, the command signal also causes the mobile robot 300 to perform the spot cleaning operation upon reaching that position.

[0173] In some examples, the remote computing system 200 identifies the room within the house 10 in which the user 100 is located based on, for example, the estimated distance and direction of the user 100 relative to the audio media device 400. If the user 100 is located in room 20B, for example, as shown in FIG. 1 , and the user 100 issues a voice command to move to the user's 100's location, the remote computing system 200 moves the mobile robot 300 to room 20B in which the user 100 is currently located. As the mobile robot 300 autonomously navigates toward room 20B, the mobile robot 300 determines that it is within room 20B upon recognizing unique features associated with room 20B. The unique features correspond, for example, to features extracted from signals of a position estimation sensor during a previous navigation operation within the house 10. These unique features are specific to properties or objects within room 20B, enabling the mobile robot 300 to distinguish room 20B from other rooms within the house 10.

[0174] In some implementations, the rooms 20A, 20B, 20C, and 20D each include objects or features that allow unique features to be extracted from signals of the position estimation sensors for each of the rooms 20A, 20B, 20C, and 20D, and the user 100 provides identification data for each of the rooms. The identification data is stored, for example, in a database accessible by the remote computing system 200 and / or the mobile robot 300. To provide identification data for the rooms, the user 100 provides each of the rooms 20A, 20B, 20C, and 20D with a spoken or typed name (e.g., entered into an editable field in a smartphone application), such as “kitchen,” “bathroom,” “dining room,” “living room,” “bedroom,” or other suitable room designation. In some examples, the mobile computing device 202 displays a user interface map corresponding to the robot map of the house 10, and the user 100 provides each room with a name by selecting it and speaking or typing the name. In some examples, the mobile robot 300 navigates to each room in the house 10. When the mobile robot 300 is in a room, the user 100 speaks the name of the room, and the name is stored as identification data for that room. The mobile robot 300 moves to each of the rooms, and as it does so, the user 100 may speak the name of each room. In some examples, the user 100 gives spoken or typed names for objects in the house 10, such as "front door," "back door," "dishwasher," "washing machine," "bed," or other indicators for objects in the house 10.

[0175] The identification data for each room allows the user 100 to control the mobile robot 300 to move to a room specified by the user or toward an object specified by the user. When the user 100 provides an utterance with a voice command (702), the user 100 specifies an audible location identifier corresponding to a predetermined location within the house 10. The audible location identifier corresponds, for example, to one of the names previously specified by the user 100, to cause the mobile robot 300 to autonomously navigate to the room corresponding to the specified name. When the mobile robot 300 receives a command signal from the remote computing system 200, it performs an action to move toward the room having the specified name (716). As described herein, when the position estimation sensor of the mobile robot 300 detects a unique feature in the room, the mobile robot 300 determines that the robot 300 is in a room having the specified name. In some examples, the audible location identifier corresponds to a name for an object, causing the autonomous mobile robot 300 to autonomously navigate toward the object upon receiving the command signal. The autonomous mobile robot 300 is controlled to stop, for example, when it is determined that the mobile robot 300 is within a predetermined distance from an object.

[0176] In some implementations, user 100 associates multiple mobile robots of the same type with audio media device 400. For example, referring to FIG. 1 , mobile robot 300 and mobile robot 301 may both be vacuum cleaning robots, and user 100 associates both of these robots with audio media device 400. User 100 provides (702) an utterance with an audible identifier that, in some cases, may refer to both mobile robots 300, 301. The audible identifier may be, for example, the type of mobile robot 300, 301. In some cases, remote computing system 200 causes audio media device 400 to issue an audible request in which user 100 specifies which of mobile robots 300, 301 user 100 intends to command.

[0177] In some examples, if the voice command indicates an action to be performed at a specified location, the remote computing system 200 determines which of the mobile robots 300, 301 is closest to the specified location and directs a command signal to the closest mobile robot 300, 301. The remote computing system 200 determines that the distance of the second mobile robot 301 to the user-specified location is less than the distance of the other mobile robot 300 to the user-specified location. For example, if the user 100 provides (702) an utterance commanding the vacuum cleaning robot to clean room 20D, the remote computing system 200 determines that the second mobile robot 301 is closer to room 20D and therefore sends a command signal to the second mobile robot 301 to cause it to clean room 20D. If the user 100 gives a voice command specifying that the mobile robot should clean at the user's 100 current location (e.g., the voice command is "Robot, come here and clean"), the remote computing system 200 may determine that the vacuum cleaning robot will be commanded to clean at the user's 100 current location.

[0178] As described herein, localization using acoustic signals allows the location of a speaking user 100 to be determined relative to the location of the mobile robot 300, and the mobile robot 300 responds to user commands such as "come here" by going in the direction of the speaking user 100. This direction and location of the user 100 is determined by a direction-sensing microphone on the audio media device 400 and communicated to the mobile robot 300 or to the remote computing system 200. In some examples, when the mobile robot 300 detects the audio media device 400 and places its location on the robot map, the audio media device 400 and / or the remote computing system 200 determines the distance to the speaking user 100. The microphone unit 402 includes, for example, multiple microphones positioned on the audio media device 400, which detect multi-path acoustic reflections caused by the user's 100's voice, for example, reflecting off surfaces within the home 10, such as walls or surfaces of objects within the home 10. The audio media device 400 and / or the remote computing system 200 use these multipath signals to calculate the origin of the detected voice command, for example, by triangulating the received multipath acoustic signals. In some implementations, the remote computing system 200 and / or the audio media device 400 compensate for noise and / or attenuation when calculating the speaking user's pose on the robot map.

[0179] Example Notification Process In addition to using voice commands to send commands to the mobile robot 300, the user 100 can provide voice commands that cause the audio media device 400 to provide audible notifications indicating the status of the mobile robot 300. Examples of such audible notifications are described with respect to notification process 800 of Figure 8. Notification process 800 of Figure 8 illustrates example actions performed by the user 100 (user actions 800A), the audio media device 400 (audio media device actions 800B), the remote computing system 200 (remote computing system actions 800C), and the mobile robot 300 (mobile robot actions 800D) to cause the audio media device 400 to issue, for example, an audible mission status update or an audible cumulative status update.

[0180] The remote computing system 200 collects data from the mobile robot 300 as it performs operations within the home 10. When the mobile robot 300 initiates (802) a mission and collects (804) data indicative of the robot's operations during the mission, the remote computing system receives the collected (804) data. The data corresponds to data collected by sensors in the sensing system 308 of the mobile robot 300, including measured conditions (e.g., temperature, humidity, lighting) of the home 10 at a location on the map, and information gathered from stationary networked devices (e.g., motion sensors, temperature sensors, humidity sensors, networked lighting, etc.).

[0181] In some examples, the user 100 requests an update on the mission currently being performed by the mobile robot 300. The user 100 provides a voice request to the audio media device 400 for a mission status update for the mobile robot 300 (806), and the audio media device 400 transmits a wireless signal corresponding to the voice request to the remote computing system 200 (808).

[0182] The remote computing system 200 polls the mobile robot 300 for data indicative of its operation during the mission and generates a mission summary based on the data from the mobile robot 300 (810). The remote computing system 200 then generates a mission status update command signal (812) that causes the audio media device to issue a mission status update (814). The mission status update corresponds to the generated summary (810). The summary may include, for example, the amount of time that has elapsed since the mobile robot 300 began the mission, the total portion of floor that has been traversed, the percentage of traversable flooring remaining (e.g., clear of obstacles and accessible by the robot 300), an estimated amount of time remaining to complete the mission, or the current room of the mobile robot 300. If the mobile robot 300 is a vacuum cleaning robot, the summary may include, for example, the amount of dirt vacuumed by the mobile robot 300 during the mission, the number of instances and map locations where spot cleaning occurred during the mission, or the number of rooms the mobile robot 300 cleaned during the mission. In some cases, the user 100 indicates specific information to be included in the mission status update, and the generated summary 810 includes the requested information.

[0183] In some examples, the user 100 may request a cumulative status update indicating multiple missions performed by the mobile robot 300. The user 100 provides a voice request for cumulative status updates for the mobile robot 300 (820), and the audio media device 400 generates a wireless signal corresponding to the voice request (822). The remote computing system 200 receives the wireless signal and generates a cumulative summary based on data from each of the missions initiated by the mobile robot 300 (824). The remote computing system 200 generates a cumulative status update command signal (826) to cause the audio media device 400 to issue an audible cumulative status update (828). The cumulative summary may include, for example, the total duration the mobile robot 300 has been operating for multiple missions, the number of missions performed by the mobile robot 300, the total cumulative distance traveled across floors, or the distance traveled by the mobile robot 300. If the mobile robot 300 is a vacuum cleaning robot, the cumulative summary may alternatively or additionally include the total amount of debris collected by the mobile robot 300 over multiple missions, or the total number of spot cleaning operations performed by the mobile robot 300. In some examples where the mobile robot 300 is operable with a docking station that ejects debris from the mobile robot 300, the cumulative summary may include the average number of ejection operations per week, the number of ejection operations performed in a certain time period, the number of ejection operations performed for an area covered by the mobile robot 300 in a certain time period, etc.

[0184] In some examples, the cumulative status update describes the cumulative status of specified missions within a user-specified time period. In a given (820) voice request, the user 100 can specify the number of previous missions from which data will be summarized in the cumulative summary, for example, the previous 2-10 missions, 10-20 missions, or all missions completed by the mobile robot 300. The user 100 can specify a date or date range for the missions from which data will be summarized in the cumulative status update.

[0185] The status update may include various information requested by the user 100 in the voice request. In some implementations, the status update includes the frequency of the mobile robot 300's operation. If the mobile robot 300 includes sensors that generate signals indicative of the operability of parts of the mobile robot 300, the remote computing system 200 receives these signals and causes the issued status update to include the estimated remaining service life of the parts of the mobile robot 300. Parts may be, for example, cleaning brushes, cleaning pads, rollers, batteries, containers, wheels, container filters, or container lids. In some cases, if the mobile robot 300 interacts with other stationary devices, such as the docking station 104 or the transmitting unit 106, the remote computing system 200 causes the status update to include information about these stationary devices. For example, a status update may indicate that the transmitting unit 106 needs a replacement battery.

[0186] In some examples, the audio media device 400 issues a mission status update after the mission is complete (814). The remote computing system 200, in some cases, determines portions of the house 10 that the mobile robot 300 did not visit during the mission and causes the audio media device 400 to issue a mission status update indicating the non-visited portions (814). The non-visited portions in the mission status update refer, for example, to rooms or portions of rooms that the mobile robot 300 did not clean during the mission. In some cases, the mission status update indicates the reason the mission was aborted, for example, that a device in the communication network 201 detected that the user 100 returned home, that an obstacle blocked a particular door in the house 10, or that the battery 2024 of the mobile robot 300 did not have enough charge for the mobile robot 300 to complete the mission.

[0187] In some examples, rather than issuing a status update (814, 828) in response to a user giving a voice request for a status update (806, 820), the audio media device 400 issues a status update when the user 100 enters the home 10. For example, when the user 100 returns to the home 10, the audio media device 400 automatically issues a status update of the missions completed by the mobile robot 300 while the user 100 was away from the home 10. The audio media device 400 issues a status update when the microphone unit 402 or other sensors of the audio media device 400 detect that the user 100 has returned to the home 10. In some cases, if the linked devices 102A, 102B include a motion sensor, the remote computing system 200 causes the audio media device 400 to issue a status update when the motion sensor detects the user 100 returning to the home 10. Alternatively or additionally, the audio media device 400 may issue a status update or detect a wireless signal, e.g., a Bluetooth or other wireless signal, emitted by a user device in response to determining that the user device, e.g., the mobile computing device 202, has connected to a wireless communications network (e.g., a WLAN) within the home 10. For example, the audio media device 400 may use its device presence sensor to determine when the user device 202 has entered a virtual perimeter, e.g., a geofence, that has been created for the home 10.

[0188] In some examples, the user 100 provides a voice request for notification of the current location of the mobile robot 300 (806). To determine the current location of the mobile robot 300, the remote computing system 200 checks the robot map and the mobile robot's estimated position within the robot map. The remote computing system 200 then causes the audio media device 400 to issue an audible notification indicating the location of the mobile robot 300. In some implementations, the audio media device 400 issues a notification indicating the location of the mobile robot 300 to the user 100 or the audio media device 400. The audible notification indicates, for example, the mobile robot 300's orientation relative to the audio media device 400. In some cases, the remote computing system 200 uses the constructed robot map or acoustic map to identify the room in which the mobile robot 300 is located and then causes the name of the room to be indicated in the audible notification. In some implementations, if the location of the mobile robot is unknown, the voice command initiates a process to determine the location of the mobile robot. The voice command, for example, causes the audio emitting system 312 of the mobile robot 300 to emit an acoustic signal to be received by the audio media device 400. The audio media device 400 then determines the location of the mobile robot 300 based on characteristics of the acoustic signal received by the microphone unit 402 of the audio media device 400. In some cases, to inform the user 100 of the current location of the mobile robot 300, the voice command causes the mobile robot 300 to emit an acoustic signal or to emit an acoustic signal periodically, for example, every 1 to 5 seconds, to help the user 100 find the location of the mobile robot 300 within the home 10.

[0189] In some examples, the audio media device 400 issues ongoing status updates of missions performed by the mobile robot 300 without the user 100 providing a voice request. For example, the ongoing status updates indicate when the mobile robot 300 is moving between rooms. In this case, the ongoing status updates indicate one or more rooms where the mobile robot 300 was previously located and where the mobile robot 300 is currently located. In some cases, the ongoing status updates indicate objects that the mobile robot 300 is passing by. The mobile robot 300 uses its sensing system 308 to detect nearby objects. If the mobile robot 300 includes an image capture system 310, the image capture system 310 captures images corresponding to nearby objects, such as a couch, refrigerator, appliances, or other stationary objects in the home 10. When the image capture system 310 captures images while the robot 300 is performing a mission, the remote computing system 200 identifies nearby objects or obstacles in the captured images. The remote computing system 200, for example, accesses an object recognition database to identify objects in the captured images, and then causes the audio media device 400 to issue a status update that the mobile robot 300 is near the particular object.

[0190] In some examples, if the mobile robot 300 stalls during its mission while the user 100 is in the house 10, the mobile robot 300 sends a signal to the remote computing system 200 indicating that the mobile robot 300 has stalled. The mobile robot 300 detects that the robot 300 has stalled using a stall sensor unit, such as, for example, an optical stationary sensor, a motor stall sensor, a mouse sensor, a gyroscope, an accelerometer, a stereo camera, and / or another sensor for motion detection. Rather than waiting for the user 100 to request a status update, the remote computing system 200 can cause the audio media device 400 to issue a status update indicating that the mobile robot 300 has stalled. If the remote computing system 200 can determine the room in which the mobile robot 300 is located, the remote computing system 200 can further cause the status update issued by the audio media device 400 to indicate the room in which the mobile robot 300 is located and stalled. If the mobile robot 300 is stalled near an object that is detectable by the sensing system 308 of the mobile robot 300 and identifiable by the remote computing system 200, the status update indicates the particular object.

[0191] Exemplary Recommendation Process In some implementations, the remote computing system 200 determines that implementation of the recommended action would benefit the mobile robot 300's actions and continues to request that the recommended action be implemented. The remote computing system 200 can cause the audio media device 400 to emit an audible signal indicating that the recommended action has been implemented or will be implemented. Examples of recommended actions are described with respect to the recommendation process 900 of FIG. 9. The recommendation process 900 of FIG. 9 illustrates example actions performed by the user 100 (user actions 900A), the audio media device 400 (audio media device actions 900B), the remote computing system 200 (remote computing system actions 900C), and the mobile robot 300 (mobile robot actions 900D) to cause the recommended action to be implemented.

[0192] The remote computing system 200 may determine that an action will be beneficial for the operation of the mobile robot based on input data from the user 100, the audio media device 400, the mobile robot 300, or a combination thereof. The user 100, for example, provides one or more user inputs indicating the operation of the mobile robot 300 (902). The user inputs may also indicate user behavior. The user inputs correspond, for example, to a user-defined schedule for the operation of the mobile robot 300, user initiation or user cessation of the operation of the mobile robot 300, etc. The audio media device 400 transmits data collected by its sensors (904), and the mobile robot 300 transmits data indicating the status of the mobile robot 300 (906).

[0193] Based on one or more of these inputs (e.g., received from operations 902, 904, 906), the remote computing system 200 determines (908) a recommended action to be performed and generates (910) a request signal to perform the recommended action. The remote computing system 200 determines (912) a destination to send the request signal for the recommended action. The recommended action corresponds, for example, to a robotic action, a user action, or an action to be performed by another computing system remote from the home 10.

[0194] If the recommended action is to be performed by the mobile robot 300, the remote computing system 200 transmits a request signal to the mobile robot 300, and the robot 300 adjusts the mobile robot operation according to the request signal, i.e., according to the recommended action (914). The recommended action may be, for example, an adjustment to a user-defined schedule for the mobile robot 300. If the mobile robot 300 is a vacuum cleaning robot, in some implementations, the remote computing system 200 determines that the vacuum cleaning robot has not cleaned the house 10 or a particular room within the house 10 for a predetermined duration, for example, one week to one month or more. The remote computing system 200 generates a request signal to perform the recommended action (910) and then transmits the request signal to the mobile robot 300 to initiate a cleaning operation within the house 10 or within the particular room within the house 10.

[0195] In some examples, the remote computing system 200 receives an indication from, for example, an occupancy sensor on the audio media device 400 or on the linked devices 102A, 102B, indicating when the occupants of the home 10 are absent. From the occupancy sensor indication, the remote computing system 200 determines, for example, a pattern of when the occupants are present or absent from the home. The remote computing system 200 causes the audio media device 400 to issue an audible request (916) to alter the user-defined schedule for the mobile robot 300 so that the mobile robot 300 operates when the occupants are typically absent from the home. If the user 100 confirms the request, the remote computing system 200 adjusts the user-defined schedule accordingly.

[0196] If the recommended action is to be performed by the user 100, the remote computing system 200 sends a request signal to the audio media device 400 to cause the audio media device 400 to issue an audible request (916) for the user 100 to perform the recommended action (918). For example, the remote computing system 200 may determine, based on data collected by the audio media device 400, that a door to a room is closed. The data from the audio media device 400 may correspond to an acoustic map that has a discrepancy with a baseline acoustic map constructed using an acoustic mapping routine. The discrepancy indicates that the door is open or closed. If the door corresponds to the door for entrance 22A and the remote computing system 200 determines, based on the acoustic mapping data, that the door is closed, the remote computing system 200 causes the audio media device 400 to issue an audible request (916) for the user to open the door, thereby allowing the mobile robot 300 to pass through entrance 22A and enter room 20B.

[0197] If the recommended action should be performed by another entity in addition to the mobile robot 300 or the user 100, the remote computing system 200 sends a request signal to that entity, e.g., another remote computing system (920). In some implementations, the data indicating the status of the mobile robot 300 indicates that the mobile robot 300 needs a replaceable part replaced soon, e.g., within three to seven days. The remote computing system 200 generates a request signal to purchase the part (910) and sends the request signal, e.g., to an online marketplace. In some cases, the request signal causes the part to be ordered and delivered to the user's 100 home 10 before the replaceable part needs to be replaced. The part corresponds to a consumable item, such as a robot battery or a floor cleaning pad. If the mobile robot 300 is a cleaning robot, the part may be a cleaning brush, a cleaning pad, a roller, a dustbin, a container lid, or other replaceable part.

[0198] In some implementations, after the remote computing system 200 determines (908) the recommended action, the remote computing system 200 causes the audio media device 400 to request confirmation from the user 100 that the recommended action should be implemented. For example, if the remote computing system 200 recommends the purchase of a replacement part, the remote computing system 200 triggers the audio media device 400 to issue an audible request to the user 100 to confirm such purchase. The user 100 then responds to the request by speaking naturally to the AMD to cause the purchase to occur or to deny the request to prevent the purchase from occurring. If the recommended action is to be implemented directly by the user 100 (e.g., the user should open a door), the remote computing system 200 may not request confirmation and instead directly issues (916) an audible notification to inform the user 100 that the user 100 should implement (918) the recommended action.

[0199] In some examples, the request signal causes a user-defined schedule for the mobile robot 300 to change according to an event recorded in the user event calendar. The user 100 can provide (902) user input indicating an upcoming event. The upcoming event may be a home get-together where the user 100 has invited guests to come to the house 10, and it may be desirable to prohibit operation of the mobile robot 300 during the event scheduled on the network-accessible calendar. If the mobile robot 300 is scheduled to perform an operation during the upcoming event, the remote computing system 200 determines that the robot operation will conflict with the upcoming event. To avoid the collision, the remote computing system 200 generates (910) a request signal to adjust the user-defined schedule for the mobile robot 300 so that the mobile robot 300 does not perform the operation during the upcoming event.

[0200] In some examples, the remote computing system 200 determines that the mobile robot 300 should perform an operation in advance of an event. For example, if the upcoming event is a house party, the remote computing system 200 generates 912 a request signal to adjust the user-defined schedule of the mobile robot 300 according to the time period during which the house party is occurring. If the mobile robot 300 is a cleaning robot, the user-defined schedule is adjusted so that the mobile robot 300 performs a cleaning operation a predetermined period of time before the upcoming event, for example, 1 hour to 24 hours before the house party. In this regard, the house 10 can be cleaned before guests arrive for the house party.

[0201] In some examples, the remote computing system 200 determines that the user 100 has initiated operation of the mobile robot 300 without defining a schedule for the mobile robot 300. The user 100 has previously initiated an operation of the mobile robot 300 without specifying a user-defined schedule. In some examples, the remote computing system 200 extracts patterns from the times at which the user 100 triggers the initiation of mobile robot operation. Based on these patterns, the remote computing system 200 generates a request signal that causes the audio media device 400 to issue an audible request to adjust the user-defined schedule. A user-defined schedule for mobile robot operation may correspond, for example, to weekly operation on a particular day of the week or daily operation at a particular time.

[0202] In some cases, the mobile robot 300 is already performing actions according to a user-defined schedule, but the remote computing system 200 determines that the user 100 has stopped some of these mobile robot actions or has caused the mobile robot 300 to perform actions in addition to those performed according to the user-defined schedule. The remote computing system 200 causes the audio media device 400 to issue an audible request to recommend adjustments to the user-defined schedule so that the mobile robot 300 is not manipulated during times when the user 100 has stopped the mobile robot 300 from moving, and so that the mobile robot 300 is manipulated during times when the user 100 has caused the mobile robot 300 to begin manipulating.

[0203] Exemplary Error Correction Process In some implementations, the detection system 308 of the mobile robot 300 determines that an error condition has occurred for the mobile robot 300. The remote computing system 200 receives an indication of the error condition from the mobile robot 300, determines recommended actions to address the root of the error condition, and causes the audio media device 400 to provide guidance to the user 100 for implementing the recommended actions. FIG. 10 shows an example of an error correction process 1000. The error correction process 1000 of FIG. 10 shows example actions performed by the user 100 (user actions 1000A), the audio media device 400 (audio media device actions 1000B), the remote computing system 200 (remote computing system actions 1000C), and the mobile robot 300 (mobile robot actions 1000D) to cause the audio media device 400 to provide guidance for correcting the error.

[0204] In some examples, a mobile robot error occurs while the mobile robot 300 performs an action (1002). While the mobile robot 300 performs an action, the mobile robot 300 determines whether an error has occurred (1004). For example, the mobile robot 300 determines that an error has occurred when sensors from the sensing system 308 detect an error condition. The error condition may correspond, for example, to a stall condition for the drive wheels 302 of the mobile robot 300, a depleted charge condition for the battery of the mobile robot 300, or other condition that may hinder or impede operation of the mobile robot 300.

[0205] If an error occurs, the mobile robot 300 transmits (1006) a wireless signal indicating the error. In some cases, the mobile robot 300 also emits an error notification signal. The remote computing system 200 determines (1008) a recommended action to address the error and then causes the audio media device to issue (1010) the error notification and initial audible instructions to address the error. The initial audible instructions correspond to the determined (1008) recommended action. The user 100 performs (1012) the recommended action in accordance with the initial audible instructions.

[0206] The mobile robot 300 determines (1014) whether the error has been addressed, for example, by checking a sensor that indicated an error condition. If the error has not been addressed, the remote computing system 200 determines (1016) a next recommended action to address the error and causes the audio media device to issue (1018) a next audible command. The user 100 again performs (1012) the next recommended action, and the mobile robot 300 again determines (1014) whether the error has been addressed. The audio media device 400 continues to issue (1018) subsequent audible commands, for example, until the mobile robot 300 determines (1014) that the error has been addressed.

[0207] If the mobile robot 300 determines (1014) that the error has been addressed, the mobile robot 300 issues (1020) a success notification. The mobile robot 300 optionally sends (1022) the success signal to the remote computing system 200 to cause the remote computing system to issue (1024) an audible success notification to the audio media device 400 indicating to the user 100 that the error has been addressed. In some implementations, the success notification includes a visual signal, for example, caused by activation of the visual indicator 406.

[0208] In some examples, as the mobile robot 300 navigates through the house 10, hair may become tangled in the drive wheel 302, thereby preventing the drive wheel 302 from rotating. The audio media device 400 issues audible instructions during the error correction process 1000 to describe to the user 100 a sequential set of actions to remove the hair from the drive wheel 302. If the mobile robot 300 is a cleaning robot, the error correction process 1000 may be executed to cause the audio media device 400 to provide audible guidance to the user 100 for addressing an error associated with a cleaning device on the mobile robot 300, such as a cleaning pad, cleaning roller, or dustbin. If the cleaning device is a cleaning pad, the audio media device 400 issues audible instructions to guide the user 100 through the steps of removing and replacing the cleaning pad. If the cleaning device is a cleaning roller, the audio media device 400 issues audible instructions to guide the user 100 through the steps of removing the cleaning roller, cleaning away debris blocking the operation of the cleaning roller, and replacing the cleaning roller at the mobile robot 300. If the cleaning device is a dustbin, the audio media device 400 issues audible instructions to guide the user 100 through the steps of removing the dustbin, emptying debris from the dustbin, and replacing the dustbin at the mobile robot 300.

[0209] In some examples, while the user 100 performs actions to address the error (1012), or after the mobile robot 300 initially determines that an error has occurred, the remote computing system 200 determines that the error cannot be addressed by user action or that assistance from a customer service representative might help the user 100 address the error more effectively. In some cases, when the number of issued audible instructions reaches this predetermined number of instructions, the remote computing system 200 may cause the audio media device 400 to issue an audible indication that the user 100 would benefit from interacting with a customer service representative. The remote computing system 200 may establish a wireless communication channel between the audio media device 400 and the customer service representative, and the customer service representative may provide further instructions to address the error. In some examples, the remote computing system 200 may cause the audio media device 400 to ask the user 100 whether they would like to connect to the customer service representative using the audio media device 400. The user 100 audibly consents, and the audio media device 400 accesses the user's contact information from the remote computing system 200 and connects the user device 202 with customer service. For example, if the error requires the mobile robot 300 to be shipped to a repair shop for maintenance, the audio media device 400 establishes a wireless communication channel with a customer service representative so that the user 100 can receive information about the repair process. In some examples, the audio media device 400 issues audible commands (1010) until a predetermined number of commands have been given.

[0210] Other alternatives While FIG. 3A is described with respect to a vacuum cleaning robot, other mobile robot types would benefit from the processes described herein. In some examples, the mobile robot is an outdoor robot that navigates in an outdoor environment. The mobile robot may be a robotic lawn mower with a rotatable blade that cuts grass or other mowable areas of the outdoor environment. During a command process 700, a user 100 can provide voice commands to the audio media device 400 to cause the robotic lawn mower to perform a mowing operation. The user 100 can issue voice commands to the audio media device 400 to specify settings for the robotic lawn mower, such as the height of the blade assembly or the height of the grass to be cut. If the outdoor environment includes multiple lawns to be cut by the robotic lawn mower, the voice command may indicate the specific lawn to be cut during a particular mowing operation. During a notification process 800, the audio media device 400 can provide status updates regarding the mowing missions performed by the robotic lawn mower. During recommendation process 900, audio media device 400 may issue an audible request to purchase replaceable parts for the robotic lawnmower, such as a snubber, blade, or blade assembly. The robotic lawnmower may therefore include sensors to detect the operability of these replaceable parts. During error correction process 1000, audio media device 400 may issue instructions to address the root of an error associated with, for example, a blade or blade assembly of the robotic lawnmower.

[0211] If the mobile robot 300 is a cleaning robot, it may be a sweeping or mopping robot that may be equipped with a cleaning pad. In some cases, the cleaning robot sprays liquid onto a floor surface as the cleaning robot navigates through the home 10. A cleaning pad is equipped on the cleaning robot so that the cleaning pad absorbs the liquid as the cleaning robot traverses the floor surface. During a command process 700, the user 100 can provide voice commands to the audio media device 400 to control cleaning operations for the cleaning robot. In some cases, the voice commands are used to adjust the amount of liquid sprayed from the cleaning robot during the cleaning operation. During a notification process 800, the audio media device 400 can provide a status update regarding the status of the cleaning pad, for example, whether the cleaning pad equipped on the cleaning robot should be replaced.

[0212] During the recommendation process 900, the audio media device 400 may recommend a particular cleaning pad to be used based on the detected floor surface type. Different types of pads that may be installed on the cleaning robot include, for example, a dry mopping pad, a wet mopping pad, or a wet mopping pad. A wet mopping pad, for example, absorbs more liquid than a wet mopping pad, while a dry mopping pad is intended for use without liquid. Based on the floor type of the room, which may be detected using sensors on the mobile robot 300 or input by the user 100, the audio media device 400 may issue an audible request to change the cleaning pad to the recommended cleaning pad. For example, if the floor surface is a wooden surface, the audio media device 400 may issue an audible request for the user 100 to attach a wet mopping pad to the cleaning robot. If the floor surface is a tile surface, the audio media device 400 may issue an audible request for the user 100 to attach a wet mopping pad to the cleaning robot. In some examples, the remote computing system 200 causes the audio media device 400 to issue an audible request to purchase an additional cleaning pad. The remote computing system 200 receives, for example, purchase dates when the user 100 previously purchased disposable cleaning pads for the mobile robot 300 and determines from the purchase dates that the user 100 is likely to need an additional cleaning pad based on the purchase dates.

[0213] In some examples, as shown in Figures 3A-3B and 11A-11G, the mobile robot 300 is capable of patrolling the house 10 and monitoring conditions therein. As shown in the examples of Figures 11A-11G, the mobile robot 300 includes driven movement members 232A, 232B that interface with the floor surface and support the robot chassis 2010. The driven movement members 232A, 232B can be commanded by a controller, for example, a microprocessor 2021, to cause the mobile robot 300 to traverse the floor surface within the house 10. The microprocessor 2021, in some examples, navigates the mobile robot 300 to or through one or more mapped locations within the house 10.

[0214] The mobile robot 300 may include a camera 2070B. In some cases, the camera 2070B is part of a detection system (e.g., the detection system 308), while in other implementations, the camera 2070B operates independently from the detection system 308. In some examples, the image capture system 310 includes the camera 2070B. In some examples, the camera 2070B operates independently from the image capture system 310. In some examples, the mobile robot 300 includes a variable height member. The variable height member is, for example, a fully retractable support column 280 that supports the camera 2070B. In some cases, the camera 2070B is mounted to the top of the support column 280, for example, at the tip of the support column 280. In some implementations, the detection system includes navigation and environmental sensors mounted on the chassis 2010 of the mobile robot 300. The detection system is configured to detect, for example, the location and status of a network entity in communication with wireless network circuitry, the network entity being included within the home 10.

[0215] In some cases, conditions imaged and / or detected by the mobile robot 300, for example by camera 2070B, are presented in a virtual reality portal that allows the remote user 100 to view the house 10 from multiple perspectives. The user 100 can interact with the presented images to simulate movement through and within multiple rooms in the virtual reality portal without having to manually drive the mobile robot 300 through the house 10 to view areas of the house 10. In some examples, the virtual reality portal is an immersive, high-definition, interactive 3D photo-reproduction of the house 10. The images are composited (e.g., constantly and / or continuously stitched) to form an interactive 3D photo-reproduction that preserves the continuity of locations imaged by the mobile robot 300 as it navigates through the house 10. In cases where the camera 2070B is mounted on a mast 280, multiple vertical perspectives of the camera 2070B are composited as the mast 280 is moved vertically. The vertical viewpoint may be adjusted as the mobile robot 300 passes through the house 10 .

[0216] In some implementations, the mobile robot 300 rotates in location about a vertical axis Z while the camera 2070B captures images of the house 10, for example, while rotating through rotation of the chassis 2010 or the support 280. In some implementations, one of the mapped locations has one or more automatically selected and / or user-selected camera elevation positions. The locations are positioned at default time lapse or distance intervals and / or at user-selected locations within the house 10. In some implementations, the camera 2070B captures between two and four images. In some implementations, the camera 2070B captures between four and eight high resolution images during a 360-degree rotation at the location of interest. In some implementations, the support 280 extends between level with the top surface of the chassis 2010 and a height of 1.5 meters (e.g., 5 feet). The location-based imagery thus provides an interactive, explorable view of the home 10 presented in a virtual representation that the remote user 100 can navigate on a human-readable display of a remote internet-accessible device without having to drive the robot 300 through the home 10.

[0217] In some implementations, by requesting the mobile robot 300 to move to a location of interest, a remote user 100 of the system views the house 10 from a first human perspective, in an immersive, interactive 3D photo-reconstruction and / or in a real-time video feed, while moving between adjacent locations and selectively examining the location of interest more thoroughly from multiple perspectives. In some examples, the mobile robot 300 selects default locations Lx1, Lx2, Lx3...Lxn for imaging the living space in a 360-degree view around the robot 300, and / or the remote user 100 of the system selects some or all of the locations from which the mobile robot collects location-based imagery of the house 10 during its passage. The system automatically generates multiple remotely accessible 360-degree views from multiple rounds of the mobile robot 300 through the house 10, in some cases each day, and the user 100 accesses these recorded, time-stamped rounds from a selection list on a human-readable display of a remote Internet-accessible device.

[0218] In some implementations, the mobile robot 300 is used to remotely monitor non-visible conditions (e.g., temperature, toxicants, humidity, and similar air quality measurements) within the home 10 and overlay visible informational icons and / or interactive menus within a virtual reality portal at displayed locations Lx1, Lx2, Lx3...Lxn where conditions are measured. In some implementations, the mobile robot 300 includes one or more on-board environmental sensors configured to collect information from the home 10 during floor plan traversal, and the monitoring system presents the information to the remote user 100 monitoring the conditions through an interactive 3D photo reproduction of the home 10. In some implementations, the system also includes one or more stationary sensors not on-board the mobile robot that are used to monitor the living space to collect data that is presented to the remote user 100 monitoring the conditions through an interactive 3D photo reproduction of the home 10. In some implementations, the mobile robot 300 detects the status of stationary sensors monitoring the house 10, and on the remote user display, the system provides references and / or interactive menus adjacent to the stationary sensors in an interactive 3D photo reproduction of the house 10.

[0219] Network entities such as thermostats, air purifiers, and humidifiers are stationary and typically located in one or two locations throughout the living space, with stationary sensors within them measuring relatively localized airflow at that specific, unchanging location. Mobile robots 300 offer the advantage of accessing locations in other compartments or rooms that are remote from or not immediately adjacent to the network entities. By monitoring and measuring temperature, air quality, and humidity throughout the living space, mobile robots 300 provide information otherwise inaccessible by static network entities.

[0220] 3A-3B and 11A-11G, an exemplary mobile robot 300 includes a chassis 2010, a controller 306, a memory 318, a battery 2024, a battery charger 2026, a drive system 2030, a mapping / navigation system 2040, a wireless communication system 316, an IR emitter 2060, sensors 2070A-N, 310, and 314, an indicator light 2074A, an audio transducer 2074B, and a structured light sensor 2070K for obstacle avoidance and detection. The controller 306 may include any suitably configured processor 2021 (e.g., a microprocessor). The microprocessor 2021 is in communication with the controller 306, the memory 318, the sensors 2070A-N, 310, and 314, and the drive system 2030. In some implementations, the camera 2070B is an imaging device that collects 2D images, panoramic views, video, and / or 3D models.

[0221] The wireless communication system 316 includes a wireless communication transmitter or module 2052 (e.g., a Wi-Fi module) and an associated antenna 2054 for enabling wireless communication between the robot 300 and the hub 110 (such as a Google OnHub wi-fi access point) and / or the private network 160 (i.e., via WAP 164). A variety of different network configurations may be utilized for the private network 160 of which the mobile robot 300 constitutes a node. In some examples, the robot 300 communicates wirelessly with the hub 110 through a router 162 via WAP 164. In some examples, the mobile robot 300 bypasses the hub 110 and communicates with the remote management server 204 via the router 162 and WAP 164.

[0222] 3B , in some implementations, the robot 300 includes environmental sensors. The exemplary robot 300 includes the following environmental sensors: an IR radiation detector 2070A, a camera 2070B, an ambient temperature sensor 2070C, an ambient light sensor 2070D, an acoustic sensor 314 (e.g., a microphone), a motion detector 2070F (e.g., a passive IR photodiode), an ultrasonic sensor 2070G, a pressure sensor 2070H, an air quality sensor 2070I, and a humidity sensor 2070J. These sensors are not exhaustive of the types of sensors that may be provided on the robot 300, and some of the sensors may be omitted depending on the environmental parameters to be detected by the robot 300.

[0223] In some implementations, the mobile robot 200 includes sensors for obstacle detection and avoidance (“ODOA”) while traversing the house 10. These sensors include an ultrasonic sensor 2070G, an infrared emitter / detector proximity sensor 2065, and a mechanical bumper switching sensor 2032 that is triggered upon contact with a stationary obstacle and a non-contact sensor, such as a structured light sensor 2070K manufactured by PixArt or the like.

[0224] In some implementations, as described in detail above, the mobile robot 300 includes a mapping / navigation system 2040 for autonomous navigation and mapping of the home 10. The mobile robot 300 includes sensors for autonomous navigation, such as a camera 310 for simultaneous visual localization and mapping (“VSLAM”), a mouse sensor 2070M, an IMU 2070L with a three-axis accelerometer and a three-axis gyroscope, and / or a wheel odometer 2070F for determining or registering the position of the robot 300 relative to the space 10 (i.e., for locating the robot 300 within the space 10). The robot 300 may localize the location of readings collected by its onboard sensors 2070A-J. Any suitable techniques and components may be used to localize and register the robot 300, such as machine vision (e.g., using the camera 310 and feature recognition or class recognition software), optical beacons, or radio frequency received signal strength indicator (RSSI) technology.

[0225] In some implementations, the mobile robot 300 includes at least one orientation sensor that observes sensor readings from objects in space to determine the current pose of the mobile robot 300 relative to the observed object. The mobile robot 300 associates the pose with a room identifier deployed in the room that is specifically associated with the observed object. In some implementations, the autonomous mobile robot 300 builds an incrementally improving map as it covers this map and transmits the map data (e.g., the entire set of map data, a simplified representation of the map data, or an abstraction of the map data) to the remote computing system 200. To reach the public Internet 205 with that data or representation and / or abstraction, the microprocessor 2021, wireless transmitter 2052, wireless receiver 2054, and / or transceivers (including those with their own embedded microprocessors) will communicate using IP (Internet Protocol) and support conventional addressing and packetization for the public Internet 205 or the remote computing system 200.

[0226] Any portion of the map database or coverage database may be transmitted to and stored at a location other than the robot 300, such as a local hub 110 or gateway in the home 10, a hub, gateway, server 204, or the like, on a remote computing system 200, or a virtualized instance thereof available on the Internet. In some cases, the mobile robot 300 transmits its map to the remote computing system 200, and an audio media device (AMD) 400 accesses this map from the remote computing system 200 to provide the user 100 with an audible summary of the measurement data. In some examples, the AMD 400 queries the user 100 regarding modification of parameters of connected devices in the home 10 in response to measured conditions. The AMD 400 affects these changes according to verbal commands from the user 100, such as closing vents or closing blinds in a relatively warm room, which causes the AMD 400 to proceed with a recommended action.

[0227] The remote computing system 200 maintains an up-to-date database of physical maps and other spatial information about the home 10 and data collection by the mobile robot 300. The map in the remote computing system 200 includes the locations of relevant objects, such as furniture, connected devices in the home, like connected lights or appliances, and layers of spatial information about the home, like temperature or lighting maps. Using this information, the remote computing system performs important spatial analysis operations. For example, if the user 100 wants a reading light on the couch in the living room, the remote computing system 200 determines which lights to turn on and how bright to make them based on the physical map of the home 10. Using knowledge of the location of the living room, couch, and nearby light sources, along with understanding how different light sources affect the illumination of the area around the couch, the remote computing system 200 adjusts the connected lights. The mobile robot 300 builds a map and persistently maintains it with each mission or periodic patrol of the home 10. The map iteratively improves and / or changes with changes in the environment's navigable floor space, e.g., the obstacle-free floor space through which the mobile robot 300 can pass. As the mobile robot 300 moves through the house 10, it identifies the structure of the space, such as where the walls are, and builds a physical map of the space. This is an important step because the physical map is the basis for understanding the space and layout of the house 10. Upon initial installation, the mobile robot 300 must explore and build a map from scratch. Over time, however, the mobile robot 300 learns the overall map, or accesses it from a remote computing system, and uses the map to systematically navigate the house 10, recognizing when something has changed and updating the map in the process.

[0228] As the mobile robot 300 moves through the house 10, it detects objects of interest, such as common furniture or appliances, which can be marked and added to the map. Spatial object understanding provides the remote computing system 200 with a richer level of understanding that is useful when performing higher-level tasks, such as responding to a request to provide a reading light next to the couch. The user 100 may also indicate other specific items of interest to the mobile robot 300, such as the AMD 400, a child's backpack, a special toy, or the family dog. The locations of these items are updated on the map as the mobile robot 300 encounters them. Furthermore, given spatial object understanding, rooms 20 may also be marked: the refrigerator is in the kitchen; the bed is in the bedroom. This provides additional semantic context to the robot map, and the AMD 400 accesses this information to provide verbal notifications to the local user 100.

[0229] The mobile robot 300 also discovers connected devices in the home through standard protocols and localizes them on a map. This includes the locations of connected lights and speakers, vents, door and window sensors, and other connected devices in the home 10. The mobile robot 300 roams the home 10 and recognizes connected devices in the home 10 using RF signatures, visual recognition, received signal strength, and other methods, and automatically places them on a robot map of the home 10. For example, the mobile robot 300 explores the home 10 and recognizes the NEST thermostat on the living room wall, the connected SAMSUNG refrigerator in the kitchen, and the HUE BLOOM lights in both the family room and the bedroom. The mobile robot 300 places the recognized devices on a map, allowing the user 100 and the AMD 400 to take advantage of this spatial knowledge of the connected devices. For example, the user can turn on the bedroom light by speaking the phrase, "Turn on the bedroom light." The interaction between the AMD 400 and the mobile robot 300 described herein enables this simple speech control via devices detected by the mobile robot 300 as it patrols the house 10 .

[0230] In some implementations, the mobile robot 300 controls various connected devices to learn how they affect the home 10. For example, the mobile robot 300 samples illuminance levels throughout the home 10 to develop a spatial illuminance map for scenarios where different lights are switched on and off at different times. This allows it to understand how to achieve several lighting profiles throughout the home 10 based on control of individual light sources. Similarly, this can be done for speakers and sound or volume profiles, airflow through climate control vents, and temperature profiles. This all leads to the mobile robot 300 providing the remote computing system 200 with a richly labeled map of the home 10. This integrated understanding of the spatial layout with the locations of key objects and devices provides the foundation for valuable spatial analysis needed to enable the smart home 10 (e.g., a home equipped with connected devices) to manage itself and deliver easy-to-use smart home capabilities to the user 100.

[0231] 12, in some implementations, the mobile robot 300 begins autonomously patrolling and mapping an enclosed space, e.g., a house 10 or a room 20 within the house 10. While navigating the floor plan of the enclosed space, the mobile robot 200 autonomously builds a map of the enclosed space, including marking waypoints along a patrol path 1200. In some implementations, the mobile robot 300 rotates in place about a vertical Z-axis (shown in FIG. 11E) at various waypoint locations L1, L2, L3...Ln throughout the enclosed space. These locations L1, L2, L3...Ln are waypoints along the robot patrol path 1200 as the robot 300 traverses the enclosed space on a surveillance mission to collect image data, data collected from stationary networked devices along the patrol path 1200, and other environmentally sensed data detected by the mobile robot's 1200 sensors, for processing and presentation to the end user 100 at the remote user terminal 144 and / or by the AMD 400, which provides audible messaging to the user 100 within the enclosed space. As shown in the schematic floor plan of the house 10 in FIG. 12, the locations have two-dimensional coordinates relative to the global coordinate system of the map. For example, in some implementations, the robot 300 begins patrol at location L0, where the robot dock is located. Location L0, for example, corresponds to the location of the robot dock and is the global origin to which all other locations L1, L2, L3...Ln are referenced on the robot map. In some implementations, the robot 300 automatically selects locations L1, L2, L3...Ln at which to stop and turn based on default time intervals and / or distance traveled. For example, in one implementation, the robot 300 stops every four feet and turns to collect images representing a 360-degree robot view at each stop location L1, L2, L3...Ln, and uploads the collected images to an image database stored in memory on or off the robot 300. In other implementations, a remote user at a remote user terminal pre-selects one or more locations Lx1, Lx2, Lx3...Lxn at which the robot 300 will stop.In some implementations, the robot 300 stops at a combination of robot default and pre-selected user locations L1, Lx1, L2, Lx2, L3...Ln.

[0232] In some implementations, the AMD 400 is separate from the mobile robot 300 and is stationary within an enclosed space or the house 10. While passing through the house 10, the mobile robot 300 detects the AMD 400 and stores the AMD location on a robot map, which can be shared with the AMD and other networked devices and robots directly or through a remote computing system 200 (e.g., the cloud). In other implementations, the AMD 400 is included on the robot chassis 2010 and moves with the mobile robot 300. In some implementations, the AMD 400 is fully embedded within the chassis of the robot 300, and in other implementations, the AMD 400 is a portable, detachable device interchangeably attached to various mobile robots 300, 301 in the house 10.

[0233] In the example where the mobile robot 300 is equipped with an AMD 400, voice interactions become location-aware in the home 10. A voice command such as "Turn off the lights" when the robot 300 is in the kitchen will turn off the connected kitchen lights, and when in the living room, will turn off the connected living room lights. Similarly, a user 100 can speak to a mobile robot 300 that has a built-in AMD 400 to set the temperature and adjust the audio volume on a connected speaker. For example, the user 100 can say "Turn the volume down," and the AMD-equipped robot 300 will turn down the volume on the speakers at the user's 100 location.

[0234] As mentioned above, in some implementations, the mobile robot 300 samples sensor readings throughout the home 10 and uses them to become more insightful in its responses / actions. For example, if the mobile robot 300 takes temperature readings in the kitchen and in the living room, the user 100 can command the AMD-equipped mobile robot 300 to adjust the temperature in both rooms evenly by saying, "Set the temperature in the living room to the same setting as the temperature in the dining room." In some implementations, the mobile robot 300 takes light level readings throughout the home as different lights are switched on and off, learning how each individual light affects the lighting of the space. Then, if the user 100 requests more light in the living room, the mobile robot 300, knowing its location in the home 10, can turn on the lights appropriately to provide more lighting to a given space in the home 10.

[0235] A user 100 may query the AMD-equipped mobile robot 300 with a query such as, "What is the Wi-Fi signal strength at the back door?" The mobile robot 300 may go to the back door, take sensor readings, and audibly and / or visually (e.g., on a smartphone app, SMS, text message, email, or banner alert) inform the user 100 of the results. In another example, the user 10 may ask the AMD-equipped mobile robot 300 a question about an environmental condition monitored within the home 10, such as by verbally asking, "Have you detected humidity above 90% anywhere in the house today?" The AMD-equipped mobile robot 300 then responds by producing an audible and / or visual response.

[0236] As described above with respect to the components of the AMD, sound localization allows the AMD-equipped mobile robot 300 to know the relative location of the speaking user 100 with respect to the robot 300, and the mobile robot 300 can respond to commands such as "come here" by facing the direction of the speaking user 100, as detected by the direction-sensing microphones on the AMD 400. Because the mobile robot 300 detects the AMD 400 and places its location on the robot map, and because the robot map is shared with remote computing devices and AMDs, the communications network 201 can determine the distance to the speaking user 100. For example, the AMD 400 may detect multi-path voice reflections from the user 100 at one or more of its microphones located on the AMD 400. Knowing the distance and orientation of the microphone relative to the wall or walls from which the user's 100 voice bounces, the AMD 400 and / or the remote computing system 200 can use these multipath signals to calculate the origin of the detected voice command based on triangulation and basic trigonometry. In some implementations, the remote computing system 200 and / or the AMD 400 compensate for noise and / or attenuation when calculating the speaking user's pose on the robot map. For example, the user 100 may say to the mobile robot 300, "Come here, stay here today," and the mobile robot 300 moves to the user's 100 location. Similarly, the user 100 may vocalize the word "follow me" to have the AMD-equipped mobile robot 300 follow behind the user 100 as he walks. The user may say, "look this way," and the AMD-equipped mobile robot 300 will reorient its front part to face the user 100.

[0237] If the user 100 gives a voice command but the AMD device 400 has trouble hearing, the AMD-equipped mobile robot 300 may reorient and / or move closer to the user 100 or away from another noise source to be more audible and ask the user 100 to repeat the voice command.

[0238] In some examples, the user 100 provides an audible command to move the mobile robot 300 toward the target. In response to the audible command, the mobile robot 300 autonomously navigates toward the target. In some cases, the mobile robot 300 moves the support 280 to a height relative to the chassis 2010 such that the camera 310 or other sensor supported by the support 280 is aimed at the target after the mobile robot 300 autonomously moves to a position close to the target.

[0239] In some implementations, the user 100 may query the AMD-equipped mobile robot 300 for items elsewhere in the home 10 that require visual or other type of sensor inspection. For example, the user 100 may ask, "Is the front door open?" (assuming the front door has no internet or LAN-connected open / close sensors). The AMD-equipped mobile robot 300 goes to the front door and visually inspects the door with the robot's camera and / or proximity or other distance sensors to acquire the data needed to answer the question. The data is sent to the remote computing system 200 for processing, and the AMD-equipped mobile robot 300 returns to the user 100 and audibly pronounces a response to the query. In this manner, the user 100 may ask the AMD-equipped mobile robot 300 to patrol for any number of status conditions requiring mobility and mapping. For example, the user may ask the AMD-equipped mobile robot 300, "Is the study window open?", "Is the kitchen light on?", or "Is someone in the living room?" In response, the AMD-equipped mobile robot 300 travels to the destination and collects data in response to the query.

[0240] Similarly, the user 100 can ask the AMD-equipped mobile robot 300 for the location of an object within the house 10. In one implementation, the user 100 speaks the words, "Where's my backpack?" and the AMD-equipped mobile robot 300 moves around the house 10 searching for the backpack. In one implementation, the AMD-equipped mobile robot 300 accesses pictures of the identified object taken in the house 10 during a recent mission or patrol, and reports the location of the identified object to the user once the object is found and verified against an on-board collection of images or a collection of images stored in the remote computing system 400. In some implementations, the user 100 asks the AMD-equipped mobile robot 300 to find a device with an RF signature. For example, if the user 100 asks the AMD-equipped mobile robot 300, "Where's my phone?", the mobile robot 300 searches for radio frequency signals throughout the home 10 to find the Bluetooth ID of the phone associated with the user 100 in a database on or accessible to the remote computing system 200 and / or the mobile robot 300. The AMD-equipped mobile robot 300 then audibly and / or visually alerts the user 100 of the phone's location. Because the mobile robot 300 is equipped with the AMD 400, the mobile robot 300 is equipped to naturally speak to the user in plain language. For example, the AMD-equipped mobile robot 300 might tell the user, "Your phone is here, on the couch, in the living room." This eliminates the need for the user 100 to log in to an application or web page on the terminal or user device 202. The user 100 need rely solely on real-time, natural speech interaction to command the mobile robot 300 and control connected devices throughout the home 10.

[0241] In some implementations, a user expresses commands and queries to the AMD-equipped mobile robot 300 in words such as "return to charging dock," "put camera in privacy mode (e.g., lower the support 280)," "go to location," "increase / decrease driving speed," "raise the camera support," "turn around," and "leave this room." The user 100 therefore does not need to log into an app or use a separate online portal or remote control to command the robot. Simple spoken words command the mobile robot 300 without requiring additional levels of inefficiency or complexity.

[0242] In some implementations, the audio media device 400 is an alternative user interface for activating and deactivating the surveillance system (e.g., cameras 310, 2070B) of the mobile robot 300. The user 100 can use voice commands to command the AMD-equipped mobile robot 300 to patrol the house when they are out, or to shut down for the night upon arriving at the home 10.

[0243] In some implementations, the AMD 400 allows the user 100 to establish an audible password for activating or deactivating the monitoring system of the mobile robot 300 .

[0244] In some implementations, the AMD 400 allows the user 100 to command the mobile robot 300 via the AMD 400 to take an image or video from the robot's camera 310, 2070B. This can be used for user identification, such as telling the robot 300 via the audio media device to recognize the user 100 standing in front of the camera 310, 2070B. As an added level of security / authorization, in some implementations the mobile robot 300 must identify the user 100 as part of unlocking a level or levels of access to functionality accessible via the audio media device 400.

[0245] In some implementations, the audio media device 400 prompts the user 100 with an audible response in response to a detection or action by the monitor robot 300. For example, if the mobile robot 300 detects a person in a house during a mission, it commands the person to identify themselves and / or provide additional information via the audio media device 400.

[0246] In some implementations, the camera 310, 2070B of the mobile robot 300 is an alternative user interface that complements the audio media device 400. One example would be to use the camera for gesture recognition so that, in response to the gesture, the mobile robot 300 relays commands to the audio media device 400 to affect the operation of the audio media device 400 and / or the operation of other devices connected to the audio media device 400 and / or the remote computing system 200. For example, the user 100 may tell the audio media device 400 via a voice command to adjust the lights, and the camera 310, 2070B on the mobile robot 300 may then detect that the user 100 is raising their hand and / or arm and command the AMD 400 and / or the remote computing system to increase or decrease the light level accordingly.

[0247] The autonomous mobile robots described herein may be controlled, at least in part, using one or more computer program products, e.g., one or more computer programs tangibly embodied in one or more information carriers, such as one or more non-transitory machine-readable media, for execution by or to control the operation of one or more data processing devices, e.g., programmable processors, computers, multiple computers, and / or programmable logic components.

[0248] Operations associated with controlling the autonomous mobile robots described herein may be performed by one or more programmable processors executing one or more computer programs to perform the functions described herein. The computer programs may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including components, subroutines, or other units suitable for use in a computing environment, either as stand-alone programs or as modules. Control of all or part of the robots described herein may be implemented by special purpose logic circuitry, e.g., FPGAs (field programmable gate arrays) and / or ASICs (application-specific integrated circuits).

[0249] The controllers described herein may include one or more processors. Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory, or both. Elements of a computer include one or more processors for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer will also include one or more machine-readable storage media, e.g., magnetic, magneto-optical, or optical disks, such as a mass PCB for storing data, or be operatively coupled to a mass storage device to receive data from, transfer data to, or both. Machine-readable storage media suitable for carrying computer program instructions and data include all forms of non-volatile storage, including, by way of example, semiconductor storage devices, e.g., EPROM, EEPROM, and flash storage devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0250] Although several implementations have been described, it will be understood that various modifications may be made and, accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0251] 100 users, remote users, end users 102 linked devices 104 Docking Station 106 Transmitting Unit 144 Remote User Terminal 200 Remote Computing System 201 Communication Network 202 Mobile Computing Devices 204 Server, Remote Management Server 205 Cloud Computing Networks, Public Internet 232 Driven moving members 280 Post 300 Mobile robot, robotic device, robot, autonomous mobile robot, AMD-equipped mobile robot 301 Second mobile robot, robot device, mobile robot 302 Drive Wheel 304 Caster Wheel 306 Controller 308 Detection System 310 Camera, Visible Light Camera, Image Capture System 312 Audio Emission System 314 Microphones, Acoustic Sensors 316 Wireless Communication Systems 317 Rotatable roller or brush 318 memory 400 Audio Media Device, AMD 402 microphone unit 404 speaker unit 406 Visual Indicators 408 Controller 410 memory 412 Wireless Communication Systems 502 Audio Media Device Identification Database 504 Mobile Robot Identification Database 506 User Identification Database 508 Command Database 2010 Robot Chassis, Chassis 2021 Microprocessors, Processors 2024 battery 2026 Battery Charger 2030 Drive System 2032 Mechanical shock absorber switching sensor 2040 Mapping / Navigation System 2052 Wireless communication transmitter or module, wireless transmitter 2054 Antenna, Wireless Receiver 2065 Infrared Emitter / Detector Proximity Sensor 2070A IR Radiation Detector 2070B Camera, Sensor 2070C Temperature Sensor, Sensor, Ambient Temperature Sensor 2070D Ambient Light Sensor, Sensor 2070F Motion Detector, Sensor, Wheel Odometer 2070G Ultrasonic Sensor, Sensor 2070H Pressure Sensor, Sensor 2070I Air Quality Sensor, Sensor, Humidity Sensor 2070J Air Moisture Content Sensor, Sensor 2070K Structured Light Sensor, Sensor 2070L IMU, Sensor 2070M Mouse Sensor, Sensor 2074A Indicator Light 2074B Audio Transducer

Claims

1. 1. A method comprising: establishing wireless communication between an autonomous cleaning robot and a stationary audio media device, the autonomous cleaning robot being capable of moving about a space, the stationary audio media device being deployed on a surface in the space and including a microphone and a speaker, and an association between identification data of the autonomous cleaning robot and identification data of the stationary audio media device being made via a remote computing system; the remote computing system identifying a robotic command corresponding to a voice command given by a user and received by the microphone of the stationary audio media device; after the remote computing system identifies the robot command, providing the robot command to the autonomous cleaning robot to cause the autonomous cleaning robot to initiate one or more actions; In response to detecting an error condition in the autonomous cleaning robot, the remote computing system transmits data to cause the stationary audio media device to issue audible instructions indicating a course of user action to address the root of the error condition; A method comprising:

2. The method of claim 1 , wherein the one or more actions include moving the autonomous cleaning robot to move from a current location of the autonomous cleaning robot to a location proximate to the user.

3. The method of claim 1 , wherein the one or more actions include moving the autonomous cleaning robot from its current location to perform a spot cleaning action at a location proximate to the user.

4. The method comprises: determining a location corresponding to identification data associated with a room within the space; The method of claim 1 , wherein the one or more actions include moving the autonomous cleaning robot from a current location to the room within the space.

5. The method of claim 4 , wherein the one or more actions include moving the autonomous cleaning robot to the room within the space to perform a cleaning action within the room.

6. 10. The method of claim 1, further comprising causing the stationary audio media device to provide an audible success notification in response to providing the robot command to the autonomous cleaning robot to cause the autonomous cleaning robot to initiate the one or more actions.

7. The method of claim 1 , wherein the one or more actions include moving the autonomous cleaning robot to dock with a docking station and causing the docking station to empty debris from a debris bin of the autonomous cleaning robot.

8. 10. The method of claim 1, further comprising generating a map of the space, the map showing locations of devices in wireless communication with a remote computing system, the autonomous cleaning robot being in wireless communication with the remote computing system.

9. The method of claim 8 , wherein the device comprises the stationary audio media device.

10. The method of claim 1 , wherein the one or more actions include navigating the autonomous cleaning robot toward the stationary audio media device.

11. The method comprises: further comprising identifying an audible identifier associated with the autonomous cleaning robot; providing the robot command after identifying the robot command includes: providing the robot command after identifying the robot command and the audible identifier. The method of claim 1.

12. 12. The method of claim 11, wherein identifying the audible identifier associated with the autonomous cleaning robot includes accessing a user account associated with the autonomous cleaning robot identification data or the stationary audio media device identification data.

13. The method of claim 1 , wherein the one or more actions include starting or pausing a cleaning action of the autonomous cleaning robot.

14. The voice command indicates a user-defined schedule, and the one or more actions include: storing the user-defined schedule for performing subsequent actions at future times in a memory of the autonomous cleaning robot; causing the autonomous cleaning robot to perform the subsequent action at the future time; The method of claim 1 , comprising:

15. The method of claim 1 , wherein the stationary audio media device does not include a display.

16. and causing the speaker of the stationary audio media device to issue a mission status update in response to the autonomous cleaning robot completing the one or more actions. The method of claim 1.

17. identifying a room identifier in a user's utterance; associating the room identifier with a room within the space in which the autonomous cleaning robot is located; further comprising: The method of claim 1.

18. 1. A method comprising: establishing wireless communication between an autonomous cleaning robot and a stationary audio media device, the autonomous cleaning robot being capable of moving about a space, the stationary audio media device being deployed on a surface in the space and including a microphone and a speaker, and an association between identification data of the autonomous cleaning robot and identification data of the stationary audio media device being made via a remote computing system; causing the speaker of the stationary audio media device to emit an audible signal representative of an operational status of the autonomous cleaning robot, the operational status corresponding to status data generated by the autonomous cleaning robot as it autonomously navigates the space; In response to detecting an error condition in the autonomous cleaning robot, the remote computing system transmits data to cause the stationary audio media device to issue audible instructions indicating a course of user action to address the root of the error condition; A method comprising:

19. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal after determining that a user has entered the space.

20. the operational status of the autonomous cleaning robot includes a schedule of previous operations, and causing the speaker of the stationary audio media device to emit the audible signal includes causing the speaker of the stationary audio media device to emit an audible signal to modify the stored schedule of operations based on the schedule of previous operations.

20. The method of claim 18.

21. 20. The method of claim 18, wherein the operational status corresponds to a resting state of the autonomous cleaning robot in the space, and wherein causing the speaker of the stationary audio media device to emit the audible signal includes causing the speaker of the stationary audio media device to emit an audible signal to indicate that the autonomous cleaning robot is in the resting state.

22. 22. The method of claim 21, wherein the audible signal indicates a location of the autonomous cleaning robot in the space.

23. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal to indicate a frequency of previous operations of the autonomous cleaning robot.

24. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal to indicate a total duration of previous operations of the autonomous cleaning robot within a predetermined period of time.

25. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal to indicate an estimated remaining service life of a part of the autonomous cleaning robot.

26. 26. The method of claim 25, wherein the part of the autonomous cleaning robot is a cleaning brush, a cleaning pad, a roller, a battery, a dust container, or a wheel module.

27. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal to identify an obstacle detected by the autonomous cleaning robot while the autonomous cleaning robot navigates through the space.

28. 20. The method of claim 18, wherein the step of causing the speaker of the stationary audio media device to emit the audible signal causes the speaker of the stationary audio media device to emit the audible signal to identify a connected device located in the space.

29. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal in response to detecting an error to guide a user to address the root of the error associated with the autonomous cleaning robot.

30. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal after the autonomous cleaning robot has autonomously navigated through the space, the audible signal identifying the space that was not traversed while the autonomous cleaning robot was autonomously navigating.

31. 20. The method of claim 18, wherein causing the speaker of the stationary audio media device to emit the audible signal comprises causing the speaker of the stationary audio media device to emit the audible signal upon receiving an audible user request for the operational status of the autonomous cleaning robot.

Images