Robotic Computing Device with Adaptive User Interaction

Abstract

Implementations described herein relate to a robotic computing device that can perform certain operations, such as communicating between users in a common space, according to certain preferences of the users. When interacting with a particular user, the robotic computing device can perform operations at a location preferred for that particular user based on the specific user's explicit or implied preferences. For example, some types of operations may be performed at a first location in a room and other types of operations may be performed at a second location in the room. When an operation involves following or guiding a user, parameters for driving the robotic computing device may be selected based on the user's preferences and / or the context in which the robotic computing device is interacting with the user (e.g., whether the context indicates a certain amount of urgency).

Description

[Technical field]

[0001] A robotic computing device with adaptive user interaction. [Background technology]

[0002] When a computing device allows interaction between an automated assistant and a user, most computing devices cannot navigate autonomously to various destinations without being manually controlled by the user. This may limit the ability of some automated assistants to provide assistance with some tasks that may involve navigating toward and / or away from the user. For example, a user who gives the user's automated assistant the task of rendering some audio content may be limited to locations where the user can perceive the audio content. This may result from the audio content being rendered through standalone speaker devices and / or other computing devices that must be manually positioned by the user, often close to an electrical outlet.

[0003] In some cases, when a user requests that an automated assistant perform a particular action that may require the automated assistant to move between geographic locations (e.g., between different rooms in a home), some tasks may be delegated to a device that is capable of performing the action. However, often such devices may not be capable of handling a variety of actions. For example, an autonomous vacuum cleaner may be capable of initiating a default cleaning action upon the request of the automated assistant, but may not be capable of performing other cleaning-related actions with specificity. This may be the result of the automated assistant and / or the autonomous vacuum cleaner not having a mechanism for translating a request presented to the automated assistant by the user. This may be particularly inefficient when the robotic home device has some interfaces (e.g., speakers, sensors, etc.) that may be necessary to fulfill a particular assistant request, but the robotic home device does not have a mechanism for translating the particular assistant request into an executable action that can be performed by the robotic home device. Summary of the Invention [Means for solving the problem]

[0004] The implementations described herein relate to a robotic device that can perform various tasks, where the robotic device can be involved in navigating to a user for a particular purpose and / or communicating certain information between users. For example, the robotic device can operate in a user's home and receive a spoken utterance from the user, such as "Robot, ask Eli if Eli is ready for school." In this situation, the user can be the person who is the subject of the inquiry from the user, "Eli's" father. In response to the spoken utterance and with prior permission from the person at home, the robotic device can navigate from the living room where the robotic device and the user were originally located and move toward Eli's room. In some implementations, the robotic device can determine Eli's probable location by accessing data that can correlate some locations in the home to some names (e.g., "Eli's room"). Alternatively or additionally, the robotic device can determine Eli's probable location based on a previous interaction between the robotic device and Eli. For example, the other user, Eli, may have been located in the office area of ​​the home during the most recent interaction between the robotic device and Eli. Based on this determination, and in response to a verbal utterance from the father, the robotic device can navigate to the home office area to find Eli, the other user.

[0005] In some implementations, the robotic computing device can determine the probable location of the other user, "Eli," by accessing interaction data that can indicate where the other user recently interacted with another device. For example, just before the user provided the above-mentioned verbal utterance, the other user may have interacted with a standalone display device in the kitchen of the user's home. The home graph data can provide a correlation between the standalone display device and the kitchen of the home based at least on the standalone display device having a label, such as "kitchen display," in the home graph data. Alternatively or additionally, a semantic label for the room can be inferred from characteristics of the room, such as, for example, that the room is labeled "kitchen" based on the room being the only room in the residence that has a dishwasher and a microwave (e.g., as determined using one or more sensors and one or more object recognition techniques). In some implementations, the robotic computing device or the standalone display device can determine that this previous interaction involved the other user, Eli, using one or more user verification techniques (e.g., voice verification, face recognition, etc.). Thus, using this data, the robotic computing device can determine that the other user recently interacted with a particular device (e.g., a kitchen display) and that the particular device is located in the kitchen of the home. Based on this determination, the robotic computing device can respond to the above-mentioned verbal utterance from the user by navigating toward the kitchen of the home to facilitate initiating a dialogue with the other user, Eli.

[0006] When the robotic device identifies the location of the other user to whom the verbal utterance is directed, the robotic device may emit an output via an output interface (e.g., a display interface, an audio interface, etc.) of the robotic device with prior permission from the other user. For example, the output emitted to the other user may be a cordial audio output such as "Hi, Eli, your father is kindly interested in knowing if you are ready for school." When the robotic device emits an audio output and / or identifies Eli's location, the robotic device may activate one or more input interfaces to receive input from Eli. For example, Eli may have heard the user ask the robotic device to ask if Eli is ready for school, and as a result, provide a responsive input to the robotic device before the robotic device has a chance to render an audio output. In such a situation, the robotic device will be able to capture the input from Eli because one or more microphones of the robotic device will be preemptively activated before arriving at the other user's location. In other cases, when the other user, Eli, provides input after the robotic device renders an audible output, the other user can provide a responsive input such as "Tell Dad I'm almost ready."

[0007] When the robotic device receives a responsive input from the other user, the robotic device can cause the responsive input to be processed and can begin navigating to the user who provided the initial verbal utterance. In some implementations, the robotic device can access a voice processing module, which can process the audio data and / or text data using one or more trained machine learning models. For example, the one or more trained machine learning models can include a transform neural network and / or other language models that can be employed to convert the responsive input into a meaningful output. For example, the robotic device can cause audio input data corresponding to a verbal utterance from the user and / or another verbal utterance from the other user to generate audio output data. The audio output data can characterize a sincere natural language output such as "Eli gently indicated that Eli is almost ready to go." In this way, a more natural type of dialogue can be created between the robotic device and each user, as opposed to solely providing a verbatim representation of what the responding user may have said. This may allow for clearer interaction between a user and their robotic device and reduce instances where a robotic device is asked to repeat something another user has said to the robotic device. Some resources, such as battery life and processing bandwidth, may be conserved as a result.

[0008] In some implementations, the robotic computing device can assist the user in identifying a particular device and / or the location of a particular device in situations where the user may not have explicitly requested the robotic computing device to locate a particular device. In some instances, the robotic computing device can perform such an action when a particular device is rendering a notification that the user cannot acknowledge because the user is not close enough to the particular device and / or because the device is operating in silent mode. For example, a cellular phone in a user's home may be operating in silent mode when the cellular phone receives an incoming telephone call. The cellular phone may vibrate in silent mode, but the user may not be able to determine that the cellular phone is vibrating when the user and the cellular phone are in different rooms. However, the robotic computing device can receive a notification that the cellular phone is receiving an incoming call via a local area network (e.g., Wi-Fi) and with prior permission from the user.

[0009] In response to receiving the notification, the robotic computing device can render an output such as "Sir, your phone is ringing on silent." This output can be rendered when the robotic computing device is located within a threshold distance of the user and / or after the robotic computing device navigates toward the user in response to the notification. When the user hears the output from the robotic computing device, the user can respond with a verbal utterance such as "Oh, thank you, I thought my phone was just here." This verbal utterance can be captured by the robotic computing device via an audio interface of the robotic computing device and converted to audio data, which can be processed at the robotic computing device and / or another specific computing device (e.g., a network device such as a server) with prior permission from the user. The audio data can be processed to determine the user's willingness to be assisted by the robotic computing device despite not having explicitly requested the assistance of the robotic computing device.

[0010] For example, the audio data and / or other data may be processed using one or more heuristic processes and / or one or more trained machine learning models (e.g., transform neural network models, convolutional neural networks, recurrent neural networks, and / or other models). In some implementations, the robotic computing device may employ a neural network-based sequence classification model to determine, with prior permission from the user, whether the user has exhibited a questioning tone and / or an amount of uncertainty. For example, the audio data may be processed to generate a metric for the amount of uncertainty the user may be exhibiting regarding the subject matter embodied in the user's verbal utterances and / or output from the robotic computing device. When the metric meets a certain metric threshold, the audio data may be further processed to determine information and / or identify actions that may assist the user in analyzing the user's uncertainty and / or questions. For example, an action that identifies the location of a cellular phone may be determined by the robotic computing device to be useful for analyzing the user's detected uncertainty.

[0011] Based on this determination, the robotic computing device can offer to perform the action by rendering another output, such as "May I take you to the phone, if you like?" In some implementations, the user can consent to allowing the robotic computing device to take the user to the cellular phone by providing an explicit response input, such as "sure." Alternatively or additionally, the user can provide approval for the robotic computing device to take the user to the cellular phone by exhibiting body language and / or other characteristics that indicate a willingness of the user to be guided by the robotic computing device to the cellular phone. For example, in response to the robotic computing device providing the other output, the user can stand up from where they are sitting and walk toward the robotic computing device. In some implementations, audio and / or image data captured by the robotic computing device with prior permission from the user can be processed to determine whether the user exhibits a positive and / or approving response (i.e., other output) to the offer from the robotic computing device. In some implementations, the audio and / or image data may be processed using one or more of the same or different trained machine learning models used to process the verbal utterances from the user. For example, with prior permission from the user, the captured images may be processed to determine that the user's trajectory is toward the robotic computing device when the user stands up from his / her seat. Based on this determination, the robotic computing device may conclude that the user has expressed a willingness to be guided by the robotic computing device to the cellular phone.

[0012] In some implementations, the speed or acceleration of the robotic computing device and / or the urgency with which the robotic computing device reacts can be based on one or more features of the context in which the robotic computing device initiated the movement. For example, the type of notification received by the robotic computing device from another computing device can indicate the urgency of the notification and thus provide a basis for the speed at which the robotic computing device proceeds. For example, the robotic computing device can operate according to a first speed when the robotic computing device is attempting to find a ringing cellular phone. However, the robotic computing device can operate according to a second speed, lower than the first speed, when the robotic computing device is attempting to find a cellular phone in response to an incoming text message. Alternatively or additionally, the robotic computing device may establish a speed for moving to a particular location based on one or more different factors, such as the urgency it detects in the user's voice, the content of the particular notification, the source of the particular notification (e.g., the robotic computing device may move faster when a spouse sends a text message compared to when an acquaintance sends a text message), the time of day the notification is received, the application that is the source of the notification (e.g., the robotic computing device may move faster when a delivery notification is received from a shopping application compared to a social media application that provides notifications), and / or any other source of data that may be the basis for the robotic computing device to move to a different location.

[0013] In some implementations, the robotic computing device may move to a particular location within a particular room depending on the operation being performed by the robotic computing device and / or one or more features of the context in which the robotic computing device is performing the operation. For example, a user preparing dinner for guests may request the robotic computing device to play music in the kitchen of the user's home. In response to an explicit request to the robotic computing device (e.g., by providing a verbal utterance such as "Play some music"), the robotic computing device may move to a particular location in the living room of the home to play music when the guest is in the home. In some implementations, this particular location may be learned by the robotic computing device and / or explicitly identified by the user (e.g., "Play music in this particular location in the living room when the guest is here").

[0014] Moreover, in various implementations, this particular location may be specific to the action being performed. For example, when performing the action "play music", the robotic computing device may move to a first location in the room, when performing the action "read me the news", the robotic computing device may move to a separate second location in the room, and when performing the action "stream video from smart camera", the robotic computing device may move to a further separate third location in the room. In some implementations, the user may specify a location in the room for performing the action, or type of action, by providing a descriptive natural language input (e.g., "Whenever you play videos, play them one meter in front of the north side of the couch."). Alternatively or additionally, the user may specify a location in the room for performing the action, or type of action, by moving the user to the desired location (e.g., "Whenever you tell me the news, just tell me the news here [user walks to the desired location and stands there]."). The robotic computing device can then, with prior permission from the user, capture image data (and / or other sensor data) and process the image data to determine the coordinates of where the user is standing and / or facing and / or inferred semantic labels therefor. Alternatively or additionally, the user can specify a location in the room to perform an action, or type of action, by interacting with a graphical user interface (GUI) of the robotic computing device or a separate device.For example, a first user can interact with the GUI to annotate a map of the first user's home to specify where some type of operation (e.g., audibly rendering the news) should be performed. In some implementations, the map can be a graphical representation of the home or other structure with semantic labels generated by processing data from one or more sensors of the robotic computing device. Alternatively or additionally, a second user can also interact with an instance of the GUI to specify where some other type of operation (e.g., enabling a video call) should be performed.

[0015] In various implementations, preferences regarding locations for the robotic computing device to perform certain actions and / or types of actions may be stored as user-specific preferences. For example, a first user may direct the robotic computing device to perform a music-type action at a first location in a living room at home, and a second user may direct the robotic computing device to perform a music-type action at a second location in the living room that is different from the first location. Thus, in response to the first user requesting that the robotic computing device "play music in the living room," the robotic computing device may verify that the first user is providing the request and move to the first location in the living room according to the first user's specified preferences.

[0016] For example, prior to the guest's arrival, the user can provide instructions for the robotic computing device to remain in the living room when performing any audio and / or video rendering. The user can provide these instructions via a verbal utterance such as "Play only music there, next to the couch, when the guest arrives." The user can optionally point to a location (e.g., when uttering "...there...") and the robotic computing device can capture image data where the user is pointing (with prior permission from the user) to determine the exact location where the user is pointing. For example, to determine the preferred "music" location where the user is pointing, geographic layout data characterizing the user's home can be compared to the estimated trajectory of the user's "pointing" finger. Alternatively or additionally, the user can interact with the GUI to annotate a map of the user's home to specify where the user would prefer certain types of actions to be performed. Then, prior to the guest's arrival, the robotic computing device can follow the user around the home (with or without playing music) as the user prepares for the guest's arrival. When a guest arrives, the robotic computing device can move to a location in the living room adjacent to the couch and either play music or wait for the user to explicitly request that the robotic computing device play music (e.g., "Play some music.").

[0017] In some implementations, this request from the user may be processed to generate preference data, which may be utilized when responding to subsequent requests from the user. For example, when the user has different guests over subsequent weekends, the user may respond to a request to play music by moving to a location adjacent to the couch in the user's living room. In some implementations, user verification may be performed by the robotic computing device before implementing some preferences for some operations. For example, voice verification and / or facial recognition may be performed in response to the user requesting to play music. When the requesting user does not correspond to a user with a particular location preference for playing music, the robotic computing device may select a location based on one or more heuristic processes and / or one or more trained machine learning models. For example, a user who did not provide an explicit preference for a location for some operations may have some preferences inferred based on a previous interaction between the user and the robotic computing device.

[0018] In some implementations, the location of the robotic computing device may be dynamic for some actions and / or for some user preferences. For example, a user may provide a request for the robotic computing device to issue an alarm after a certain duration (e.g., 10 minutes). In response, the robotic computing device may not initially follow the user, but may initialize a "follow" action after the duration before the alarm reaches a certain value. In some implementations, this certain value may be based on one or more features of the alarm request, the reason for the requested alarm, the estimated relative location of the user to the robotic computing device, and / or the context in which the user requested the alarm. In some implementations, a user may provide a request to the robotic computing device that may cause the robotic computing device to follow (e.g., follow) the user without the user explicitly requesting that the robotic computing device follow the user. This behavior may be learned over time through interactions with the robotic computing device and / or based on an explicit request from the user. For example, a user may explicitly request that the robotic computing device follow the user in some situations or stop following the user in other situations. Such instances can provide feedback data that can be utilized to train the robotic computing device to follow and / or not follow the user in some situations without the user explicitly providing a request for the robotic computing device to follow the user.

[0019] For example, a user may request that the robotic computing device play music, and in response, the robotic computing device may initialize a following behavior and a music rendering behavior. In some implementations, the following behavior may be initialized in some contexts where the user is determined not to have another audio device at hand and / or is not walking towards an audio device. For example, when a user is walking along a hallway and provides a verbal utterance such as "play some music," the robotic computing device may determine that the hallway does not have an audio device present. Alternatively or additionally, the robotic computing device may determine that the user is walking along the hallway in the direction of a room that does not have an audio device. In response to the verbal utterance, the robotic computing device may initialize music rendering and also initialize a following behavior so that the user can hear music while the user is walking along the hallway into the room. Instead, when a room is determined to have an audio device, the robotic computing device may initialize playing music while the user is in the hallway, but delegate playing music to an audio device in the room once the user enters the room. Thereafter, when the user is inside the room, the robotic computing device may optionally remain outside the room and cease following actions, at least according to the user's determined preferences.

[0020] In some implementations, the characteristics of the following behavior may depend on the action being performed by the robotic computing device and / or the type of action being performed, and / or the particular user providing the request for the robotic computing device to perform the action. For example, the robotic computing device may follow the user a distance "x" when performing an action of playing music, but follow the user at a different distance "y" when enabling a video or audio call. Alternatively or additionally, the robotic computing device may follow the user at a particular speed according to the preferences of the particular user and / or the action or type of action being performed. In some implementations, the characteristics of the following behavior may be based on where the robotic computing device is located and / or whether the robotic computing device is located in a room with a particular inferred semantic label. For example, the robotic computing device may perform a following behavior according to a certain distance and / or speed when in the "kitchen", but a different distance and a different speed when in the "garage".

[0021] The above description is provided as an overview of some implementations of the present disclosure, further descriptions of which and other implementations are described in more detail below.

[0022] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) for performing a method, such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more computer systems including one or more processors operable to execute stored instructions for performing a method, such as one or more of the methods described above and / or elsewhere herein.

[0023] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0024] [Figure 1A] FIG. 1 illustrates a diagram in which a user interacts with a robotic computing device that can infer, without an explicit request, whether the user wants to be directed to a device that is providing a notification. [Figure 1B] FIG. 1 illustrates a diagram in which a user interacts with a robotic computing device that can infer, without an explicit request, whether the user wants to be directed to a device that is providing a notification. [Figure 2A] FIG. 1 illustrates a diagram in which users interact with a robotic computing device that can communicate between and interact with users at learned preferred locations. [Figure 2B] FIG. 1 illustrates a diagram in which users interact with a robotic computing device that can communicate between and interact with users at learned preferred locations. [Figure 2C] FIG. 1 illustrates a diagram in which users interact with a robotic computing device that can communicate between and interact with users at learned preferred locations. [Diagram 3] FIG. 1 illustrates a system in which a robotic computing device can be operated to enable communication between users and proceed to a particular location to complete an operation without an explicit request from the user. [Figure 4] FIG. 1 illustrates a method for operating a robotic computing device to paraphrase user input and / or user responses that may be relayed by the robotic computing device from a first user to a second user. [Diagram 5] FIG. 1 illustrates a method for operating a robotic computing device to autonomously assign semantic labels to areas of a space or structure to help ensure that certain actions are performed in certain areas according to user preferences. [Figure 6] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] 1A and 1B show diagrams 100 and 120 in which a user 102 interacts with a robotic computing device 104 that can infer whether the user wants to be guided to a device providing a notification. Alternatively or additionally, the robotic computing device can move to a device and / or a specific location according to drive operating parameters selected based on one or more characteristics of the context in which the notification is being provided. For example, the robotic computing device 104 can determine that the user 102 has received a text message on a separate device, such as a cellular phone, when the cellular phone is operating in silent mode (e.g., vibrate only mode). Each of the cellular phone and the robotic computing device 104 can access a local area network, which may be wirelessly accessible to the robotic computing device 104 in the room 106 of the user 102. In some implementations, and with prior permission from the user 102, the robotic computing device 104 can determine whether the user 102 has acknowledged the text message and / or whether the user 102 is located in an area where the user 102 can detect receipt of a text message. When the robotic computing device 104 predicts that the user 102 has not acknowledged the text message and / or is not located in an area where a text message can be detected, the robotic computing device 104 can render an audible output 108, such as "You received a text message from Julian." Alternatively or additionally, the robotic computing device 104 can proactively render the audible output 108 when the cellular phone receives a text message.

[0026] In some implementations, the robotic computing device 104, with prior permission from the user 102, can generate a prediction regarding whether the user 102 is aware of where the cellular phone is located. The prediction can be based on, for example, a verbal utterance 110 indicating that the user 102 is not aware of the location of the cellular phone. The verbal utterance 110 can include content such as, "I don't even know where my phone is." Alternatively or additionally, the image data and / or audio data can be processed to determine whether the user 102, with prior permission from the user 102, knows where the cellular phone is located. For example, in response to the audible output 108, the user 102 can look around (e.g., to the left of the user 102 and to the right of the user 102) for the user's cellular phone, which can be an indication that the user 102 is not aware of where the user's cellular phone is located. Based on one or more of these contextual features, the robotic computing device 104 may determine that the user 102 is unaware of where the user's 102 cellular phone is located and provide another audible output 112, such as "If you like, I can show you."

[0027] In some implementations, the robotic computing device 104 can infer that the user 102 is interested in being guided to the user's cellular phone without the user 102 providing an input that explicitly guides the robotic computing device 104 to lead the user 102 to the cellular phone. For example, the contextual feature 114 can include the user 102 standing up from the user's 102 seat after another audible output from the robotic computing device 104. This feature 114 can be a positive indication that the user 102 is willing to be guided by the robotic computing device 104 to the user's cellular phone. For example, as shown in the diagram 120 of FIG. 1B, the robotic computing device 104 can exit the room 106 and enter another room 126 to facilitate guiding the user 102 to a device 124, such as the user's cellular phone. In this way, the user 102 does not necessarily have to engage in an explicit conversation with the user's 102 assistant device to achieve some benefit. This can conserve computational resources of the robot computing device 104 and / or other devices, since less processing and storage space can be consumed before initializing the performance of an operation.

[0028] 2A, 2B, and 2C show diagrams 200, 220, and 240 in which a user 202 interacts with a robotic computing device 204 that can communicate between and interact with users at learned preferred locations. For example, the user 202 can provide a verbal utterance 208, such as "Go see if Jimmy cleaned his room." The verbal utterance can be directed to the robotic computing device 204, which can be located in the room 206 with the user 202. In response to detecting the verbal utterance 208, the robotic computing device 204 can render an audible output 210, such as "Okay, I'll go to his room and check." In some implementations, the robotic computing device 204 can determine that the verbal utterance 208 embodies one or more requests for the robotic computing device 204 to perform one or more actions. The one or more actions can include determining a location for "Jimmy's" room and determining if the room has been "cleaned."

[0029] In some implementations, the robotic computing device 204 can determine the location of Jimmy's room by communicating with other computing devices located within a threshold distance of the robotic computing device 204 and / or connected to a common network with the robotic computing device 204. For example, in response to determining a requested action, the robotic computing device 204 can cause one or more devices in and outside the room 206 to render one or more different types of output (e.g., visual, audio, antenna, etc.) that can be detected by the robotic computing device 204. Based on these outputs, the robotic computing device 204 can determine that one or more of the devices correspond to "Jimmy's" room. Alternatively or additionally, the robotic computing device 204 can determine the relative distance of the robotic computing device 204 from one or more smart devices based on one or more signal metrics (e.g., signal quality) optionally detected within a time window or duration. In some implementations, a user-specified name for a particular device may be "Jimmy's speaker" and / or "Jimmy's smart light," thus providing evidence that the location of the particular device corresponds to Jimmy's room. Alternatively or additionally, one or more signal metrics associated with communication between a particular device and the robotic computing device 204 may indicate a relative distance and / or relative location of the particular device from the robotic computing device 204.

[0030] In some implementations, such a label for the device may be identified via a smart home graph, an assistant application, and / or any other application or module that may be associated with the user 202 that is accessible to the robot computing device 204. Alternatively or additionally, the robot computing device 204 may determine, with prior permission from one or more users, a particular room in which Jimmy was previously located and / or is currently located. For example, historical interaction data accessible to the robot computing device 204 may indicate that the majority of interactions between the robot computing device 204 and the other user (i.e., Jimmy) occurred in another room 226, as shown in FIG. 2B. Based on this determination and the absence of conflicting data (e.g., data indicating that the room 226 is not Jimmy's room), the robot computing device 204 may navigate to the room 226 to fulfill the request from the user 202.

[0031] When the robotic computing device 204 arrives at the other room 226, the robotic computing device 204 can gather data regarding the other room 226 using one or more interfaces of the robotic computing device 204 and / or interacting with one or more other devices in the other room 226. For example, the robotic computing device 204 can utilize one or more cameras to capture one or more images of the other room 226. The one or more images can be processed using one or more trained machine learning models to determine whether the other room 226 should be classified as "clean". In some implementations, the robotic computing device 204 can position itself in the other room 226 based on preferences of the user 202 and / or the other user 222. For example, a first location preference for the robotic computing device 204 in the other room 226 can correspond to a location where the robotic computing device 204 should be positioned to gather data. Alternatively or additionally, a second location preference for the robotic computing device 204 in the other room 226 may correspond to another location where the robotic computing device 204 should be placed to interact with the other user 222.

[0032] In some implementations, the robotic computing device 204 can infer location preferences based on the frequency with which the user engages with the robotic computing device 204 at some locations in a room. Alternatively or additionally, the robotic computing device 204 can determine the user's location preferences based on explicit instructions from the user. For example, the user can provide an explicit request for the robotic computing device 204 to enable visual output and / or video calling only at some preferred distances and / or only in certain parts of the room. Alternatively or additionally, the user can provide an explicit request for the robotic computing device 204 to enable visual output and / or video calling only at other distances and / or only in other parts of the room. These instructions can be received and generated into preference data, which can then be utilized by the robotic computing device 204 when interacting with one or more users.

[0033] According to the above example, the robotic computing device 204 can enter another room 226 and optionally render an output 228 to the other user 222 to facilitate fulfilling the request from the user 202. For example, the other output 228 can be a request for information that can assist the robotic computing device 204 in fulfilling the request from the user 202, such as "Did you clean up your room?" In response, the other user 222 (i.e., Jimmy) can provide a verbal utterance 230, such as "Yes, I'm working on it." This information can optionally be utilized in combination with other data generated by the robotic computing device 204 to fulfill the request from the user 202. For example, upon receiving this information and / or data, the robotic computing device 204 can navigate to the user 202, as shown in FIG. 2C.

[0034] In some implementations, the robotic computing device 204 can navigate to a particular location 246 in the room 206 based on the preferences of the user 202. For example, the user 202 can prefer that the robotic computing device 204 should provide its response at a particular location 246 when the robotic computing device 204 is entering the room 206 to provide an audible response. Alternatively or additionally, the user 202 can have a history of guiding the robotic computing device 204 to the particular location 246 by providing a fetch command (e.g., "come here") in combination with a pointing gesture (e.g., a pointing movement toward the particular location 246). Based on these historical interactions, the robotic computing device 204 can determine that the user 202 prefers to hear a response from the robotic computing device 204 at a particular location 246 in the room 206. In some implementations, the preferred distance and / or preferred location may vary from room to room and / or for different actions and / or different types of output. For example, the user 202 may prefer that the robot computing device 204 provide “news” from the end of the bed when the robot computing device 204 is in the user's 202 bedroom, but may allow audio telephone calls from the side of the bed when the robot computing device 204 is in the user's 202 bedroom.

[0035] When the robotic computing device 204 arrives at a particular location 246 to fulfill the request, the robotic computing device 204 can render an audible output 242, such as "The room looks clean, Jimmy says he's working on it." This audible output 242 can be generated using one or more language models (e.g., recurrent neural networks, transformer network models, etc.). In some implementations, the audible output 242 can include content that is different from that provided by the other user 222, but still conveys a similar conclusion. Alternatively or additionally, the audible output 242 can include content based on responses from the user 222 and data generated using one or more interfaces (e.g., cameras) of the robotic computing device 204. In this manner, the rendered content from the robotic computing device 204 can embody natural language content that characterizes the data and information obtained to facilitate fulfilling the request from the user 202.

[0036] FIG. 3 illustrates a system 300 in which a robotic computing device can be operated to enable communication between users and proceed to a specific location to complete an operation without an explicit request from the user. The system 300 can include a computing device 302, which can be a robotic computing device, that includes one or more applications to enable the robotic computing device to interface with a user. For example, the robotic computing device can include an automated assistant 304. The automated assistant 304 can operate as part of an assistant application provided in one or more other computing devices and / or a server device. A user can interact with the automated assistant 304 via an assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 304 by providing linguistic, textual, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to initialize one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automated assistant 304 may be initialized based on processing the context data 336 using one or more trained machine learning models.

[0037] The context data 336 can characterize one or more features of an environment to which the automated assistant 304 is accessible and / or one or more features of a user predicted to be intended to interact with the automated assistant 304. The computing device 302 can include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to enable a user to control the application 334 of the computing device 302 via a touch interface. In some implementations, the computing device 302 can provide an audible user interface output without a display device and thereby providing a graphical user interface output. Additionally, the computing device 302 can provide a user interface, such as a microphone, for receiving spoken natural language input from a user. In some implementations, the computing device 302 can include a touch interface and may lack a camera, but may optionally include one or more other sensors.

[0038] The computing device 302 and / or other third party client devices may be in communication with the server device over a network, such as the Internet. Additionally, the computing device 302 and any other computing devices may be in communication with each other over a local area network (LAN), such as a Wi-Fi network. The computing device 302 may offload computational tasks to the server device to conserve computational resources at the computing device 302. For example, the server device may host the automated assistant 304 and / or the computing device 302 may send inputs received at one or more assistant interfaces 320 to the server device. However, in some implementations, the automated assistant 304 may be hosted at the computing device 302 and various processes that may be associated with automated assistant operations may be implemented at the computing device 302.

[0039] In various implementations, all or less than all aspects of the automated assistant 304 may be implemented on the computing device 302. In some of those implementations, aspects of the automated assistant 304 may be implemented via the computing device 302 and interface with a server device that may implement other aspects of the automated assistant 304. The server device may optionally service multiple users and their associated auxiliary applications via multiple threads. In implementations in which all or less than all aspects of the automated assistant 304 are implemented via the computing device 302, the automated assistant 304 may be an application that is separate from (e.g., installed "on") the operating system of the computing device 302, or may alternatively be implemented directly by (e.g., considered an application of, but integral with, the operating system).

[0040] In some implementations, the automated assistant 304 can include an input processing engine 306, which can employ multiple different modules to process input and / or output for the computing device 302 and / or the server device. For example, the input processing engine 306 can include a speech processing engine 308, which can process audio data received at the assistant interface 320 to identify text embodied in the audio data. The audio data can be transmitted from the computing device 302 to the server device, for example, to conserve computational resources at the computing device 302. Additionally or alternatively, the audio data can be processed exclusively at the computing device 302.

[0041] The process for converting audio data to text can include speech recognition algorithms that can employ neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data can be parsed by the data parsing engine 310 and made available to the automated assistant 304 as text data that can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 310 can be provided to the parameter engine 312 to determine whether the user has provided input corresponding to a particular intent, action, and / or routine that can be performed by the automated assistant 304 and / or an application or agent that can be accessed via the automated assistant 304. For example, the assistant data 338 can be stored at the server device and / or the computing device 302 and can include data defining one or more actions that can be performed by the automated assistant 304, as well as parameters necessary to perform those actions. The parameter engine 312 can generate one or more parameters for the intent, action, and / or slot value and provide the one or more parameters to the output generation engine 314. The output generation engine 314 can use the one or more parameters to communicate with the assistant interface 320 to provide output to the user and / or to communicate with one or more applications 334 to provide output to the one or more applications 334.

[0042] In some implementations, the automated assistant 304 may be an application that may be installed "on top of" the operating system of the computing device 302 and / or may itself form part of (or the entirety of) the operating system of the computing device 302. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be implemented using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally in the computing device 302. The on-device speech recognition generates recognized text for the verbal utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) may be implemented using an on-device NLU module that processes the recognized text generated using the on-device speech recognition and, optionally, contextual data to generate NLU data.

[0043] The NLU data may include an intent corresponding to the verbal utterance and, optionally, parameters for that intent (e.g., slot values). On-device fulfillment may be implemented using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and, optionally, other local data, to determine actions to take to analyze the intent of the verbal utterance (and, optionally, parameters for that intent). This may include determining local and / or remote responses (e.g., answers) to the verbal utterance, interactions with locally installed applications to perform based on the verbal utterance, commands to send to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the verbal utterance, and / or other analytical actions to perform based on the verbal utterance. The on-device fulfillment may then initiate local and / or remote implementation / execution of the actions determined to analyze the verbal utterance.

[0044] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be utilized at least selectively. For example, recognized text may be at least selectively sent to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, recognized text may be sent for remote fulfillment, optionally in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the latency reduction they provide when analyzing spoken speech (by not requiring a client-server round trip to analyze spoken speech). Furthermore, on-device functionality may be the only functionality available in situations where there is no or limited network connectivity.

[0045] In some implementations, the computing device 302 may include one or more applications 334, which may be provided by a third party entity different from the entity that provided the computing device 302 and / or the automated assistant 304. The application state engine of the automated assistant 304 and / or the computing device 302 may access the application data 330 to determine one or more actions that may be performed by the one or more applications 334, as well as the state of each application of the one or more applications 334 and / or the state of each device associated with the computing device 302. The device state engine of the automated assistant 304 and / or the computing device 302 may access the device data 332 to determine one or more actions that may be performed by the computing device 302 and / or one or more devices associated with the computing device 302. Additionally, the application data 330 and / or any other data (e.g., device data 332) may be accessed by the automated assistant 304 to generate context data 336, which may characterize the context in which a particular application 334 and / or device is executing and / or the context in which a particular user is accessing the computing device 302, the application 334, and / or any other device or module.

[0046] While one or more applications 334 are executing on the computing device 302, the device data 332 can characterize a current operational state of each application 334 executing on the computing device 302. Additionally, the application data 330 can characterize one or more features of the executing application 334, such as the content of one or more graphical user interfaces being rendered in the orientation of the one or more applications 334. Alternatively or additionally, the application data 330 can characterize action schemas, which can be updated by the respective applications and / or by the automated assistant 304 based on the current operational status of the respective applications. Alternatively or additionally, the one or more action schemas for the one or more applications 334 can remain static but can be accessed by the application state engine to determine suitable actions to be initiated via the automated assistant 304.

[0047] The computing device 302 can further include an assistant invocation engine 322, which can use one or more trained machine learning models to process the application data 330, the device data 332, the context data 336, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 can process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 304, or the data can be considered to indicate an intent by the user to invoke an automated assistant instead of requiring the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models can be trained using training data examples based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating conditions. The training data examples can be generated to capture training data that characterize contexts in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant. When one or more trained machine learning models are trained according to these examples of training data, the assistant invocation engine 322 can cause the automated assistant 304 to detect or limit detection of spoken invocation phrases from the user based on contextual and / or environmental features.

[0048] In some implementations, the assistant invocation engine 322 can process data generated using one or more assistant interfaces 320 to determine whether the user has expressed a willingness to benefit from the automated assistant 304 and / or the robotic computing device's actions. For example, data captured via one or more assistant interfaces 320 can be processed by the assistant invocation engine 322 to determine whether the user's verbal utterances, non-verbal gestures, disfluencies, and / or other movements can be considered as an invocation to the robotic computing device. Alternatively or additionally, the data can be processed to determine a context in which the user provided such input and / or movements. Based on the determined context and / or expressions of the user, the robotic computing device can determine whether the user is willing to allow the robotic computing device to perform a particular action. The action can be, for example, to provide information to the user and / or to guide the user to a particular location despite the user not providing an explicit request for the robotic computing device to do so.

[0049] In some implementations, the system 300 can include a driving parameter engine 316 that can determine one or more parameters for moving the robotic computing device in some context and / or based on some data. For example, data that provides a basis for a particular operation to be performed by the robotic computing device can be processed by the driving parameter engine 316 to determine how the robotic computing device should be moved when fulfilling the particular operation. For example, the application data 330, the device data 332, and / or the contextual data 336 can be processed to determine whether there is an urgency and / or time limit associated with a particular request from a user. In some implementations, this can be determined using one or more heuristic processes and / or one or more trained machine learning models. Alternatively or additionally, data associated with the request can be processed by the driving parameter engine 316 to generate an embedding that can be mapped to a latent space, and a distance in the latent space to a point and / or area can indicate whether the request is urgent or not. This processing by the drive parameter engine 316 may be utilized to determine drive parameters, such as speed, acceleration, travel time, power limits, and / or any other parameters that may be associated with driving a robotic device.

[0050] For example, the application data 330 may indicate that the user has requested to be guided to the location of a device that has provided an urgent notification. The application data 330 may be processed by the driving parameter engine 316 to determine that the application is exhibiting a particular status that is predicted to be particularly urgent relative to other notifications and / or other application statuses. In some implementations, this determination may be based on whether the application status has a time quality (e.g., the phone is ringing, an important contact is calling, and therefore there is only a certain amount of time to answer the phone call). Based on this determination, the driving parameter engine 316 may identify speed parameters for controlling one or more motors of the robotic computing device when fulfilling the request to be guided to the location of the device. In some implementations, the system 300 may include a layout detection engine 318 that may enable the robotic computing device to determine the relative location and / or other features of a room within a space or structure in which the robotic computing device is located. For example, the layout detection engine 318 may be utilized by a robotic computing device in response to input from a user to attempt to identify devices, users, room locations, and / or characteristics of a space and / or structure.

[0051] In some implementations, the layout detection engine 318 can cause other devices to provide output to assist in determining the current location of the robotic computing device relative to other portions of the space in which the robotic computing device is located. For example, the layout detection engine 318 can process the application data 330 to determine that a device in a particular room has a user-defined label (e.g., “laundry room speaker”) that can indicate a name for the particular room (e.g., “laundry room”). When the robotic computing device is directed to enter a particular room, the layout detection engine 318 can cause the device to provide an output (e.g., illuminate a light or display, render audio, transmit an antenna signal, etc.). The layout detection engine 318 can identify one or more different characteristics (e.g., signal metrics) of the output (e.g., signal quality, amplitude, magnitude, audio frequency, light frequency, etc.) from one or more different devices received within a time window or duration to determine the relative location of the particular room compared to the current location of the robotic computing device.

[0052] For example, the robotic computing device may determine whether the robotic computing device is co-located with one or more devices having a semantic label associated with a room specified by or inferred from a user request. Based on this determination, the layout detection engine 318 may determine how to move the robotic computing device from its current location to the desired room location. In some implementations, locations within a structure (e.g., a home or business) may be mapped using public knowledge graph analysis and / or personal knowledge graph analysis. The public knowledge graph may be generated based on prior interactions between one or more users, people, and one or more other applications. Alternatively or additionally, the private knowledge graph may be generated based on prior interactions between a user and one or more applications (e.g., an assistant application and / or an IoT application).

[0053] In some implementations, the system 300 can include a location preference engine 326 that can determine a preferred location of the robotic computing device for performing some operations, interacting with a particular user, and / or possibly positioning the robotic computing device. The location preference engine 326 can process data using one or more heuristic processes and / or trained machine learning models to determine whether the current location of the robotic computing device is suitable for fulfilling a request from a user. In some implementations, the preferred location can be requested directly by a user and / or inferred from one or more prior interactions with one or more users. For example, a user can explicitly or indirectly request that the robotic computing device perform some operations (e.g., enabling a phone call or a video call) at a first location in a room and other operations (e.g., playing music) at a second location in the room. These preferred locations can be different for different users and / or for different rooms.

[0054] For example, when a first user receives a voice call through the robot computing device, the robot computing device may navigate to a first location in the room. However, when a second user receives a voice call, the robot computing device may navigate to a second location in the room so that the second user can receive the voice call through the robot computing device. In some implementations, the location preference engine 326 can determine a dynamic location for the robot computing device. For example, a first user may prefer that the robot computing device follow the first user by a distance x (where x is any distance value) when playing music. However, a second user may prefer that the robot computing device follow the second user by a different distance y (where y is any distance value) when rendering news reports and conference calls.

[0055] FIG. 4 illustrates a method 400 for operating a robotic computing device to paraphrase user input and / or user responses that may be relayed by the robotic computing device from a first user to a second user. The robotic computing device may relay messages between locations at different rates according to the type of input explicitly provided to the robotic computing device and / or inferred by the robotic computing device. The method 400 may be implemented by one or more computing devices, applications, and / or any other devices or modules that may be associated with an automated assistant. The method 400 may include an operation 402 for determining whether spoken speech and / or other user input has been provided to the robotic computing device. The robotic computing device may be a computing device that can move to various locations within a user's home and connect to one or more different networks to transmit and receive data. When spoken speech or other user input is detected at the robotic computing device, the method 400 may proceed to an operation 404. In other cases, the robotic computing device may continue to determine whether a user has provided input.

[0056] Operation 404 may include determining whether a user has directed the robotic computing device to perform an action that may require progression. An action that may require progression may include relaying a message to another user, who may not be located within a distance that allows the robotic computing device to emit audible and / or visible audio. For example, a verbal utterance from a first user may include a request for the robotic computing device to provide an audible message to a second user, who may be located in a different room than the first user. The verbal utterance from the first user may be, for example, "tell Phoenix that I need to leave because Sheri just pulled up in front of my house." This verbal utterance may embody a request for the robotic computing device to perform at least an action of moving to a location of the second user (e.g., "Phoenix") and another action of providing a message to the second user (e.g., "Mark is leaving now because Sheri just pulled up in front of my house."). When it is determined that a request for the robotic computing device to proceed to another location has been received, method 400 may proceed from operation 404 to operation 406. Otherwise, the method 400 may proceed from operation 404 to operation 410.

[0057] Operation 406 may include determining whether the request from the user has an increased relative importance. The increased relative importance may refer to the priority or severity of the request relative to other requests submitted by the user and / or one or more other users. For example, in some implementations, vocal characteristics of the verbal utterance (e.g., cadence, words per second, etc.) may be identified and processed to determine whether the verbal utterance is intended to have an increased relative importance. In some implementations, an action request corresponding to a time event may indicate the relative importance of the request. For example, a request associated with a situation that has at least a certain probability of not changing within a threshold duration may be considered to be of non-increased relative importance. However, a different request associated with a situation that has at least a certain probability of changing within a threshold duration may be considered to be of increased relative importance.

[0058] When the verbal utterance "Tell Phoenix I need to leave because Sheri just pulled up in front of my house" is received by the robotic computing device, the verbal utterance may be determined to embody a request of increased relative importance. This determination may be based at least in part on one or more heuristic processes and / or one or more trained machine learning models. For example, the robotic computing device and / or other computing devices may determine that the request embodied in the verbal utterance is provided in a tone that characterizes a time event (e.g., that the user needs to leave) and / or indicates urgency (e.g., a tone of the user's voice such as "...I...need...to leave..."). Based on historical interactions between the user and the robotic computing device and / or one or more other applications, the robotic computing device may determine that this tone is distinct from the user's general tone and indicates a sense of urgency.

[0059] When it is determined that the user has provided a request with increased relative importance, the method 400 may proceed to operation 408. Otherwise, the method 400 may proceed to operation 412. Operation 408 may include causing the robotic computing device to proceed according to a first driving parameter. For example, the first driving parameter may include, but is not limited to, an amount of acceleration, an amount of velocity, and / or an amount of energy utilized to fulfill the request from the user. When the first driving parameter is selected, the amount of energy and / or velocity consumed by the robotic computing device may be selected to be increased relative to other values ​​selected for requests of non-increased relative importance. For example, operation 412 may include causing the robotic computing device to proceed according to a second driving parameter, which may be different from the first driving parameter. The second driving parameter may characterize a speed setting and / or an acceleration setting that may be lower than the settings corresponding to the first driving parameter. When the first actuation parameter and / or the second actuation parameter are employed to control the navigation movement of the robotic computing device, the method 400 may proceed from operation 408 or operation 412 to operation 410.

[0060] Operation 410 may include causing the robotic computing device to perform a requested action. Operation 410 may be performed when the user arrives at the destination and / or en route to the corresponding destination. The requested action may be, for example, rendering an output to another user, identifying the location of another computing device, searching for information available at the destination, and / or any other action that the computing device may be requested to perform. To further the above example, the robotic computing device may render an audible output to a second user (i.e., Phoenix), such as "Sheri just pulled up in front of the house, Mark is about to leave now." In some implementations, the output from the robotic computing device may differ from a verbal utterance from the first user, but may convey information provided by the first user. Method 400 may proceed from operation 410 to operation 414, which determines whether the first user, the second user, or another user is expected to provide an additional request to the robotic computing device.

[0061] The robotic computing device may determine whether the user is predicted to provide an additional request to the robotic computing device based on one or more direct and / or indirect gestures performed by the user. For example, the robotic computing device may determine that, with prior permission from the user, the user has gazed at the device, directed the user's voice to the device, moved toward the device, and / or performed a gesture indicating an interest in providing an additional request to the robotic computing device. To further the above example, operation 414 may be performed with respect to the first user and / or the second user. For example, when the robotic computing device reaches the second user and provides an audible output, the robotic computing device may determine, with prior permission from the second user, whether the second user is predicted to provide an input to the robotic computing device. Alternatively or additionally, operation 414 may be performed when the robotic computing device returns to the first user after providing an audible output to the second user. When the user is predicted to provide an additional input to the robotic computing device, the method 400 may proceed to operation 416. Otherwise, the method 400 may return to operation 402 to determine whether an input has been provided to the robotic computing device.

[0062] Operation 416 may include having the robot computing device follow or possibly move with the user according to the type of predicted input. For example, when the robot computing device predicts that the second user will provide an input in response to the audible output, the robot computing device may move with the second user. In some implementations, the robot computing device may move with the second user for a duration that corresponds to the type of predicted input and / or the reliability score for the input prediction. For example, when the robot computing device predicts an upcoming input with a first reliability score, the robot computing device may follow the user for a first duration. However, when the robot computing device predicts another upcoming input with a second reliability score that is greater than the first reliability score, the robot computing device may follow the user for a second duration that is greater than the first duration. The method 400 may then return to operation 402 for determining whether the user has provided an input to the robot computing device.

[0063] FIG. 5 illustrates a method 500 for operating a robotic computing device to autonomously assign semantic labels to one or more areas within a space or structure occupied by one or more users and / or the robotic computing device. The semantic labels may be assigned to several locations to facilitate ensuring that the robotic computing device performs several actions in several areas according to user preferences. The method 500 may be performed by any computing device, application, and / or any apparatus or module capable of interacting with the robotic computing device. The method 500 may include an operation 502 of determining whether an area of ​​a space occupied by the robotic computing device is not associated with a semantic label. For example, the robotic computing device may operate in a home occupied by one or more users (e.g., a mother user and a daughter user), and the home may include various different areas (e.g., different parts of a living room, a kitchen, an office, a bedroom, etc.). The operation 502 may be initialized when the robotic computing device is located in the kitchen of the home prior to or in response to a user providing input to the robotic computing device. In this manner, each semantic label can assist the robotic computing device in performing certain actions that can be more effectively implemented with information related to locations at and / or near the robotic computing device. For example, when the robotic computing device is guided by a first user to communicate with a second user in a home, the robotic computing device, with prior authorization from the second user, can access semantic labels for different parts of the home to predict the location of the second user and, optionally, thereafter, the location of the first user.

[0064] When the robot computing device determines that the space occupied by or near the robot computing device is not associated with a semantic label, the method 500 may proceed from operation 502 to operation 504. In other cases, the robot computing device may continue to determine whether the space at or near the robot computing device is associated with a semantic label. Operation 504 may include causing a first set of smart devices to emit one or more first outputs during a first time window. The first set of smart devices may be selected to emit one or more first outputs based on the first set of smart devices being assigned a label with related content (e.g., "kitchen counter speaker", "fridge smart display", etc.). Alternatively or additionally, the first set of smart devices may include several devices located at one or more specific locations on a map that is generated over time using data from one or more sensors of the robot computing device. For example, as the robot computing device moves to different locations in the home of a first user, the robot computing device may capture data (with prior permission from the user in the home) that characterizes several locations of several devices in the home. This data can be utilized to correlate these devices with locations on mapping generated by the robotic computing device and / or one or more other devices.

[0065] Method 500 may proceed from operation 504 to operation 506, which may include causing the second set of smart devices to emit one or more second outputs during a second time window. In some implementations, the one or more first outputs and the one or more second outputs may include audio and / or visual outputs. For example, the one or more first outputs may include one or more characteristics that are the same or different from one or more other characteristics of the one or more second outputs. For example, the one or more first outputs may embody one or more frequencies (e.g., audio and / or visual) that are different from one or more other frequencies (e.g., audio and / or visual) embodied by the one or more second outputs. In some implementations, the second set of smart devices may be selected to emit one or more second outputs based on the robotic computing device determining that the second set of smart devices is located in a different room and / or area of ​​the space than the first set of smart devices.

[0066] From operation 506, the method 500 may proceed to operation 508, which may include processing sensor data generated by one or more sensors of the robotic computing device. In some implementations, the first and second time windows may be at least partially overlapping or non-overlapping in duration. For example, sensor data captured during the first time window may have the same or different timestamp as other sensor data captured during the second time window. In some implementations, the sensor data captured by the robotic computing device and / or one or more other computing devices may be processed to determine whether a first set of smart devices and / or a second set of smart devices are co-located with the robotic computing device. For example, a magnitude of a detected output during the first time window may be compared to a magnitude of another detected output during the second time window. This comparison may indicate whether the first set of smart devices or the second set of smart devices are co-located at or near an area occupied by the robotic computing device. Alternatively or additionally, one or more characteristics of the detected output may be determined and compared to the first set of characteristics and / or the second set of characteristics. When a first set of characteristics is detected as embodied in the output detected by the robotic computing device, the robotic computing device can determine that a first set of smart devices are co-located at or near the robotic computing device. Alternatively or additionally, when a second set of characteristics is detected as embodied in the output detected by the robotic computing device, and optionally when the second set of characteristics is absent, the robotic computing device can determine that a second set of smart devices are co-located at or near the robotic computing device.

[0067] In some implementations, the detected characteristics may include a magnitude of the output, a frequency of the output, a change in the magnitude of the output (e.g., compared to a set magnitude), a change or modulation in the frequency of the output (e.g., compared to a set frequency), and / or any other characteristic that may be associated with the rendered output. In some implementations, the audio and / or visual output may be rendered at one or more frequencies that may not be visually detectable and / or audibly detectable by a natural human and / or a human unaided by non-inherent features. In some implementations, the output rendered by a set of devices may include a combination of one or more different outputs (e.g., output rendered by different interface modalities) that may be distinguished from other outputs rendered by another set of devices.

[0068] When the robotic computing device is determined to be co-located with a particular set of smart devices (e.g., the first set and / or the second set), the method 500 may proceed from operation 510 to operation 512. Operation 512 may include generating a semantic label for an area in the space currently occupied by the robotic computing device. In some implementations, the set of smart devices may include, but is not limited to, smart lights, smart televisions, smart thermostats, smart speakers, and / or any other devices that may be associated with a user and controlled via a separate device and / or application. In some implementations, the semantic label may be based on an existing descriptor of a room in the space or structure, which descriptor was previously assigned to the detected set of smart devices. For example, the detected set of smart devices may be assigned a descriptor (e.g., “Sam’s Room [device type]”) by a user in response to an explicit user input to an automated assistant and / or other device or application. Alternatively or additionally, the detected set of smart devices may be assigned a descriptor, which may be generated based on processing the data using one or more heuristic processes and / or one or more machine learning models. When an area occupied by a robotic computing device is assigned a semantic label, the semantic label may be stored in association with the location on a generated map that may be utilized by the robotic computing device to navigate between locations within a space or structure.

[0069] When a user provides a verbal input that is, for example, synonymous with a semantic label, the robotic computing device can correlate the verbal input with the area to which the semantic label is assigned. In this way, the robotic computing device can utilize existing data and / or other information to map the user's home without the user having to explicitly identify the area in the home to the robotic computing device. This allows the robotic computing device to fulfill some requests where less information is explicitly provided by the user, thereby conserving the computational resources of the robotic computing device. For example, when a semantic label (e.g., "Sheri's room") is assigned to an area of ​​the map, the robotic computing device can navigate to that area when, in some instances, the semantic label (or synonymous terms and / or portions of the semantic label) is identified in the natural language input provided to the robotic computing device (e.g., "Go tell Sheri that dinner is ready") When the robotic computing device determines that no set of smart devices is co-located with the robotic computing device, method 500 can proceed from operation 510 to operation 514. Operation 514 may include causing the robotic computing device to move to a different area of ​​a space occupied by a user and / or the robotic computing device (e.g., a home) and optionally perform operation 502.

[0070] 6 is a block diagram 600 of an exemplary computer system 610. The computer system 610 generally includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including memory 626 and a file storage subsystem 626, user interface output devices 620, user interface input devices 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computer system 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0071] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or onto a communications network.

[0072] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 610 to a user or to another machine or computer system.

[0073] The storage subsystem 624 stores programming and data configurations that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the method 400, the method 500, and / or for implementing one or more of the system 300, the robotic computing device 104, the robotic computing device 204, the automated assistant, and / or any other applications, devices, apparatus, and / or modules described herein.

[0074] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 626 used in the storage subsystem 624 can include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files and can include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations can be stored by the file storage subsystem 626, in the storage subsystem 624, or in other machines accessible by the processor 614.

[0075] Bus subsystem 612 provides a mechanism for allowing the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0076] The computer system 610 can be of different types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 shown in Figure 6 is intended only as a specific example to illustrate some implementations. Numerous other configurations of the computer system 610 are possible, having more or fewer components than the computer system shown in Figure 6.

[0077] In situations where the systems described herein may collect or utilize personal information about users (or, as is sometimes referred to herein as "participants"), the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how to receive content from a content server that may be more relevant to the user. Also, certain data may be handled in one or more ways before being stored or used such that personally identifiable information is removed. For example, the user's identifying information may be handled such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized, in which case geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be determined. Thus, the user may have control over how information is collected and / or used about the user.

[0078] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications are deemed to be within the scope of the implementations described herein. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, it should be understood that the above implementations are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be practiced other than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

[0079] In some implementations, a method implemented by one or more processors is provided, including determining, by the robotic computing device, that a user has uttered a verbal utterance indicating that the user is unsure of the location of the particular computing device. The verbal utterance does not embody an explicit request for the robotic computing device to identify the location of the particular computing device. The method may further include causing an output interface of the robotic computing device to provide an indication to the user that the robotic computing device is capable of determining the location of the particular computing device. The method may further include processing input data from one or more input interfaces of the robotic computing device to facilitate determining, by the robotic computing device, whether the user is willing to allow the robotic computing device to guide the user toward the location of the particular computing device. The method may further include, when the robotic computing device determines that the user is willing to allow the robotic computing device to guide the user toward the location of the particular computing device, causing the robotic computing device to communicate with the particular computing device to facilitate estimating a relative location of the particular computing device relative to the robotic computing device, and moving the robotic computing device toward the relative location of the particular computing device.

[0080] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0081] In some implementations, processing the input data to facilitate determining whether the user is willing to allow the robotic computing device to guide the user toward the location of the particular computing device includes processing image data indicative of the user's movement toward the robotic computing device. In some implementations, the input data is free of audio data characterizing an explicit request from the user for the robotic computing device to determine the relative location of the particular computing device. In some implementations, causing the robotic computing device to move toward the relative location of the particular computing device includes causing the robotic computing device to move toward the relative location of the particular computing device at a speed selected based on a status of an application accessible via the particular computing device. In some of those implementations, the application includes a voice call application, and the status of the application indicates that the user has missed a call from a particular contact. In some implementations, causing the robotic computing device to communicate with the particular computing device to facilitate estimating the relative location of the particular computing device relative to the robotic computing device includes determining a signal metric based on communication between the robotic computing device and the particular computing device, the signal metric indicating a relative distance of the particular computing device from the robotic computing device. In some of these implementations, the signal metric includes the audio amplitude of the audio output rendered by a particular computing device.

[0082] In some implementations, a method implemented by one or more processors is provided and includes receiving, by the robotic computing device, a verbal utterance from a first user located in a space with the robotic computing device and a second user. The method may further include determining, based on the verbal utterance, that the first user has directed the robotic computing device to communicate with the second user. The second user is located at a second user location different from the first user location of the first user. The method may further include, in response to the verbal utterance, causing the robotic computing device to move to the second user location and render an output for the second user. The output embodies a natural language question based on the verbal utterance from the first user. The method may further include receiving, by the robotic computing device, a responsive input from the second user. The responsive input embodies natural language content responsive to the natural language question embodied in the output from the robotic computing device. The method may further include, after the robotic computing device provides the output for the second user, causing the robotic computing device to move to the first user location and render another output for the first user, the another output characterizing the responsive input from the second user and embodying other natural language content different from the natural language content embodied in the responsive input from the second user.

[0083] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0084] In some implementations, causing the robot computing device to move to the second user location includes determining a location preference associated with the second user. In some versions of those implementations, causing the robot computing device to move to the second user location includes moving the robot computing device to a specific location corresponding to a preferred location indicated by the location preference. The location preference can indicate a preferred location for the robot computing device when the robot computing device communicates with the second user. In some of those versions, the preferred location indicates a preferred distance of the robot computing device from the second user, and the specific location is at least a preferred distance away from the second location of the second user. In some implementations, causing the robot computing device to move to the first user location includes determining a location preference associated with the first user and moving the robot computing device to a specific location corresponding to a preferred location indicated by the location preference. The location preference can indicate a preferred location for the robot computing device when the robot computing device renders a particular type of output for the first user. For example, a particular type of output includes an audible output provided through an audio output interface of the robotic computing device, or a visual output provided through a display interface of the robotic computing device. For example, a particular type of output may be an audible output with content that characterizes a message from another user.

[0085] In some implementations, a method implemented by one or more processors is provided and includes determining, at a robotic computing device, that a user has requested that the robotic computing device perform an operation in a particular room located in a space including a plurality of different rooms. The method may further include causing, by the robotic computing device, one or more devices in one or more of the plurality of different rooms to provide one or more respective outputs detectable by the robotic computing device. The method may further include determining whether a current location of the robotic computing device corresponds to the particular room based on the one or more respective outputs. When the current location of the robotic computing device does not correspond to the particular room, the method may further include causing the robotic computing device to move to the particular room based on the current location of the robotic computing device not corresponding to the particular room, and causing the robotic computing device to perform the operation when the robotic computing device is located in the particular room.

[0086] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0087] In some implementations, causing the robot computing device to move to a particular room includes determining that a particular part of a particular room is preferred by a user for performing a particular type of operation corresponding to the operation, and causing the robot computing device to move to the particular part of the particular room. In some implementations, causing the robot computing device to move to a particular room includes determining that a particular part of a particular room is preferred by a user for performing a particular type of operation not corresponding to the operation, and causing the robot computing device to move to a different part of the particular room. In some implementations, the method may further include, when a current location of the robot computing device corresponds to a particular room, causing the robot computing device to identify a part of the current room in which the robot computing device is located that is a preferred part for performing the operation. In some of those implementations, causing the robot computing device to identify a part of the current room that is a preferred part of the room for performing the operation includes determining, based on the user requesting the operation, that the user has previously requested the robot computing device to perform a particular type of operation corresponding to the operation in the preferred part of the room. In some implementations, the method may further include, when the current location of the robotic computing device does not correspond to a particular room, causing the robotic computing device to render an output requesting a user to confirm that the current location of the robotic computing device is approved for performing the above-mentioned action before the robotic computing device performs the above-mentioned action.In some implementations, the method may further include, when a current location of the robotic computing device corresponds to a particular room, causing the robotic computing device to identify a relative distance to follow a user within the current room in which the robotic computing device is located when performing the above-mentioned operation while the user moves to another part of the particular room.

[0088] In some implementations, a method implemented by one or more processors is provided and includes determining that the mobile robotic computing device is currently located within a particular area of ​​a structure (e.g., within a room of a home) based on a map generated by the mobile robotic computing device and based at least in part on sensor observations of the mobile robotic computing device. The method may further include causing a first subset of the smart devices to each emit one or more first outputs and causing a second subset of the smart devices to each emit one or more second outputs while the mobile robotic computing device is located within the particular area. The one or more first outputs are audible and / or visual, and the one or more first outputs are emitted during a first time window and / or with one or more first characteristics in response to the first subset of the smart devices each being assigned a first semantic label in the home graph. The one or more second outputs are audible and / or visual, and the one or more second outputs are emitted during a second time window and / or with one or more second characteristics in response to the second subset of smart devices each being assigned a second semantic label in the home graph. The method may further include acquiring sensor data during the emission of the one or more first outputs and the one or more second outputs. The sensor data is generated by one or more sensors of the mobile robot computing device. The method may further include determining that the first subset of smart devices are co-located with the robot in a particular area based on an analysis of the sensor data.Determining that the first subset of smart devices are co-located with the robot in a particular area is based on (1) the analysis indicating a detected output during a first time window and / or matching one or more first characteristics, and / or (2) a magnitude of the detected output during the first time window and / or matching one or more first characteristics. The method may further include assigning an inferred semantic label to the particular area in response to determining that the first subset of smart devices are co-located with the robot in a given room. The inferred semantic label may be the same as or derived from the first semantic label assigned to the first subset of smart devices in the home graph.

[0089] These and other implementations of the techniques disclosed herein can optionally include one or more of the following features.

[0090] In some implementations, the one or more first outputs are emitted during a first time window, and the one or more second outputs are emitted during a second time window. In some versions of those implementations, determining that the first subset of smart devices are co-located with the robot in the particular area includes determining that the detected output occurs during the first time window, and determining that there is no detected output occurring during the second time window. In some additional or alternative versions of those implementations, determining that the first subset of smart devices are co-located with the robot in the particular area includes determining that a magnitude of the detected output occurring during the first time window is greater than an additional magnitude of an additional detected output occurring during the second time window. In some implementations, the one or more first outputs have a first characteristic, and the one or more second outputs have a second characteristic. In some versions of those implementations, determining that the first subset of smart devices are co-located with the robot in the particular area includes determining that the detected output matches the first characteristic and determining that there is no detected output that matches the second characteristic. In some of those versions, the one or more first characteristics include a first frequency and the one or more second characteristics comprise a second frequency. For example, the first output can include a visual output, the first frequency can be a first visual frequency, the second output can include a second visual output, and the second frequency can be a second visual frequency. In some implementations, the one or more first outputs have the first characteristic and the one or more second outputs have the second characteristic, and determining that the first subset of smart devices are co-located with the robot in the particular area includes determining that a magnitude of the first characteristic in the detected output is greater than an additional magnitude of the second characteristic in the detected output. In some versions of those implementations, the one or more first characteristics include a first frequency and the one or more second characteristics include a second frequency.For example, the first output can include an audible output, where the first frequency is a first audible frequency outside the range of human hearing, and the second output can include an audible output, where the second frequency is a second audible frequency outside the range of human hearing.

[0091] In some implementations, the first subset of smart devices includes a standalone automated assistant device, and the one or more first outputs include a first audible output via a hardware speaker of the automated assistant device. In some implementations, the first subset of smart devices includes a standalone automated assistant device, and the one or more first outputs include a first visual output via a hardware display of the automated assistant device or via a light emitting diode of the automated assistant device. In some implementations, the first subset of smart devices includes a smart light, a smart television, and / or a smart thermostat. In some implementations, the first semantic label in the home graph is a first descriptor of a first room in the structure previously assigned to the first subset of smart devices based on a first explicit user input, and / or the second semantic label in the home graph is a second descriptor of a second room in the structure previously assigned to the second subset of smart devices based on a second explicit user input. In some implementations, assigning the inferred semantic label to the particular area includes automatically assigning the inferred semantic label to the particular area in the map for use by the mobile robotic device. In some versions of those implementations, the method can further include using the inferred semantic label in controlling navigation of the mobile robotic device after automatically assigning the inferred semantic label to the particular area in the map for use by the mobile robotic device.In some of those versions, using the inferred semantic label in controlling navigation of the mobile robotic device includes determining, based on processing a verbal input detected at one or more microphones of the mobile robotic device, that one or more terms of the verbal input match the inferred semantic label, and causing the robot to navigate to the particular area based on determining that the one or more terms match the inferred semantic label and based on assigning the inferred semantic label to the particular area in the map. In some implementations, assigning the inferred semantic label to the particular area includes suggesting to a user in a graphical user interface that the inferred semantic label be assigned to the particular area in the map for use by the mobile robotic device, and in response to receiving a positive user interface input of the user responsive to the suggestion, assigning the inferred semantic label to the particular area in the map for use by the mobile robotic device. [Explanation of symbols]

[0092] 102 users 104 Robotic Computing Devices 106 rooms 108 Audible Output 110 Oral Speech 112 Alternate Audible Outputs 114 Features 124 devices 126 Another Room 202 users 204 Robot Computing Devices 206 rooms 208 Oral Speech 210 Audible Output 222 The other user, the user 226 Another room, room 228 output, other output 230 Oral Speech 242 Audible Output 246 Specific Location 300 Systems 302 Computing Devices 304 Automated Assistants 306 Input Processing Engine 308 Voice Processing Engine 310 Data Parsing Engine 312 Parameter Engine 314 Output Generation Engine 316 Driving Parameter Engine 318 Layout Detection Engine 320 Assistant Interface 322 Assistant Call Engine 326 Location Preference Engine 330 Application Data 332 Device Data 334 Applications 336 Context Data 338 Assistant Data 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 626 Memory, File Storage Subsystem 630 Random Access Memory (RAM) 632 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, the method comprising: determining, based on a map generated by the mobile robotic computing device and based at least in part on sensor observations of the mobile robotic computing device, that the mobile robotic computing device is currently located within a particular area of ​​a structure; While the mobile robotic computing device is located within the particular area, causing a first subset of smart devices to each emit one or more first outputs, the one or more first outputs being audible and / or visual, the one or more first outputs being emitted during a first time window and / or with one or more first characteristics in response to the first subset of smart devices each being assigned a first semantic label in a home graph; causing a second subset of smart devices to each emit one or more second outputs, the one or more second outputs being audible and / or visual, the one or more second outputs being emitted during a second time window and / or with one or more second characteristics in response to the second subset of smart devices each being assigned a second semantic label in the home graph; acquiring sensor data during the emission of the one or more first outputs and the one or more second outputs, the sensor data being generated by one or more sensors of the mobile robot computing device; determining, based on the analysis of the sensor data, that the first subset of smart devices are co-located with a robot in the particular area, the determining that the first subset of smart devices are co-located with the robot in the particular area comprising: the analysis indicating detected outputs during the first time window and / or matching the one or more first characteristics; and / or a magnitude of the detected output during the first time window and / or matching the one or more first characteristics; and In response to determining that the first subset of smart devices are co-located with the robot in a given room, assigning an inferred semantic label to the particular area, the inferred semantic label being the same as or derived from the first semantic label assigned to the first subset of smart devices in the home graph; A method comprising:

2. determining that the one or more first outputs are emitted during the first time window and the one or more second outputs are emitted during the second time window and that the first subset of smart devices are co-located with the robot in the particular area, comprising: determining that the detected output occurs during the first time window; and determining that no detected output occurs during the second time window.

2. The method of claim 1, comprising:

3. determining that the one or more first outputs are emitted during the first time window and the one or more second outputs are emitted during the second time window and that the first subset of smart devices are co-located with the robot in the particular area, comprising: determining that the magnitude of the detected output occurring during the first time window is greater than an additional magnitude of an additional detected output occurring during the second time window; 2. The method of claim 1, comprising:

4. determining that the one or more first outputs have the first characteristic and the one or more second outputs have the second characteristic and that the first subset of smart devices are co-located with the robot in the particular area, comprising: determining that the detected output matches the first characteristic; and determining that no detected output matches the second characteristic.

3. The method of claim 1 or claim 2, comprising:

5. The method of claim 4 , wherein the one or more first characteristics comprise a first frequency and the one or more second characteristics comprise a second frequency.

6. the first output comprises a visual output and the first frequency is a first visual frequency; the second output comprises a visual output and the second frequency is a second visual frequency. The method of claim 5.

7. determining that the one or more first outputs have the first characteristic and the one or more second outputs have the second characteristic and that the first subset of smart devices are co-located with the robot in the particular area, comprising: determining that a magnitude of the first characteristic in the detected output is greater than an additional magnitude of the second characteristic in the detected output; The method of claim 1 or claim 3, comprising:

8. The method of claim 7 , wherein the one or more first characteristics comprise a first frequency and the one or more second characteristics comprise a second frequency.

9. the first output comprises an audible output, and the first frequency is a first audible frequency outside the range of human hearing; the second output comprises an audible output and the second frequency is a second audible frequency outside the range of human hearing. The method of claim 8.

10. 10. The method of claim 1, wherein the first subset of smart devices comprises a standalone automated assistant device, and the one or more first outputs comprise a first audible output via a hardware speaker of the standalone automated assistant device.

11. 11. The method of claim 1, wherein the first subset of smart devices comprises a standalone automated assistant device, and the one or more first outputs comprise a first visual output via a hardware display of the standalone automated assistant device or via a light emitting diode of the standalone automated assistant device.

12. 12. The method of claim 1, wherein the first subset of smart devices comprises a smart light, a smart television, or a smart thermostat.

13. the first semantic label in the home graph is a first descriptor of a first room in a structure that was previously assigned to the first subset of smart devices based on a first explicit user input; the second semantic label in the home graph is a second descriptor of a second room within a structure previously assigned to the second subset of smart devices based on a second explicit user input; 13. The method according to any one of claims 1 to 12.

14. assigning the inferred semantic label to the particular area, automatically assigning the inferred semantic labels to the particular areas in the map for use by the mobile robotic computing device.

14. The method of any one of claims 1 to 13, comprising:

15. after automatically assigning the inferred semantic labels to the particular areas in the map for use by the mobile robotic computing device; using the inferred semantic labels in controlling navigation of the mobile robotic computing device.

15. The method of claim 14, further comprising:

16. using the inferred semantic labels in controlling navigation of the mobile robotic computing device, determining, based on processing a verbal input detected at one or more microphones of the mobile robot computing device, that one or more terms of the verbal input match the inferred semantic labels; causing the robot to navigate to the particular area based on determining that the one or more terms match the inferred semantic label and based on assigning the inferred semantic label to the particular area in the map; 16. The method of claim 15, comprising:

17. The step of assigning the inferred semantic label to the particular area comprises: suggesting to a user in a graphical user interface that the inferred semantic label be assigned to the particular area in the map for use by the mobile robotic computing device; in response to receiving a positive user interface input from the user responsive to the suggestion, assigning the inferred semantic label to the particular area in the map for use by the mobile robotic computing device; 14. The method of any one of claims 1 to 13, comprising:

18. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform a method according to any one of claims 1 to 17.

19. 18. A system comprising one or more computing devices configured to perform the method of any one of claims 1 to 17.

20. The system of claim 19 , wherein the one or more computing devices comprise a mobile robotic computing device.

21. 18. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to implement a method according to any one of claims 1 to 17.

Images