Selective interaction between a robotic device and an additional computing device

Abstract

The embodiments relate to a robotic computing device that can request additional information from nearby other device(s) to fulfill a request and / or delegate a particular operation to nearby other device(s). Delegating a particular operation may include requesting assistance from the nearby device by the robotic computing device moving to the location of the nearby device and providing input from the robotic computing device to the nearby device. In some cases, the input may include an audible rendering of an invocation phrase and a command phrase to invoke an automated assistant accessible from the nearby device. The decision to delegate a particular operation or request additional information may be based on a variety of factors, such as the predicted efficiency of performing the particular operation and the estimated accuracy of the execution.

Description

[Background technology]

[0001] Automated assistants have been widely deployed in many homes to provide hands-free access to information from the Internet and to control other peripherals in and outside the home. Often, assistant-enabled devices are static devices and may not have the ability to move to different locations in the home. As a result, operations performed by an automated assistant may be geographically restricted depending on where a user places the assistant-enabled device during installation. This may not be particularly inefficient when performing certain tasks, such as when a user issues an Internet search query (e.g., "Who is Joseph Fourier?"). However, other tasks involving communication between users and / or rendering output to a user on the move may prove inefficient or ineffective when performed by a static device or network of static devices. For example, a user listening to a phone call audio rendered by a kitchen assistant device may lose the audio when he or she moves from the kitchen to a hallway where there is no assistant-enabled device. As a result, the user may have to ask the person on the phone to repeat what he or she said and / or pause the call when he or she needs to leave the room, wasting time and computational resources.

[0002] In some cases, an assistant-enabled device may be able to respond to a query from a user by having another assistant-enabled device render an output responsive to the query. However, delegating the execution of a particular operation in this manner may not be effective in a multi-assistant environment where multiple assistant devices are associated with different software providers and / or user accounts. As a result, the execution of a particular operation may be delegated to a less suitable assistant device, even though other more suitable devices (e.g., devices with better sound quality, stronger signal, more efficient power usage, etc.) are available. For example, a communication task between users may be performed by default by the assistant device that received the corresponding request from the user. However, if other more suitable devices are available to fulfill such requests and / or if the default device is not positioned to effectively communicate with any of the particular users, the lack of ability of the device to interact may result in further wasted resources. Summary of the Invention [Means for solving the problem]

[0003] The embodiments described herein relate to a robotic computing device that can interact with other nearby devices (possibly using techniques common to human users) to facilitate fulfilling requests from a user to the robotic computing device. The robotic computing device can, for example, render voice commands to nearby assistant-enabled devices to fulfill requests issued by the user to the robotic computing device. For example, a user may make a voice utterance to the robotic computing device such as "Can you help Emma find her toy car?", which may be a request to the robotic computing device to identify a particular object in the user's home. In response to receiving the voice utterance, the robotic computing device may determine that a search for information stored on the robotic computing device to identify an appropriate description of "toy car" does not result in an accurate description. For example, the robotic computing device has an internet search capability, but search results for the phrases "Emma's toy car" or "toy car" may not provide an accurate description of the object the user is referring to. Thus, the robotic computing device, with prior or current permission from the user, may in turn generate output commands to be provided to one or more other computing devices within the user's home to obtain more precise details about the object the user is referring to.

[0004] In some embodiments, the robot computing device may generate natural language content, such as "Assistant, what is 'Emma's car toy'?" and then render the natural language content as an audio output to a nearby computing device. In response, the nearby computing device (e.g., another Assistant-enabled device) may cause an image of "Emma's car toy" to be rendered on a display panel. One or more camera images of the displayed image may be captured by a camera of the robot computing device with prior permission from the user, and the robot computing device may utilize these camera images to estimate a likely location of "Emma's car toy". For example, prior images captured by the robot computing device and associated with a location on the home graph may be processed to determine whether an object in the camera image is one that has been captured in any of the prior images. If it is determined that a particular prior image includes an object (e.g., "Emma's car toy"), the robot computing device may identify and utilize a saved map location associated with the particular prior image. The robot computing device may then navigate from the robot computing device's current location (e.g., the location where the robot computing device emitted the audible output to the nearby device) to a map location that corresponds to the object in the particular camera image. Alternatively or additionally, the robotic computing device may generate additional natural language content representing the location on the map and render another output representing the additional natural language content to the user (e.g., an audible or visual output saying, "The toy car is under the kitchen table").

[0005] In some embodiments, the robotic computing device may determine whether to delegate an action to another nearby device and perform such delegation using one or more techniques similar to human-computer interaction. That is, if the robotic computing device determines that another computing device may be better suited to perform a requested action, the robotic computing device may issue a human-recognizable command to the other computing device. For example, a user may make a vocal utterance to the robotic computing device, such as "play some music while I cook." In response, the robotic computing device may determine that a plugged-in device is better suited to fulfill this request, at least in part because the robotic computing device is operating using battery power. Based on this determination, the robotic computing device may identify a nearby device that can more efficiently fulfill the request from the user.

[0006] For example, the robot computing device may identify kitchen smart displays (i.e., standalone display devices) that are within a threshold distance from the user to render audio content. In some embodiments, the robot computing device may identify the device name or type of kitchen smart display and render commands for the robot computing device based on the name or type of the kitchen smart display. For example, based on identifying the type of device in the vicinity, the robot computing device may identify a call phrase for that particular type of device (e.g., "OK, smart device..."). Using this call phrase and with prior permission from the user, the robot computing device may render an audible output such as "OK, smart device, play some music while I cook." In response, the kitchen smart display may begin rendering music. In this way, the robot computing device can act as an interface between the user and all of its smart devices, even if the kitchen smart display is not specifically pre-configured to be delegated tasks from the robot computing device. When task delegation is performed based on capabilities and / or efficiency, the smart devices in the user's home can follow the instructions of the robot computing device to conserve resources such as bandwidth and power.

[0007] In some embodiments, the robotic computing device may leverage the capabilities of another device if it determines that another device can complete a requested task faster than the robotic computing device. Such an instance may occur when the standalone device and / or other robotic computing devices are determined to be closer to the location of the task than the robotic computing device that originally received the request to complete the task. For example, a user ("Kaiser") may speak an utterance to the robotic computing device such as "Tell Karma to turn off the heater upstairs." The robotic computing device may be located on the first floor of the user's home with the standalone computing device, and another standalone computing device may be located on the second floor of the home closer to another user ("Karma"). Upon receiving the voice utterance from the user, the robotic computing device may determine, with prior permission from a person(s) in the home, that the other user is on a different floor of the home than the robotic computing device. Based on this determination, the robotic computing device may cause another standalone computing device on the second floor to send a message to the other user.

[0008] In some embodiments, the robotic computing device may cause another standalone computing device to send a message in a variety of different ways. For example, the robotic computing device may communicate with a standalone computing device on the first floor to send a message (e.g., "Kaiser wants you to turn off the heater.") to another standalone computing device on the second floor. The communication between the robotic computing device and the standalone computing device may be via a local area network (LAN), a wide area network (WAN) such as the Internet, audible or inaudible frequencies, Bluetooth communication, and / or any other medium used for communication between devices. For example, the robotic computing device may generate natural language content corresponding to a command to be sent to a standalone speaker device on the first floor. The natural language content may be embodied in an audible or inaudible message to the standalone speaker device (e.g., "OK smart device, send a message to Karma telling her to turn off the heater on the second floor."). In response, another standalone computing device on the second floor may render an audible output and / or a visual message (e.g., "Please turn off the heater on the second floor."). Providing a robotic computing device that can interface with other computers in a user's home can allow for more efficient utilization of resources within the home while minimizing the time required to complete a requested task.

[0009] The above description is provided as a summary of some embodiments of the present disclosure. These and other embodiments are described in more detail below.

[0010] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., central processing unit (CPU)(s), graphics processing unit (GPU)(s), and / or tensor processing unit (TPU)(s)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include one or more computer systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0011] As will be apparent, all combinations of the foregoing concepts and additional concepts described in more detail herein are intended to be part of the subject matter disclosed herein, for example, all combinations of subject matter recited in the claims at the end of this disclosure are intended to be part of the subject matter disclosed herein. [Brief description of the drawings]

[0012] [Figure 1A] FIG. 1 illustrates a robotic computing device that operates to assist a user with the assistance of other nearby computing devices. [Figure 1B] FIG. 1 illustrates a robotic computing device that operates to assist a user with the assistance of other nearby computing devices. [Figure 1C] FIG. 1 illustrates a robotic computing device that operates to assist a user with the assistance of other nearby computing devices. [Figure 1D] FIG. 1 illustrates a robotic computing device that operates to assist a user with the assistance of other nearby computing devices. [Figure 1E]FIG. 1 illustrates a robotic computing device that operates to assist a user with the assistance of other nearby computing devices. [Diagram 2] FIG. 1 illustrates a system for operating a robotic computing device that can seek additional information from other nearby devices in order to fulfill requests and / or delegate certain tasks to the other nearby devices. [Diagram 3] FIG. 1 illustrates a method for controlling a robotic computing device to fulfill a request from a user by delegating performance to one or more other computing devices and / or by collecting additional information from one or more other computing devices. [Figure 4] FIG. 1 is a block diagram of an example computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] 1A, 1B, 1C, 1D, and 1E show views 100, 120, 140, 160, and 180 of a robotic computing device 104 operating to assist a user 102 with the assistance of other nearby computing devices. The robotic computing device 104 may determine whether to seek assistance from additional computing devices based on a variety of factors, such as the confidence that the robotic computing device 104 can accurately fulfill a request from a user, the amount of energy expected to be consumed by one or more devices, the time it takes to fulfill a request by one or more devices, the amount of processing bandwidth consumed by one or more devices, and / or other factors that may be considered when delegating a task to and / or from a particular device. For example, the robotic computing device 104 and the user 102 may be located within an environment 106 (e.g., the user's 102 home) when the user 102 provides a voice utterance 110 (e.g., "Can you help me find Luke's diary?"). The robotic computing device 104 can provide a response output 108 in response to the voice utterance 110 (e.g., "Yes, I can help you find Luke's diary").

[0014] In some embodiments, in response to receiving the voice utterance 110, the robot computing device 104 may process audio data corresponding to the voice utterance 110 to identify one or more operations to perform in response to the voice utterance 110. For example, the robot computing device 104 may determine that the user 102 has requested assistance in locating an object in the environment 106. In some embodiments, this request may be fulfilled by at least executing an intent having a slot value associated with the object to be located. For example, the intent's slot value may include an image of the object. However, even though the robot computing device 104 has access to multiple images that may be similar to and / or associated with the object, the robot computing device 104 may determine that the multiple images are not adequate for a threshold confidence level. As a result, the robot computing device 104 may determine whether to delegate fulfillment of the request to another device and / or whether to seek additional information from another device and / or a user in the environment 106 (or outside the environment 106).

[0015] For example, in response to determining that the robot computing device 104 may not be able to fulfill the request with a threshold confidence, the robot computing device 104 may determine whether another device is available to assist in fulfilling the request. For example, the robot computing device 104 may use sensor data and / or other data available to the robot computing device 104 to determine whether the environment 106 includes one or more other devices that can be invoked to provide additional information. In some embodiments, such data may include image data and / or audio data captured by the robot computing device 104 with prior permission from a user in the home. For example, the image captured by the robot computing device 104 may be processed to determine whether the environment 106 includes one or more other computing devices and / or to determine the type of one or more other computing devices in the environment 106. Once the robot computing device 104 identifies a smart display device 124 and / or a smart speaker device 122 ("smart" may indicate that the device has the capability to access the Internet and respond to user input), the robot computing device 104 may determine whether an invocation phrase is required to invoke the identified device. For example, the robotic computing device 104 may perform an internet search and / or other database search to determine how to request additional information from a particular device.

[0016] As shown in view 120 of FIG. 1B, the robotic computing device 104 may navigate to an area of ​​the environment 106 in which the smart display device 124 resides and may request additional information from the smart display device 124 using the determined call phrase. For example, the robotic computing device 104 may utilize its wheel(s) and / or track(s) in navigating to an area of ​​the environment, possibly utilizing a pre-created environmental map and / or navigation method(s) (e.g., Simultaneous Map and Aerial Localization (SLAM)). Although a particular robotic computing device 104 is illustrated in FIG. 1B and other figures, it will be appreciated that the embodiments disclosed herein may be implemented in additional or alternative mobile robotic computing devices. For example, other mobile computing device(s) capable of self-navigating using wheel(s), track(s), leg(s) (e.g., multi-legged robots), rotors (e.g., unmanned aerial robots), and / or wings. Additionally, the robot computing device 104 may generate content for the robot input provided by the robot computing device 104 to the smart display device 124. The content may be generated using one or more heuristic processes and / or one or more trained machine learning models (e.g., language models). For example, the robot computing device 104 may generate content that embodies a request to describe the characteristics of a particular object because the robot computing device 104 may not have a threshold confidence regarding the appearance of the particular object. Following the previous example, the robot computing device 104 may render audio output 130 including content such as "Assistant, ..." which may mean an invocation phrase or "...what does Luke's diary look like?" which may mean a command requesting additional information from the smart display device 124.

[0017] In response to the audio output 130, the smart display device 124 may render search results 142 on a display interface 144 of the smart display device 124, as shown in view 140 of FIG. 1C. Alternatively or additionally, the smart display device 124 may provide an audio response 146, such as "here are the search results for 'Luke's Diary'", followed by an audio description of the characteristics of the object for which the robot computing device 104 is querying. In some embodiments, as the smart display device 124 is rendering the search results 142 and / or the audio description of the object, the robot computing device 104 may capture data based on the output that the smart display device 124 is rendering. For example, the robot computing device 104 may use one or more cameras 166 and / or one or more microphones to capture the output that the smart display device 124 is rendering. The smart display device 124 may capture an image of the search results 142. The image may include an image of a diary that one or more users recently searched for on the Internet. This image may be used by the robotic computing device 104 to fulfill a request from the user 102 .

[0018] In some embodiments, data captured by the robotic computing device 104 may be compared with private and / or public home knowledge graph data, with prior permission from the user(s), to determine the location of the object queried by the user 102. Values ​​stored in the knowledge graph may include text values ​​(e.g., names of objects, names of places, other textual descriptors of an entity), numerical values ​​(e.g., type of entity, usage data, age, height, weight, other characteristic data, other numerical information associated with an entity), or pointers to user-specific values ​​(e.g., locations in memory for entities in the user's knowledge graph, locations in memory that associate two or more entities in the user's knowledge graph, etc.). That is, values ​​specific to a user and / or environment (e.g., a particular home) may take various forms and may be specific to a field of a personal record defined by a record schema. The value may indicate actual information specific to the user, or may be a designation of a memory location and / or device from which information specific to the user and / or environment can be obtained.

[0019] In some cases, a comparison of the data presented by the smart display device 124 with the data graphed in the personal knowledge graph and / or the home knowledge graph may identify information that may assist the robot computing device 104 in fulfilling a request from the user 102. For example, the robot computing device 104 may determine a location of the identified object based on similar objects that have been captured and location data stored in the home knowledge graph of the user 102. Once the location is identified, the robot computing device 104 may optionally determine whether the location was determined with a threshold certainty or confidence. If the robot computing device 104 determines with a threshold confidence that the object identified by the user (e.g., Luke's diary) is at the location, the robot computing device 104 may render instructions to the user 102.

[0020] For example, as shown in view 160 of FIG. 1D , the robotic computing device 104 may navigate from an area in the environment 106 where the user 102 requested the robotic computing device 104 for help to another area. The other area may be a location where the user 102 has traveled that the robotic computing device 104 predicted with prior permission from the user 102. Depending on the output rendered to the user 102, the robotic computing device 104 may navigate to the location of the user 102 and / or request another computing device to render the output to the user 102. For example, if the user 102 is predicted to be in a location close to another computing device, the robotic computing device 104 may communicate with the other computing device (e.g., via a Wi-Fi network) to have the other computing device render the output to the user 102. Otherwise, the robotic computing device 104 may navigate to a location close to the user 102 and render the audible output 162 and / or visual output. For example, once the robot computing device 104 has determined the information to fulfill the request from the user 102, the robot computing device 104 may render an audible output 1062, such as "Luke's diary is in the kitchen. Let me take you to Luke's diary." In response, the user 102 may provide an affirmative answer 164, such as "Okay, thanks," confirming that the user 102 wishes to be guided to the location of the specified object.

[0021] In response to the user 102 confirming the offer from the robot computing device 104 to guide to the object, the robot computing device 104 may move to another area in the environment 106. This other area may be, for example, a kitchen that includes a counter on which the desired object is located. In some cases, as shown in view 180 of FIG. 1E, the user 102 may issue another request to the robot computing device 104. The other request may be, for example, a voice utterance 182 such as "Tell Luke that his diary is in the kitchen." The request may be the user 102 asking the robot computing device 104 to convey a message to another user. In some embodiments, the robot computing device 104 may process this input from the user 102 and determine whether to delegate fulfillment of the request. For example, using the home knowledge graph data and / or other data, the robot computing device 104 may determine whether one or more other devices can more effectively fulfill the request from the user 102.

[0022] In some cases, the robotic computing device 104 may, with prior permission from the user(s), determine a predicted location of the user Luke and estimate whether the robotic computing device 104 can effectively communicate with Luke or whether another device can more effectively communicate with Luke. For example, the robotic computing device 104 may determine, based on the personal knowledge graph data, that the user Luke is playing with his mobile phone in his room and that a smart speaker device is in his room. Based on this determination, the robotic computing device 104 may determine that without the assistance of another device, the robotic computing device 104 would consume more energy and / or take longer to communicate with Luke. Thus, the robotic computing device 104 may determine that the smart speaker device 188 is nearby and, based on the type of smart device, may invoke the smart speaker device 188 to send a message to the user Luke. For example, the robotic computing device 104 may generate content for an audible output such as, "Assistant, tell Luke that his diary is in the kitchen." This audible output may enable the smart speaker device 188 and / or one or more other computing devices to render output 190 such as, "Luke, your diary is in the kitchen."

[0023] In some embodiments, the robot computing device 104 may use the smart speaker device 188, the smart display device 124, and / or other computing devices to verify that the object 186 identified by the robot computing device 104 (e.g., Luke's diary) is the object that the user 102 was trying to find. For example, the robot computing device 104 may identify the object 186, manipulate an arm or other portion of the robot computing device 104 to pick up the object 186, and bring the object 186 to a location of the smart display device 124 or the smart speaker device 188 (e.g., if the smart speaker device 188 includes a camera). The robot computing device 104 may then request the smart display device 124 (or other device) to verify whether the object 186 is the object that the user 102 was referring to.

[0024] For example, the robotic computing device 104 may generate a command including an invocation phrase and content requesting the smart display device 124 to capture an image of what the robotic computing device 104 is holding. For example, the command may be "Smart device, is this Luke's diary?" In some embodiments, the command may be generated based on a prior interaction between the user 102 and the smart display device 124. Such a prior interaction informs the robotic computing device 104 of the capabilities of the smart display device 124. Alternatively or additionally, the command may be generated based on a search of a database, the Internet, or other information source to determine the type of command to which the smart display device 124 will respond.

[0025] In some embodiments, the robot computing device 104 may confirm to the user 102 or another user that the object 186 identified by the robot computing device 104 is the object the user 102 refers to. For example, the robot computing device 104 may initiate a request to have another user (e.g., Luke) confirm the name of the object 186 by providing a command to the smart display device 124 to perform a video call with the robot computing device 104. For example, the robot computing device 124 may generate a command such as "Smart device, call Luke on a video call" and render the command as an audio output to the smart display device 124. In response, the smart display device 124 may initiate a video call between the robot computing device 124 and the other user (with prior permission from the other user). In some cases, the other user may not be in the environment 106, but the robot computing device 104 may still initiate communication with the other user via another computing device in the environment 106. When a video call is initiated on the smart display device 124, the robotic computing device 104 may hold up the object 186 and request the other user to confirm the name of the object 186 (e.g., "Hi Luke. Is this your diary?"). If in response the other user confirms the name of the object 186, the robotic computing device 104 may consider the request from the user 102 to be fulfilled.

[0026] FIG. 2 illustrates a system 200 for operating a robotic computing device (e.g., computing device 202) that can request additional information from other nearby devices to fulfill requests and / or delegate certain tasks to the other nearby devices. The robotic computing device may include an automated assistant 204. The automated assistant 204 may operate as part of an assistant application provided on one or more computing devices, such as the computing device 202 and / or a server device. A user can interact with the automated assistant 204 through assistant interface(s) 220. The assistant interface(s) 220 can be a microphone, a camera, a touchscreen display, a user interface, and / or other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 204 and cause the automated assistant 204 to initiate one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.) by providing verbal, textual, and / or graphical input to the assistant interface 220. Alternatively, the automated assistant 204 may be initialized based on processing the context data 236 using one or more trained machine learning models. The context data 236 may characterize one or more characteristics of an environment in which the automated assistant 204 is accessible and / or one or more characteristics of a user who is predicted to intend to interact with the automated assistant 204. The computing device 202 may include a display device. The display device may be a display panel including a touch interface for receiving touch inputs and / or gestures that enable a user to control the applications 234 of the computing device 202 via a touch interface.In some embodiments, the computing device 202 may not have a display device and therefore may provide an audible user interface output without providing a graphical user interface output. Additionally, the computing device 202 may provide a user interface such as a microphone for receiving spoken natural language input from a user. In some embodiments, the computing device 202 may include a touch interface and may lack a camera, but may possibly include one or more other sensors.

[0027] The computing device 202 and / or other third-party client devices may communicate with the server device over a network such as the Internet. Additionally, the computing device 202 and other computing devices may communicate with each other over a local area network (LAN) such as a Wi-Fi network. The computing device 202 may offload computational tasks to the server device to conserve computing resources of the computing device 202. For example, the server device may host the automated assistant 204 and / or the computing device 202 may send inputs received at one or more assistant interfaces 220 to the server device. However, in some embodiments, the automated assistant 204 may be hosted on the computing device 202 and various processes that may be associated with automated assistant operation may be executed on the computing device 202.

[0028] In various embodiments, all or less than all of the automated assistant 204 may execute on the computing device 202. In some of these embodiments, some portions of the automated assistant 204 may execute via the computing device 202 and may interface with a server device that can execute other portions of the automated assistant 204. The server device may serve multiple users and their associated assistant applications, possibly via multiple threads. In embodiments in which all or less than all of the automated assistant 204 executes via the computing device 202, the automated assistant 204 may be an application separate from the operating system of the computing device 202 (e.g., installed "on top of" the operating system) or may be executed directly by the operating system of the computing device 202 (e.g., an operating system application but considered integral to the operating system).

[0029] In some embodiments, the automated assistant 204 may include an input processing engine 206. The input processing engine 206 may employ multiple different modules to process input and / or output of the computing device 202 and / or the server device. For example, the input processing engine 206 may include a voice processing engine 208, which may process voice data received at the assistant interface 220 to identify text contained in the voice data. The voice data may be transmitted from the computing device 202 to the server device, for example, to conserve computational resources of the computing device 202. Additionally or alternatively, the voice data may be processed exclusively at the computing device 202.

[0030] The process of converting the voice data to text may include speech recognition algorithms that may employ neural networks and / or statistical models to identify groups of voice data that correspond to words and phrases. The text converted from the voice data may be parsed by the data parsing engine 210 and made available to the automated assistant 204 as text data. Such text data may be used to generate and / or identify command phrase(s), intent(s), action(s), slot value(s), and / or other content specified by the user. In some embodiments, the output data provided by the data parsing engine 210 may be provided to the parameter engine 212 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be executed by the automated assistant 204 and / or an application or agent accessible via the automated assistant 204. For example, the assistant data 238 may be stored on the server device and / or the computing device 202, and may include data defining one or more actions that may be executed by the automated assistant 204, and parameters required to execute the actions. The parameter engine 212 may generate one or more parameters for the intent, action, and / or slot value and may provide the one or more parameters to the output generation engine 214. The output generation engine 214 may use the one or more parameters to communicate with the assistant interface 220 to provide output to a user and / or to communicate with one or more applications 234 to provide output to the one or more applications 234.

[0031] In some embodiments, the automated assistant 204 may be an application that may be installed "on" the operating system of the computing device 202 and / or may itself form part (or the entirety) of the operating system of the computing device 202. The automated assistant application may include and / or have access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, the on-device speech recognition may be performed using an on-device speech recognition module that processes the voice data (detected by the microphone(s)) using an end-to-end speech recognition machine learning model stored locally on the computing device 202. The on-device speech recognition generates recognized text for the voice utterances (if any) present in the voice data. Also, for example, the on-device natural language understanding (NLU) may be performed using an on-device NLU module. The on-device NLU module processes recognized text generated using on-device speech recognition and optionally processes contextual data to generate NLU data.

[0032] The NLU data may include the intent(s) corresponding to the voice utterance and possibly parameter(s) of the intent(s) (e.g., slot values). On-device fulfillment may be performed using an on-device fulfillment module that uses the NLU data (from the on-device NLU) and possibly other local data to determine the action(s) to perform to resolve the intent(s) of the voice utterance (and possibly the parameter(s) of the intent). This may include: determining local and / or remote responses (e.g., answers) to the voice utterance, determining interaction(s) with locally installed application(s) to perform based on the voice utterance, determining command(s) to send (directly or via corresponding remote system(s)) to Internet of Things (IoT) device(s) based on the voice utterance, and / or determining other resolution action(s) to perform based on the voice utterance. On-device fulfillment may then initiate local and / or remote execution / execution of the determined action(s) to resolve the voice utterance.

[0033] In various embodiments, remote speech processing, remote NLU, and / or remote fulfillment may be utilized at least selectively. For example, recognized text may be at least selectively sent to a component(s) of a remote automated assistant for remote NLU and / or remote fulfillment. For example, recognized text may possibly be sent for remote fulfillment in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to reduced latency incurred when resolving a voice utterance (since no client-server round trip(s) are required to resolve the voice utterance). Additionally, on-device functionality may be the only functionality available in situations where there is no network connectivity or where network connectivity is limited.

[0034] In some embodiments, the computing device 202 may include one or more applications 234. The one or more applications 234 may be provided by a third party entity different from the entity that provided the computing device 202 and / or the automated assistant 204. The application state engine of the automated assistant 204 and / or the computing device 202 may access the application data 230 to determine one or more actions that may be performed by the one or more applications 234 and may further determine the state of each application of the one or more applications 234 and / or the state of each device associated with the computing device 202. The device state engine of the automated assistant 204 and / or the computing device 202 may access the device data 232 to determine one or more actions that may be performed by the computing device 202 and / or the one or more devices associated with the computing device 202. Additionally, the application data 230 and / or other data (e.g., the device data 232) may be accessed by the automated assistant 204 to generate the context data 236. The contextual data 236 may characterize the context in which a particular application 234 and / or device is executing and / or the context in which a particular user is accessing the computing device 202 (accessing the application 234 and / or other devices or modules).

[0035] While one or more applications 234 are executing on the computing device 202, the device data 232 may characterize a current operational state of each application 234 executing on the computing device 202. Additionally, the application data 230 may characterize one or more features of the executing applications 234, such as the content of one or more graphical user interfaces rendered at the direction of the one or more applications 234. Alternatively or additionally, the application data 230 may characterize action schemas that may be updated by the respective applications and / or automated assistant 204 based on the respective application's current operational state. Alternatively or additionally, one or more action schemas of the one or more applications 234 may remain static but may be accessed by the application state engine to determine appropriate actions to initiate via the automated assistant 204.

[0036] The computing device 202 may further include an assistant invocation engine 222. The assistant invocation engine 222 may process the application data 230, the device data 232, the context data 236, and / or other data accessible to the computing device 202 using one or more trained machine learning models. The assistant invocation engine 222 may process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 204, or whether to consider the data to indicate the user's intent to invoke the automated assistant instead of having the user explicitly speak the invocation phrase. For example, the one or more trained machine learning models may be trained using instances of training data. The instances of training data are based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating states. The instances of training data may be generated to obtain training data. The training data characterizes a context in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant. Once one or more trained machine learning models have been trained based on these training data instances, the assistant invocation engine 222 may cause the automated assistant 204 to detect or limit detection of voice invocation phrases from the user based on contextual and / or environmental features.

[0037] In some embodiments, the assistant invocation engine 222 may process data to determine how to invoke a nearby device that provides access to an instance of the automated assistant 204 and / or another assistant application. For example, image data captured by the camera of the computing device 202 may be processed to determine the type of computing device that is located in the same environment as the computing device 202. The computing device 202 may then determine that this type of computing device is one that provides access to a particular automated assistant that can be invoked using a particular invocation phrase. Based on this determination, the computing device 202 may initialize the output generation engine 214 to render an invocation phrase when the computing device 202 determines to seek additional information from a nearby device and / or to delegate a task to a nearby device. The output generation engine 214 may then render an output (e.g., an audible output, a Bluetooth command, a non-audible request, etc.) that includes the invocation phrase and the content of the request generated by the computing device 202.

[0038] In some embodiments, the system 200 may include a delegation engine 216 that may determine whether to delegate the fulfillment of a request and / or a portion of a request to another computing device. In some embodiments, the delegation engine 216 may process data to determine the state of devices and / or applications associated with a user that may delegate a particular task with prior permission from the user. The data processed by the delegation engine 216 may be selected in response to a request provided by the user, allowing the computing device 202 to customize the delegation of a particular task for each request. For example, when the delegation engine 216 is determining whether to delegate a task of rendering music for a user, the data may indicate whether a particular interface of the device is being used. Alternatively or additionally, the delegation engine 216 may process power usage data to identify nearby devices that are connected to utility power or using battery power when determining whether to delegate various types of requests. In this manner, the computing device 202 may avoid delegating a particular task to a nearby device that may not have the optimal power source to complete the task.

[0039] In some embodiments, the delegation engine 216 may access personal knowledge graph data and / or home graph data of one or more different users (with prior permission from the users) to make a decision regarding delegating a particular task to fulfill a request. For example, the home graph data may indicate the location of a particular object and / or feature in the user's home, the status of a particular device (e.g., processing bandwidth, signal strength, battery charge), and / or other properties of the device or object, with prior permission from the user. Alternatively or additionally, the home graph data may indicate the status of a particular device and / or application being accessed in the home. The personal knowledge graph data may also be utilized by the delegation engine 216 to determine whether to delegate a particular task to a particular device. For example, the personal knowledge graph data may indicate a user's preferences regarding having a particular device perform a particular operation, and / or other preferences that may be stored on the device. The computing device 202 may use such data to delegate a task that the user prefers to be performed on another device (even though the user has made a corresponding request to the computing device 202).

[0040] In some embodiments, data generated by the robot computing device may be used to update the home graph data and / or the personal knowledge graph data. For example, image data and / or audio data captured by the robot computing device with prior permission from the user as the robot computing device moves through the environment may be utilized to update the home graph data and / or the personal knowledge graph data. In some embodiments, the robot computing device may proactively generate such data if a particular portion of the home graph data and / or the personal knowledge graph data has not been updated for at least a threshold period of time. Alternatively or additionally, data regarding a particular object and / or a particular portion of the environment may be updated if the robot computing device has not observed the particular object and / or a particular portion of the environment for at least a threshold period of time.

[0041] In some embodiments, the computing device 202 may include an information request engine 218. The information request engine 218 may process data to determine whether to request additional information to fulfill a request from a user. If the information request engine 218 determines that a particular request cannot be fulfilled with a threshold level of confidence without at least requesting the additional information, the additional information may be requested from another device and / or application. In some embodiments, the decision to request additional information and / or to delegate a particular task may be based on one or more heuristic processes and / or one or more trained machine learning models. For example, when a user issues a request to the computing device 202, the delegation engine 216 and / or the information request engine 218 may generate a confidence metric that indicates whether a task needs to be delegated and / or whether additional information needs to be requested to fulfill the request.

[0042] For example, inputs embodying the request may be processed to determine a reliability metric for fulfilling the request. Alternatively or additionally, candidate responses may be generated by the computing device 202 and processed to determine a reliability metric for the candidate responses. If any of the reliability metrics do not meet the reliability metrics for delegating the task, the delegation engine 216 may initiate a process to determine an appropriate device to delegate the task to. If any of the reliability metrics do not meet the reliability metrics for requesting additional information, the information request engine 218 may initiate a process to determine an appropriate device to ask for additional information. In some embodiments, the information request engine 218 may first determine whether additional information needs to be requested for a particular request, and then the delegation engine 216 may determine whether one or more tasks for the request need to be delegated to another device. For example, the computing device 202 may receive a request to send a message to another device, and the information request engine 218 may initially determine, with a threshold confidence, that no additional information needs to be requested to fulfill the request. Subsequently or simultaneously, the delegation engine 216 may determine that a particular computing device is better suited to perform the task of sending a message. The delegation engine 216 may then generate data used by the assistant invocation engine 222 to output requests from the computing device 202 to a particular computing device to cause the particular computing device to carry out the request (e.g., send a message).

[0043] FIG. 3 illustrates a method 300 for controlling a robotic computing device to fulfill a request from a user by delegating performance to one or more other computing devices and / or by collecting additional information from one or more other computing devices. The method 300 may be performed by one or more computing devices, applications, and / or any other apparatus or module that may be associated with an automated assistant. The method 300 may include an operation 302 for determining whether the robotic computing device receives a user input. The user input may be, for example, a request to the robotic computing device to provide a location of an object in the user's home. For example, the user may provide a voice utterance such as "Where is Maggie's drone?", which refers to a toy drone owned by another user who is in the same environment (e.g., home) as the user. The user may provide a voice utterance to the robotic computing device to actively recruit the robotic computing device to assist the user and other users in finding the object (e.g., "Maggie's drone").

[0044] The method 300 may proceed from operation 302 to operation 304. Operation 304 may include determining whether the robot computing device needs to delegate fulfillment of the request to another device in the environment. In some cases, the environment may be a user's home and may include multiple different devices that provide access to one or more different automated assistant applications. For example, the environment may include an office with a first assistant device and a kitchen with a second assistant device. When the robot computing device receives a user input, the robot computing device may process the user input and determine that the user has requested that the robot computing device locate an object. Based on this determination, the robot computing device may estimate a location of the object and a magnitude of a reliability metric for the location estimate.

[0045] In some embodiments, the location estimation may be based on an image search performed in response to receiving user input from a user. One or more image results identified from the image search may be compared with one or more images captured by the robotic computing device as the robotic computing device moves through the environment, with prior permission from the user. The robotic computing device and / or another device (e.g., a server device) may determine whether the object identified by the user corresponds to the item identified in the captured image with a threshold confidence. For example, a confidence metric may be generated based on the comparison of the image search result(s) and the captured image(s) to quantify the confidence for identifying the particular object. If the confidence does not meet a certain confidence threshold, the robotic computing device may decide to seek additional information regarding the particular object requested to be identified. However, if the confidence meets a certain threshold, the robotic computing device may perform the requested action of identifying the particular object or may delegate the task to another device.

[0046] In some embodiments, the delegation of an action can be based on various factors, such as, for example, whether delegating the action reduces power consumption, alleviates network and / or processing bandwidth limitations at the device, whether the action completes faster, and / or other factors suitable for consideration when delegating actions between devices. For example, the robotic computing device may determine that an additional device (e.g., another robotic device, or another smart device) is closer to a predicted location of the object. Based on this determination, the robotic computing device may determine that delegating the action of checking the location of the object will consume less energy than at least performing the action itself.

[0047] If the robotic computing device determines to delegate one or more actions based on one or more different factors, the method 300 may proceed from operation 304 to operation 314. Operation 314 may include the robotic computing device providing an input therefrom to the additional computing device. Following the previous example, the input may be a communication (e.g., audio, visual, wireless, Bluetooth, etc.) between the robotic computing device and another computing device. The content of the communication may embody a request for the other computing device to navigate to and / or confirm the location of an object (e.g., Maggie's drone). The method 300 may proceed from operation 314 to operation 316. Operation 316 may include causing the additional computing device to perform an action to facilitate fulfillment of the request.

[0048] For example, once the robot computing device determines with a threshold confidence that the location of the object can be determined by the robot computing device, the robot computing device may communicate with the additional computing device to determine the location of the object. The communication from the robot computing device may be audio communication, image communication, wireless communication, and / or any other communication that can be provided between devices. In some embodiments, the robot computing device may provide an audio output at a frequency higher or lower than frequencies that can be naturally detected by humans. Alternatively or additionally, the communication from the robot computing device may be provided to the additional computing device as a communication via a wireless communication protocol (e.g., Bluetooth, Wi-Fi, LTE, and / or any other communication protocol). The communication may be processed by the additional computing device, such that the additional computing device may search for the location of the object according to one or more different processes. For example, the additional computing device may be another robot computing device that can navigate different environmental areas than the robot computing device upon receiving a communication from the robot computing device. Once the additional computing device determines the location of the object, the additional computing device may communicate the location of the object to the robot computing device.

[0049] If the robot computing device determines not to delegate the action to another computing device in operation 304, the method 300 may proceed from operation 304 to operation 306. Operation 306 may include determining whether to request additional information from another computing device and / or another user (with prior permission from the other user). In some embodiments, the determination may be based on whether the confidence meets a threshold confidence and / or one or more other thresholds. For example, if the magnitude of the confidence metric does not meet another confidence metric, the robot computing device may determine to seek information from the additional computing device (e.g., moving from operation 306 to operation 308). Alternatively or additionally, the robot computing device may rely on one or more heuristic processes and / or one or more trained machine learning models to determine whether to request additional information. For example, input from a user may be processed using one or more trained machine learning models to generate an embedding that can be compared to existing embeddings mapped to the latent space. If the embedding distance between the generated embedding and the existing embedding meets a threshold, the method 300 may proceed from operation 306 to operation 312.

[0050] The operation 312 may include causing the robot computing device to perform one or more actions (i.e., operations) to facilitate fulfillment of the request. For example, following the previous example, the robot computing device may locate the object with a threshold confidence and indicate the location to the user. In some embodiments, the robot computing device may indicate the location by providing an audible and / or visual output at one or more interfaces of the robot computing device and / or another computing device. Alternatively or additionally, the robot computing device may indicate the location of the object by suggesting to move to the object's location (e.g., "Okay, let's take you to Maggie's toy") and then move to the object's location.

[0051] If the robotic computing device determines to request additional information from another computing device, the method 300 may proceed from operation 306 to operation 308. Operation 308 may include moving the robotic computing device toward and communicating with the additional computing device. For example, the additional computing device may be a stand-alone computing device, such as a display device or a speaker device. The robotic computing device may select the additional computing device to communicate with based on a determination that the user has previously interacted with the additional computing device. For example, with prior permission from the user, the robotic computing device may identify one or more devices with which the user has previously interacted and determine whether the one or more devices provide access to an automated assistant. If the robotic computing device determines that a particular additional device provides access to an automated assistant, it may initialize communication with the automated assistant. For example, the robotic computing device may identify a type of automated assistant accessible via the additional computing device and determine an invocation phrase for communicating with the automated assistant based on the type of automated assistant. For example, the robotic computing device may use an application programming interface (API) and / or perform an internet search to identify the invocation phrase of the additional computing device.

[0052] Once the invocation phrase is identified, the robotic computing device may generate natural language content including the invocation phrase and a specific request for the automated assistant. The invocation phrase may be, for example, "Assistant..." and the specific request may be, for example, "What does Maggie's drone look like?" The content of the specific request may be based on features of the user input that may be uncertain to the robotic computing device. The function identified as the basis for obtaining additional information may be selected based on one or more parameters and / or slot values ​​of an action to be performed by the robotic computing device to facilitate the fulfillment of the request from the user. For example, the specific slot value may be based on an identifier of the object to be located. Thus, to generate the identifier, the robotic computing device may request additional information about the object to be located from the additional computing device. Next, the method 300 may proceed from operation 308 to operation 310.

[0053] The operation 310 may include receiving information regarding the request from the additional computing device. The additional information may be communicated via the same modality in which the robotic computing device communicated with the additional computing device. Alternatively or additionally, the additional computing device may communicate with the robotic computing device via a modality different from the modality in which the robotic computing device communicated with the additional computing device. For example, the robotic computing device may convey the request for the additional information via an audio output to the additional computing device. In some embodiments, audio output from the robotic computing device may be provided to the additional computing device via beamforming and / or by selectively directing audio and / or other output to a particular additional computing device. In some embodiments, the audio output of the robotic computing device may be rendered to embody a voice that is acceptable to a speaker identification ("ID") process utilized by the additional computing device with prior permission from the user. The voice may be selected by the user and / or based on prior interactions between the robotic computing device and the additional computing device. In some embodiments, the technique of beamforming output may be performed using one or more microphones and / or antennas that can detect the output provided by the robotic computing device. Based on the relative amplitude and / or phase of the signal detected at each microphone and / or antenna, each individual output interface (e.g., speaker, transmitter, etc.) may be adjusted to facilitate transmission to additional computing devices through constructive interference. In some embodiments, this beamforming technique may be utilized in response to the robotic computing device deciding to delegate a task (i.e., an action) to and / or seek additional information from another computing device.

[0054] In response to the audio output from the robotic computing device, the additional computing device may render an image on its display panel. The robotic computing device may then capture the image of the display panel and / or download the image via a wireless communication protocol (e.g., Bluetooth, Wi-Fi, etc.). Based on the additional information provided by the additional computing device, the method 300 may then proceed from operation 310 and return to operation 304 to determine whether to delegate an action to another device, perform an action, and / or request further information from another device.

[0055] 4 is a block diagram 400 of an example computer system 410. Computer system 410 typically includes at least one processor 414 that communicates with a number of peripheral devices via a bus subsystem 412. These peripheral devices may include, for example, a storage subsystem 424 including memory 425 and a file storage subsystem 426, a user interface output device 420, a user interface input device 422, and a network interface subsystem 416. The input and output devices allow a user to interact with computer system 410. Network interface subsystem 416 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0056] The user interface input devices 422 may include a keyboard, a pointing device (e.g., a mouse, a trackball, a touchpad, a graphics tablet, etc.), a scanner, a touch screen integrated into a display, a voice input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to encompass all possible types of devices and methods for inputting information into the computer system 410 or a communications network.

[0057] The user interface output devices 420 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a flat panel device, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or other mechanism for producing a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. In general, use of the term "output device" is intended to encompass all possible types of devices and methods for outputting information from computer system 410 to a user or to another machine or computer system.

[0058] The storage subsystem 424 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 424 may include logic to perform selected aspects of the method 300 and / or logic to implement one or more of the robotic computing device 104, the system 200, and / or any other applications, devices, apparatus, and / or modules described herein.

[0059] These software modules typically execute on the processor 414 alone or in combination with other processors. The memory 425 used by the storage subsystem 424 may include multiple memories, such as a main random access memory (RAM) 430 for storing instructions and data during program execution, and a read-only memory (ROM) 432 in which fixed instructions are stored. The file storage subsystem 426 may provide persistent storage of program files and data files, and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules that perform the functions of a particular implementation may be stored by the file storage subsystem 426 in the storage subsystem 424, or may be stored on other machines accessible by the processor(s) 414.

[0060] Bus subsystem 412 provides a mechanism that allows the various components and subsystems of computer system 410 to communicate with each other as intended. Although bus subsystem 412 is shown generally as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0061] Computer system 410 can be of many different types, such as a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Because computer and network capabilities are constantly changing, the illustration of computer system 410 shown in Figure 4 is intended as only one example to illustrate some embodiments. Many other configurations of computer system 410 are possible, having more or fewer components than the computer system shown in Figure 4.

[0062] Where the systems described herein collect or may utilize personal information about users (or "participants" as they are often referred to herein), the user may be provided with an opportunity to control whether the program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location, etc.) or may be provided with an opportunity to control whether and / or how the user receives content from content servers that is more relevant to the user. Additionally, certain data may be processed in one or more ways before storage or use to remove any personally identifiable information. For example, the user's identity may be processed to make personal information non-identifiable to the user, or, where geographic location information (e.g., to the city, zip code, or state level, etc.) may be obtained, the user's geographic location may be generalized to make it non-identifiable to the user's specific geographic location. Thus, users may control how information about them is collected and used.

[0063] Although several embodiments have been described and illustrated herein, various other means and / or structures can be utilized to perform the functions described herein and / or obtain the results and / or one or more advantages described herein, and each such variation and / or modification is deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are for illustrative purposes, and the actual parameters, dimensions, materials, and / or configurations will vary depending on the particular application(s) for which the disclosure is utilized. Those skilled in the art will recognize, or be able to ascertain, using no more than routine experimentation, many equivalents to the specific embodiments described herein. It should thus be understood that the above-described embodiments are presented by way of example only, and that, within the scope of the appended claims and their equivalents, embodiments other than as specifically described and claimed may be practiced. Embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is within the scope of the disclosure, unless such features, systems, articles, materials, kits, and / or methods are mutually inconsistent.

[0064] In some embodiments, a method is provided that is executed by one or more processors, the method including receiving a user input at the robot computing device requesting the robot computing device to locate a particular object within an environment of the user and the robot computing device. The method may further include determining whether the robot computing device is able to locate the particular object within the environment with a threshold level of confidence in response to the user input. If the robot computing device is unable to locate the particular object within the environment with the threshold level of confidence, the method may further include moving the robot computing device within the environment towards another location closer to an additional computing device based on the robot computing device not locating the particular object with the threshold level of confidence. The method may further include generating a robotic input by the robot computing device and having the robotic input provided by the robotic computing device as an input to the additional computing device, the robotic input including content requesting information associated with the particular object from the additional computing device. The method may further include receiving, by the robotic computing device, a response output from the other computing device in response to the robotic input, characterizing particular information associated with the particular object. The method may further include having the robotic computing device locate the particular object within the environment for the user based on the particular information.

[0065] These and other embodiments of the technology disclosed herein may include one or more of the following features.

[0066] In some embodiments, causing the robotic computing device to locate a particular object may include moving the robotic computing device to another location within the environment to facilitate locating the particular object.

[0067] In some embodiments, receiving the response output from the additional computing device may include capturing an image of a display panel of the additional computing device, where the display panel is rendering specific information to the robotic computing device when the image is captured.

[0068] In some embodiments, determining whether the robotic computing device can locate a particular object in the environment with a threshold confidence level includes processing home graph data characterizing the positions of various features of the environment in which the user and the robotic computing device reside, and generating a confidence metric characterizing the probability that a particular one of the various features of the environment corresponds to a particular object, the confidence metric being compared to the threshold confidence level.

[0069] In some embodiments, the method may further include causing the robotic computing device to locate the particular object within the environment if the robotic computing device is able to locate the particular object within the environment with a threshold confidence.

[0070] In some embodiments, a method executed by one or more processors is provided, the method including receiving a user input at the robotic computing device embodying a request for the robotic computing device to assist in performing an operation. The robotic computing device is present in an environment with a user and one or more other computing devices. The method may further include determining, in response to the user input, whether an additional computing device of the one or more other computing devices indicates a state more suitable for initializing execution of the operation compared to a current state of the robotic computing device. The method may further include causing the robotic computing device to generate and provide as an input to the additional computing device a robotic input by the robotic computing device if it is determined that the additional computing device of the one or more other computing devices indicates a state more suitable for initializing execution of the operation. The robotic input includes content requesting the additional computing device to initialize execution of the operation. Providing the robotic input causes the additional computing device to initialize execution of the operation based on the robotic input.

[0071] These and other embodiments of the technology disclosed herein may include one or more of the following features.

[0072] In some embodiments, an additional computing device among the other computing device(s) is determined to be indicative of a more suitable state for initiating execution of an operation based on (a) the additional computing device being connected to a utility power source, and (b) the current state of the robotic computing device being a battery-powered state.

[0073] In some embodiments, an additional computing device among the other computing device(s) is determined to be indicative of a more suitable state for initiating execution of an operation based on the additional computing device indicating a signal strength that is greater than the signal strength indicated by the robotic computing device in the current state.

[0074] In some embodiments, an additional computing device among the other computing device(s) is determined to be indicative of a more suitable state for initiating execution of an operation based on the additional computing device indicating a processing bandwidth that is greater than the processing bandwidth indicated by the robot computing device in its current state.

[0075] In some embodiments, if it is determined that the additional computing devices do not indicate a state more suitable for initiating execution of the operation, the method further includes causing the robotic computing device to initialize execution of the operation.

[0076] In some embodiments, having the robotic input generated by the robotic computing device and provided as an input to the additional computing device includes having the robotic computing device render audio output via an audio interface of the robotic computing device.

[0077] In some embodiments, having the robotic computing device generate the robotic input and provide it as an input to the additional computing device includes determining a location of the additional computing device and having the robotic computing device render an audio output toward the location of the additional computing device. In some aspects of these embodiments, having the robotic computing device render the audio output via an audio interface of the robotic computing device includes repositioning the robot and / or one or more components of the robot such that one or more speakers of the robot that emit the audio output are oriented toward the location of the additional computing device. In some cases, having the robotic computing device render the audio output via an audio interface of the robotic computing device includes rendering the audio output at a frequency above the human audible frequency range.

[0078] In some embodiments, a method is provided that is executed by one or more processors, the method including receiving a user input at the robotic computing device embodying a request for the robotic computing device to assist in performing an operation. The robotic computing device is present in an environment with a user and one or more other computing devices. The method may further include, in response to the user input, determining whether an additional computing device of the one or more other computing devices can perform the operation more effectively than the robotic computing device due to a location of the additional computing device. If the robotic computing device determines that the additional computing device can perform the operation more effectively due to a location of the additional computing device, the method may further include providing a robotic input from the robotic computing device to the additional computing device to facilitate having the additional computing device perform the operation. The robotic input causes the additional computing device to initiate execution of the operation and perform the request embodied in the user input.

[0079] These and other embodiments of the technology disclosed herein may include one or more of the following features.

[0080] In some embodiments, determining whether a position of the additional computing device allows the additional computing device to perform an operation more effectively than the robotic computing device includes determining that a position of the additional computing device is closer to a particular area of ​​the environment than the robotic computing device, i.e., determining that a position of the additional computing device is closer to a particular area than a robot position of the robotic computing device.

[0081] In some embodiments, determining whether the location of the additional computing device allows the additional computing device to perform the operation more effectively than the robotic computing device includes determining that the location of the additional computing device is closer to another user in the environment than the robotic computing device, i.e., that the location of the additional computing device is closer to the other user than the robot location of the robotic computing device (i.e., that the robot location is farther from the other user than the location of the additional computing device). In these embodiments, fulfilling the request includes communicating with the other user.

[0082] In some embodiments, providing a robotic input from the robotic computing device to the additional computing device to facilitate causing the additional computing device to perform an operation includes rendering an audible output at the robotic computing device embodying natural language content that instructs the additional computing device to perform an operation. In some aspects of these embodiments, the additional computing device provides access to an automated assistant, and the natural language content includes an invocation phrase that invokes the automated assistant. In some of the invocation phrase embodiments, the method further includes identifying, by the robotic computing device, a particular type of the additional computing device, and including, by the robotic computing device, an invocation phrase in the natural language content in response to identifying the invocation phrase, if any, for the particular type. Identifying the particular type of the additional computing device may optionally include processing image data captured by a camera of the robotic computing device, and identifying the particular type based on processing the image data. In some additional or alternative aspects of these embodiments, the operation includes sending a message to another user, and the natural language content characterizes the message sent by the additional computing device to the other user.

Claims

1. 1. A method executed by one or more processors, comprising: Receiving user input at the robotic computing device requesting the robotic computing device to locate a particular object within an environment of the user and the robotic computing device; determining whether the robotic computing device is able to identify the location of the particular object within the environment with a threshold confidence in response to the user input; if the robotic computing device is unable to identify the location of the particular object within the environment with the threshold confidence; based on the robotic computing device not identifying the location with the threshold confidence, moving the robotic computing device within the environment towards another location closer to an additional computing device; causing, by the robotic computing device, a robotic input to be generated by the robotic computing device and provided by the robotic computing device as an input to the additional computing device, the robotic input including content requesting information associated with the particular object from the additional computing device; receiving, by the robotic computing device, a response output from the other computing device in response to the robotic input, the response output characterizing particular information associated with the particular object; causing the robotic computing device to determine, for the user, the location of the particular object within the environment based on the particular information; A method comprising:

2. Having the robotic computing device identify the location of the particular object comprises: The method of claim 1 , comprising moving the robotic computing device to another location within the environment to facilitate locating the particular object.

3. Receiving the response output from the additional computing device includes: capturing an image of a display panel of the additional computing device; The method of claim 1 , wherein the display panel is rendering the particular information to the robotic computing device when the image is captured.

4. Determining whether the robotic computing device is able to identify the location of the particular object within the environment with the threshold confidence comprises: Processing home graph data characterizing the positions of various features of the environment in which the user and the robotic computing device reside; generating a confidence metric based on processing the home graph data, the confidence metric characterizing the probability that a particular one of the various features of the environment corresponds to the particular object; comparing said confidence metric to said threshold confidence; The method according to any one of claims 1 to 3, comprising:

5. if the robotic computing device is able to identify the location of the particular object within the environment with the threshold confidence, causing the robotic computing device to determine the location of the particular object within the environment; The method of any one of claims 1 to 4, further comprising:

6. 1. A method executed by one or more processors, comprising: receiving a user input at a robotic computing device embodying a request for assistance in performing an operation from the robotic computing device, the robotic computing device being present in an environment along with the user and one or more other computing devices; determining, in response to the user input, whether additional computing devices of the one or more other computing devices indicate a state that is more suitable for initiating execution of the operation as compared to a current state of the robotic computing device; if it is determined that the additional computing device of the one or more other computing devices exhibits the state more suitable for initiating execution of the operation; causing, by the robotic computing device, a robotic input to be generated by the robotic computing device and provided as an input to the additional computing device, the robotic input includes content requesting the additional computing device to initiate execution of the operation; and Providing the robotic input includes causing the additional computing device to initiate and provide execution of the operation based on the robotic input; A method comprising:

7. 7. The method of claim 6, wherein the additional computing device of the one or more other computing devices is determined to indicate a more suitable state for initiating execution of the operation based on the additional computing device being connected to a utility power source and the current state of the robotic computing device being a battery-powered state.

8. 8. The method of claim 6 or 7, wherein the additional computing devices of the one or more other computing devices are determined to be indicative of a more suitable state for initiating execution of the operation based on the additional computing devices indicating a signal strength greater than a signal strength indicated by the robotic computing device in the current state.

9. 9. The method of claim 6, wherein the additional computing device of the one or more other computing devices is determined to indicate a more suitable state for initializing execution of the operation based on the additional computing device indicating a processing bandwidth greater than a processing bandwidth indicated by the robot computing device in the current state.

10. if it is determined that the additional computing devices of the one or more other computing devices do not exhibit a state more suitable for initiating execution of the operation, causing said robotic computing device to initiate execution of said operation. The method of any one of claims 6 to 9, further comprising:

11. Having the robotic input generated by the robotic computing device and provided as an input to the additional computing device comprises: A method according to any one of claims 6 to 10, comprising causing the robotic computing device to render an audio output via an audio interface of the robotic computing device.

12. Having the robotic input generated by the robotic computing device and provided as an input to the additional computing device comprises: determining a location of the additional computing device; and causing the robotic computing device to render the audio output toward the location of the additional computing device; The method of claim 11 , comprising:

13. 13. The method of claim 12, wherein causing the robotic computing device to render the audio output through the audio interface of the robotic computing device includes rendering the audio output at a frequency above a human audible frequency range.

14. Having the robotic computing device render the audio output directed to the location of the additional computing device comprises:

13. The method of claim 12, comprising repositioning the robot and / or one or more components of the robot so that one or more speakers of the robot emitting the audio output face towards the location of the additional computing device.

15. 1. A method executed by one or more processors, comprising: receiving a user input at a robotic computing device embodying a request for assistance in performing an operation with the robotic computing device, the robotic computing device being present in an environment along with a user and one or more other computing devices; determining, in response to the user input, whether an additional computing device of the one or more other computing devices can perform the operation more effectively than the robotic computing device depending on a position of the additional computing device; if the robotic computing device determines that the position of the additional computing device enables the additional computing device to more effectively perform the operation; providing a robotic input from the robotic computing device to the additional computing device to facilitate causing the additional computing device to perform the operation; providing the robot input to cause the additional computing device to initiate execution of the operation and carry out the request embodied in the user input; The method comprising:

16. Determining whether the position of the additional computing device allows the additional computing device to perform the operation more effectively than the robotic computing device, The method of claim 15 , comprising determining that the location of the additional computing device is closer to a particular region of the environment than the robotic computing device.

17. Determining whether the position of the additional computing device allows the additional computing device to perform the operation more effectively than the robotic computing device, determining that the location of the additional computing device is closer to another user in the environment than the robotic computing device; The method of claim 15 or 16, wherein fulfilling the request includes communicating with the other user.

18. providing the robotic input from the robotic computing device to the additional computing device to facilitate causing the additional computing device to perform the operation, The method of any one of claims 15 to 17, comprising rendering an audible output at the robotic computing device embodying natural language content that instructs the additional computing device to perform the operation.

19. the additional computing device providing access to an automated assistant; The method of claim 18 , wherein the natural language content includes an invocation phrase that invokes the automated assistant.

20. identifying, by the robotic computing device, a particular type of the additional computing device; in response to identifying the call phrase for the particular type, if any, including, by the robotic computing device, the call phrase in the natural language content; 20. The method of claim 18, further comprising:

21. Identifying the particular type of the additional computing device comprises: processing image data captured by a camera of the robotic computing device; identifying the particular type based on processing the image data; and 21. The method of claim 20, comprising:

22. 22. The method of claim 18, wherein the operation includes sending a message to another user, and the natural language content characterizes the message sent by the additional computing device to the other user.

23. one or more processors; A memory storing instructions which, when executed, cause the one or more processors to perform a method according to any one of claims 1 to 22; A mobile robot computing device comprising:

24. One or more computer readable storage media storing instructions that, when executed, cause one or more processors to perform the method of any one of claims 1 to 22.

Images