Assistant device arbitration using wearable device data

Abstract

Implementations presented herein relate to using data available from a wearable computing device, such as computerized eyewear, to implement device arbitration in a multi-device environment. The computerized eyewear can include a camera that can be used to provide image data to address issues related to device arbitration. In some implementations, the direction in which a user is directing their computerized eyewear and / or their gaze (as detected by the computerized eyewear with prior permission from the user) can be used to prioritize a particular device in a multi-device environment. The detected orientation of the computerized eyewear can also be used to determine how to simultaneously allocate content between a graphical display of the computerized eyewear and another graphical display of another client device. When content is allocated to the computerized eyewear, content-specific gestures can be enabled and actionable at the computerized eyewear.

Description

Background Technology

[0001] Humans can engage in human-computer dialogue with interactive software applications referred to herein as “automated assistants” (also known as “digital agents,” “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “assistant applications,” “conversational agents,” etc.). For example, humans (who may be referred to as “users” when interacting with automated assistants) can give commands / requests to automated assistants using spoken natural language input (i.e., utterances) that can in some cases be converted into text and then processed, and / or by providing textual (e.g., typed) natural language input.

[0002] Users can interact with the automation assistant using multiple client devices. For example, some users may have a coordinated "ecosystem" of client devices, comprising one or more smartphones, one or more tablets, one or more vehicle computing systems, one or more wearable computing devices, one or more smart TVs and / or one or more standalone interactive speakers, and other client devices. Users can use any of these client devices (assuming the automation assistant client is installed) to engage in human-machine dialogue with the automation assistant. In some cases, these client devices may be distributed around the user's primary residence, secondary residence, workplace, and / or other structures. For example, mobile client devices such as smartphones, tablets, smartwatches, etc., can be on the user and / or wherever the user last placed them. Other client devices, such as traditional desktop computers, smart TVs, and standalone interactive speakers, can be more fixed but may be located in various locations (e.g., rooms) within the user's home or workplace.

[0003] When a user has multiple automated assistant devices in their home, each device may operate in different states as a result of performing different actions. In such cases, a user might request modification to a specific action being performed at an assistant device, inadvertently causing different devices to modify different actions. This is possible in part because some assistant devices may depend solely on whether the user has heard the command to modify a specific action. Therefore, the adaptability of assistant devices to certain multi-assistant environments can be limited when the user does not speak directly to the device they are intended to interact with. For example, a user might accidentally initialize an action at an assistant device, potentially requiring the user to repeat previous verbal commands to re-invoke the action at the desired device.

[0004] Therefore, in response to an unexpected invocation of a specific assistant device, the memory and processing bandwidth of that assistant device may be temporarily consumed. Such seemingly redundant consequences can waste network resources, as some assistant input may be processed by a natural language model accessible only via a network connection. Furthermore, any data related to the unintentionally affected action must be re-downloaded to the intended device to facilitate the completion of the affected action, and any energy wasted in canceling energy-intensive actions (such as controlling display backlights, heating elements, and / or motorized equipment) may not be recoverable. Summary of the Invention

[0005] The implementation described herein relates to device arbitration technology, which involves processing data from computerized glasses worn by a user to identify the appropriate client device to which user input is directed. Allowing device arbitration to be performed using data from the computerized glasses can minimize the number of instances in which client devices are accidentally activated. In this way, memory, power, and network bandwidth can be reserved for those devices most likely to be accidentally activated by some detected user input.

[0006] In some implementations, a user can be positioned in an environment including multiple assistant-enabled devices, such as in the living room of a user's home. The assistant-enabled devices can be activated in response to user input, such as verbal input. Furthermore, the assistant-enabled devices can assist in device arbitration to identify specific computing devices that the user may have intended to invoke using the user input. When verbal input is provided, the user may be wearing computerized glasses, and the computerized glasses may include one or more cameras that can provide image data for detecting the direction the user may be facing. The identified direction can then be used during device arbitration to prioritize certain devices over others based on the direction the user is facing.

[0007] In some implementations, computerized glasses may include circuitry for detecting the position of a user's pupils to determine the user's gaze relative to a region and / or object in the environment. For example, computerized glasses may include a front-facing camera that can be used to identify the area the user is facing and a back-facing camera that can be used to identify the user's gaze. When a user provides input to an automated assistant accessible via multiple devices in the environment, the computerized glasses can provide information about the user's gaze to assist in device arbitration. For example, when a user is facing an area of ​​the environment that includes multiple assistant-enabled devices, the user may provide verbal utterances to the automated assistant. Data generated at the computerized glasses may be used to determine whether the user's gaze is more directed toward a particular assistant-enabled device compared to other assistant-enabled devices. When a particular device is selected based on the user's gaze, the automated assistant may respond to the user's verbal utterances at that particular device.

[0008] In some implementations, the assistant-enabled device, including a camera, can provide processable image data, as well as additional image data from the computerized glasses, to perform device arbitration. For example, visual characteristics of the user and / or environment can be determined from one or more cameras detached from the computerized glasses to determine whether a particular device should be prioritized during device arbitration. As an example, a user's appendage can be directed toward a particular device, but the appendage may not be visible in the viewing window of the computerized glasses' camera. However, the orientation of the appendage may be visible in the viewing window of a camera on another computing device (e.g., a standalone display device). In some instances, the user may be facing a specific area comprising two or more assistant-enabled devices and may be providing verbal utterances. While the user is providing verbal utterances, the user may concurrently have appendages (e.g., a hand or foot) directed toward a particular device among the two or more assistant-enabled devices. In such instances, image data from other computing devices (e.g., a camera from a standalone display device) and additional image data from the computerized glasses can be processed to select a particular device for responding to verbal utterances.

[0009] In some implementations, the computerized glasses can detect one or more different outputs (e.g., a first output, a second output, etc.) from one or more different assistant-enabled devices to determine the location and / or arrangement of the computerized glasses relative to one or more different devices. For example, to calibrate the computerized glasses for a specific user, the user could provide spoken words such as "Assistant, I'm looking at the kitchen display" while looking at the display interface of a computing device in their kitchen. In response, image data captured by the camera of the kitchen computing device and / or other image data captured by the computerized glasses can be processed to calibrate the computerized glasses for that specific user.

[0010] In some instances, this calibration operation can enhance the implementation of computerized glasses and / or other assistant-enabled devices, particularly when the user is looking at the assistant-enabled device, as the user may not typically position their head and / or face fully towards the device. Additionally, this calibration operation can enhance interaction between the user and other assistant-enabled devices that may not have an integrated camera and therefore may not be able to provide image data during device arbitration. For example, the user may be looking at a particular assistant-enabled device, but the device may not be within the external camera-facing viewing window of the computerized glasses. During device arbitration, with prior permission from the user, the gaze detected by the internal camera of the computerized glasses can be used to prioritize the assistant-enabled device the user is looking at over other devices (e.g., another device that may be in the external camera-facing viewing window).

[0011] In some implementations, calibration and / or device arbitration can be performed via one or more different modalities using communication between one or more assistant-enabled devices and the computerized glasses. For example, a standalone speaker device may include lights that can illuminate the front of the computerized glasses for a camera to detect the location of the standalone speaker relative to the computerized glasses. Alternatively or additionally, ultrasonic sounds may be emitted by one or more devices, such as the computerized glasses and / or one or more other assistant-enabled devices, to determine the location of the device relative to other devices. In some implementations, one or more lights on the device may be detected by the camera of the computerized glasses to determine whether the device has: lost connection, is no longer synchronized with another device, and / or is otherwise exhibiting a specific state that allows communication via said one or more lights. In this way, the computerized glasses can detect changes in the corresponding state of one or more devices while the user is wearing them.

[0012] In some implementations, the location of the device relative to the computerized glasses can be used to control certain features of the computerized glasses and / or one or more assistants to enable the device. For example, content being viewed on a computing device (e.g., a television) can be associated with content being rendered at the display interface of the computerized glasses. In some instances, a user may be watching a live stream of a sporting event on their television and may also be viewing comments from friends at the display interface of the computerized glasses. When the user moves away from the television and / or otherwise removes their gaze from the television, the content being rendered at the television can be rendered at the display interface of the computerized glasses, depending on the user's preference. For example, when a user moves away from the television in their living room to change their laundry in another room of their home, a change in the relative location of the computerized glasses and / or a change in the user's gaze can be detected. Based on this change in user location, additional content data can be rendered at the display interface of the computerized glasses. Alternatively or additionally, based on this change in user location, content reduction can be implemented at the television to conserve power and other computing resources, such as network bandwidth.

[0013] The above description is provided as an overview of some embodiments of this disclosure. Further descriptions of those and other embodiments are given below in more detail.

[0014] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more methods described above and / or elsewhere herein. Other implementations may include a system of one or more computers including one or more processors operable to execute stored instructions to perform methods such as one or more methods described above and / or elsewhere herein.

[0015] It should be understood that all combinations of the foregoing and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. Attached Figure Description

[0016] Figure 1A and 1B The illustration shows a view of a user invoking an automated assistant while wearing computerized glasses that assist in device arbitration.

[0017] Figure 2 The illustration shows a view of a user wearing computerized glasses according to some of the embodiments discussed herein.

[0018] Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E The illustration shows a view of a user interacting with an automated assistant that relies on computerized glasses for device arbitration.

[0019] Figure 4 The illustration depicts a system for performing device arbitration using data available from a device such as computerized glasses, which has the functionality to perform augmented reality.

[0020] Figure 5 The illustration depicts a method for enabling device arbitration in a multi-device environment using data available from wearable computing devices such as computerized glasses.

[0021] Figure 6 This is a block diagram of an example computer system. Detailed Implementation

[0022] Figure 1A and 1B Views 100 and 102 illustrate user 102 invoking an automation assistant while wearing computerized glasses 104 that assist in device arbitration. The computerized glasses 104 can assist in device arbitration by providing at least image data characterizing the user 102 and / or the computerized glasses 104's field of view 112 and / or the user 102's gaze. In this way, when assistant input is detected at multiple devices, data from the computerized glasses 104 can be used to identify the specific device to which the user 102 is directing assistant input. For example, the user 102 may be watching television 106 while seated in an environment 108 such as the user 102's living room. While watching television 106, the user 102's field of view 112 may include television 106, display device 118, and tablet device 110. In some embodiments, television 106 may include a computing device providing access to the automation assistant, or alternatively, a radar detector 126 (i.e., a detachable accessory device) may be attached to television 106 to render certain content on television 106. Display device 118 and tablet device 110 can also provide user 104 with access to the automation assistant.

[0023] When wearing computerized glasses 104, user 102 can provide verbal commands 114, such as “Assistant, play the movie I was watching last night.” User 102 can provide verbal commands 114 to modify the operation of television 106 and / or electronic dog 126. However, since user 102 is in an environment 108 with multiple assistant-enabled devices (e.g., tablet device 110 and display device 118), multiple different devices can detect verbal commands 114 from user 102. For example, each of tablet device 110, display device 118, television 106, and computerized glasses 104 can detect verbal commands 114 from user 102. Therefore, a device arbitration process can be initiated at one or more devices and / or remote computing devices to identify the specific device on which user 102 is directing verbal commands 114.

[0024] In some implementations, the device arbitration process may include identifying the device that detected the verbal utterance 114 from user 102 and determining whether any of the identified devices is associated with user 102's field of view 112. For example, the computing device performing the arbitration process may determine that television 106, display device 118, radar detector 126, and / or tablet device 110 are associated with user 102's field of view 112. This determination may be based on image data generated by one or more cameras of computerized glasses 104. The computing device may then determine that television 106 occupies a portion of the field of view 112 that is farther from the periphery or outer boundary of the field of view 112 compared to tablet device 110 and display device 118. Alternatively or additionally, the computing device performing the arbitration process may determine that image data from another camera on television 106 and / or tablet device 110 indicates that user 102's orientation is directed more towards television 106 compared to tablet device 110 and display device 118.

[0025] Based on one or more of these determinations, the computing device performing the device arbitration process can determine that the television 106 is the device to which the user 102 is directing the verbal utterance 114, instead of the display device 118. In some embodiments, the device arbitration process can result in the electronic dog 126 being selected as being influenced by the verbal utterance 114 when the television 106 is influenced by an automated assistant input due to the electronic dog 126. Therefore, even if the electronic dog 126 may be invisible to the user 102 and / or its field of view 112 may be hidden by the television 106, the electronic dog 126 can still respond to the verbal utterance 114 when the user 102 provides assistant input and when the user 102 is directing their gaze in the direction of the electronic dog 126. For example, the audio data corresponding to the verbal utterance 114 can be captured by the microphone of the tablet device 110 and another microphone of the display device 118, but the operation requested by the user 102 can be performed at the television 106 and / or the electronic dog 126 based on image data from the computerized glasses 104. For example, based on identifying a specific computing device in response to spoken words 114, television 106 can perform operation 116 of playing a movie according to spoken words 114 from user 102 instead of playing a movie on display device 118 or tablet device 110.

[0026] Figure 1B The illustration shows a view 120 of a user 102 changing their field of view 124 to face a computing device 122 located in a different area of ​​environment 108. The user 102 can change their orientation to direct spoken words from a television 106 to a different device. For example, when the user initially provides previous spoken words 114, the computing device 122 may be performing an operation 128 of playing music. Therefore, in order to turn off the music without affecting the movie on the television 106, the user 102 can direct their face and computerized glasses 104 more towards the computing device 122 than towards the television 106. In some embodiments, the computerized glasses 104 may include an outward-facing camera having a field of view 124 (i.e., a viewing window or visual angle), and the outward-facing camera may generate image data that can be used during device arbitration proceedings. Alternatively or additionally, the computerized glasses 104 may include an inward-facing camera, and the inward-facing camera may also generate image data that can be used during device arbitration proceedings.

[0027] For example, image data generated using an inward-facing camera can characterize that user 102's gaze is slightly directed upward toward computing device 122 and away from tablet device 110. In this way, although the microphone of computing device 122 does not detect verbal input as clearly as tablet device 110 or display device 118 (at least when user 102...), Figure 1BWhen positioned as depicted, image data from the inward-facing camera can direct the user 102's gaze at the computing device 122. For example, when the television 106 is playing a movie and the computing device 122 is playing music, the user 102 can provide verbal utterances 130, such as "Assistant, stop." To determine whether the computing device 122 is intended to be the target of verbal utterances 130, image data from one or more cameras of the computerized glasses 104 and / or data from one or more other devices can be processed during device arbitration.

[0028] In some implementations, during the device arbitration process, heuristic processes and / or one or more trained machine learning models may be used to select a particular device in response to user input. For example, one or more trained machine learning models may be used to process image data from an inward-facing camera and other image data from an outward-facing camera to identify the device to which the user input is directed. Alternatively or additionally, heuristic processes may be used to determine whether to prioritize a particular device over other candidate devices based on data from one or more sources. For example, a device not located in the user's and / or the computerized glasses' field of view may be considered to have a lower priority than another device identified as being in the user's and / or the computerized glasses' field of view. For example, when computing device 122 takes precedence over other devices in environment 108, one or more operations may be performed at computing device 122 to satisfy a user request embodied in spoken utterance 130. For example, computing device 122 may perform operation 132 to stop music from playing at computing device 122.

[0029] Figure 2The illustration shows a view 200 of a user 202 wearing computerized glasses 204 according to some embodiments discussed herein. The computerized glasses 204 may include a computer 208, which may include one or more processors and / or one or more memory devices, and may receive power from one or more power sources (e.g., batteries, wireless power transmission, etc.). The computer 208 may be implemented at least partially by a housing 214 and / or may be detachable from the housing 214. The housing 214 may resemble the frame of one or more different styles of glasses and may have one or more lenses 206 attached to the housing 214. In some embodiments, the computerized glasses 204 may include one or more front-facing cameras 210, which may be arranged to have a field of view corresponding to the user 202's field of view. In some embodiments, the computerized glasses 204 may include one or more inward-facing cameras 212, which may be arranged to have another field of view including one or more of the user 202's eyes. For example, one or more inward-facing cameras 212 may be arranged to capture image data characterizing the position of the user 202's left and / or right eyes. In some implementations, computer 208 may be connected to one or more antennas and / or other communication hardware that allows computer 208 to communicate with one or more other computing devices. For example, computer glasses 204 may be connected to a Wi-Fi network, an LTE network, and / or may communicate via Bluetooth and / or any other communication mode.

[0030] In some embodiments, the one or more lenses 206 may operate as a display interface for rendering graphical content visible to a user wearing computerized glasses 204. The graphical content rendered at the lenses 206 may assist device arbitration in response to multiple devices detecting input from the user 202. For example, the user 202 may guide their head and computerized glasses 204 in a direction that causes a first computing device and a second computing device to be within the field of view of the forward-facing camera 210. While the user 202 is guiding the computerized glasses 204 in this direction, the user may provide verbal commands, such as instructing a specific computing device to play music from a music application. An automated assistant may detect the verbal commands and, in response, render multiple instances of the music application's icon in the lenses 206. For example, a first instance of the music application icon may be rendered in the lens 206 above the first computing device, and a second instance of the music application icon may be rendered in the lens 206 above the second computing device.

[0031] In some implementations, each instance of the music app icon may be rendered in response to spoken words in a manner that indicates to the user that a particular device has not yet been selected. For example, each instance of the music app icon may be "grayed out," blurred, blinking, and / or otherwise have one or more features indicating that a device should be selected by the user. To select one of the devices, the user 202 may adjust the direction of their gaze and / or computer glasses 204 toward the first or second computing device. In response, the automation assistant may detect the adjustment of the user's gaze and / or orientation and cause the first or second instance of the music app icon to provide feedback that it has been selected. For example, when the user 202 directs their gaze and / or computer glasses 204 more toward the first computing device, the first instance of the music app may blink, jitter, become idle, no longer grayed out, no longer blurred, and / or otherwise indicate that the first computing device has been selected. In this way, the user 202 can receive feedback that they have selected a particular device and, if they prefer the second computing device, redirect their gaze and / or computer glasses 204. In some implementations, if user 202 is satisfied with their choice, user 202 may continue to watch the first computing device for a threshold time period, or move away from both computing devices, in order to confirm their choice and make the first computing device respond to verbal utterances.

[0032] In some implementations, the graphical content rendered at lens 206 can help clarify parameters of a specific request submitted by the user to an automation assistant and / or another application. For example, user 202 can provide verbal statements such as "Play some music," and in response, the automation assistant can render a first icon of a first music application and a second icon of a second music application in lens 206. The icons can be rendered at or near the specific computing device to which user 202 is directing their attention, and can be rendered to provide feedback to encourage user 202 to select the specific music application used to render the music. In some implementations, a timer can also be rendered at lens 206 to indicate the amount of time the user has before a specific music application is selected. For example, if user 202 does not provide additional input indicating whether they prefer one application over another, the automation assistant can render a specific icon to provide visual feedback indicating that the music application corresponding to that specific icon is selected by default.

[0033] In some implementations, the graphical content rendered at lens 206 may correspond to parameters provided to the application in response to assistant input from user 202. For example, in response to the spoken phrase "Play the new song," the automated assistant may render a first graphical element and a second graphical element in lens 206 at or near a specific audio device. The first graphical element may include text identifying the name of a first song, and the second graphical element may include text identifying the name of a second song. In this way, user 202 may notice that the spoken phrase they provide is somewhat ambiguous and that additional input may be needed to select a specific song. User 202 may then provide additional input (e.g., adjust their gaze, rotate their head, perform a gesture, tap shell 214 while gazing at a specific icon, provide another spoken phrase, and / or provide any other input) to specify a particular song. In some implementations, the graphical content rendered in lens 206 may be adjusted according to the change in orientation when a change in user 202's orientation is detected. For example, an icon rendered to appear above a computing device viewed by user 202 can be shifted in lens 206 in the opposite direction to the direction in which user 202 has rotated their head. Similarly, when the computing device is no longer in the field of view of user 202 and / or computer glasses 204, the icon can no longer be rendered in lens 206.

[0034] Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E Views 300, 320, 340, 360, and 380 illustrate user 302 interacting with an automated assistant that may rely on computerized glasses 304 for device arbitration. These figures illustrate at least one instance where user 302 causes certain operations to be performed on a computing device in a room and then repositions to a separate room, while maintaining the ability to provide assistant input to the computerized glasses 304 to control the operations. For example, user 302 may provide verbal utterances 312, such as “Assistant, play footage from the security camera from last night.” Verbal utterances 312 may include requests for the automated assistant to access a security application and to render video data from the security application on a display device accessible to the automated assistant.

[0035] In some implementations, to determine the specific device on which user 302 intends video data to be rendered, the automation assistant may cause one or more devices to provide one or more different outputs. When the user provides spoken words 312, the output can be detected by computerized glasses 304 worn by user 302. For example, the automation assistant may identify one or more candidate devices that have detected spoken words 312. In some implementations, the automation assistant may cause each candidate device to provide the output. The output used to provide at each candidate device may be different from the output used to provide at other candidate devices. In other words, each candidate device may be used to provide a corresponding unique output. In some implementations, the automation assistant causes the output at a candidate device to be rendered once or multiple times at the candidate device, and causes the duration of each rendering to be less than a given duration, such as less than a tenth of a second or less than fifty milliseconds. In some implementations of those and / or others, the output rendered at the candidate device may be output that might not be detectable by a human without a human modality. For example, the automation assistant may cause television 306 and tablet device 310 to incorporate images into one or more graphic content frames rendered at television 306 and tablet device 310, respectively. Frames including images can be rendered at a frequency greater than that detectable by humans (e.g., greater than or equal to 60 frames per second). In this way, if user 302 is facing television 306 when they provide verbal utterance 312, the front-facing camera of computerized glasses 304 can detect images in one or more frames without interrupting user 302. Alternatively or additionally, when both television 306 and tablet device 310 are within the front-facing camera and / or user 302's field of view, the automation assistant can determine that television 306 occupies more focus of user 302 and / or computerized glasses 304 than tablet device 310.

[0036] In some implementations, when user 302 provides verbal utterances, the automation assistant can determine that television 306 and tablet device 310 are close to computer glasses 304 and / or user 302. Based on this determination, the automation assistant can cause television 306 and tablet device 310 to display different images to identify devices within the field of view of the camera and / or computer glasses 304. The automation assistant can then determine whether one or more cameras on computer glasses 304 have detected one image instead of another to identify the specific device that user 302 is guiding their input to.

[0037] In some implementations, one or more devices in the environment may include LEDs, which may be controlled by each corresponding device and / or automation assistant during device arbitration. Light emitted from the LEDs can then be detected to select a specific device to respond to input from user 302. For example, in response to the detection of spoken words from user 302 at a first device and a second device, the automation assistant may illuminate a first LED of the first device and a second LED of the second device. When the automation assistant determines that light emitted from the first LED but not the second LED is being detected by computerized glasses 304, the automation assistant may select the first device in response to the spoken words. Alternatively or additionally, the automation assistant may illuminate each LED to exhibit certain characteristics to assist device arbitration. For example, the first LED may be illuminated such that the characteristics of the light emitted by the first LED differ from the characteristics of other light emitted by the second LED. Such characteristics may include color, amplitude, duration, frequency, and / or any other characteristics of the light that may be controlled by the application and / or device. For example, the automation assistant may illuminate the first LED for 0.1 seconds every 0.5 seconds and the second LED for 0.05 seconds every 0.35 seconds. In this way, one or more cameras of the computerized glasses 304 can detect these patterns of light, and the automated assistant can associate each detected pattern with a corresponding device. The automated assistant can then identify the LEDs that the user 302 directs their gaze and / or the computerized glasses 304 toward in order to select a specific device that will respond to the user's input.

[0038] In some instances, the automation assistant can determine that spoken words 312 are directed to the television 306 and can cause the television 306 to perform an operation 314 of playing camera footage from a security camera at the television 306. For example, the automation assistant can cause a security application to be accessed at the television 306 to render a security footage requested by the user 302. When the security application is launched, icons 330 identifying the security application and optional GUI elements 332 (i.e., graphical elements) can be rendered at the television 306. To identify certain features of the camera footage, the user 302 can listen to audio 316 from the television 306 and view video rendered at the display interface of the television 306. Furthermore, the user 302 can perform various physical gestures via computerized glasses 304 to control the television 306 and / or the security application. For example, the computerized glasses 304 may include an external camera 322 that can capture image data for processing by the automation assistant. For example, when user 302 performs a swipe gesture 326, the automation assistant can detect the swipe gesture 326 and cause the security application to perform a specific operation corresponding to the swipe gesture 326 (such as fast forward).

[0039] In some implementations, the user 302 can be repositioned to another room in their home, causing the computerized glasses 304 to operate in a manner that reflects the user 302's repositioning 344. For example, as Figure 3C and Figure 3D As illustrated, user 302 can reposition from environment 308 to a separate environment 362 to change their clothing 364 while also listening to audio 316 from a security application. Computerized glasses 304 can provide an interface for controlling the security application and / or an automated assistant in response to user 302 repositioning away from television 306. In some embodiments, the determination that user 302 has been repositioned may be based on data generated at one or more other devices in the computerized glasses and / or environment 308. In some embodiments, computerized glasses 304 may indicate that user 302 can control applications and / or devices they previously observed through computerized glasses 304. For example, an icon 330 representing a security application may be rendered in the display interface of computerized glasses 304. Alternatively or additionally, in response to user 302 redirecting their gaze and / or face away from television 306, optional GUI elements 332 and graphical elements 384 rendered at television 306 may be rendered in the display interface of computerized glasses 304.

[0040] In some implementations, device arbitration can be performed using computerized glasses 304 when user 302 may not be viewing the device and / or otherwise directs their attention to another computer device. For example, and as... Figure 3E Provided, user 302 can perform gesture 382 to provide input to an automation assistant, a specific application, and / or a specific computing device. One or more cameras of the computer glasses (e.g., outward-facing camera 322) can detect physical gestures, such as user 302 moving their hand from left position 368 to right. In response to detecting physical gesture 382, ​​the automation assistant can operate to identify one or more devices that detected the physical gesture. In some instances, the automation assistant can determine that only computer glasses 304 detected physical gesture 382. In any case, the automation assistant can determine whether physical gesture 382 is intended to initiate one or more operations at computer glasses 304 and / or separate devices.

[0041] In some instances, user 302 may be viewing content from a security application on the display interface of computer glasses 304. In such instances, the automation assistant can determine that physical gesture 382 is intended to influence the operation of the security application. In response to physical gesture 382, ​​the automation assistant can cause the security application to quickly forward certain content being rendered at computer glasses 304. Alternatively or additionally, the automation assistant can perform heuristic processes to identify the application and / or device to which user 302 is directing physical gesture 382. In some implementations, and utilizing prior permission from user 302, the automation assistant can determine that user 302 was recently looking at television 306, and prior to looking at television 306, user 302 was looking at tablet device 310. This determination can cause the automation assistant to prioritize television 306 over tablet device 310 when selecting the device that will respond to physical gesture 382.

[0042] Alternatively or additionally, the automation assistant can determine, based on image data from the in-face camera 324 of the computerized glasses 304, that the user 302 was recently looking at a security application on the television 306, and prior to looking at the security application, that the user 302 was looking at a social media application on the tablet device 310. Based on the identification of the security application and the social media application, the automation assistant can determine that a physical gesture 382 is acceptable as input to the security application but not as input to the social media application. Therefore, according to this process, the automation assistant can select a security application in response to a physical gesture input from the user 302.

[0043] In some implementations, device arbitration can be performed using one or more trained machine learning models that can be used to process application data and / or context data. For example, when user 302 provides physical gesture 382, ​​application data can characterize the operational status of one or more applications that may be associated with user 302. Alternatively or additionally, context data can characterize the features of the context in which user 302 provides physical gesture 382. Such features may include, but are not limited to, user 302's location, time of day, one or more of the user's activities (with prior permission from the user), and / or any other information that can be associated with user 302 when user 302 provides physical gesture 382. For example, audio data captured by one or more microphones of computerized glasses 304 and / or one or more other devices can be processed to identify contextual features of the environment with prior permission from user 302. For example, audio data captured from a movie can be used to assist an automation assistant in determining whether physical gesture 382 should affect an application rendering the movie. When the automation assistant determines that a physical gesture 382 is intended to affect a movie (e.g., fast-forwarding a part of the movie that the user 302 does not want to hear), the automation assistant can generate command data that can be passed to the application rendering the movie, without the user having to look at the TV 306 that is displaying the movie.

[0044] Figure 4 The illustration depicts a system 400 for performing device arbitration using data available from a device such as computerized glasses, the device having functionality for performing augmented reality. An automation assistant 404 may operate as part of an assistant application provided at one or more computing devices, such as computing device 402 and / or server device. A user can interact with the automation assistant 404 via an assistant interface 420, which may be a microphone, camera, touchscreen display, user interface, and / or any other device capable of providing an interface between the user and the application. For example, a user can initialize the automation assistant 404 by providing language, text, and / or graphical input to the assistant interface 420, causing the automation assistant 404 to initialize one or more actions (e.g., providing data, controlling peripheral devices, accessing agents, generating input and / or output, etc.). Alternatively, the automation assistant 404 may be initialized based on the processing of contextual data 436 using one or more trained machine learning models. Contextual data 436 may characterize one or more features of the environment accessible to the automation assistant 404 and / or be predicted as one or more features of the user intended to interact with the automation assistant 404.

[0045] Computing device 402 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control applications 434 of computing device 402 via the touch interface. In some embodiments, computing device 402 may lack a display device, thereby providing audible user interface output without providing graphical user interface output. Furthermore, computing device 402 may provide a user interface, such as a microphone, for receiving spoken natural language input from a user. In some embodiments, computing device 402 may include a touch interface and may not have a camera, but may optionally include one or more other sensors. In some embodiments, computing device 402 may provide augmented reality functionality and / or may be a wearable device, such as, but not limited to, computerized glasses, contact lenses, watches, clothing, and / or any other wearable device. Therefore, although various embodiments are described herein with respect to computerized glasses, the techniques disclosed herein can be implemented in conjunction with other electronic devices, including augmented reality functionality, such as other wearable devices that are not computerized glasses.

[0046] Computing device 402 and / or other third-party client devices can communicate with the server device via a network such as the Internet. Additionally, computing device 402 and any other computing devices can communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 402 can offload computing tasks to the server device to save computing resources at computing device 402. For example, the server device can host an automation assistant 404, and / or computing device 402 can transmit input received at one or more assistant interfaces 420 to the server device. However, in some embodiments, the automation assistant 404 can be hosted at computing device 402, and various processes associated with the operation of the automation assistant can be executed at computing device 402.

[0047] In various implementations, all or fewer aspects of the automation assistant 404 may be implemented on the computing device 402. In some of those implementations, aspects of the automation assistant 404 are implemented via the computing device 402 and may interface with a server device that may implement other aspects of the automation assistant 404. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations where all or fewer aspects of the automation assistant 404 are implemented via the computing device 402, the automation assistant 404 may be an application separate from the operating system of the computing device 402 (e.g., installed "on top" of the operating system), or alternatively, may be implemented directly by the operating system of the computing device 402 (e.g., considered an application of the operating system, but integrated with the operating system).

[0048] In some implementations, the automation assistant 404 may include an input processing engine 406, which may employ multiple different modules to process inputs and / or outputs from the computing device 402 and / or the server device. For example, the input processing engine 406 may include a voice processing engine 408, which can process audio data received at the assistant interface 420 to identify text embodied in the audio data. The audio data may be transferred from, for example, the computing device 402 to the server device to conserve computing resources at the computing device 402. Additionally or alternatively, the audio data may be processed exclusively at the computing device 402.

[0049] The process for converting audio data into text may include a speech recognition algorithm that employs neural networks and / or statistical models to identify groups of audio data corresponding to words or phrases. The text converted from the audio data may be parsed by a data parsing engine 410 and made available to the automation assistant 404 as text data that can be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some embodiments, the output data provided by the data parsing engine 410 may be provided to a parameter engine 412 to determine whether the user has provided input corresponding to a specific intent, action, and / or routine that can be performed by the automation assistant 404 and / or an application or agent that can be accessed via the automation assistant 404. For example, assistant data 438 may be stored at a server device and / or computing device 402 and may include data defining one or more actions that can be performed by the automation assistant 404, as well as parameters necessary to perform the actions. The parameter engine 412 may generate one or more parameters for intents, actions, and / or slot values ​​and provide said one or more parameters to an output generation engine 414. The output generation engine 414 can use one or more parameters to communicate with the automation assistant 420 to provide output to the user, and / or communicate with one or more applications 434 to provide output to one or more applications 434.

[0050] In some implementations, the automation assistant 404 may be an application that can be installed "on top" of the operating system of the computing device 402, and / or may form part (or all) of the operating system of the computing device 402 itself. The automation assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device performance. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model locally stored at the computing device 402. On-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. Furthermore, on-device natural language understanding (NLU) may be performed, for example, using an on-device NLU module that processes the recognized text generated using on-device speech recognition, along with optional context data, to generate NLU data.

[0051] NLU data may include the intent corresponding to the spoken utterance and optionally parameters (e.g., slot values) of the intent. On-device execution can be performed using an on-device execution module that utilizes NLU data (from on-device NLU) and optionally other local data to determine the action to be taken to resolve the intent of the spoken utterance (and optionally the parameters for resolving the intent). This may include determining local and / or remote responses (e.g., answers) to the spoken utterance, interactions with locally installed applications performed based on the spoken utterance, commands transmitted to Internet of Things (IoT) devices based on the spoken utterance (directly or via a corresponding remote system), and / or other resolving actions performed based on the spoken utterance. On-device execution can then initiate local and / or remote implementation / execution of the determined actions to resolve the spoken utterance.

[0052] In various implementations, remote speech processing, remote NLU, and / or remote execution can be used at least selectively. For example, recognized text can be transmitted at least selectively to a remote automation assistant component for remote NLU and / or remote execution. For example, recognized text can be optionally transmitted in parallel with on-device execution for remote execution, or in response to a failure of on-device NLU and / or on-device execution. However, on-device speech processing, on-device NLU, on-device execution, and / or on-device execution can be prioritized at least due to the reduced latency they provide when resolving spoken utterances (since no client-server round trips are required to resolve spoken utterances). Furthermore, on-device functionality can be the only functionality available in situations where there is no network connectivity or limited network connectivity.

[0053] In some implementations, computing device 402 may include one or more applications 434 that may be provided by a third-party entity different from the entity providing computing device 402 and / or automation assistant 404. The application state engine of automation assistant 404 and / or computing device 402 may access application data 430 to determine one or more actions that can be performed by the one or more applications 434, as well as the state of each of the one or more applications 434 and / or the state of the corresponding device associated with computing device 402. The device state engine of automation assistant 404 and / or computing device 402 may access device data 432 to determine one or more actions that can be performed by computing device 402 and / or one or more devices associated with computing device 402. Furthermore, application data 430 and / or any other data (e.g., device data 432) may be accessed by automation assistant 404 to generate context data 436, which may characterize the context in which a particular application 434 and / or device is performing and / or the context in which a particular user is accessing computing device 402, accessing application 434 and / or any other device or module.

[0054] While one or more applications 434 are executing at computing device 402, device data 432 can characterize the current operational state of each application 434 executing at computing device 402. Furthermore, application data 430 can characterize one or more features of the executing application 434, such as the content of one or more graphical user interfaces rendered in the direction of the one or more applications 434. Alternatively or additionally, application data 430 can characterize action patterns, which can be updated by the respective application and / or automation assistant 404 based on the current operational state of the respective application. Alternatively or additionally, one or more action patterns of one or more applications 434 can remain static but can be accessed by the application state engine to determine the appropriate action to be initialized via automation assistant 404.

[0055] The computing device 402 may further include an assistant invocation engine 422, which may use one or more trained machine learning models to process application data 430, device data 432, context data 436, and / or any other data accessible to the computing device 402. The assistant invocation engine 422 may process this data to determine whether to wait for the user to explicitly utter an invocation phrase to invoke the automation assistant 404, or to consider the data to indicate the user's intention to invoke the automation assistant—instead of requiring the user to explicitly utter an invocation phrase. For example, the one or more trained machine learning models may be trained using instances of training data based on scenarios in which the user is in various operational states across multiple devices and / or applications. Instances of training data may be generated to capture training data that characterizes scenarios in which the user invokes the automation assistant and other scenarios in which the user does not invoke the automation assistant.

[0056] When the one or more trained machine learning models are trained based on these instances of training data, the assistant invocation engine 422 can cause the automation assistant 404 to detect or restrict the detection of verbal invocation phrases from the user based on features of the context and / or environment. Additionally or alternatively, the assistant invocation engine 422 can cause the automation assistant 404 to detect or restrict the detection of one or more assistant commands from the user based on features of the context and / or environment. In some embodiments, the assistant invocation engine 422 can be disabled or restricted based on the computing device 402 detecting assistant suppression output from another computing device. In this way, when the computing device 402 is detecting assistant suppression output, the automation assistant 404 will not be invoked based on the context data 436—otherwise, if assistant suppression output is not being detected, the automation assistant 404 would be invoked.

[0057] In some implementations, system 400 may include a device arbitration engine 416 that can assist in performing device arbitration when one or more devices and / or applications detect input from a user. For example, in some implementations, device arbitration engine 416 may process data from one or more different devices to determine whether to initiate a device arbitration process. In some implementations, the data may be received via a network connection, one or more interfaces of system 400, and / or by any other mode through which the computing device receives data. For example, device arbitration engine 416 may determine that, in response to assistant input, multiple different devices are projecting ultrasonic sound and / or light. Based on this determination, device arbitration engine 416 may initiate a process for selecting a specific device among the multiple different devices that will respond to assistant input from the user.

[0058] In some implementations, system 400 may include a gaze detection engine 418 that can determine a user's gaze relative to one or more different objects in the user's environment. For example, system 400 may be computerized glasses that include an inward-facing camera directed to one or more of the user's eyes. Based on image data generated using the inward-facing camera, system 400 can identify a specific area in the environment that the user is directing their eyes towards. In some implementations, gaze detection engine 418 may determine the user's gaze direction based on data from one or more different devices, such as separate computing devices including cameras. Image data from the separate computing devices may indicate the user's posture and / or the direction in which the user is directing their attention towards one or more objects, with the user's prior permission. In this way, gaze detection engine 418 can determine the direction in which the user is directing their attention before, during, and / or after the user provides input to automation assistant 404.

[0059] In some embodiments, system 400 includes a field-of-view engine 426 that can process data characterizing the field of view of one or more cameras representing the user and / or device. For example, field-of-view engine 426 can process image data from one or more cameras of the computerized glasses to identify one or more objects and / or devices located within the camera's field of view at one or more temporal instances. In some embodiments, field-of-view engine 426 can also process device data 432 to identify certain objects that can be associated with certain devices in the user's field of view. For example, a kitchen sink can be an object associated with the user's independent display device. Therefore, when the user provides user input, and when the kitchen sink is identified in the user's field of view, field-of-view engine 426 can determine that the independent computing device is affected by the user input.

[0060] In some implementations, system 400 may include an interface content engine 424 for rendering content on one or more interfaces of system 400 based on output from device arbitration engine 416. For example, when device arbitration engine 416 identifies a computing device 402 affected by input from a user, interface content engine 424 may render content at one or more interfaces of computing device 402 (e.g., the display interface of computerized glasses). When device arbitration engine 416 determines that the user has directed input to a separate computing device, and the separate computing device is within the user's field of view, interface content engine 424 may render notifications for the separate computing device. For example, when computing device 402 is computerized glasses, interface content engine 424 may render graphical content at the display interface of computing device 402. Graphical content may be rendered such that it appears "on top" and / or near the separate computing device in the user's field of view (e.g., in an area of ​​the glasses lens corresponding to the location of the separate device). Graphical content may include, but is not limited to, one or more icons, colors and / or other graphic features, which may indicate that the device arbitration engine 416 has selected a separate computing device as a response to input from the user.

[0061] In some implementations, the device arbitration engine 416 may request additional input from the user to help identify a specific device the user intends to influence in response to the input. The device arbitration engine 416 may pass the identifier and / or location of candidate devices to the interface content engine 424, which may render a graphical indication in the lenses of the computerized glasses at or near the relative location of the candidate device. For example, a first optional element may be rendered at the leftmost portion of the computerized glasses' display interface to indicate that a computing device in the leftmost portion of the user's field of vision is a candidate device. A second optional element may be rendered simultaneously with the first optional element at a more central location in the display interface to indicate that another computing device in the central portion of the user's field of vision is also a candidate device. The user may then perform a gesture (e.g., raising their index finger) in front of their face, which will be captured by one or more cameras on the computerized glasses. This gesture may indicate to the automation assistant 404 that a specific device (e.g., the central device) is the device the user intends to respond to in response to user input.

[0062] Figure 5The illustration depicts a method 500 for implementing device arbitration in a multi-device environment using data available from wearable computing devices such as computerized glasses. Method 500 can be performed by one or more computing devices, applications, and / or any other means or modules that may be associated with an automated assistant. Method 500 may include an operation 502 determining whether assistant input has been detected. Assistant input may be user input provided by a user to one or more computing devices that provide access to the automated assistant. In some embodiments, assistant input may be provided to computing devices with augmented reality functionality, such as computerized glasses, computerized contact lenses, telephones, tablets, portable computing devices, smartwatches, and / or any other computing devices that can enhance the perception of one or more users. It should be noted that in embodiments including computerized glasses discussed herein, computerized glasses can be any computing device that provides augmented reality functionality. When assistant input is detected, method 500 can proceed from operation 502 to operation 504. Otherwise, the automated assistant may continue to determine whether the user has provided input to the automated assistant.

[0063] Operation 504 may be an optional operation that includes determining whether assistant input is detected at multiple devices. For example, a user may be wearing a pair of computerized glasses while simultaneously looking at a separate speaker device that provides access to an automated assistant. Therefore, when the user provides assistant input, the assistant input may be detected at the computerized glasses, the separate speaker device, and one or more other devices in the user's environment. When a single computing device has specifically detected the assistant input, method 500 may proceed from operation 504 to operation 512, where the specific computing device that detected the assistant input is used to initiate the execution of one or more operations. However, when multiple devices have detected the assistant input, method 500 may proceed from operation 504 to operation 506 to initialize device arbitration for selecting one or more devices to respond to the assistant input.

[0064] Operation 506 may include identifying candidate devices that detect assistant input from the user. For example, when the assistant input is spoken utterance, the automated assistant may identify multiple devices that capture audio data corresponding to the spoken utterance. In some implementations, each candidate device may capture audio data and process the audio data to generate a score for the assistant input. Device arbitration may be initialized for those candidate devices when each score on each candidate device meets a threshold. For example, candidate devices may include computer glasses, stand-alone speaker devices, and tablet devices that the user may have placed on a table near the user. Method 500 may proceed from operation 506 to operation 508 to further determine whether the assistant input is directed to a specific computing device.

[0065] Operation 508 may include processing data generated at the candidate devices and / or computerized glasses. For example, the computerized glasses may include an outward-facing camera and / or an inward-facing camera that can be used to generate image data to identify a specific device that the user may be directing assistant input to. The image data generated using the outward-facing camera may capture images including one of the candidate devices and / or objects associated with the candidate device. For example, a stand-alone speaker device may be supported by a decorative table that the user is viewing when providing assistant information. Therefore, when the decorative table and / or stand-alone speaker device is determined to be located in the user's and / or the computerized glasses' viewing window, the automated assistant may determine that assistant input is directed to the stand-alone speaker device. Alternatively or additionally, one or more of the candidate devices may provide output that can be detected by one or more sensors of the computerized glasses. Therefore, when the computerized glasses detect output from a specific device among the candidate devices, the automated assistant may determine that assistant input is directed to that specific device.

[0066] Method 500 can proceed to operation 510, which may include determining whether the assistant input is directed to a specific candidate device based on the processing at operation 508. When the automation assistant determines that the assistant input is directed to a specific candidate device other than computer glasses, method 500 can proceed from operation 510 to operation 512. Alternatively, when the automation assistant and / or another application determines that the assistant input is not directed to a specific candidate device, method 500 can proceed from operation 510 to operation 514.

[0067] Operation 514 may include determining whether system input is directed to the computerized glasses and / or another computing device providing augmented reality functionality. For example, while a user is handwriting in a notebook while wearing the computerized glasses, the user may provide spoken words. Therefore, although other candidate devices may have detected the user's spoken words, no other candidate device may be visible in the viewing window of the computerized glasses. Thus, the automation assistant can determine the user's intent and respond to the spoken words via the computerized glasses. For example, while the user is looking at a notebook and wearing the computerized glasses, the user may request the automation assistant to check the spelling of words written in the notebook. In response, the automation assistant may cause the computerized glasses to render one or more graphical elements indicating whether the word is correctly spelled in the notebook via augmented reality and / or provide audible output via one or more speakers of the computerized glasses.

[0068] When the automated assistant determines that assistant input has been directed to the computerized glasses, method 500 can proceed from operation 514 to operation 512. Otherwise, method 500 can proceed from operation 514 to operation 516. Operation 516 may be an optional operation, which includes requesting additional input from the user to assist the automated assistant in identifying the specific computing device to which the user's intent assistant input has been directed. Thereafter, method 500 can proceed from operation 516 to operation 502 and / or operation 504.

[0069] Figure 6 This is a block diagram 600 of an example computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow users to interact with the computer system 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0070] User interface input device 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or drawing tablet, a scanner, a touchscreen incorporated into a display, audio input devices such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to include all possible types of devices and methods for inputting information into computer system 610 or into a communication network.

[0071] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or other mechanisms for creating visible images. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of devices and methods for outputting information from computer system 610 to a user or to another machine or computer system.

[0072] Storage subsystem 624 stores the functional programming and data construction of some or all of the modules described herein. For example, storage subsystem 624 may include selected aspects of performing method 500 and / or implementing system 400, computer glasses 104, television 106, tablet device 110, computing device 122, computer glasses 204, television 306, computer glasses 304, tablet device 310, computing device 342 and / or one or more of any other applications, devices, apparatuses and / or modules discussed herein.

[0073] These software modules are typically executed by processor 614 alone or in combination with other processors. Memory 625 in storage subsystem 624 may include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 where fixed instructions are stored. File storage subsystem 626 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Functional modules implementing specific embodiments may be stored by file storage subsystem 626 in storage subsystem 624 or in other machines accessible to processor 614.

[0074] Bus subsystem 612 provides mechanisms for allowing various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0075] Computer system 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly changing nature of computers and networks, for the purposes of illustrating some embodiments, Figure 6 The description of the computer system 610 depicted is intended only as a concrete example. Figure 6 Compared to the computer system described in the diagram, computer system 610 has many other configurations with more or fewer components.

[0076] In situations where the system described herein collects or can utilize personal information about a user (or generally referred to herein as a "participant"), the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Furthermore, before specific data is stored or used, said specific data may be disposed of in one or more ways such that personally identifiable information is removed. For example, the user's identifier may be disposed of such that no personally identifiable information can be used to identify the user, or the geographic location information obtained from it may be generalized (e.g., to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Therefore, the user may have control over how information is collected and / or used regarding the user.

[0077] While several embodiments have been described and illustrated herein, various other components and / or structures may be utilized for performing functions and / or obtaining results and / or one or more of the advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications in which the teachings are used. Many equivalents of the specific embodiments described herein will be recognized or determined by those skilled in the art using no more than conventional experimentation. Therefore, it is to be understood that the foregoing embodiments are presented by way of example only, and that embodiments may be practiced in ways different from those specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of this disclosure pertain to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

[0078] In some embodiments, a method implemented by one or more processors is stated to include operations such as determining that a user has directed assistant input to an automated assistant accessible via any one of a plurality of computing devices connected to a network, wherein the user is wearing computerized glasses, which are computing devices including one or more cameras, and wherein the user is in an environment including the plurality of computing devices. The method may further include identifying two or more candidate devices among the plurality of computing devices that detected the assistant input from the user, based on the assistant input from the user, wherein the two or more candidate devices are separate from the computerized glasses. The method may further include determining, based on image data generated by processing the one or more cameras using the computerized glasses, whether the assistant input was directed to a specific computing device among the two or more candidate devices or to the computerized glasses. In some embodiments, the method may further include causing the specific computing device to perform one or more operations corresponding to the assistant input when the assistant input is determined to be directed to a specific computing device among the two or more candidate devices.

[0079] In some implementations, determining whether assistant input is directed to a specific computing device among the two or more candidate devices or to the computer glasses includes: determining, based on image data, whether the specific computing device among the two or more candidate devices is positioned within the viewing window of the one or more cameras. In some implementations, the cameras of the one or more cameras of the computer glasses are directed toward the user's eyes, and determining that the specific computing device among the two or more candidate devices is associated with the user's visual perspective includes: determining that the user's gaze is directed more toward the specific computing device than toward any other device among the two or more candidate devices.

[0080] In some embodiments, determining whether assistant input is directed to a specific computing device among the two or more candidate devices or to the computer glasses includes: determining whether a specific object in the viewing window of the one or more cameras is associated with the relative position of the specific computing device among the two or more candidate devices. In some embodiments, the specific computing device is not located in the viewing window of the one or more cameras of the computer glasses. In some embodiments, assistant input is a physical gesture performed by a user, and the physical gesture is detected by the computer glasses. In some embodiments, the computer glasses include a graphical display interface that is rendering content when the user provides assistant input, and the method further includes: when the assistant input is determined to be directed to the computer glasses: modifying the content being rendered at the graphical display interface of the computer glasses according to the physical gesture.

[0081] In some embodiments, the method further includes: when assistant input is determined to be directed to computerized glasses and a specific computing device, rendering a first portion of the content at the specific computing device and rendering a second portion of the content at the display interface of the computerized glasses. In some embodiments, the specific computing device is a detachable accessory device connected to the display device, and the specific computing device is not visible in the viewing windows of one or more cameras of the computerized glasses. In some embodiments, the computerized glasses include a graphical display interface that is at least partially transparent while the graphical display interface is rendering content, and wherein the method further includes: when assistant input is determined to be directed to a specific computing device among the two or more candidate devices, rendering graphical elements at positions in the graphical display interface corresponding to the specific computing device.

[0082] In some implementations, assistant input includes a request for an automated assistant to initialize a specific application, and the graphical elements are application-specific. In some implementations, identifying two or more candidate devices among a plurality of computing devices that have detected assistant input from a user includes determining that a particular computing device is rendering a first output, and another computing device among the plurality of computing devices is rendering a second output, wherein the first and second outputs are detected by computerized glasses.

[0083] In some embodiments, a particular computing device includes a graphical user interface (GUI), and the first output includes graphical elements rendered at the GUI, and the graphical elements are embodied in one or more graphical content frames rendered at a frequency of 60 frames per second or greater. In some embodiments, the first output is different from the second output, and determining whether the assistant input is directed to a particular computing device among the two or more candidate devices or to computer glasses includes determining that the first output is detected within the viewing window of the computer glasses, and the second output is not detected within the viewing window of the computer glasses.

[0084] In other embodiments, a method implemented by one or more processors is stated to include operations such as determining by a computing device that a user has provided input to an automation assistant, the automation assistant being accessible via one or more computing devices located in an environment with the user, wherein the input corresponds to a request from the automation assistant to provide content to the user, and wherein said one or more computing devices include computerized glasses worn by the user when the input is provided. The method may further include operations such as identifying a specific device for rendering content for the user based on the input from the user, wherein the specific device is detached from the computerized glasses. The method may further include operations such as causing the specific device to render content for the user based on the identified specific device. The method may further include operations such as processing context data provided by said one or more computing devices in the user's environment when the user provides input. The method may further include operations such as determining, based on the context data, whether to provide the user with additional content associated with the request. The method may further include operations such as causing the computerized glasses to perform one or more additional operations to facilitate the rendering of the additional content via one or more interfaces of the computerized glasses when the automation assistant determines to provide additional content to the user.

[0085] In some implementations, the context data includes image data provided by one or more cameras of the computerized glasses, and determining whether to provide the user with additional content associated with the request includes: determining whether the user is viewing a particular device while the particular device is performing the one or more operations. In some implementations, causing the computerized glasses to render the additional content includes: causing the computerized glasses to access the content data via a network connection, and causing the display interface of the computerized glasses to render one or more graphical elements based on the content data.

[0086] Still in other embodiments, the method implemented by one or more processors is stated to include operations such as determining by a computing device that a user provides input to an automated assistant, which is accessible via the computing device, wherein the input corresponds to a request for the automated assistant to perform one or more operations. The method may further include receiving by the computing device context data indicating that the user is wearing computerized glasses, wherein the computerized glasses are detached from the computing device. The method may further include rendering output from an interface of the computing device based on the context data, which may be detected at another interface of the computerized glasses. The method may further include determining whether the computerized glasses have detected output from the computing device. The method may further include, when the computing device determines that the computerized glasses have detected output: causing the computing device to perform the one or more operations to facilitate the fulfillment of the request.

[0087] In some embodiments, determining whether the computerized glasses have detected output from a computing device includes processing additional contextual data indicating whether one or more cameras of the computerized glasses have detected output from the computing device. In some embodiments, the method may further include the following operation: when the computing device determines that the computerized glasses have detected output: causing the computerized glasses to render one or more graphical elements, which can be selected in response to a physical gesture from a user.

Claims

1. A method implemented by one or more processors, the method comprising: It is determined that the user has directed assistant input to an automated assistant, which can be accessed via any of a plurality of computing devices connected to the network. The user is wearing computerized glasses, which are computing devices including one or more cameras. The user is located in an environment that includes the plurality of computing devices; Based on the assistant input from the user, identify two or more candidate devices among the plurality of computing devices that detected the assistant input from the user. Wherein, the two or more candidate devices are separated from the computerized glasses; Based on processing image data generated by the one or more cameras using the computerized glasses, determine whether the assistant input is directed to a specific computing device among the two or more candidate devices or to the computerized glasses; When the assistant input is determined to be directed to the specific computing device among the two or more candidate devices: To cause the specific computing device to perform one or more operations corresponding to the assistant input; When the assistant input is determined to be directed to the computerized glasses and the specific computing device: The first part of the content is rendered on the specific computing device, and The second part of the content is rendered on the display interface of the computerized glasses.

2. The method according to claim 1, wherein, Determining whether the assistant input is directed to the specific computing device among the two or more candidate devices or the computerized glasses includes: Based on the image data, determine whether the specific computing device among the two or more candidate devices is located in the viewing window of the one or more cameras.

3. The method according to claim 1, in, The camera in one or more of the computerized glasses is directed toward the user's eyes, and Specifically, determining that the assistant input is directed to a particular computing device among the two or more candidate devices includes: It is determined that the user's gaze is directed more towards the specific computing device than towards any other device among the two or more candidate devices.

4. The method according to claim 1, wherein, Determining whether the assistant input is directed to the specific computing device among the two or more candidate devices or the computerized glasses includes: Determine whether a specific object in the viewing window of the one or more cameras is associated with the relative position of the specific computing device among the two or more candidate devices.

5. The method according to claim 4, wherein, The specific computing device is not positioned within the viewing window of the one or more cameras of the computerized glasses.

6. The method according to claim 1, wherein, The assistant input is a physical gesture performed by the user, and the physical gesture is detected by the computerized glasses.

7. The method according to claim 6, in, The computerized glasses include a graphical display interface that renders content as the user provides input to the assistant, and The method further includes: When the assistant input is determined to be directed to the computerized glasses: The content being rendered at the graphical display interface of the computerized glasses is modified according to the physical gesture.

8. The method according to claim 1, in, The specific computing device is a detachable accessory device connected to the display device, and The specific computing device is not visible in the viewing window of the one or more cameras of the computerized glasses.

9. The method according to claim 1, in, The computerized glasses include a graphical display interface that is at least partially transparent while rendering content, and The method further includes: When the assistant input is determined to be directed to the specific computing device among the two or more candidate devices: The graphical elements are rendered at the positions in the graphical display interface corresponding to the specific computing device.

10. The method according to claim 9, in, The assistant input includes a request from the automated assistant to initialize a specific application, and The graphic elements are based on the specific application.

11. The method according to claim 1, wherein, The two or more candidate devices that identify the detection of the assistant input from the user among the plurality of computing devices include: It is determined that the specific computing device is rendering the first output, and another computing device among the plurality of computing devices is rendering the second output. The first output and the second output are detected by the computerized glasses.

12. The method according to claim 11, in, The specific computing device includes a graphical display interface, and the first output includes graphical elements rendered at the graphical display interface, wherein the graphical elements are embodied in one or more graphical content frames, the one or more graphical content frames being rendered at a frequency of 60 frames per second or greater.

13. The method according to claim 11, in, The first output is different from the second output, and Determining whether the assistant input is directed to a specific computing device among the two or more candidate devices or to the computerized glasses includes: It is determined that the first output is detected within the viewing window of the computerized glasses, and the second output is not detected within the viewing window of the computerized glasses.

14. A method implemented by one or more processors, the method comprising: It is determined that the user has directed assistant input to an automated assistant, which can be accessed via any of a plurality of computing devices connected to the network. The user is wearing computerized glasses, which are computing devices including one or more cameras. The user is located in an environment that includes the plurality of computing devices; Based on the assistant input from the user, identify two or more candidate devices among the plurality of computing devices that detected the assistant input from the user. Wherein, the two or more candidate devices are separated from the computerized glasses, Among them, the two or more candidate devices that identify the assistant input from the user among the plurality of computing devices include: It is determined that a specific computing device is rendering a first output and another computing device among the plurality of computing devices is rendering a second output. Wherein, the first output and the second output are detected by the computerized glasses, and Wherein, the first output is different from the second output; Based on processing image data generated by the one or more cameras using the computerized glasses, determining whether the assistant input is directed to a specific computing device among the two or more candidate devices or the computerized glasses, wherein determining whether the assistant input is directed to the specific computing device among the two or more candidate devices or the computerized glasses includes: It is determined that the first output is detected within the viewing window of the computerized glasses, and the second output is not detected within the viewing window of the computerized glasses; When the assistant input is determined to be directed to the specific computing device among the two or more candidate devices: The specific computing device is made to perform one or more operations corresponding to the assistant input.

15. The method according to claim 14, in, The camera in one or more of the computerized glasses is directed toward the user's eyes, and Specifically, determining that the assistant input is directed to a particular computing device among the two or more candidate devices includes: It is determined that the user's gaze is directed more towards the specific computing device than towards any other device among the two or more candidate devices.

16. The method according to claim 14, in, Determining whether the assistant input is directed to the specific computing device among the two or more candidate devices or the computerized glasses includes: Determine whether a specific object in the viewing window of the one or more cameras is associated with the relative position of the specific computing device among the two or more candidate devices.

17. The method according to claim 16, in, The specific computing device is not positioned within the viewing window of the one or more cameras of the computerized glasses.

18. The method according to claim 14, in, The assistant input is a physical gesture performed by the user, and the physical gesture is detected by the computerized glasses.

19. The method according to claim 18, in, The computerized glasses include a graphical display interface that renders content as the user provides input to the assistant, and The method further includes: When the assistant input is determined to be directed to the computerized glasses: The content being rendered at the graphical display interface of the computerized glasses is modified according to the physical gesture.

20. The method according to claim 14, in, The specific computing device is a detachable accessory device connected to the display device, and The specific computing device is not visible in the viewing window of the one or more cameras of the computerized glasses.

21. The method according to claim 14, in, The computerized glasses include a graphical display interface that is at least partially transparent while rendering content, and The method further includes: When the assistant input is determined to be directed to the specific computing device among the two or more candidate devices: The graphical elements are rendered at the positions in the graphical display interface corresponding to the specific computing device.

22. The method according to claim 21, in, The assistant input includes a request from the automated assistant to initialize a specific application, and The graphic elements are based on the specific application.

23. The method according to claim 14, in, The specific computing device includes a graphical display interface, and the first output includes graphical elements rendered at the graphical display interface, and The graphic elements are embodied in one or more graphic content frames, which are rendered at a frequency of 60 frames per second or higher.

24. A computer program comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 23.

25. A computing system configured to perform the method of any one of claims 1 to 23.

26. The computing system according to claim 25, wherein, The computing system includes computerized glasses.

Images