Assistant device arbitration using wearable device data
Computerized glasses with cameras enhance device arbitration in multi-assistant environments by determining the user's intended device based on gaze direction, reducing redundant processing and energy consumption.
Patent Information
- Application Number
- JP2024111040
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-02-03
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2041-12-06
AI Technical Summary
In environments with multiple automated assistant devices, user inputs can inadvertently activate incorrect devices due to reliance on audio detection, leading to redundant processing, network resource waste, and energy consumption, as well as the need for repeated verbal commands to correct actions.
Implementing computerized glasses that use image data from forward- and inward-facing cameras to determine the user's intended device by analyzing gaze direction and environmental context, facilitating device arbitration to prioritize correct device activation.
Reduces accidental device activation, conserves memory, power, and network bandwidth by accurately identifying the intended device, minimizing redundant processing and energy waste.
Smart Images

Figure 2024156688000001_ABST
Abstract
Description
[Technical field]
[0001] Assistant device reconciliation using wearable device data. [Background technology]
[0002] Humans may engage in human-to-computer interactions with interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant, in some cases using verbal natural language input (i.e., speech) that may be converted to text and then processed, and / or by providing textual (e.g., typed) natural language input.
[0003] A user may engage with an automated assistant using multiple client devices. For example, some users may own a coordinated "ecosystem" of client devices that comprises a combination of one or more smartphones, one or more tablet computers, one or more vehicle computing systems, one or more wearable computing devices, one or more smart televisions, and / or one or more independent interactive speakers, among other client devices. A user may engage in human-to-computer interactions with an automated assistant using any of these client devices (assuming an automated assistant client is installed). In some cases, these client devices may be scattered around the user's primary residence, secondary residence, workplace, and / or other buildings. For example, mobile client devices such as smartphones, tablets, smart watches, etc. may be on the user's body and / or where the user last left them. Other client devices such as traditional desktop computers, smart televisions, and independent interactive speakers may be more stationary but may be located in various locations (e.g., rooms) within the user's home or workplace.
[0004] When a user has multiple automated assistant devices in his / her home, each assistant device may have a different operating state as a result of performing different actions. At such times, the user may request to change a particular action in progress on the assistant device, but may mistakenly cause different assistant devices to change different actions. This may be in part because some assistant devices may only depend on whether the respective assistant devices heard the user say a command to change a particular action. As a result, the adaptability of the assistant device to a particular multi-assistance environment may be limited when the user is not speaking directly to the assistant device that is intended to interact with it. For example, the user may mistakenly initialize an action on the assistant device, thereby potentially requiring the user to repeat a previous verbal utterance to reinvoke the action on the desired device.
[0005] As a result, memory and processing bandwidth for a particular assistant device may be consumed momentarily in response to an erroneous invocation of the particular assistant device. Such seemingly redundant results may waste network resources, for example, because some assistant inputs may be processed by natural language models accessible only via a network connection. Furthermore, any data associated with the inadvertently affected action must be re-downloaded to the desired device in facilitating the completion of the affected action, and energy wasted from canceling energy-intensive actions (e.g., controlling display backlights, heating elements, and / or powered home appliances) may not be recoverable. Summary of the Invention [Means for solving the problem]
[0006] The implementations described herein relate to device arbitration techniques that involve processing data from computerized glasses worn by a user to identify the appropriate client device to which user input is directed. Allowing device arbitration to be performed using data from the computerized glasses can minimize the number of instances in which client devices are erroneously activated. In this way, memory, power, and network bandwidth can be conserved for those devices that are most susceptible to accidental activation from a particular detected user input.
[0007] In some implementations, a user may be in an environment that includes multiple assistant-enabled devices, such as the living room of the user's home. The assistant-enabled devices may be activated in response to a user input, such as a verbal utterance. Additionally, the assistant-enabled devices may assist device arbitration to identify a particular computing device that the user may have intended to invoke with the user input. The user may be wearing computerized glasses when providing the verbal utterance, and the computerized glasses may include one or more cameras that may provide image data to detect a direction in which the user may be facing. The identified direction may be used during device arbitration to prioritize a particular device over other devices based on the direction in which the user is facing.
[0008] In some implementations, the computerized glasses can include circuitry for detecting the position of the user's pupils to determine the user's gaze toward an area and / or object in the environment. For example, the computerized glasses can include a forward-facing camera that can be used to identify an area toward which the user is facing and a reverse-facing camera that can be used to identify the user's gaze. When a user provides input to an automated assistant accessible through multiple devices in the environment, the computerized glasses can provide information regarding the user's gaze to assist in device arbitration. For example, a user can provide verbal utterances to the automated assistant when the user is facing toward an area of the environment that includes multiple assistant-enabled devices. Data generated in the computerized glasses can be used to determine whether the user's gaze is more directed toward a particular assistant-enabled device compared to other assistant-enabled devices. Once a particular device is selected based on the user's gaze, the automated assistant can respond to the user's verbal utterances at the particular device.
[0009] In some implementations, an assistant-enabled device including a camera can provide image data that can be processed together with other image data from the computerized glasses to perform device arbitration. For example, visual features of the user and / or the environment can be determined from one or more cameras separate from the computerized glasses to determine whether to prioritize a particular device during device arbitration. As an example, a user's accessory may be pointed toward a particular device, but the accessory may not be visible within the field of view window of the camera of the computerized glasses. However, the orientation of the accessory may be visible within the field of view window of the camera of another computing device (e.g., a stand-alone display device). In some cases, a user may be facing toward a particular area that includes two or more assistant-enabled devices and may provide verbal utterances. A user may have an accessory (e.g., a hand and / or a foot) pointed toward a particular device of the two or more assistant-enabled devices at the same time that the user is providing verbal utterances. In such cases, image data from the other computing device (e.g., from the camera of the stand-alone display device) and other image data from the computerized glasses can be processed to select a particular device to respond to the verbal utterance.
[0010] In some implementations, the computerized glasses can detect one or more different outputs (e.g., a first output, a second output, etc.) from one or more different assistant-enabled devices to determine a position and / or location of the computerized glasses relative to the one or more different devices. For example, to calibrate the computerized glasses for a particular user, the user can provide a verbal utterance such as "Assistant, I'm looking at the kitchen display" while gazing at a display interface of a computing device in his or her kitchen. In response, images captured via the camera of the kitchen computing device and / or other image data captured via the computerized glasses can be processed to calibrate the computerized glasses for that particular user.
[0011] In some cases, this calibration operation can enhance the performance of the computerized glasses and / or other assistant-enabled devices, especially when the user gazes at the assistant-enabled device, typically when the user does not position his / her head and / or face completely toward the assistant-enabled device. In addition, this calibration operation can enhance the interaction between the user and other assistant-enabled devices that may not have an integrated camera and therefore may not be able to provide image data during device arbitration. For example, the user may be gazing at a particular assistant-enabled device, but the assistant-enabled device may not be within the field of view window of the outward-facing camera of the computerized glasses. The gaze detected by the inward-facing camera of the computerized glasses may be used, with prior permission from the user, to prioritize the assistant-enabled device that the user gazes at over other devices (e.g., another device that may be within the field of view window of the outward-facing camera) during device arbitration.
[0012] In some implementations, the calibration and / or device arbitration may be performed using communication between one or more assistant-enabled devices and the computerized glasses via one or more different modalities. For example, the independent speaker device may include a light that can be illuminated for the forward-facing camera of the computerized glasses to detect the position of the independent speaker relative to the computerized glasses. Alternatively or additionally, ultrasound may be emitted by one or more devices, such as the computerized glasses and / or one or more other assistant-enabled devices, to determine the position of the device relative to the other devices. In some implementations, one or more lights on the device may be detected by the camera of the computerized glasses to determine whether the device has lost connection, is no longer synchronized with the other devices, and / or otherwise indicates a particular state that may be communicated via one or more lights. In this manner, the computerized glasses may detect changes in the respective states of one or more devices when a user is wearing the computerized glasses.
[0013] In some implementations, the location of the device relative to the computerized glasses may be used to control certain functions of the computerized glasses and / or one or more assistant-enabled devices. For example, content being viewed on a computing device (e.g., a television) may be associated with content being rendered on the display interface of the computerized glasses. In some cases, a user may be watching a live stream of a sporting event on his / her television and may also be viewing comments from friends on the display interface of the computerized glasses. When the user moves away from the vicinity of the television and / or otherwise looks away from the television, the content that was being rendered on the television may be rendered on the display interface of the computerized glasses according to the user's preferences. For example, when a user moves away from the vicinity of the television in his / her living room to change his / her laundry in another room of his / her house, a change in the relative location of the computerized glasses and / or a change in the user's gaze may be detected. Based on this change in the user's location, additional content data may be rendered on the display interface of the computerized glasses. Alternatively or additionally, based on this change in the user's location, content reduction may be implemented on the television to conserve power and other computational resources such as network bandwidth.
[0014] The above description is provided as a summary of some implementations of the present disclosure. Further descriptions of these and other implementations are described in more detail below.
[0015] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more computer systems including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.
[0016] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are considered to be part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Brief description of the drawings]
[0017] [Figure 1A] FIG. 1 illustrates a user's view of invoking an automated assistant while wearing computerized glasses that can assist with device arbitration. [Figure 1B] FIG. 1 illustrates a user's view of invoking an automated assistant while wearing computerized glasses that can assist with device arbitration. [Diagram 2] FIG. 2 illustrates a view of a user wearing computerized glasses according to certain implementations discussed herein. [Figure 3A] FIG. 1 illustrates a view of a user interacting with an automated assistant that can rely on computerized glasses for device arbitration. [Figure 3B] FIG. 1 illustrates a view of a user interacting with an automated assistant that can rely on computerized glasses for device arbitration. [Figure 3C]FIG. 1 illustrates a view of a user interacting with an automated assistant that can rely on computerized glasses for device arbitration. [Figure 3D] FIG. 1 illustrates a view of a user interacting with an automated assistant that can rely on computerized glasses for device arbitration. [Figure 3E] FIG. 1 illustrates a view of a user interacting with an automated assistant that can rely on computerized glasses for device arbitration. [Figure 4] FIG. 1 illustrates a system for performing device arbitration using data available from a device, such as computerized glasses, having capabilities for performing augmented reality. [Diagram 5] FIG. 1 illustrates a method for performing device arbitration in a multi-device environment using data available from a wearable computing device, such as computerized glasses. [Figure 6] FIG. 1 is a block diagram of an exemplary computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] 1A and 1B show views 100 and 120, respectively, of a user 102 invoking an automated assistant while wearing computerized glasses 104 that can assist with device arbitration. The computerized glasses 104 can assist with device arbitration by providing at least image data characterizing the field of view 112 of the user 102 and / or the computerized glasses 104 and / or image data characterizing the gaze of the user 102. In this manner, when assistant input is detected at multiple devices, data from the computerized glasses 104 can be used to identify the particular device to which the user 102 is directing the assistant input. For example, the user 102 can be watching a television 106 while sitting in an environment 108, such as the user's 102 living room. While watching the television 106, the field of view 112 of the user 102 can include the television 106, a display device 118, and a tablet device 110. In some implementations, the television 106 can include a computing device that provides access to an automated assistant, or alternatively, a dongle 126 (i.e., a removable accessory device) can be attached to the television 106 to render certain content on the television 106. The display device 118 and the tablet device 110 can also provide the user 104 with access to an automated assistant.
[0019] While wearing the computerized glasses 104, the user 102 can provide verbal utterances such as, "Assistant, play the movie I was watching last night." The user 102 can provide verbal utterances 114 to modify the operation of the television 106 and / or the dongle 126. However, because the user 102 is located in an environment 108 having multiple assistant-enabled devices (e.g., tablet device 110 and display device 118), multiple different devices may detect the verbal utterances 114 from the user 102. For example, each of the tablet device 110, the display device 118, the television 106, and the computerized glasses 104 may detect the verbal utterances 114 from the user 102. As a result, a device arbitration process may be initiated in one or more of the devices and / or remote computing devices to identify the particular device to which the user 102 is directing the verbal utterances 114.
[0020] In some implementations, the device arbitration process may also include identifying devices that have detected verbal utterances 114 from the user 102 and determining whether any of the identified devices are associated with the field of view 112 of the user 102. For example, a computing device performing the arbitration process may determine that the television 106, the display device 118, the dongle 126, and the tablet device 110 are associated with the field of view 112 of the user 102. This determination may be based on image data generated by one or more cameras of the computerized glasses 104. The computing device may then determine that the television 106 occupies a portion of the field of view 112 that is further from the periphery or outer boundary of the field of view 112 than the tablet device 110 and the display device 118. Alternatively or additionally, the computing device performing the arbitration process may determine that image data from another camera of the television 106 and / or the tablet device 110 indicates that the orientation of the user 102 is more oriented toward the television 106 than the tablet device 110 and the display device 118.
[0021] Based on one or more of these determinations, the computing device performing the device arbitration process may determine that the television 106 is the device to which the user 102 directed the verbal utterance 114, instead of the display device 118. In some implementations, if the television 106 is the target of automated assistant input for the dongle 126, the device arbitration process may result in the selection of the dongle 126 as the target of the verbal utterance 114. Thus, even though the dongle 126 may be invisible to the user 102 and / or may be obscured from the field of view 112 by the television 106, the dongle 126 may nevertheless respond to the verbal utterance 114 when the user 102 is directing gaze in the direction of the dongle 126 when providing assistant input. For example, audio data corresponding to the verbal utterance 114 may be captured by a microphone of the tablet device 110 or another microphone of the display device 118, while an action requested by the user 102 may be performed at the television 106 and / or the dongle 126 based on image data from the computerized glasses 104. For example, based on identifying a particular computing device responsive to the verbal utterance 114, the television 116 may perform an action 116 of playing a movie per the verbal utterance 114 from the user 102, instead of the movie being played on the display device 118 or the tablet device 110.
[0022] FIG. 1B illustrates a view 120 of a user 102 changing the user's field of view 124 to face a computing device 122 located in a different area of the environment 108. The user 102 can reorient himself to direct verbal utterances to a different device than the television 106. For example, the computing device 122 may have been performing an operation 128 of playing music when the user first provided a previous verbal utterance 114. Thus, to stop the music without affecting the movie on the television 106, the user 102 can turn his face and the computerized glasses 104 toward the computing device 122 rather than the television 106. In some implementations, the computerized glasses 104 can include an outward-facing camera having a field of view 124 (i.e., a viewing window or visual viewpoint), the outward-facing camera can generate image data that can be used during the device arbitration process. Alternatively or additionally, the computerized glasses 104 can include an inward-facing camera, the inward-facing camera can also generate image data that can be used during the device arbitration process.
[0023] For example, image data generated using an inward-facing camera may characterize the gaze of user 102 as being directed slightly upward toward the direction of computing device 122 and away from tablet device 110. In this manner, image data from the inward-facing camera may indicate that the gaze of user 102 is directed toward computing device 122, even though the microphone of computing device 122 does not detect verbal input as clearly as tablet device 110 or display device 118 (at least when user 102 is positioned as shown in FIG. 1B ). For example, user 102 may provide verbal utterance 130, such as "Assistant, stop," when television 106 is playing a movie and computing device 122 is playing music. Image data from one or more cameras of computerized glasses 104 and / or data from one or more other devices may be processed during a device arbitration process to determine that computing device 122 is intended to be the subject of verbal utterance 130.
[0024] In some implementations, a heuristic process and / or one or more trained machine learning models may be used during the device arbitration process to select a particular device that responds to input from a user. For example, one or more trained machine learning models may be used to process image data from an inward-facing camera or other image data from an outward-facing camera to identify a device to which a user input is directed. Alternatively or additionally, a heuristic process may be used to determine whether to prioritize a particular device over other candidate devices based on data from one or more sources. For example, a device that is not located within the field of view of the user and / or computerized glasses may be considered of lower priority than another device that is determined to be within the field of view of the user and / or computerized glasses. For example, if the computing device 122 is prioritized over other devices in the environment 108, one or more operations may be performed at the computing device 122 to fulfill the user request embodied in the verbal utterance 130. For example, the computing device 122 may perform an operation 132 to cause music to no longer be played at the computing device 122.
[0025] 2 illustrates a view 200 of a user 202 wearing computerized glasses 204 according to some implementations discussed herein. The computerized glasses 204 can include a computer 208, which can include one or more processors and / or one or more memory devices, and can receive power from one or more energy sources (e.g., batteries, wireless power transmission, etc.). The computer 208 can be at least partially embodied by and / or separate from a housing 214. The housing 214 can resemble one or more different styles of eyeglass frames and can have one or more lenses 206 attached to the housing 214. In some implementations, the computerized glasses 204 can include one or more forward-facing cameras 210, which can be positioned to have a field of view that corresponds to the field of view of the user 202. In some implementations, the computerized glasses 204 can include one or more inward-facing cameras 212, which can be positioned to have another field of view that includes one or more eyes of the user 202. For example, one or more inward-facing cameras 212 may be positioned to capture image data characterizing the position of the left and / or right eye of the user 202. In some implementations, the computer 208 may be connected to one or more antennas and / or other communications hardware that enable the computer 208 to communicate with one or more other computing devices. For example, the computerized glasses 204 may be connected to a Wi-Fi network, an LTE network, and / or communicate via Bluetooth protocols, and / or any other communications modality.
[0026] In some implementations, one or more lenses 206 can operate as a display interface for rendering graphical content visible to a user wearing the computerized glasses 204. The graphical content rendered in the lenses 206 can assist with device arbitration in response to multiple devices detecting input from the user 202. For example, the user 202 can orient his or her head and the computerized glasses 204 in a direction that causes a first computing device and a second computing device to be within the field of view of the forward-facing camera 210. Once the user 202 has the computerized glasses 204 pointed in this direction, the user 202 can provide verbal utterances to cause a particular computing device to play music from a music application, for example. The automated assistant can detect the verbal utterances and, in response, cause multiple instances of an icon for the music application to be rendered in the lenses 206. For example, a first instance of a music application icon can be rendered in the lenses 206 on the first computing device and a second instance of the music application icon can be rendered in the lenses 206 on the second computing device.
[0027] In some implementations, each instance of the music application icon may be rendered in a manner that indicates to the user that a particular device has not been selected to respond to a verbal utterance. For example, each instance of the music application icon may be "grayed out," blurred, flashing, and / or may otherwise have one or more characteristics that indicate that one of the devices should be selected by the user. To select one of the devices, the user 202 may adjust his / her gaze and / or the direction of the computerized glasses 204 more toward the first computing device or the second computing device. In response, the automated assistant may detect the adjustment in the direction of the user's gaze and / or orientation and cause the first instance or the second instance of the music application icon to provide feedback that one has been selected. For example, if the user 202 directs his / her gaze and / or the computerized glasses 204 more toward the first computing device, the first instance of the music application may flash, rock, idle, no longer be grayed out, no longer be blurred, and / or otherwise indicate that the first computing device has been selected. In this manner, the user 202 receives feedback that the user has selected a particular device and can redirect his or her gaze and / or the computerized glasses 204 if the user prefers the second computing device. In some implementations, if the user 202 is satisfied with their selection, the user 202 can continue to look at the first computing device or look away from both computing devices for a threshold period of time to confirm their selection and cause the first computing device to respond to a verbal utterance.
[0028] In some implementations, the graphical content rendered in the lens 206 can help clarify the parameters of a particular request submitted by the user to the automated assistant and / or another application. For example, the user 202 can provide a verbal utterance such as "play some music," and in response, the automated assistant can cause a first icon for a first music application and a second icon for a second music application to be rendered in the lens 206. The icons can be rendered at or near the particular computing device to which the user 202 is directing their attention, and the icons are rendered to provide feedback to prompt the user 202 to select a particular music application for rendering music. In some implementations, a timer can also be rendered in the lens 206 to indicate how much time the user has before a particular music application is selected. For example, the automated assistant can cause a particular icon to be rendered to provide visual feedback indicating that the music application corresponding to that icon will be selected by default if the user 202 does not provide additional input indicating whether the user prefers one application over another.
[0029] In some implementations, the graphical content rendered in the lens 206 can correspond to parameters to be provided to the application in response to an assistant input from the user 202. For example, in response to a verbal utterance of "play a new song," the automated assistant can cause a first graphical element and a second graphical element to be rendered in the lens 206 at or near a particular audio device. The first graphical element can include text identifying the name of the first song, and the second graphical element can include text identifying the name of the second song. In this manner, the user 202 can realize that there was some ambiguity in the verbal utterance provided by the user and that additional input may be required to select a particular song. The user 202 can then provide additional input (e.g., adjust the user's gaze, rotate the user's head, perform a gesture, tap the housing 214 while gazing at a particular icon, provide another verbal utterance, and / or provide any other input) to specify the particular song. In some implementations, when a change in the orientation of the user 202 is detected, the graphical content rendered in the lens 206 may be adjusted according to the change in orientation. For example, an icon rendered to appear above the computing device the user 202 is looking at may be shifted in the lens 206 in a direction opposite to the direction the user 202 has turned his head. Similarly, when the computing device is no longer within the field of view of the user 202 and / or the computerized glasses 204, the icon may no longer be rendered in the lens 206.
[0030] 3A, 3B, 3C, 3D, and 3E respectively show views 300, 320, 340, 360, and 380 of a user 302 interacting with an automated assistant that can rely on computerized glasses 304 for device arbitration. These figures illustrate at least one example in which a user 302 causes a particular action to be performed on a computing device in a room and then moves to another room, but maintains the ability to provide assistant input to the computerized glasses 304 to control the action. For example, the user 302 can provide a verbal utterance 312, such as "Assistant, play the footage from the security camera from last night." The verbal utterance 312 can include a request for the automated assistant to access a security application and render video data from the security application on a display device accessible to the automated assistant.
[0031] In some implementations, to determine the particular device on which the user 302 intends the video data to be rendered, the automation assistant can cause one or more devices to each provide one or more different outputs. The output can be detected by computerized glasses 304 worn by the user 302 when the user provides the verbal utterance 312. For example, the automation assistant can identify one or more candidate devices that detected the verbal utterance 312. In some implementations, the automation assistant can cause each candidate device to provide an output. The output provided at each candidate device can be differentiated relative to the output provided at other candidate devices. In other words, each of the candidate devices can be caused to provide a corresponding unique output. In some implementations, the automation assistant can cause the candidate devices to render the output at the candidate device one or more times, each rendering for less than a given duration, such as less than a tenth of a second or less than 50 milliseconds. In some of these and / or other implementations, the output rendered at the candidate devices can be an output that may not be detectable by a human without an artificial modality. For example, the automated assistant can cause the television 306 to incorporate an image into one or more frames of graphical content being rendered at the television 306 and the tablet device 310, respectively. The frames containing the image can be rendered at a higher frame frequency (e.g., 60 frames per second or higher) than is detectable by humans. In this way, if the user 302 faces the television 306 when the user provides a verbal utterance 312, the forward-facing camera of the computerized glasses 304 can detect the image in one or more frames without the user 302 being interrupted.Alternatively or additionally, if both the television 306 and the tablet device 310 are within the field of view of the forward-facing camera and / or user 320, the automated assistant may determine that the television 306 is occupying more of the focus of the user 302 and / or computerized glasses 304 than the tablet device 310.
[0032] In some implementations, the automation assistant can determine that the television 306 and tablet device 310 are near the computerized glasses 304 and / or the user 302 when the user 302 provides a verbal utterance. Based on this determination, the automation assistant can cause the television 306 and tablet device 310 to render different images to identify devices that are within the field of view of the cameras and / or computerized glasses 304. The automation assistant can then determine whether one or more cameras of the computerized glasses 304 detected one image but not another image to identify the particular device to which the user 302 is directing his or her input.
[0033] In some implementations, one or more devices in the environment may include LEDs that may be controlled by each corresponding device and / or the automation assistant during device arbitration. The light emitted from the LEDs may then be detected to select a particular device to respond to input from the user 302. For example, in response to a verbal utterance from the user 302 being detected at the first device and the second device, the automation assistant may illuminate a first LED of the first device and a second LED of the second device. If the automation assistant determines that light emitted from the first LED but not from the second LED is being detected by the computerized glasses 304, the automation assistant may select the first device to respond to the verbal utterance. Alternatively or additionally, the automation assistant may illuminate each LED to exhibit a particular characteristic to aid in device arbitration. For example, the first LED may be illuminated such that the characteristic of the light emitted by the first LED is different from the characteristic of the other light emitted by the second LED. Such characteristics may include color, amplitude, duration, frequency, and / or any other characteristics of light that may be controlled by an application and / or device. For example, the automation assistant may turn on a first LED for 0.1 seconds every 0.5 seconds and a second LED for 0.05 seconds every 0.35 seconds. In this manner, one or more cameras of the computerized glasses 304 may detect these light patterns and the automation assistant may associate each detected pattern with a respective device. The automation assistant may then identify the LED to which the user 302 is looking and / or pointing the computerized glasses 304 in order to select a particular device that responds to the user input.
[0034] In some cases, the automated assistant can determine that a verbal utterance 312 is directed to the television 306 and can cause the television 306 to perform an operation 314 of playing camera footage from a security camera on the television 306. For example, the automated assistant can cause a security application to be accessed on the television 306 to render the security footage requested by the user 302. When the security application is launched, an icon 330 identifying the security application and selectable GUI elements 332 (i.e., graphical elements) can be rendered on the television 306. To identify particular features of the camera footage, the user 302 can hear audio 316 from the television 306 and view rendered video on the display interface of the television 306. Additionally, the user 302 can perform various physical gestures to control the television 306 and / or the security application via the computerized glasses 304. For example, the computerized glasses 304 can include an outward-facing camera 322 that can capture image data for processing by the automated assistant. For example, when the user 302 performs a swipe gesture 326, the automated assistant can detect the swipe gesture 326 and cause a security application to perform a particular action corresponding to the swipe gesture 326 (e.g., fast forward).
[0035] In some implementations, the user 302 may move to another room in the user's home to operate the computerized glasses 304 in a manner that reflects the user's 302 movement 344. For example, the user 302 may move from an environment 308 to another environment 362 to change his / her laundry 364 while still hearing audio 316 from the security application, as shown in FIGS. 3C and 3D. The computerized glasses 304 may provide an interface for controlling a security application and / or an automated assistant in response to the user 302 moving away from the television 306. In some implementations, the determination that the user 302 has moved may be based on data generated in the computerized glasses and / or one or more other devices in the environment 308. In some implementations, the computerized glasses 304 may indicate that the user 302 can control applications and / or devices that the user 302 was previously viewing through the computerized glasses 304. For example, an icon 330 representing a security application may be rendered in a display interface of the computerized glasses 304. Alternatively or additionally, in response to the user 302 redirecting his / her gaze and / or face away from the television 306, the selectable GUI elements 332 and graphical elements 384 rendered on the television 306 may be rendered in the display interface of the computerized glasses 304.
[0036] In some implementations, device arbitration may be performed using the computerized glasses 304 when the user 302 may not be looking at the device and / or may otherwise be directing his / her attention towards another computing device. For example, as provided in FIG. 3E, the user 302 may perform a gesture 382 to provide input to the automation assistant, a particular application, and / or a particular computing device. One or more cameras (e.g., the outward-facing camera 322) of the computerized glasses may detect a physical gesture, which may be a physical gesture in which the user 302 manipulates the user's hand from a left position 368 toward the right. In response to detecting the physical gesture 382, the automation assistant may operate to identify one or more devices that detected the physical gesture. In some cases, the automation assistant may determine that only the computerized glasses 304 detected the physical gesture 382. Nonetheless, the automation assistant may determine whether the physical gesture 382 was intended to initialize one or more operations in the computerized glasses 304 and / or another device.
[0037] In some cases, the user 302 may be viewing content from a security application in a display interface of the computerized glasses 304. In such a case, the automation assistant may determine that the physical gesture 382 is intended to affect the operation of the security application. In response to the physical gesture 382, the automation assistant may cause the security application to fast forward through certain content being rendered in the computerized glasses 304. Alternatively or additionally, the automation assistant may perform a heuristic process to identify an application and / or device to which the user 302 is pointing the physical gesture 382. In some implementations, with prior permission from the user 302, the automation assistant may determine that the user 302 has recently gazed at the television 306, and that prior to gazing at the television 306, the user 302 gazed at the tablet device 310. This determination may cause the automation assistant to prioritize the television 306 over the tablet device 310 when selecting a device to respond to the physical gesture 382.
[0038] Alternatively or additionally, the automation assistant may determine, based on image data from the inward-facing camera 324 of the computerized glasses 304, that the user 302 recently gazed at a security application on the television 306, and prior to gazing at the security application, the user 302 gazed at a social media application on the tablet device 310. Based on distinguishing between the security application and the social media application, the automation assistant may determine that the physical gesture 382 is acceptable as an input to the security application, but is not acceptable as an input to the social media application. Thus, according to this process, the automation assistant may select a security application to respond to a physical gesture input from the user 302.
[0039] In some implementations, device arbitration may be performed using one or more trained machine learning models that may be used to process application data and / or context data. The application data may, for example, characterize the operational state of one or more applications that may be associated with the user 302 when the user 302 provides the physical gesture 382. Alternatively or additionally, the context data may characterize features of the context in which the user 302 provided the physical gesture 382. Such features may include, but are not limited to, the location of the user 302, the time of day, (with prior permission from the user) one or more activities of the user, and / or any other information that may be associated with the user 302 when the user 302 provides the physical gesture 382. For example, audio data captured by one or more microphones of the computerized glasses 304 and / or one or more other devices may be processed to identify contextual features of the environment, with prior permission from the user 302. For example, audio data capturing audio from a movie may be used to assist the automated assistant in determining whether the physical gesture 382 should affect the application that is rendering the movie. If the automated assistant determines that the physical gesture 382 is intended to affect the movie (e.g., fast-forward the movie through a portion of the movie that the user does not want to hear), the automated assistant can generate command data that can be communicated to an application that is rendering the movie without the user necessarily having to gaze at the television 306 that is displaying the movie.
[0040] FIG. 4 illustrates a system 400 for performing device arbitration using data available from a device, such as computerized glasses, having capabilities for performing augmented reality. The automation assistant 404 can operate as part of an assistant application provided in one or more computing devices, such as the computing device 402 and / or a server device. A user can interact with the automation assistant 404 via an assistant interface 420, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automation assistant 404 by providing verbal, textual, and / or graphical input to the assistant interface 420 to cause the automation assistant 404 to initialize one or more actions (e.g., provide data, control a peripheral device, access an agent, generate input and / or output, etc.). Alternatively, the automation assistant 404 may be initialized based on processing of contextual data 436 using one or more trained machine learning models, where the contextual data 436 may characterize one or more features of an environment in which the automation assistant 404 has access and / or one or more features of a user that is expected to intend to interact with the automation assistant 404.
[0041] The computing device 402 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to enable a user to control applications 434 of the computing device 402 via a touch interface. In some implementations, the computing device 402 may lack a display device, thereby providing an audible user interface output without providing a graphical user interface output. Additionally, the computing device 402 may provide a user interface, such as a microphone, for receiving verbal natural language input from a user. In some implementations, the computing device 402 may include a touch interface and may lack a camera, but may optionally include one or more other sensors. In some implementations, the computing device 402 may provide augmented reality functionality and / or may be a wearable device, such as, but not limited to, computerized glasses, contact lenses, watches, clothing, and / or any other wearable device. Thus, although various implementations are described herein with respect to computerized glasses, the techniques disclosed herein may be implemented in conjunction with other electronic devices that include augmented reality functionality, such as other wearable devices that are not computerized glasses.
[0042] The computing device 402 and / or other third party client devices can communicate with the server device via a network such as the Internet. In addition, the computing device 402 and any other computing devices can communicate with each other via a local area network (LAN), such as a Wi-Fi network. The computing device 402 can offload computational tasks to the server device to conserve computational resources at the computing device 402. For example, the server device can host the automation assistant 404 and / or the computing device 402 can send inputs received at one or more assistant interfaces 420 to the server device. However, in some implementations, the automation assistant 404 can be hosted at the computing device 402 and various processes that can be associated with the automation assistant operations can be executed at the computing device 402.
[0043] In various implementations, all or less than all aspects of the automation assistant 404 may be implemented in the computing device 402. In some of those implementations, aspects of the automation assistant 404 may be implemented via the computing device 402 and interface with a server device that may implement other aspects of the automation assistant 404. The server device may optionally provide services to multiple users and their associated assistant applications via multiple threads. In implementations in which all or less than all aspects of the automation assistant 404 are implemented via the computing device 402, the automation assistant 404 may be an application separate from (e.g., installed "on") the operating system of the computing device 402, or alternatively, may be implemented directly by (e.g., considered an application of the operating system but integral to the operating system).
[0044] In some implementations, the automation assistant 404 can include an input processing engine 406 that can use multiple different modules to process input and / or output of the computing device 402 and / or the server device. For example, the input processing engine 406 can include a voice processing engine 408 that can process audio data received at the assistant interface 420 to identify text embodied in the audio data. The audio data can be transmitted from the computing device 402 to a server device, for example, to conserve computational resources at the computing device 402. Additionally or alternatively, the audio data can be processed exclusively at the computing device 402.
[0045] The process for converting audio data to text may include speech recognition algorithms that may employ neural networks and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be analyzed by the data analysis engine 410 and made available to the automation assistant 404 as text data that may be used to generate and / or identify command phrases, intents, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data analysis engine 410 may be provided to the parameter engine 412 to determine whether the user has provided input that corresponds to a particular intent, action, and / or routine that may be performed by the automation assistant 404 and / or an application or agent that may be accessed via the automation assistant 404. For example, assistant data 438 may be stored at the server device and / or computing device 402 and may include data defining one or more actions that may be performed by the automation assistant 404, as well as parameters necessary to perform the action. The parameter engine 412 may generate one or more parameters for the intents, actions, and / or slot values and provide the one or more parameters to the output generation engine 414. The output generation engine 414 can use one or more parameters to communicate with an assistant interface 420 to provide output to a user and / or to communicate with one or more applications 434 to provide output to the one or more applications 434.
[0046] In some implementations, the automation assistant 404 may be an application that may be installed "on" the computing device 402 and / or that may itself form part (or the entirety) of the operating system of the computing device 402. The automation assistant application may include and / or access on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally in the computing device 402. The on-device speech recognition generates recognized text for the verbal utterances (if any) present in the audio data. Also, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes the recognized text generated using the on-device speech recognition and, optionally, contextual data to generate NLU data.
[0047] The NLU data can include an intent corresponding to the verbal utterance and optionally parameters for the intent (e.g., slot values). On-device fulfillment can be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine actions to take to resolve the intent (and optionally parameters for the intent) of the verbal utterance. This can include determining local and / or remote responses (e.g., answers) to the verbal utterance, interactions with locally installed applications to perform based on the verbal utterance, commands to send to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the verbal utterance, and / or other resolution actions to perform based on the verbal utterance. The on-device fulfillment can then initiate local and / or remote fulfillment / execution of the determined actions to resolve the verbal utterance.
[0048] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be utilized, at least selectively. For example, recognized text may be at least selectively sent to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, recognized text may be optionally sent for remote fulfillment in parallel with on-device fulfillment or in response to failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the reduced latency they provide when resolving spoken utterances (as client-server round trips are not required to resolve the spoken utterance). Additionally, on-device functionality may be the only functionality available in situations where network connectivity is absent or limited.
[0049] In some implementations, the computing device 402 may include one or more applications 434 that may be provided by a third party entity different from the entity that provided the computing device 402 and / or the automation assistant 404. The application state engine of the automation assistant 404 and / or the computing device 402 may access the application data 430 to determine one or more actions that may be performed by the one or more applications 434, as well as the state of each application of the one or more applications 434 and / or the state of each device associated with the computing device 402. The device state engine of the automation assistant 404 and / or the computing device 402 may access the device data 432 to determine one or more actions that may be performed by the computing device 402 and / or one or more devices associated with the computing device 402. Additionally, the application data 430 and / or any other data (e.g., device data 432) may be accessed by the automation assistant 404 to generate context data 436 that can characterize the context in which a particular application 434 and / or device is running, and / or the context in which a particular user is accessing the computing device 402, accessing the application 434, and / or accessing any other device or module.
[0050] While one or more applications 434 are executing on the computing device 402, the device data 432 can characterize a current operational state of each application 434 executing on the computing device 402. Additionally, the application data 430 can characterize one or more features of the executing applications 434, such as the content of one or more graphical user interfaces being rendered in the direction of the one or more applications 434. Alternatively or additionally, the application data 430 can characterize action schemas that can be updated by the respective applications and / or by the automation assistant 404 based on the current operational state of the respective applications. Alternatively or additionally, the one or more action schemas for the one or more applications 434 can remain static but can be accessed by the application state engine to determine suitable actions to initialize via the automation assistant 404.
[0051] The computing device 402 may further include an assistant invocation engine 422 that may use one or more trained machine learning models to process the application data 430, the device data 432, the context data 436, and / or any other data accessible to the computing device 402. The assistant invocation engine 422 may process this data to determine whether to wait for the user to explicitly speak an invocation phrase to invoke the automated assistant 404, or to determine whether to consider the data as indicative of the user's intent to invoke an automated assistant instead of requiring the user to explicitly speak an invocation phrase. For example, the one or more trained machine learning models may be trained using instances of training data based on scenarios in which a user is in an environment in which multiple devices and / or applications exhibit various operating states. Instances of training data may be generated to capture training data that characterize contexts in which a user invokes an automated assistant and other contexts in which a user does not invoke an automated assistant.
[0052] When one or more trained machine learning models are trained according to these instances of the training data, the assistant invocation engine 422 can cause the automated assistant 404 to detect or limit detection of spoken invocation phrases from the user based on the context and / or environmental features. Additionally or alternatively, the assistant invocation engine 422 can cause the automated assistant 404 to detect or limit detection of one or more assistant commands from the user based on the context and / or environmental features. In some implementations, the assistant invocation engine 422 can be disabled or limited based on the computing device 402 detecting an assistant suppression output from another computing device. In this way, if the computing device 402 detects an assistant suppression output, the automated assistant 404 is not invoked based on the context data 436 that would otherwise cause the automated assistant 404 to be invoked if the assistant suppression output was not detected.
[0053] In some implementations, the system 400 can include a device arbitration engine 416 that can assist one or more devices and / or applications in performing device arbitration when they detect an input from a user. For example, in some implementations, the device arbitration engine 416 can process data from one or more different devices to determine whether to initiate a device arbitration process. In some implementations, the data can be received via a network connection, one or more interfaces of the system 400, and / or any other modality through which a computing device can receive data. For example, the device arbitration engine 416 can determine that a plurality of different devices are projecting ultrasound and / or light in response to an assistant input. Based on this determination, the device arbitration engine 416 can initiate a process for selecting a particular device among the plurality of different devices that responds to the assistant input from the user.
[0054] In some implementations, the system 400 can include a gaze detection engine 418 that can determine the user's gaze toward one or more objects in the user's environment. For example, the system 400 can be computerized glasses that include an inward-facing camera that is directed toward one or more eyes of the user. Based on image data generated using the inward-facing camera, the system 400 can identify a particular area in the environment to which the user is directing his or her eyes. In some implementations, the gaze detection engine 418 can determine the direction of the user's gaze based on data from one or more different devices, such as a separate computing device that includes a camera. The image data from the separate computing device can indicate, with prior permission from the user, the user's posture and / or the direction in which the user is directing one or more of the user's appendages. In this manner, the gaze detection engine 418 can determine the direction in which the user is directing the user's attention before, during, and / or after the user provides input to the automated assistant 404.
[0055] In some implementations, the system 400 can include a field of view engine 426 that can process data characterizing the field of view of one or more cameras of a user and / or a device. For example, the field of view engine 426 can process image data from one or more cameras of computerized glasses to identify one or more objects and / or devices located within the field of view of the cameras at one or more instances of time. In some implementations, the field of view engine 426 can also process device data 432 to identify specific objects that can be associated with a particular device within the field of view of the user. For example, a kitchen sink can be an object associated with the user's stand-alone display device. Thus, the field of view engine 426 can determine that the stand-alone computing device is the subject of the user input if a kitchen sink is identified within the field of view of the user when the user provides the user input.
[0056] In some implementations, the system 400 may include an interface content engine 424 that causes one or more interfaces of the system 400 to render content according to output from the device arbitration engine 416. For example, if the device arbitration engine 416 identifies the computing device 402 as a target for input from a user, the interface content engine 424 may cause content to be rendered in one or more interfaces of the computing device 402 (e.g., a display interface of computerized glasses). If the device arbitration engine 416 determines that the user has directed input to a separate computing device and that the separate computing device is within the user's field of view, the interface content engine 424 may cause a notification to be rendered for the separate computing device. For example, the interface content engine 424 may cause graphical content to be rendered in a display interface of the computing device 402 if the computing device 402 is a computerized pair of glasses. The graphical content may be rendered such that the graphical content appears "on" and / or adjacent to the separate computing device within the user's field of view (e.g., within an area of the glasses lenses that corresponds to the location of the separate device). The graphical content may include, but is not limited to, one or more icons, colors, and / or other graphical characteristics that may indicate that the device arbitration engine 416 has selected a separate computing device as responsive to input from a user.
[0057] In some implementations, the device arbitration engine 416 may request additional input from the user to assist the user in identifying the particular device to which the input is intended. The device arbitration engine 416 may communicate the identifiers and / or locations of the candidate devices to the interface content engine 424, which may render a graphical display at or near the relative location of the candidate devices within the lenses of the computerized glasses. For example, a first selectable element may be rendered in the leftmost portion of the display interface of the computerized glasses to indicate that a computing device in the leftmost portion of the user's field of view is a candidate device. A second selectable element may be rendered in a more central location of the display interface simultaneously with the first selectable element to indicate that another computing device in the central portion of the user's field of view is also a candidate device. The user may then perform a gesture (e.g., hold up his / her index finger) in front of his / her face for the gesture to be captured by one or more cameras of the computerized glasses. The gesture can indicate to the automation assistant 404 that a particular device (eg, the most central device) is the device that the user intends to respond to user input.
[0058] FIG. 5 illustrates a method 500 for performing device arbitration in a multi-device environment using data available from a wearable computing device, such as computerized glasses. Method 500 may be executed by one or more computing devices, applications, and / or any other apparatus or module that may be associated with an automated assistant. Method 500 may include operation 502 of determining whether an assistant input is detected. The assistant input may be a user input provided by a user to one or more computing devices that provide access to the automated assistant. In some implementations, the assistant input may be provided to a computing device with augmented reality capabilities, such as computerized glasses, computerized contact lenses, a phone, a tablet device, a portable computing device, a smart watch, and / or any other computing device that may augment the perception of one or more users. It should be noted that in the implementations discussed herein, including computerized glasses, the computerized glasses may be any computing device that provides augmented reality capabilities. If an assistant input is detected, method 500 may proceed from operation 502 to operation 504. Otherwise, the automated assistant may continue to determine whether the user has provided an input to the automated assistant.
[0059] Operation 504 may be an optional operation that includes determining whether an assistant input is detected at multiple devices. For example, a user may be wearing a pair of computerized glasses while gazing at a standalone speaker device that provides access to an automated assistant. Thus, when a user provides an assistant input, the assistant input may be detected at the computerized glasses, the standalone speaker device, and one or more other devices in the user's environment. If a single computing device exclusively detects the assistant input, method 500 may proceed from operation 504 to operation 512, where the particular computing device that detected the assistant input initiates the performance of one or more operations. However, if multiple devices detect the assistant input, method 500 may proceed from operation 504 to operation 506 to initiate device arbitration to select one or more devices to respond to the assistant input.
[0060] Operation 506 may include identifying a candidate device that detected an assistant input from a user. For example, if the assistant input is a verbal utterance, the automated assistant may identify multiple devices that captured audio data corresponding to the verbal utterance. In some implementations, each candidate device may capture audio data and process the audio data to generate a score for the assistant input. If each score for each candidate device meets a threshold, device arbitration may be initiated for the candidate devices. For example, the candidate devices may include computerized glasses, a stand-alone speaker device, and a tablet device that the user may have placed on a table near the user. Method 500 may proceed from operation 506 to operation 508 for further determination of whether the assistant input was directed to a particular computing device.
[0061] Operation 508 may include processing the data generated at the candidate devices and / or the computerized glasses. For example, the computerized glasses may include an outward-facing camera and / or an inward-facing camera that may be used to generate image data to identify a particular device to which the user may be directing the assistant input. The image data generated using the outward-facing camera may capture an image including one of the candidate devices or an object associated with the candidate device. For example, the stand-alone speaker device may be supported by a decoration table that the user was looking at when the user provided the assistant information. As a result, if the decoration table and / or the stand-alone speaker device are determined to be within the field of view window of the user and / or the computerized glasses, the automated assistant may determine that the assistant input was directed to the stand-alone speaker device. Alternatively or additionally, one or more of the candidate devices may provide an output that may be detected by one or more sensors of the computerized glasses. Thus, when the computerized glasses detect an output from a particular one of the candidate devices, the automated assistant may determine that the assistant input was directed to that particular device.
[0062] Method 500 may proceed to operation 510, which may include determining that the assistant input was directed to a particular candidate device based on the processing at operation 508. If the automated assistant determines that the assistant input is directed to a particular candidate device separate from the computerized glasses, method 500 may proceed from operation 510 to operation 512. Alternatively, if the automated assistant and / or another application determines that the assistant input is not directed to a particular candidate device, method 500 may proceed from operation 510 to operation 514.
[0063] Operation 514 may include determining whether the system input is directed to the computerized glasses and / or another computing device that provides an augmented reality function. For example, a user may provide verbal utterances while handwriting in a notebook while wearing the computerized glasses. Thus, even if other candidate devices detect verbal utterances from the user, none of the other candidate devices may be visible within the field of view window of the computerized glasses. As a result, the automated assistant may determine that the user intended for the automated assistant to respond to the verbal utterances via the computerized glasses. For example, the user may request that the automated assistant check the spelling of words written in the notebook while the user is gazing at the notebook and wearing the computerized glasses. In response, the automated assistant may cause the computerized glasses to render one or more of the graphical elements indicating whether the words are spelled correctly in the notebook via augmented reality and / or provide an audible output via one or more speakers of the computerized glasses.
[0064] If the automated assistant determines that the assistant input is directed to the computerized glasses, method 500 may proceed from operation 514 to operation 512. Otherwise, method 500 may proceed from operation 514 to operation 516. Operation 516 may be an optional operation involving the automated assistant requesting additional input from the user to assist the automated assistant in identifying the particular computing device to which the user intended the assistant input to be directed. Method 500 may then proceed from operation 516 to operation 502 and / or operation 504.
[0065] 6 is a block diagram 600 of an exemplary computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computer system 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.
[0066] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphical tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or a communications network.
[0067] The user interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computer system 610 to a user or to another machine or computer system.
[0068] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the method 500 and / or for implementing one or more of the system 400, the computerized glasses 104, the television 106, the tablet device 110, the computing device 122, the computerized glasses 204, the television 306, the computerized glasses 304, the tablet device 310, the computing device 342, and / or any other applications, devices, apparatus, and / or modules discussed herein.
[0069] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include several memories including a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of a particular implementation may be stored by the file storage subsystem 626 in the storage subsystem 624 or by other machines accessible by the processor 614.
[0070] The bus subsystem 612 provides a mechanism for allowing the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is shown diagrammatically as a single bus, alternative implementations of the bus subsystem may use multiple buses.
[0071] The computer system 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 shown in FIG. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of the computer system 610 can have more or fewer components than the computer system shown in FIG.
[0072] In situations where the systems described herein may collect or utilize personal information about users (or, as is often referred to herein as "participants"), the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from content servers that may be more relevant to the user. Also, certain data may be processed in one or more ways before it is stored or used, such that personally identifiable information is removed. For example, the user's identification information may be processed such that personally identifiable information cannot be determined about the user, or the user's geographic location may be generalized where the geographic location information is obtained (such as to the city, zip code, or state level) such that the user's specific geographic location cannot be identified. Thus, the user may have control over how information is collected and / or used about the user.
[0073] Although several implementations have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each such change and / or modification is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the teachings are used. Those skilled in the art will recognize or be able to ascertain, using no more than routine experimentation, many equivalents to the specific implementations described herein. Thus, the foregoing implementations are presented by way of example only, and it should be understood that within the scope of the appended claims and their equivalents, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. Additionally, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.
[0074] In some implementations, a method implemented by one or more processors is shown as including an operation such as determining that a user has directed an assistant input to an automated assistant accessible through any one of a plurality of computing devices connected to a network, where the user is wearing computerized glasses, the computerized glasses being a computing device including one or more cameras, and the user is located in an environment including the plurality of computing devices. The method may further include an operation of identifying two or more candidate devices of the plurality of computing devices that have detected the assistant input from the user based on the assistant input from the user, where the two or more candidate devices are separate from the computerized glasses. The method may further include an operation of determining whether the assistant input is directed to a particular computing device of the two or more candidate devices or to the computerized glasses based on processing image data generated using one or more cameras of the computerized glasses. In some implementations, the method may further include an operation of causing the particular computing device to perform one or more operations corresponding to the assistant input if it is determined that the assistant input is directed to a particular computing device of the two or more candidate devices.
[0075] In some implementations, determining whether the assistant input is directed to a particular computing device of the two or more candidate devices or to the computerized glasses includes determining whether the particular computing device of the two or more candidate devices is located within a field of view window of one or more cameras based on the image data. In some implementations, a camera of one or more cameras of the computerized glasses is directed toward the user's eye, and determining that the particular computing device of the two or more candidate devices is associated with the user's visual viewpoint includes determining that the user's gaze is directed toward the particular computing device rather than toward any other device of the two or more candidate devices.
[0076] In some implementations, determining whether the assistant input is directed to a particular computing device of the two or more candidate devices or to the computerized glasses includes determining whether a particular object in a field of view of one or more cameras is associated with a relative position of the particular computing device of the two or more candidate devices. In some implementations, the particular computing device is not located within a field of view of one or more cameras of the computerized glasses. In some implementations, the assistant input is a physical gesture performed by a user, and the physical gesture is detected by the computerized glasses. In some implementations, the computerized glasses include a graphical display interface that is rendering content when the user provides the assistant input, and the method further includes, if the assistant input is determined to be directed to the computerized glasses, changing the content rendered in the graphical display interface of the computerized glasses according to the physical gesture.
[0077] In some implementations, the method further includes, when it is determined that the assistant input is directed to the computerized glasses and the specific computing device, rendering a first portion of the content on the specific computing device and rendering a second portion of the content on the display interface of the computerized glasses. In some implementations, the specific computing device is a removable accessory device connected to the display device, and the specific computing device is not visible within a field of view window of one or more cameras of the computerized glasses. In some implementations, the computerized glasses include a graphical display interface that is at least partially transparent when the graphical display interface is rendering the content, and the method further includes, when it is determined that the assistant input is directed to a specific computing device of the two or more candidate devices, rendering a graphical element at a position in the graphical display interface corresponding to the specific computing device.
[0078] In some implementations, the assistant input includes a request for the automated assistant to initialize a particular application, and the graphical element is based on the particular application. In some implementations, identifying two or more candidate devices of the plurality of computing devices that detected the assistant input from the user includes determining that a particular computing device is rendering a first output and another computing device of the plurality of computing devices is rendering a second output, and the first output and the second output are detected by computerized glasses.
[0079] In some implementations, the particular computing device includes a graphical display interface, and the first output includes a graphical element rendered in the graphical display interface, the graphical element being embodied in one or more graphical content frames rendered at a frequency of 60 frames per second or greater. In some implementations, the first output is different from the second output, and determining whether the assistant input is directed to a particular computing device of the two or more candidate devices or to the computerized glasses includes determining that the first output is detected within a field of view window of the computerized glasses and that the second output is not detected within a field of view window of the computerized glasses.
[0080] In other implementations, a method implemented by one or more processors is shown as including operations such as determining, by a computing device, that a user has provided input to an automated assistant accessible via one or more computing devices located in the user's environment, the input corresponding to a request for the automated assistant to provide content to the user, the one or more computing devices including computerized glasses worn by the user when the user provided the input. The method may further include an operation of identifying a particular device for rendering content for the user based on the input from the user, the particular device being separate from the computerized glasses. The method may further include an operation of causing the particular device to render content for the user based on identifying the particular device. The method may further include an operation of processing contextual data provided by one or more computing devices that were in the user's environment when the user provided the input. The method may further include an operation of determining whether to provide the user with additional content associated with the request based on the contextual data. If the method determines that the automated assistant provides the additional content to the user, the method may further include an operation of causing the computerized glasses to perform one or more additional operations to facilitate rendering the additional content via one or more interfaces of the computerized glasses.
[0081] In some implementations, the contextual data includes image data provided by one or more cameras of the computerized glasses, and determining whether to provide the user with the additional content associated with the request includes determining whether the user is viewing the particular device while the particular device is performing the one or more operations. In some implementations, causing the computerized glasses to render the additional content includes causing the computerized glasses to access content data via a network connection and causing a display interface of the computerized glasses to render one or more graphical elements based on the content data.
[0082] In yet another implementation, a method implemented by one or more processors is shown as including an act such as determining, by a computing device, that a user has provided an input to an automated assistant accessible via the computing device, the input corresponding to a request for the automated assistant to perform one or more actions. The method may further include an act of receiving, by the computing device, contextual data indicating that the user is wearing computerized glasses, the computerized glasses being separate from the computing device. The method may further include an act of causing an interface of the computing device to render an output that may be detected in another interface of the computerized glasses based on the contextual data. The method may further include an act of determining whether the computerized glasses have detected an output from the computing device. The method may further include an act of causing the computing device to perform one or more actions to facilitate fulfilling the request if the computing device determines that the computerized glasses have detected an output.
[0083] In some implementations, determining whether the computerized glasses detected output from the computing device includes processing other contextual data indicative of whether one or more cameras of the computerized glasses detected output from the computing device. In some implementations, the method may further include an act of causing the computerized glasses to render one or more graphical elements that may be selected in response to a physical gesture from the user if the computing device determines that the computerized glasses detected output. [Explanation of symbols]
[0084] 100 Views 102 users 104 Computerized Glasses 106 Television 108 Environment 110 Tablet Devices 112 Field of view 114 Oral Speech 116 Motion, TV 118 Display Devices 120 Views 122 Computing Devices 124 Field of View 126 Dongle 128 operation 130 Oral Speech 200 Views 202 users 204 Computerized Glasses 206 Lens 208 Computer 210 Forward Facing Camera 212 Inward-facing camera 214 Housing 300 Views 302 users 304 Computerized Glasses 306 Television 308 Environment 310 Tablet Devices 312 Oral Speech 314 operation 316 Audio 320 Views 322 Outward Facing Camera 326 Swipe Gestures 330 Icons 332 Selectable GUI Elements 340 Views 344 Move 360 Views 362 Environment 364 Laundry 368 left position 380 Views 382 Gestures, Physical Gestures 384 Graphical Elements 400 Systems 402 Computing Devices 404 Automation Assistant 406 Input Processing Engine 408 Voice Processing Engine 410 Data Analysis Engine 412 Parameter Engine 414 Output Generation Engine 416 Device Arbitration Engine 418 Gaze Detection Engine 420 Assistant Interface 422 Assistant Call Engine 424 Interface Content Engine 426 Vision Engine 430 Application Data 432 Device Data 434 Applications 436 Context Data 438 Assistant Data 600 Block Diagram 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory 626 File Storage Subsystem 630 Main Random Access Memory (RAM) 632 Read Only Memory
Claims
1. 1. A method implemented by one or more processors, the method comprising: determining that a user has directed an assistant input to an automated assistant accessible via any one of a plurality of computing devices connected to the network, the user is wearing computerized glasses, the computerized glasses being a computing device including one or more cameras; the user is located within an environment that includes the plurality of computing devices; Steps and Identifying two or more candidate devices of the plurality of computing devices that have detected the assistant input from the user based on the assistant input from the user, the two or more candidate devices are separate from the computerized glasses. Steps and determining whether the assistant input is directed to a particular computing device of the two or more candidate devices or to the computerized glasses based on processing image data generated using the one or more cameras of the computerized glasses; If it is determined that the assistant input is directed to the particular computing device of the two or more candidate devices, causing the particular computing device to perform one or more actions corresponding to the assistant input; A method comprising:
2. determining whether the assistant input is directed to the particular computing device of the two or more candidate devices or to the computerized glasses, determining, based on the image data, whether the particular computing device of the two or more candidate devices is located within a field of view window of the one or more cameras; The method of claim 1.
3. a camera of the one or more cameras of the computerized glasses is directed toward an eye of the user; determining that the particular computing device of the two or more candidate devices is associated with a visual viewpoint of the user, determining that the user's gaze is directed toward the particular computing device over any other device of the two or more candidate devices; The method of claim 1.
4. determining whether the assistant input is directed to the particular computing device of the two or more candidate devices or to the computerized glasses, determining whether a particular object within a field of view window of the one or more cameras is associated with a relative position of the particular computing device of the two or more candidate devices; The method of claim 1.
5. The method of claim 4 , wherein the particular computing device is not located within the field of view window of the one or more cameras of the computerized glasses.
6. The method of claim 1 , wherein the assistant input is a physical gesture performed by the user, the physical gesture being detected by the computerized glasses.
7. the computerized glasses include a graphical display interface that is rendering content as the user provides the assistant input; The method further comprising: If it is determined that the assistant input is directed to the computerized glasses, modifying the content rendered on the graphical display interface of the computerized glasses in accordance with the physical gesture. The method of claim 6.
8. If it is determined that the assistant input is directed to the computerized glasses and the particular computing device, causing a first portion of content to be rendered on the particular computing device; causing a second portion of the content to be rendered at a display interface of the computerized glasses; The method of claim 1, further comprising:
9. the particular computing device is a removable accessory device connected to a display device; the particular computing device is not visible within a viewing window of the one or more cameras of the computerized glasses; The method of claim 1.
10. the computerized glasses include a graphical display interface that is at least partially transparent when the graphical display interface is rendering content; The method further comprising: If it is determined that the assistant input is directed to the particular computing device of the two or more candidate devices, causing a graphical element to be rendered at a location within the graphical display interface corresponding to the particular computing device. The method of claim 1.
11. the assistant input comprises a request for the automated assistant to initialize a particular application; the graphical elements are based on the particular application; The method of claim 10.
12. Identifying the two or more candidate devices of the plurality of computing devices that have detected the assistant input from the user, determining that the particular computing device is rendering a first output and that another computing device of the plurality of computing devices is rendering a second output; the first output and the second output are detected by the computerized glasses. The method of claim 1.
13. the particular computing device includes a graphical display interface, and the first output includes graphical elements rendered in the graphical display interface; the graphical element is embodied in one or more frames of graphical content that are rendered at a frequency of 60 frames per second or greater; The method of claim 12.
14. the first output is different from the second output; determining whether the assistant input is directed to the particular computing device of the two or more candidate devices or to the computerized glasses, determining that the first output is detected within a field of view window of the computerized glasses and that the second output is not detected within the field of view window of the computerized glasses. The method of claim 12.
15. 1. A method implemented by one or more processors, the method comprising: determining, by a computing device, that a user has provided input to an automated assistant accessible via one or more computing devices located in the user's environment; the input corresponds to a request for the automated assistant to provide content to the user; the one or more computing devices include computerized glasses worn by the user when the user provides the input; Steps and identifying a particular device for rendering the content for the user based on the input from the user; the particular device being separate from the computerized glasses; Steps and based on identifying the particular device, causing the particular device to render the content for the user; processing contextual data provided by the one or more computing devices that were in the user's environment when the user provided the input; determining whether to provide additional content associated with the request to the user based on the context data; If the automated assistant determines to provide the additional content to the user, causing the computerized glasses to perform one or more additional operations to facilitate rendering the additional content via one or more interfaces of the computerized glasses; A method comprising:
16. the contextual data includes image data provided by one or more cameras of the computerized glasses; determining whether to provide the user with the additional content associated with the request, determining whether the user is looking at the particular device when the particular device is performing the one or more operations; The method of claim 15.
17. causing the computerized glasses to render the additional content, causing the computerized glasses to access content data via a network connection; causing a display interface of the computerized glasses to render one or more graphical elements based on the content data; Including, The method of claim 15.
18. 1. A method implemented by one or more processors, the method comprising: determining, by a computing device, that a user has provided input to an automated assistant accessible via said computing device; the input corresponds to a request for the automation assistant to perform one or more actions; receiving, by the computing device, contextual data indicating that the user is wearing computerized glasses; the computerized glasses being separate from the computing device; causing an interface of the computing device to render an output that can be detected in another interface of the computerized glasses based on the context data; determining whether the computerized glasses detect the output from the computing device; If the computing device determines that the computerized glasses detect the output, causing the computing device to perform the one or more actions in furtherance of fulfilling the request; A method comprising:
19. determining whether the computerized glasses detect the output from the computing device, processing other contextual data indicative of whether one or more cameras of the computerized glasses detected the output from the computing device; The method of claim 18.
20. If the computing device determines that the computerized glasses detect the output, and causing the computerized glasses to render one or more graphical elements that can be selected in response to a physical gesture from the user. The method of claim 18.
21. A computer program comprising instructions which, when executed by one or more processors of a computing system, cause the computing system to perform the method of any one of claims 1 to 20.
22. 21. A computing system configured to carry out the method of any one of claims 1 to 20.
23. The computing system of claim 22 , wherein the computing system comprises computerized glasses.
Citation Information
Patent Citations
Gaze Application Launcher
JP2017538218A
Object identification method, device and virtual reality device in virtual reality communication
JP2018526690A
Controller and agent cooperation method
JP2019133216A
Gesture recognition system and glasses with gesture recognition function
US20140009623A1
Display device including touch sensor and touch sensing method for the same
US20180113549A1
