Adapting assistant suggestions rendered at computerized glasses according to changes in user gaze and / or other user inputs

By rendering the automation assistant suggestions based on field data on computerized glasses, the problem of inefficient use of automation assistants in unfamiliar environments is solved, efficient task processing and resource utilization are achieved, and user experience and security are improved.

CN119072673APending Publication Date: 2024-12-03GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280095753.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-10-18
Filing Date
2022-12-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When using automation assistants, it is difficult for users to efficiently utilize their functions in unfamiliar environments, resulting in low task processing efficiency and waste of resources. Especially when wearing computerized glasses, they need to manually browse a large number of suggestions to complete the task.

Method used

By wearing computerized glasses, using scene data and user input, the automation assistant renders selectable suggestions on the glasses interface, limits relevant suggestions within the current visual range, uses directional GUI elements to indicate external objects, and dynamically adjusts the set of suggestions to Improve user experience and resource utilization.

Benefits of technology

It improves the efficiency of users interacting with automation assistants in unfamiliar environments, saves time and resources, ensures the effective utilization of computerized glasses' computing and display resources, and improves user experience and security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072673A_ABST
    Figure CN119072673A_ABST
Patent Text Reader

Abstract

Implementations set forth herein relate to an automated assistant that may render selectable suggestions at a display interface of computerized glasses and may adapt the suggestions according to changes in a user's gaze direction and / or other further input from the user. The selectable suggestions may initially be rendered based on contextual data that may be associated with a user that directs its gaze into an environment containing different environmental characteristics. The automated assistant may identify particular environmental features to predict interest to the user, and may adapt the selectable suggestions when the user expresses interest in the particular features. A user's interest in a particular environmental feature may be expressed by redirecting its gaze toward the particular feature and / or providing further input related to the particular feature.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Humans may engage in human-computer conversations with interactive software applications, referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when they interact with automated assistants) may provide commands and / or requests to automated assistants using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] As automated assistants become more common in everyday user lives, the functionality of automated assistants is evolving to cover an ever-expanding range of capabilities. While such evolution is often desirable to end users, such users may not have an easy way to discover how certain new features of automated assistants may benefit them. As a result, certain tasks undertaken by users may be more efficiently handled by automated assistants, but the users are unaware that the assistant functionality can help complete the tasks.

[0003] For example, a user exploring a foreign city may attempt to engage with third-party entities (such as a public transportation authority and / or a food delivery service) but may not realize that such third-party entities may be engaged through their automated assistant. Thus, a user who may be able to skillfully interact with their automated assistant in certain contexts (e.g., in their home) may not have any idea of ​​how to most efficiently interact with their automated assistant in unfamiliar contexts. In some cases, this may result in a waste of time and resources when a user manually interacts with their respective personal computing device (e.g., with a touch keyboard) without invoking their automated assistant to automatically handle certain tasks. Summary of the invention

[0004] The implementation described herein relates to an automated assistant that can be accessed via wearable computerized glasses and can render selectable suggestions at the interface of the computerized glasses based on the characteristics of the user's environment. For example, a user can explore a foreign city on a sidewalk while wearing computerized glasses. When the user looks across the street beside the sidewalk, various different locations and objects will appear within the visual range of the computerized glasses. While looking at various locations and objects, the user can provide at least partial input to the automated assistant, and in response, receive suggestions based on objects and locations within the current visual range of the computerized glasses. Although the objects and locations that are obvious within the current visual range of the computerized glasses may be extremely large in number (e.g., hundreds), the automated assistant can limit the number of suggestions to the number of suggestions that may be particularly relevant. Limiting the number of suggestions can save often constrained computing resources (e.g., constrained battery capacity) of computerized glasses and / or can solve the limited display interface of computerized glasses (e.g., limiting the effective area for displaying suggestions to prevent excessive blocking of the user's line of sight). As an example, the automated assistant can process context data to determine suggestions to be presented to the user via the interface of the computerized glasses. The contextual data may indicate location, time, gaze, and / or other information associated with the moment when a user solicited suggestions from an automated assistant while wearing the computerized glasses and gazing in a particular direction. In this way, time and effort may be saved by providing more relevant suggestions to a user who may be soliciting from their automated assistant, rather than having the user manually scroll through a larger number of suggestions in response to input to the automated assistant. Additionally, this may improve the experience with the computerized glasses, and suggestions in certain contexts may be limited in order to promote safety of the user and conserve resources of the computerized glasses.

[0005] In some implementations, suggestions may be rendered for a user at a display interface of the computerized glasses, and each suggestion may be selectable via input to the computerized glasses (e.g., verbal speech, input gesture, etc.). For example, a determination may be made to render N suggestions, where N is greater than 1, and only one suggestion may be rendered at a time via the computerized glasses (e.g., by verbal and / or touch input that enables cycling through suggestions rendered at a given time). When suggestions are generated for objects outside of a current visual range of the computerized glasses, a directional graphical user interface (GUI) element may be rendered at a display interface of the computerized glasses. The directional GUI element may optionally be selectable and may provide an indication that the suggestion is available for objects outside of a current visual range of the computerized glasses. In some implementations, the directional GUI element may be rendered simultaneously with one or more other selectable suggestions corresponding to objects within and / or outside of a current visual range of the computerized glasses. The directional GUI element may be smaller and less obtrusive than any suggestions being rendered, thereby enabling the user to be informed of the additional suggestions available and their direction relative to the user - while preventing multiple suggestions from being displayed simultaneously given the limited display interface of, for example, computerized glasses.

[0006] In some implementations, the set of selectable suggestions generated to be rendered at and / or rendered at the computerized glasses may change based on changes in user context, changes in the visual range of the computerized glasses, and / or additional input from one or more users. For example, while the user is wearing the computerized glasses and walking around in a foreign city, the user may provide invocation input (e.g., saying "Assistant...", tapping the computerized glasses, etc.) and / or other partial input to the automated assistant. In response, a first set of one or more selectable suggestions may be rendered for the user using suggestion data generated using currently available contextual data. Thereafter, when a change in context, subsequent input, gaze and / or focus direction, and / or a change in the visual range of the computerized glasses is detected, a second set of one or more other selectable suggestions may be rendered in response.

[0007] For example, a user who directs his gaze across the street to a row of restaurants can ask the automated assistant via his computerized glasses to provide help in the current context. The user can ask the automated assistant by providing a call input and / or by otherwise looking in a specific direction in a manner indicating that the user wants to receive suggestions about what they are looking at. In response, the automated assistant can render a first set of selectable suggestions, which can be rendered at the display interface of the computerized glasses. The selectable suggestions can be selected and / or generated based on the user's context and / or one or more objects present within the visual range of the computerized glasses. For example, the one or more objects may include 3 different restaurants, and the first selectable suggestion in the selectable suggestions may include a selectable GUI element for accessing the website of each corresponding restaurant. When the user redirects his gaze, and / or otherwise causes the visual range of the computerized glasses to move to a specific restaurant, the automated assistant can render a second set of selectable suggestions at the display interface. The second set of selectable suggestions can be selected and / or generated based on the updated context in which the user is more focused on a specific restaurant. Thus, the second set of selectable suggestions may include one or more selectable GUI elements that provide additional information about a particular restaurant - rather than providing information for three different restaurants.

[0008] In some implementations, the automated assistant may cause changes to the set of suggestions to appear at the computerized glasses and / or other assistant-enabled devices in response to the user continuing to provide spoken utterances. For example, while the user is looking at 3 different restaurants, the user may invoke the automated assistant by providing a partial spoken utterance (such as "Assistant..."). In response, the automated assistant may proactively provide a first set of selectable suggestions at the display interface of the computerized glasses. When the user continues to provide another portion of the spoken utterance, such as "tell me about Syria Palace", the automated assistant may process the audio corresponding to the additional spoken utterance to facilitate updating the suggestions being rendered. For example, and in response to the additional spoken utterance, the automated assistant may cause a second set of selectable suggestions to be rendered. The second set of selectable suggestions may include a link to a digital menu for a particular restaurant (e.g., Syria Palace), a link to a "Syrian Food" webpage, and a link to a blog reviewing a particular restaurant.

[0009] In some implementations, changes to selectable suggestions may occur in response to changes in the user's context. For example, with the user's prior permission, the automated assistant may determine that the user has received a calendar invitation with a specified location for an event. The calendar invitation may be received when the user directs the visual range of the computerized glasses to 3 restaurants, and in response, the automated assistant may cause a second set of selectable suggestions to be rendered. In this example, the second set of selectable suggestions may include content based on the calendar invitation and one or more objects in the environment with the user with the user's prior permission. For example, the user may be looking at 3 restaurants, but the calendar invitation may mention a location that is behind the user when the user is looking at the 3 restaurants. Based on the context, the automated assistant may cause a directional GUI element to be rendered at the edge of the display interface, indicating that a certain basis of the subject matter of the second set of selectable suggestions is behind the user or otherwise outside the visual range of the computerized glasses.

[0010] Alternatively or additionally, a second set of selectable suggestions may be rendered with a directional GUI element, thereby associating the second set of selectable suggestions with a location behind the user. Alternatively or additionally, there may be multiple directional GUI elements indicating the locations of certain objects that the user may be interested in in the current scene. A specific directional GUI element pointing behind the user may have one or more features that distinguish the specific directional GUI element from other directional GUI elements. In some implementations, when the user redirects his or her gaze to a different directional GUI element and / or a different object corresponding to the different directional GUI element (e.g., a no parking sign), a third set of selectable suggestions (e.g., a link to a local government website) may be rendered to supplement any user interest expressed in the different objects.

[0011] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those implementations and other implementations will be described in more detail below.

[0012] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform a method such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include a system of one or more computers including one or more processors operable to execute the stored instructions to perform a method such as one or more of the methods described above and / or elsewhere herein.

[0013] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Illustrated is a view of a user wearing computerized glasses according to some implementations discussed herein.

[0015] Figure 2A , Figure 2B , Figure 2C and Figure 2D Illustrated is a view of a user interacting with an automated assistant via computerized glasses such that the automated assistant renders various suggestions as the user changes their gaze and / or provides further input.

[0016] Figure 3 A system is illustrated that provides an automated assistant for rendering suggestions that can be rendered at a display interface of computerized glasses and that can dynamically adapt based on changes in user gaze and / or other input to the automated assistant.

[0017] Figure 4 A method for providing selectable suggestions at a display interface of computerized glasses based on characteristics of an environment of a user wearing the computerized glasses and / or contextual data associated with the user is illustrated.

[0018] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0019] Figure 1A view 100 of a user 102 wearing computerized glasses 104 according to some implementations discussed herein is illustrated. The computerized glasses 104 may include a computer 108, which may include one or more processors and / or one or more memory devices, and may receive power from one or more energy sources (e.g., batteries, wireless power transmission, etc.). The computer 108 may be at least partially embodied by a housing 114 and / or may be detachable from the housing 114. The housing 114 may resemble a frame of one or more different types of glasses and may have one or more lenses 106 attached to the housing 114. In some implementations, the computerized glasses 104 may include one or more front-facing cameras 110, which may be arranged to have a visual range that may include the field of view of the user 102. In some implementations, the computerized glasses 104 may include one or more inward-facing cameras 112, which may be arranged to have another visual range that includes one or more eyes of the user 102, with prior permission of the user 102. For example, one or more inward-facing cameras 112 may be arranged to capture image data representative of the position of the left eye and / or right eye of the user 102. In some implementations, the computer 108 may be connected to one or more antennas and / or other communication hardware that allows the computer 108 to communicate with one or more other computing devices. For example, the computerized glasses 104 may be connected to a Wi-Fi network, an LTE network, and / or may communicate via a Bluetooth protocol and / or any other communication modality.

[0020] In some implementations, one or more lenses 106 may operate as a display interface for rendering graphical content visible to a user wearing the computerized glasses 104. The graphical content rendered at the lenses 106 may assist the user 102 when the user is in certain environments. For example, the user 102 may direct his head and the computerized glasses 104 in a direction such that various environmental features (e.g., objects) are within the visual range of the front-facing camera 110. When the user 102 directs the computerized glasses 104 in the direction, the user 102 may provide a spoken utterance, for example, to cause an automated assistant to render one or more selectable suggestions related to the various environmental features. The automated assistant may detect the spoken utterance and, in response, cause a plurality of selectable suggestions to be rendered at the lenses 106. For example, a first selectable suggestion (e.g., a link to a city website) may be rendered in the lenses 106 to appear above a first environmental feature (e.g., a parking meter), and a second selectable suggestion (e.g., a link to a taxi company website) may be rendered in the lenses 106 to appear above a second environmental feature (e.g., a taxi parked near a parking meter).

[0021] In some implementations, each selectable suggestion may be "greyed out," blurred, flashing, and / or otherwise have one or more features indicating that the user 102 has not yet selected any of the selectable suggestions. In order to select one of the selectable suggestions, the user 102 may adjust the direction of their gaze and / or computerized glasses 104 to be more toward the first selectable suggestion or the second selectable suggestion. In response, the automated assistant may detect the adjustment of the gaze and / or facing direction of the user 102, and cause the first selectable suggestion or the second selectable suggestion to provide feedback that one has been selected. For example, when the user 102 directs their gaze and / or computerized glasses 104 more toward the first environmental feature, the first selectable suggestion may flash, shake, become idle, no longer gray, no longer blurry, and / or otherwise indicate that the first selectable suggestion has been selected. In this way, the user 102 can receive feedback that they have selected a particular selectable GUI element, and if they prefer to select a different selectable GUI element, they can redirect their gaze and / or computerized glasses 104. In some implementations, if user 102 is satisfied with their selection, user 102 may continue to look at the first selectable suggestion for a threshold period of time, or look away from both selectable suggestions, in order to confirm their selection and cause the first selectable suggestion to become activated.

[0022] Figure 2A , Figure 2B , Figure 2C and Figure 2D The user 202 is illustrated interacting with the automated assistant via the computerized glasses 206 so that the automated assistant renders various suggested views 200, 220, 240, and 260 as the user 202 changes his gaze and / or provides further input. The user 202 may be located in an environment 212, such as a busy city street, that includes various environmental features that the user 202 may be interested in receiving additional information about. Determining specific environmental features that the user 202 may be interested in relative to other features may be performed based on contextual data, which may be processed during operations 208 performed during or before the user 202 is in the environment 212. In some implementations, the contextual data may be generated by the computerized glasses 206 and / or another computing device in communication with the computerized glasses 206. For example, another portable computing device (e.g., a cellular phone 204) may communicate with the computerized glasses 206 with the prior permission of the user 202 to provide location data, time data, assistant data, application data, and / or any other data that may be relevant to the current context of the user 202.

[0023] In some implementations, the contextual data may indicate that the user 202 recently browsed for restaurants via their cell phone 204, which may alert the automated assistant that the user 202 may be looking for a particular dining location. Based on this determination, the automated assistant may cause any food-related environmental features that are apparent within the visual range of the computerized glasses 206 to be highlighted or otherwise identified via a GUI element 226 rendered at the display interface of the computerized glasses 206. For example, a front-facing camera 222 of the computerized glasses 206 may capture an image of the environment that the user 202 is facing, and optionally, a rear-facing camera 224 may be used to determine a gaze direction 228 of the user 202. The image may be processed as part of the contextual data to identify specific environmental features that may be of interest to the user 202. When a specific environmental feature is identified, the automated assistant may cause the GUI element 226 to indicate that it has been identified as an environmental feature that the user 202 is expected to be interested in. In some implementations, the specific GUI element 226 to which the gaze direction 228 is estimated to be most directed may have a visual feature that is different from another visual feature shared by other GUI elements 226. For example, and as Figure 2B As illustrated, the fill color of the leftmost GUI element 226 may be different than the fill colors of the other GUI elements 226 because the gaze direction 228 of the user 202 is directed more toward the leftmost GUI element 226 .

[0024] In some implementations, contextual data may be processed to generate suggestion data, which may characterize a suggestion set 232 that may be rendered as a selectable suggestion element 230 at a display interface of computerized glasses 206. In some implementations, suggestion set 232 may be generated based on gaze direction 228 of user 202 and / or environmental features present within the visual range of computerized glasses 206. For example, contextual data (such as location data and time data) may indicate that a plurality of different Italian restaurants are open near the location of user 202, and additionally, contextual data may also indicate that user 202 has previously listened to an Italian restaurant podcast. Based on this information, and that user 202 is looking at an Italian restaurant, the automated assistant may cause GUI element 226 to appear above and / or near the line of sight of the restaurant, and also cause suggestion set 232 to appear, the suggestion set having suggestions about Italian restaurants and / or Italian podcasts. In some implementations, the suggestion set 232 can appear in both lenses of the computerized glasses 206 such that the user 202 can view the suggestion set 232 as occupying a three-dimensional (3D) perspective of the environment, optionally along with the GUI elements 226 .

[0025] In some implementations, a particular location (e.g., a first location) that is the subject of a selectable suggestion element 230 can be assigned a GUI element 226 that is different from GUI elements 226 assigned to other locations (e.g., a second location) that are within and / or not within the visual range of the computerized glasses 206. In some implementations, a GUI element 234 corresponding to a location outside the visual range of the computerized glasses 206 can be rendered to indicate directions to the shortest route to the location. The location can be identified based on contextual data, the gaze of the user 202, and / or other input from the user 202, thereby allowing the user 202 to be aware of potential locations of interest that may be outside the current field of view. Figure 2B As illustrated, the GUI element 234 may be rendered with a directional symbol that points to the right of the user 202 and outside of the current visual range of the computerized glasses 206, for example.

[0026] In some implementations, the suggestion set 232 can be updated based on updated contextual data being processed by the automated assistant, subsequent input being received by the automated assistant, changes in the gaze direction 228 of the user 202, changes in the visual range of the computerized glasses 206, and / or any other changes that the automated assistant can determine with the prior permission of the user 202. For example, the user 202 can provide a spoken utterance 246, which can be embodied in audio data processed by the automated assistant with the prior permission of the user 202. The spoken utterance 246 can be at least a partial utterance that may or may not include an invocation phrase (e.g., "Assistant...") and / or may not include an explicit command (e.g., "Find movies."). In contrast, if Figure 2C As illustrated, spoken utterance 246 may be at least a partial utterance that may provide further context and help the automated assistant refine the suggestions rendered at the display interface of computerized glasses 206 .

[0027] Alternatively or additionally, user 202 may change their gaze direction 228 toward a different direction 242, and the different direction may be used as a basis for refining the suggestions rendered by the automated assistant. For example, user 202 may redirect their gaze from being focused on the Italian restaurant to being focused on the movie theater. In response, the automated assistant may cause GUI element 226 to no longer be rendered at the display interface of computerized glasses 206. Additionally, the automated assistant may cause additional GUI elements 244 to be rendered to appear at or near an environmental feature (e.g., a movie theater) to which user 202 may have directed their gaze and / or to which the user may have provided input (e.g., verbal utterance 246) related to the environmental feature.

[0028] In some implementations, in response to the spoken utterance 246 and / or the user 202 redirecting their gaze, the automated assistant can cause a different set of suggestions 264 to be generated and rendered at the display interface of the computerized glasses 206. For example, the different set of suggestions can be generated based on the spoken utterance 246 and / or the environmental features and / or the lack of previous environmental features within the visual range of the computerized glasses 206. For example, because the user 202 adjusted their gaze to be more directed toward the Italian cinema (e.g., indicated by the additional GUI element 244), the different set of suggestions 264 can be rendered with cinema-related suggestions. In some implementations, the different set of suggestions can include selectable suggestions that can be selected via another spoken utterance (e.g., "Show me what movies are playing tonight") that may not include an invocation phrase for invoking the automated assistant. Alternatively or additionally, the user 202 may select a particular selectable suggestion (e.g., “Call the Italian movie theater”) by gesturing without directly contacting the computerized glasses 206 (e.g., by gesturing with their hand in front of the front-facing camera 222 in a manner indicating interest in the particular selectable suggestion). It should be noted that while the selectable suggestions 232 and the distinct selectable suggestions 264 are illustrated as being remote from the computerized glasses 206, these selectable suggestions are intended to appear within the field of view of the user 202 as a result of being rendered at the display interface of the computerized glasses 206.

[0029] Figure 3A system 300 is illustrated that provides an automated assistant 304 for rendering suggestions that can be rendered at a display interface of computerized glasses and can be dynamically adapted based on changes in user gaze and / or other inputs to the automated assistant 304. The automated assistant 304 can operate as part of an assistant application provided at one or more computing devices (such as computing device 302 and / or server device). A user can interact with the automated assistant 304 via an assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 304 by providing speech, text, and / or graphical input to the assistant interface 320 so that the automated assistant 304 initializes one or more actions (e.g., providing data, controlling peripheral devices, accessing an agent, generating input and / or output, etc.). Alternatively, the automated assistant 304 can be initialized based on processing context data 336 using one or more trained machine learning models. Context data 336 may characterize one or more characteristics of an environment accessible to automated assistant 304, and / or one or more characteristics of a user predicted to be intended to interact with automated assistant 304. Computing device 302 may include a display device, which may be a display panel, including a touch interface for receiving touch input and / or gestures to allow a user to control applications 334 of computing device 302 via the touch interface. In some implementations, computing device 302 may lack a display device, thereby providing audible user interface output rather than graphical user interface output. In addition, computing device 302 may provide a user interface, such as a microphone, for receiving spoken natural language input from a user. In some implementations, computing device 302 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0030] Computing device 302 and / or other third-party client devices may communicate with the server device via a network such as the Internet. Additionally, computing device 302 and any other computing devices may communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 302 may offload computing tasks to a server device in order to conserve computing resources at computing device 302. For example, a server device may host automated assistant 304, and / or computing device 302 may transmit input received at one or more assistant interfaces 320 to a server device. However, in some implementations, automated assistant 304 may be hosted at computing device 302, and various processes associated with automated assistant operations may be performed at computing device 302.

[0031] In various implementations, all or some aspects of automated assistant 304 may be implemented on computing device 302. In some of these implementations, aspects of automated assistant 304 are implemented via computing device 302 and may interface with a server device that may implement other aspects of automated assistant 304. The server device may optionally serve multiple users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of automated assistant 304 are implemented via computing device 302, automated assistant 304 may be an application separate from an operating system of computing device 302 (e.g., installed “on top” of an operating system), or alternatively, may be implemented directly by an operating system of computing device 302 (e.g., treated as an application of the operating system, but integrated with the operating system).

[0032] In some implementations, automated assistant 304 may include an input processing engine 306 that may employ a number of different modules to process input and / or output of computing device 302 and / or server devices. For example, input processing engine 306 may include a speech processing engine 308 that may process audio data received at assistant interface 320 to identify text embodied in the audio data. The audio data may be transmitted from, for example, computing device 302 to a server device in order to conserve computing resources at computing device 302. Additionally or alternatively, the audio data may be processed exclusively at computing device 302.

[0033] The process for converting audio data to text may include a speech recognition algorithm (neural network may be used) and / or a statistical model for identifying an audio data group corresponding to a word or phrase. The text converted from the audio data may be parsed by the data parsing engine 310 and made available as text data for the automated assistant 304, which may be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by the user. In some implementations, the output data provided by the data parsing engine 310 may be provided to the parameter engine 312 to determine whether the user has provided input corresponding to a specific intention, action, and / or routine that the automated assistant 304 can perform and / or an application or agent that can be accessed via the automated assistant 304. For example, the assistant data 338 may be stored at a server device and / or a computing device 302, and may include data defining one or more actions that the automated assistant 304 can perform, and the parameters required to perform these actions. The parameter engine 312 may generate one or more parameters for the intention, action, and / or slot value, and provide one or more parameters to the output generation engine 314. Output generation engine 314 may use the one or more parameters to communicate with assistant interface 320 to provide output to a user, and / or to communicate with one or more applications 334 to provide output to one or more applications 334 .

[0034] In some implementations, automated assistant 304 may be an application that may be installed “on top of” an operating system of computing device 302, and / or may itself form part of (or the entirety of) the operating system of computing device 302. The automated assistant application includes and / or has access to on-device speech recognition, on-device natural language understanding, and on-device fulfillment. For example, on-device speech recognition may be performed using an on-device speech recognition module that processes audio data (detected by a microphone) using an end-to-end speech recognition machine learning model stored locally at computing device 302. On-device speech recognition generates recognized text for spoken utterances (if any) present in the audio data. Additionally, for example, on-device natural language understanding (NLU) may be performed using an on-device NLU module that processes the recognized text generated using on-device speech recognition and, optionally, context data to generate NLU data.

[0035] The NLU data may include an intent corresponding to the spoken utterance and optionally parameters for the intent (e.g., slot values). The on-device fulfillment may be performed using an on-device fulfillment module that utilizes the NLU data (from the on-device NLU) and optionally other local data to determine an action to be taken to resolve the intent (and optionally parameters for the intent) of the spoken utterance. This may include determining a local and / or remote response (e.g., answer) to the spoken utterance, an interaction with a locally installed application performed based on the spoken utterance, a command sent to an Internet of Things (IoT) device (directly or via a corresponding remote system) based on the spoken utterance, and / or other resolution actions performed based on the spoken utterance. The on-device fulfillment may then initiate a local and / or remote performance / execution of the determined action to resolve the spoken utterance.

[0036] In various implementations, remote speech processing, remote NLU, and / or remote fulfillment may be at least selectively utilized. For example, the recognized text may be at least selectively transmitted to a remote automated assistant component for remote NLU and / or remote fulfillment. For example, the recognized text may be optionally transmitted for remote execution in parallel with on-device execution or in response to a failure of on-device NLU and / or on-device fulfillment. However, on-device speech processing, on-device NLU, on-device fulfillment, and / or on-device execution may be prioritized at least due to the reduced latency they provide when resolving spoken utterances (attributed to the fact that no client-server round trip is required to resolve spoken utterances). In addition, in situations where there is no network connection or a limited network connection, on-device functionality may be the only functionality available.

[0037] In some implementations, computing device 302 may include one or more applications 334, which may be provided by a third-party entity that is different from the entity providing computing device 302 and / or automated assistant 304. Automated assistant 304 and / or an application state engine of computing device 302 may access application data 330 to determine one or more actions that one or more applications 334 are capable of performing, as well as a state of each of one or more applications 334 and / or a state of a corresponding device associated with computing device 302. Automated assistant 304 and / or a device state engine of computing device 302 may access device data 332 to determine one or more actions that computing device 302 and / or one or more devices associated with computing device 302 are capable of performing. Additionally, automated assistant 304 can access application data 330 and / or any other data (e.g., device data 332) to generate context data 336, which can characterize the context in which a particular application 334 and / or device is executing, and / or the context in which a particular user is accessing computing device 302, accessing application 334, and / or any other device or module.

[0038] While one or more applications 334 are executing at computing device 302, device data 332 may characterize the current operating state of each application 334 being executed at computing device 302. Additionally, application data 330 may characterize one or more features of executing applications 334, such as the content of one or more graphical user interfaces being rendered at the direction of one or more applications 334. Alternatively or additionally, application data 330 may characterize action paradigms based on the current operating state of the respective applications, which action paradigms may be updated by the respective applications and / or automated assistant 304. Alternatively or additionally, one or more action paradigms of one or more applications 334 may remain static, but may be accessed by the application state engine in order to determine appropriate actions to be initiated via automated assistant 304.

[0039] The computing device 302 may further include an assistant invocation engine 322 that can use one or more trained machine learning models to process application data 330, device data 332, context data 336, and / or any other data accessible to the computing device 302. The assistant invocation engine 322 can process the data to determine whether to wait for the user to explicitly say an invocation phrase to invoke the automated assistant 304, or to treat the data as indicating the user's intention to invoke the automated assistant—instead of requiring the user to explicitly say an invocation phrase. For example, one or more trained machine learning models can be trained using examples of training data that are based on scenarios in which the user is in an environment where multiple devices and / or applications are exhibiting various operating states. Instances of training data can be generated to capture training data that characterizes contexts in which the user invokes the automated assistant and other contexts in which the user does not invoke the automated assistant. When training one or more trained machine learning models based on these instances of training data, the assistant invocation engine 322 can cause the automated assistant 304 to detect or limit detection of spoken invocation phrases from the user based on features of the context and / or environment. Additionally or alternatively, assistant invocation engine 322 can cause automated assistant 304 to detect or limit detection of one or more assistant commands from a user based on characteristics of the context and / or environment. In some implementations, assistant invocation engine 322 can be disabled or limited based on computing device 302 detecting an assistant suppression output from another computing device. In this way, when computing device 302 detects an assistant suppression output, automated assistant 304 is not invoked based on context data 336—otherwise, automated assistant 304 is caused to be invoked if no assistant suppression output is detected.

[0040] In some implementations, the system 300 may include a context processing engine 316 that may process context data 336, device data 332, application data 330, and / or assistant data 338 to determine characteristics of a user's environment. The context processing engine 316 may also process data to facilitate determining an estimated level of interest of the user for each specific environmental feature in the environment. When the estimated level of interest in a specific environmental feature satisfies a threshold level of interest, the specific environmental feature may be identified for the environmental feature engine 318. In some implementations, one or more heuristic processes and / or one or more trained machine learning models may be used to determine the level of interest. For example, one or more trained machine learning models may be used to process data accessible to the context processing engine 316 to facilitate determining specific topics that may be of interest to the user (with prior permission of the user) at any particular time, location, and / or in a particular context. As a result of the processing, a specific topic may be identified, and an estimated level of interest of the user for a specific environmental feature may be determined based at least on the specific topic. Particular environmental features may be identified to environmental features engine 318 when such features are determined to have an estimated user level of interest that satisfies a threshold level of interest (eg, as determined by automated assistant 304 based on a given context).

[0041] Environmental feature engine 318 may process data characterizing each identified environmental feature to determine the type of environmental feature that the user may be interested in when in a particular context. For example, while estimating that the user may be interested in a variety of different environmental features, automated assistant 304 may generate comparison data to compare the characteristics of a particular environmental feature (e.g., determining that the particular environmental feature is a restaurant, a car, a street sign, etc.). One or more heuristic processes and / or one or more trained machine learning models may be used to classify these characteristics to identify a characteristic type for a set of environmental features. Selectable suggestion engine 326 may then use this identified type to render selectable suggestions that may correspond to the type of environmental feature (e.g., suggestions for only restaurants and / or restaurant menu items that can be found near the user). For example, when a user directs their gaze to a neighborhood with multiple restaurants, environmental feature engine 318 may determine that the user may be interested in the "restaurant" environmental feature type. Based on this determination, as well as data from context processing engine 316, selectable suggestion engine 326 may generate selectable suggestions corresponding to the "restaurant" environmental feature type. The selectable suggestions may, for example, provide links (e.g., web pages, application interfaces, etc.) to information about multiple restaurants on a block (such as menu items available at each particular restaurant). In some implementations, the selectable suggestions may be rendered as selectable GUI elements at a display interface of the computerized glasses. When the user directs their gaze to a particular environmental feature, one or more selectable suggestions and / or GUI elements corresponding to the particular environmental feature may be highlighted and / or otherwise made to appear different from other selectable suggestions and / or other GUI elements.

[0042] In some implementations, the system 300 may include a suggestion adaptation engine 324 that can cause selectable suggestions rendered by the automated assistant 304 to adapt based on any changes in context, changes in gaze, and / or changes in other information that may be available to the automated assistant 304 with the user's prior permission. For example, data captured by the inward-facing camera or the rear-facing camera can be processed to determine whether the user has changed their gaze to redirect to another environmental feature. When it is determined that the user has redirected their gaze, data associated with another environmental feature can be processed to generate another set of selectable suggestions that can be rendered at the display interface of the computerized glasses. Alternatively or additionally, when the user does not redirect their gaze but otherwise provides subsequent input to the automated assistant 304 and / or demonstrates a change in context, the selectable suggestions can be adapted. In this way, the user does not have to manually select suggestions that are not of particular interest to the user. Conversely, the user can influence the fluidity of the suggestions being rendered via automated assistant 304 by providing relatively brief inputs to automated assistant 304 and / or expressing interest in various objects as they might normally express (e.g., looking at and / or moving away from objects that they are less interested in, and looking at and / or moving toward objects that they are more interested in).

[0043] Figure 4 A method 400 for providing selectable suggestions at a display interface of computerized glasses based on characteristics of an environment of a user wearing computerized glasses and / or contextual data associated with the user is illustrated. The method 400 may be performed by one or more computing devices, applications, and / or any other device or module that may be associated with an automated assistant. The method 400 may include an operation 402 of processing contextual data associated with a user located in an environment and wearing computerized glasses. The contextual data may be based on the user's current context, the user's expected context, and / or the user's previous context. For example, the contextual data may include data generated by one or more sensors integrated with the computerized glasses and / or integrated with another computing device that communicates with the computerized glasses. The contextual data may include images, audio, video, location data, time data, and / or any other information of the user's environment with the user's prior permission, and thus may capture various characteristics of the environment. Alternatively or additionally, the contextual data may include data based on one or more previous interactions between the user and the automated assistant, and / or between one or more other users and their respective automated assistants (with the user's prior permission).

[0044] Method 400 may proceed from operation 402 to operation 404: determining environmental features of the user environment based on processing context data. In some implementations, one or more heuristic processes and / or machine learning models may be used to determine features of the user environment. For example, a sensor of the computerized glasses may be used to determine the user's gaze direction, another sensor may be used to determine the user's coordinate location, and / or other data may be used to determine the current time when the user is looking at his environment. Based on this information, an automated assistant accessible via the computerized glasses may identify certain features of interest that are within the visual range of the computerized glasses. For example, the automated assistant may process information using one or more trained machine learning models to determine natural language understanding of signs that may be within the visual range of the computerized glasses. As an example, the sign may include a variety of different kitchenware that is being advertised for sale at a store across the street from the user (e.g., the user may be staring at the store across the street). Therefore, the automated assistant can identify each specific kitchenware (e.g., a food processor, a kitchen robot, an espresso machine).

[0045] Method 400 may proceed from operation 404 to operation 406, which may include generating suggestion data characterizing one or more assistant operations associated with environmental features. In some implementations, the suggestion data may include links, and each link may correspond to a corresponding feature within the environment. When a particular link is selected, the automated assistant may cause information stored in association with the link to be rendered at a display interface of the computerized glasses. The information may include, for example, additional selectable suggestions that may be activated by the user and / or the automated assistant, content of one or more web pages that may be accessible via the computerized glasses, content generated by one or more applications associated with the automated assistant, content generated by one or more devices communicating with the computerized glasses, and / or any other information that may be associated with the automated assistant. For example, when a user is looking at various kitchenware advertisements, the suggestion data may include website links for various kitchenware observed within the visual range of the computerized glasses.

[0046] Method 400 may optionally proceed from operation 406 to operation 408, which may include determining whether the user solicits help from the automated assistant when or before directing their gaze to the environmental feature. The user may solicit help from the automated assistant by providing direct and / or indirect input to the automated assistant via computerized glasses and / or another computing device. For example, the user may provide an invocation input (e.g., a tap gesture, verbal input, etc.) to cause the automated assistant to be invoked and provide any help that may be necessary in the current context to the user. For example, when the user is looking at a kitchenware advertisement across the street, the user may tap the computerized glasses with their finger as an invocation input. Alternatively or additionally, with the user's prior permission, the automated assistant may interpret the user's gaze at one or more specific objects as an invocation input to which the automated assistant may respond. For example, when the user looks at the kitchenware advertisement through the lenses of the computerized glasses for a threshold duration, the automated assistant may be invoked and suggestions may be provided to the user.

[0047] When it is determined that the user has asked the automated assistant for help when directing his gaze to the environmental feature or before, the method 400 may proceed from operation 408 to operation 410. Otherwise, the method 400 may proceed from operation 408 to operation 416: waiting for further input and / or selection timeout for the automated assistant. Operation 410 may include rendering a specific set of one or more selectable suggestions at the display interface of the computerized glasses. In some implementations, a set of one or more selectable suggestions may be rendered so that each selectable suggestion is rendered to appear near and / or on top of a corresponding object visible within the visual range of the computerized glasses associated with the selectable suggestion. Alternatively or in addition, a set of one or more selectable suggestions may be rendered in an organized array at a portion of the display interface of the computerized glasses. The user may select a specific selectable suggestion by providing an input gesture, a spoken word, and / or any other input for a specific selectable suggestion. However, in some implementations, the user may decide to provide further input and / or adjust his gaze, thereby causing the automated assistant to change the set of selectable suggestions being rendered.

[0048] From operation 410, method 400 may proceed to operation 412: determining whether the user has changed the direction of their gaze and / or providing further input to the automated assistant. The user may change the direction of their gaze with or without moving their head and with or without moving the position of their eyes. For example, one or more sensors of the computerized glasses may detect a change in the visual range of the computerized glasses without determining the user's gaze, and infer that the user's gaze direction has changed. Alternatively or in addition, one or more sensors of the computerized glasses may determine that the position of the user's eyes has changed, thereby indicating that the user's gaze has changed. In some implementations, one or more additional environmental features to which the user directed their gaze may be identified, and the additional environmental features may be used as a basis for rendering a different set of one or more selectable suggestions. Alternatively or in addition, the user may provide subsequent input, such as a spoken utterance (e.g., a partial spoken utterance, such as "show me where..."), and the subsequent input may be used as a basis for rendering a different set of one or more selectable suggestions.

[0049] When it is determined that the user has changed their gaze and / or provided further input, method 400 may proceed from operation 412 to operation 414: causing the currently rendered set of one or more selectable suggestions to be changed at the display interface. For example, in response to the user changing their gaze direction and / or providing further input, the automated assistant may cause one or more selectable suggestions in the set of selectable suggestions to no longer be rendered at the display interface. In some implementations, specific selectable suggestions may be selected for removal from the display interface based on one or more features of the environment no longer appearing within the visual range of the computerized glasses. Method 400 may proceed from optional operation 414 to operation 404: determining an environmental feature of the environment that may be apparent within the altered visual range of the computerized glasses and / or may be within the gaze direction of the user.

[0050] In this way, a user may explicitly and / or implicitly solicit selectable suggestions from their automated assistant while wearing the computerized glasses, and these suggestions may be updated based on the user's changing interests. This may eliminate the need for the user to provide additional input to receive updated suggestions, thereby saving computing resources and / or power resources of the computerized glasses. Additionally, as the user interacts with suggestions rendered via the computerized glasses, the machine learning models may (with the user's prior permission) be further trained using training data generated during such interactions. Further training certain machine learning models in this manner may allow the automated assistant to provide more accurate suggestions, thereby additionally reducing the number of suggestions that the user may have to "scroll" through before identifying a suitable suggestion to select. This may further save computing resources and / or power resources at the computerized glasses. Such resources may be essential when the user is traveling away from home and may not have resources immediately available to charge their device and / or wait for certain processes to complete.

[0051] Figure 5 5 is a block diagram 500 of an example computer system 510. The computer system 510 generally includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input devices and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0052] The user interface input device 522 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), an interface for computerized glasses, a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system, a microphone), and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system 510 or onto a communication network.

[0053] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanisms for generating visible images. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 510 to a user or another machine or computer system.

[0054] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400 and / or implementing one or more of system 300, automated assistant, computerized glasses, and / or any other applications, devices, apparatuses, and / or modules discussed herein.

[0055] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored by the file storage subsystem 526 in the storage subsystem 524, or in other machines accessible to the processor 514.

[0056] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0057] Computer system 510 may be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The depicted description of computer system 510 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 510 are possible, and these configurations are similar to those of FIG. Figure 5 The computer systems depicted may have more or fewer components than those shown.

[0058] In situations where the systems described herein collect or make available personal information about users (or "participants" as they are often referred to herein), users may be provided with the opportunity to control whether a program or feature collects information about the user (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Additionally, certain data may be processed in one or more ways prior to its storage or use so that personally identifiable information is removed. For example, the user's identity may be processed so that personally identifiable information about the user cannot be determined, or, in the case of geographic location information, the user's geographic location may be generalized (such as to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may control how information about the user is collected and / or how that information is used.

[0059] Although several implementations have been described and illustrated herein, various other devices and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications of the teachings used. Those skilled in the art will recognize or be able to confirm many equivalents of the specific implementations described herein using only routine experiments. Therefore, it should be understood that the aforementioned implementations are presented only by way of example, and it should be understood that within the scope of the attached claims and their equivalents, implementations may be practiced in a manner other than specifically described and claimed. The implementations disclosed herein relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not contradict each other.

[0060] In some implementations, a method implemented by one or more processors is described as including operations such as: processing context data generated at computerized glasses being worn by a user in an environment, the environment having features characterized by the context data. The computerized glasses provide access to an automated assistant that the user can invoke via an interface of the computerized glasses and generates context data using one or more sensors integrated with the computerized glasses and / or integrated with a different computing device in communication with the computerized glasses. The method may further include: generating suggestion data representing one or more operations based on processing the context data, the one or more operations being associated with the user's environment and being initialized by the automated assistant when the user is in the environment. The method may further include: determining that the user has provided an invocation input to the automated assistant while the user is wearing the computerized glasses and in the environment. Receiving the invocation input at the interface of the computerized glasses. The method may further include: causing a set of one or more selectable suggestions to be rendered at a display interface of the computerized glasses based on the suggestion data and in response to the user providing the invocation input to the computerized glasses. User selection of a particular selectable suggestion in the set of one or more selectable suggestions causes the automated assistant to initialize execution of one or more operations associated with the user's environment.

[0061] These and other implementations disclosed herein may include one or more of the following features.

[0062] In some implementations, the method may further include: determining that the user has provided additional input to the automated assistant after the user provides the invocation input, wherein the additional input includes a spoken utterance that embodies a portion of a command phrase that the automated assistant can implement; and causing a distinct set of one or more selectable suggestions to be rendered at a display interface of the computerized glasses based on the portion of the command phrase. In some implementations, the contextual data further characterizes a specific time when the user wears the computerized glasses in the environment, and the suggestion data is generated further based on the specific time. In some implementations, the method may further include: determining that the user has adjusted a gaze direction while wearing the computerized glasses after the user provides the invocation input, wherein the gaze direction is determined using one or more sensors integrated with the computerized glasses; and causing a distinct set of one or more selectable suggestions to be rendered at a display interface of the computerized glasses based on the user adjusting the gaze direction.

[0063] In some implementations, the feature includes a first location and a second location, processing the context data includes determining that a previous interaction between the user and the automated assistant is associated with the first location, and generating the suggestion data further based on the previous interaction between the user and the automated assistant. In some implementations, causing the set of one or more selectable suggestions to be rendered at a display interface of the computerized eyewear includes causing a particular selectable suggestion to be rendered with an indication that the particular selectable suggestion is associated with the first location. In some implementations, causing the particular selectable suggestion to be rendered with an indication that the particular selectable suggestion is associated with the first location includes causing the particular selectable suggestion to be rendered at a location in the display interface that appears to be closer to the first location than the second location.

[0064] In some implementations, a method implemented by one or more processors is described as including operations such as: processing context data generated at a computerized eyewear being worn by a user in an environment, the environment having features characterized by the context data. The computerized eyewear provides access to an automated assistant that the user can invoke via an interface of the computerized eyewear. The method may further include: determining that the user has provided a verbal utterance to the automated assistant to facilitate receiving a suggestion about one or more places to go to in the environment. The method may further include: generating location data representing a corresponding location for each of the one or more places in the environment. Each corresponding location is determined based on the context data and a verbal utterance from the user. The method may further include: determining that a specific location of the location in the environment is outside the current visual range of the computerized eyewear. The current visual range of the computerized eyewear corresponds to the direction the user is facing when wearing the computerized eyewear. The method may further include: causing the automated assistant to render a graphical user interface (GUI) element at a display interface of the computerized eyewear. The GUI element indicates that the specific location of the location is outside the current visual range of the computerized eyewear.

[0065] These and other implementations disclosed herein may include one or more of the following features.

[0066] In some implementations, the GUI element indicates directions for a route starting from an edge of a display interface of the computerized glasses to a specific location. In some implementations, the method may further include: causing the automated assistant to render other GUI elements corresponding to other locations in one or more locations within the environment at the display interface, wherein the other locations are within the current visual range of the computerized glasses. In some implementations, the method may further include: determining based on the contextual data that the user's gaze is directed to a specific location in the one or more locations within the environment, wherein causing the other GUI elements to be rendered at the display interface includes: causing a specific GUI element in the other GUI elements to be rendered with a visual feature that is different from another visual feature of the GUI element. In some implementations, the method may further include: generating suggestion data representing selectable suggestions based on the location data and the spoken utterance, wherein the selection of a specific suggestion in the selectable suggestions causes the automated assistant to provide additional information about the specific suggestion; and causing one or more of the selectable suggestions to be rendered at the display interface of the computerized glasses together with the other GUI elements. In some implementations, the method may further include: determining, based on contextual data, that the user's gaze has changed after rendering one or more selectable suggestions at the display interface; and rendering one or more other selectable suggestions at the display interface based on determining that the user's gaze has changed, wherein the one or more other selectable suggestions are associated with a specific location to which the user directed his or her gaze.

[0067] In some implementations, a method implemented by one or more processors is described as including operations such as: processing context data generated at a computerized eyewear being worn by a user, the user being in an environment and directing their gaze toward a portion of the environment. The computerized eyewear provides access to an automated assistant, which the user can invoke via an interface of the computerized eyewear. The method may further include: generating suggestion data characterizing one or more assistant operations based on the processing of the context data, the one or more assistant operations being initialized by the automated assistant when the user is in the environment. When the user wears the computerized eyewear and directs their gaze toward the portion of the environment, the one or more assistant operations are associated with one or more environmental features visible within a current visual range of the computerized eyewear. The method may further include: causing a set of one or more selectable suggestions to be rendered at a display interface of the computerized eyewear based on the suggestion data and when the one or more environmental features are within the current visual range of the computerized eyewear. User selection of a particular selectable suggestion in the set of one or more selectable suggestions causes the automated assistant to initialize execution of a particular operation in the one or more assistant operations associated with the one or more environmental features visible within the current visual range of the computerized eyewear.

[0068] These and other implementations disclosed herein may include one or more of the following features.

[0069] In some implementations, processing the contextual data generated at the computerized glasses includes: determining an estimated interest level of the user in an environmental feature type visible within a current visual range of the computerized glasses based on the contextual data, wherein when the estimated interest level satisfies a threshold interest level, one or more environmental features correspond to the environmental feature type. In some implementations, generating suggestion data characterizing one or more assistant operations includes: generating comparison data that provides information for comparing characteristics of the one or more environmental features for each environmental feature corresponding to the environmental feature type. In some implementations, the environmental feature type includes a restaurant type, and the comparison data indicates menu items for each restaurant visible within the current visual range of the computerized glasses. In some implementations, the contextual data includes interaction data generated during an interaction between the user and the automated assistant before the user directs his gaze to the portion of the environment. In some implementations, the interaction data indicates that the user recognized the environmental feature type during the interaction between the user and the automated assistant. In some implementations, the contextual data includes interaction data generated during an interaction between the user and another application before the user directs his gaze to the portion of the environment.

Claims

1. A method implemented by one or more processors, the method comprising: processing contextual data generated at the computerized eyewear being worn by a user located in an environment having features characterized by the contextual data, wherein the computerized glasses provide access to an automated assistant, the user being able to invoke the automated assistant via an interface of the computerized glasses, and wherein the contextual data is generated using one or more sensors integrated with the computerized glasses and / or integrated with a distinct computing device in communication with the computerized glasses; generating suggestion data representative of one or more actions based on processing the contextual data, the one or more actions being associated with the environment of the user and initiatable by the automated assistant when the user is in the environment; determining that the user has provided an invocation input to the automated assistant while the user is wearing the computerized glasses and located in the environment, wherein receiving said call input at said interface of said computerized eyewear; as well as Based on the suggestion data and in response to the user providing the invocation input to the computerized glasses, causing a set of one or more selectable suggestions to be rendered on a display interface of the computerized glasses, Wherein user selection of a particular selectable suggestion from the set of the one or more selectable suggestions causes the automated assistant to initiate performance of the one or more operations associated with the environment of the user.

2. The method of claim 1, further comprising: determining that the user has provided additional input to the automated assistant after the user provided the invocation input, wherein the additional input comprises a spoken utterance embodying a portion of a command phrase executable by the automated assistant; and Causing a distinct set of one or more selectable suggestions to be rendered at the display interface of the computerized eyewear based on the portion of the command phrase.

3. The method according to claim 1 or claim 2, wherein the contextual data further characterizes a specific time when the user wears the computerized glasses in the environment, and The suggestion data is further generated based on the specific time.

4. The method of claim 1, further comprising: After the user provides the invocation input, determining that the user has adjusted a gaze direction while wearing the computerized glasses, wherein the gaze direction is determined using the one or more sensors integrated with the computerized glasses; and Based on the user adjusting the gaze direction, a distinct set of one or more selectable suggestions is rendered at the display interface of the computerized glasses.

5. A method as claimed in any preceding claim, wherein The feature includes a first position and a second position, Processing the contextual data includes determining that a previous interaction between the user and the automated assistant is associated with the first location, and The suggestion data is generated further based on the previous interactions between the user and the automated assistant.

6. The method of claim 5, wherein causing the set of one or more selectable suggestions to be rendered at the display interface of the computerized glasses comprises: Causes a particular selectable suggestion to be rendered with an indication that the particular selectable suggestion is associated with the first location.

7. The method of claim 6, wherein causing the particular selectable suggestion to be rendered with the indication that the particular selectable suggestion is associated with the first location comprises: The specific selectable suggestion is rendered in the display interface at a position that appears closer to the first position than the second position.

8. A method implemented by one or more processors, the method comprising: processing contextual data generated at the computerized eyewear being worn by a user located in an environment having features characterized by the contextual data, wherein the computerized glasses provide access to an automated assistant, the user being able to invoke the automated assistant via an interface of the computerized glasses; determining that the user has provided a spoken utterance to the automated assistant to facilitate receiving a suggestion regarding one or more places to travel to in the environment; generating, for each of the one or more locations in the environment, location data representative of the corresponding location, wherein each corresponding position is determined based on the contextual data and the spoken utterance from the user; determining that a particular location of a place in the environment is outside of a current visual range of the computerized glasses, wherein the current visual range of the computerized glasses corresponds to the direction the user is facing when wearing the computerized glasses; and causing the automated assistant to render graphical user interface (GUI) elements on a display interface of the computerized glasses, Wherein the GUI element indicates that the particular location of the place is outside the current visual range of the computerized glasses.

9. The method of claim 8, wherein the GUI element indicates directions for a route starting from an edge of the display interface of the computerized eyewear to the specific location.

10. The method of claim 8 or claim 9, further comprising: causing the automated assistant to render, at the display interface, other GUI elements corresponding to other of the one or more locations in the environment, Wherein the other location is within the current visual range of the computerized glasses.

11. The method of claim 10, further comprising: determining, based on the contextual data, that the user's gaze is directed toward a particular location among the one or more locations in the environment, The rendering of the other GUI elements on the display interface includes: A specific GUI element among the other GUI elements is caused to be rendered with a visual characteristic that is different from another visual characteristic of the GUI element.

12. The method of claim 10, further comprising: generating suggestion data representing a selectable suggestion based on the position data and the spoken utterance, wherein selection of a particular suggestion among the selectable suggestions causes the automated assistant to provide additional information about the particular suggestion; as well as The one or more of the selectable suggestions are caused to be rendered at the display interface of the computerized glasses along with the other GUI elements.

13. The method of claim 12, further comprising: determining, based on the contextual data, that a gaze of the user has changed after rendering the one or more selectable suggestions at the display interface; as well as Based on determining that the gaze of the user has changed, causing one or more other selectable suggestions to be rendered on the display interface, The one or more other selectable suggestions are associated with a particular location toward which the user directed his or her gaze.

14. A method implemented by one or more processors, the method comprising: processing contextual data generated at the computerized glasses being worn by a user who is located in an environment and directs his gaze toward a portion of the environment, wherein the computerized glasses provide access to an automated assistant, the user being able to invoke the automated assistant via an interface of the computerized glasses; generating suggestion data representative of one or more assistant operations based on processing the context data, the one or more assistant operations being initiatable by the automated assistant while the user is in the environment, wherein the one or more assistant operations are associated with one or more environmental features visible within a current visual range of the computerized glasses when the user wears the computerized glasses and directs their gaze toward the portion of the environment; as well as causing a set of one or more selectable suggestions to be rendered on a display interface of the computerized glasses based on the suggestion data and when the one or more environmental features are within the current visual range of the computerized glasses, Wherein a user selection of a particular selectable suggestion from the set of the one or more selectable suggestions causes the automated assistant to initiate performance of a particular one of the one or more assistant operations associated with the one or more environmental features visible within the current visual range of the computerized glasses.

15. The method of claim 14, wherein processing the contextual data generated at the computerized eyewear comprises: determining, based on the contextual data, an estimated level of interest of the user in types of environmental features visible within the current visual range of the computerized glasses, Wherein when the estimated interest level meets a threshold interest level, the one or more environmental features correspond to the environmental feature type.

16. The method of claim 15, wherein generating the suggestion data characterizing the one or more assistant operations comprises: Comparison data is generated that provides information for comparing characteristics of the one or more environmental features for each environmental feature corresponding to the environmental feature type.

17. The method of claim 16, wherein the environmental feature type comprises a restaurant type, and the comparison data indicates menu items for each restaurant visible within the current visual range of the computerized eyewear.

18. A method as claimed in any one of claims 14 to 17, wherein the contextual data comprises interaction data generated during an interaction between the user and the automated assistant before the user directs their gaze towards the portion of the environment.

19. The method of claim 18, wherein the interaction data indicates that the user recognized the environmental characteristic type during the interaction between the user and the automated assistant.

20. A method as claimed in any one of claims 14 to 17 or 19, wherein the contextual data comprises interaction data generated during an interaction between the user and another application before the user directs his gaze towards the part of the environment.

21. A system comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to perform operations corresponding to any one of claims 1 to 20.

22. The system of claim 21, wherein the at least one processor comprises a computerized eyewear processor incorporated into the computerized eyewear.

23. The system of claim 21 or 22, wherein the at least one process comprises a smartphone processor incorporated into a smartphone that communicates with the computerized glasses.

24. At least one computer readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 20.