Initializing non-assistant background actions by automatic assistant when accessing non-assistant applications

Abstract

The invention relates to initializing non-assistant background actions by an automatic assistant when accessing non-assistant applications. Implementations set forth herein relate to a system that employs an automated assistant to facilitate interaction between a user and another application, which can provide the automated assistant with permission to initialize related application actions while the user interacts with the other application. Further, the system may allow an automatic assistant to initialize the actions of different applications, although it is actively operating a particular application. The automatic assistant may collect available actions using various application-specific schemes, which may be compared to incoming requests from the user to the automatic assistant. Additional data, such as contextual and historical interactions, may also be used to rank and identify suitable application actions to be initialized by an automated assistant.

Description

Description of the case This application is a divisional application of Chinese invention patent application No. 201980093732.0, filed on June 13, 2019. Technical Field The present disclosure generally relates to initiating non-assistant background actions through an automated assistant when a non-assistant application is accessed. Background Art

[0001] A human may engage in a human-machine conversation with what is referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may use spoken natural language input (i.e., spoken utterances), which in some cases may be converted to text and then processed, and / or provide commands and / or requests to the automated assistant by providing textual (e.g., typed) natural language input. The automated assistant may respond to the request by providing responsive user interface output, which may include audible and / or visual user interface output.

[0002] When the user is operating other applications, the availability of the automatic assistant may be limited. As a result, the user may try to call the automatic assistant to perform certain functions associated with other applications by the user, but when the automatic assistant cannot continue, the conversation session with the automatic assistant is finally terminated. For example, the limited availability or limited functionality of the automatic assistant may mean that the user cannot control other applications through voice commands processed by the automatic assistant. This wastes computing resources, such as network and processing bandwidth, because any processing of spoken words during the conversation session will not result in the execution of any action. In addition, due to this defect, when the user interacts with their respective automatic assistants, the user may avoid operating other applications, which may otherwise provide efficiency for various tasks performed by the user. An example of such a task is to control a separate hardware system, such as a heating system, an air conditioning system, or other climate control system, by an application installed on the user's computing device. Such avoidance may lead to inefficiency in any device that may otherwise be assisted by such applications (such as smart thermostats and other application control devices within the hardware system) and anyone who may benefit from such applications or benefit from their associated control.

[0003] Furthermore, when the user does choose to interact with their automated assistant to perform certain tasks, the limited functionality of the automated assistant may inadvertently negate an ongoing conversation session due to the user opening another application. The opening of another application may be assumed by many systems as an indication that the user is no longer interested in advancing the ongoing conversation session, which wastes computing resources when the user actually intended to perform an action related to the conversation session. In such cases, the conversation session may be cancelled, causing the user to repeat any prior spoken utterances to re-invoke the automated assistant, which can hinder any progress the user may have made in initiating certain application actions using such systems. Summary of the invention

[0004] Embodiments described herein relate to one or more systems for allowing a user to invoke an automated assistant to initiate the execution of one or more actions of a particular application (or separate applications), while allowing the user to interact with the particular application. The particular application with which the user is interacting can be associated with data characterizing a variety of different actions that can be performed by the particular application. In addition, separate applications can also be associated with this data, which can also characterize the various actions that can be performed by those separate applications. While the user is interacting with a particular application, the user can provide spoken words to facilitate the completion of one or more actions that the particular application or one or more separate applications can complete.

[0005] As an example, a specific application may be an alarm system application that a user is accessing through a computing device. For example, an alarm system application may be installed on a computing device. In the following discussion, a computing device will be mentioned in the context of a tablet computing device, but it should be understood that a computing device may be a smart phone, a smart watch, etc. instead. A user may use a tablet computing device to access an alarm system application to view a video captured by one or more security cameras communicating with the alarm system application. While watching a video, a user may want to ensure their alarm system. To do this, a user may provide a verbal utterance while interacting with the video interface of the alarm system application, i.e., without closing the video they are viewing or otherwise navigating to a separate interface of the alarm system application to ensure the alarm system. For example, a user may provide a verbal utterance such as "secure the alarm system". In some embodiments, the verbal utterance may be processed at a tablet computing device and / or a remote computing device (e.g., a server device) to identify one or more actions that a user is requesting an automated assistant to perform to initialize it. For example, automatic voice recognition and / or natural language understanding of the verbal utterance may be performed on the device. Automatic speech recognition (ASR) may be performed on the device to detect certain terms that may correspond to specific actions that can be performed by the device. Alternatively or additionally, natural language understanding (NLU) may be performed on the device to identify certain intents that can be performed by the device.

[0006] Input data characterizing natural language content can be used to determine one or more specific actions that the user is requesting to be initiated by the automated assistant. For example, the tablet computing device can access application data for each application accessible through the tablet computing device and / or store application data. In some embodiments, the application can be accessed in response to the user through non-voice activity (e.g., button presses, physical interaction with the device, indirect input such as gestures) and / or through voice activity (e.g., hot words, invocation phrases, detecting specific terms in spoken utterances, and / or detecting that spoken utterances correspond to intents).

[0007] The automated assistant may access application data only in response to an invocation gesture (e.g., the voice / non-voice activity mentioned above), before the assistant receives and / or processes the remainder of the verbal user request / utterance following the invocation gesture. This may allow the assistant to obtain application data for any application currently running on the tablet computing device before the device finishes receiving / processing the full request from the user (i.e., after the invocation gesture). Thus, once the full request is received and processed, the assistant is able to immediately determine whether the request can be executed by the application currently running on the device. In some cases, the application data obtained in this manner may be limited to application data for the application currently running in the foreground of a multitasking execution environment on the device.

[0008] To access application data, the automated assistant application may send a request to one or more applications in response to non-voice activity and / or voice activity (e.g., through an operating system). In response, the one or more applications may provide application data characterizing contextual actions and / or global actions. Contextual actions may be identified and / or executable based on the current state of the corresponding application that performs the contextual action (e.g., the active application that the user is accessing), and global actions may be identified and / or executable regardless of the current state of the corresponding application. By allowing applications to provide application data in this manner, the automated assistant may operate based on an accurate index of the action, thereby enabling the automated assistant to initialize related actions in response to specific inputs. In addition, in some embodiments, since ASR and / or NLU are performed on the device and combined with action selection, the automated assistant may access application data without any network transmission.

[0009] The input data may be compared to application data for one or more different applications to identify one or more actions that the user wants to initialize. In some embodiments, the one or more actions may be ranked and / or otherwise prioritized to identify the most appropriate action to initialize in response to the spoken utterance. The one or more actions may be prioritized based on the content of the spoken utterance, current, past, and / or expected use of one or more applications, contextual data characterizing the context in which the user provides the spoken utterance, whether the action corresponds to an active application, and / or any other information that can be used to prioritize the one or more actions over one or more other operations. In some embodiments, actions corresponding to active applications (i.e., applications executing in the foreground of a graphical user interface) may be prioritized and / or ranked higher than actions corresponding to inactive applications.

[0010] In some embodiments, application data may include structured data, such as in the form of a scheme, which may represent various different actions that can be performed by a specific application to which the application data corresponds. For example, an alarm system application may be associated with specific application data representing a scheme that includes various different entries representing one or more different actions that can be performed by the alarm system application. In addition, a thermostat application may be associated with other application data representing another scheme that includes entries representing one or more other actions that can be performed by the thermostat application. In some embodiments, although the two applications are different, the scheme for each application may include an entry representing an "on" action.

[0011] Each entry may include attributes of an "on" action, a natural language description of the "on" action, a file path to data associated with the "on" action, and / or any other information that may be relevant to an application action. For example, an entry in a scheme for an alarm system application may include a path to an execution file to ensure the alarm system. Additionally, a separate entry in a scheme for a thermostat application may include information characterizing the current state of the thermostat application and / or a thermostat corresponding to the thermostat application. The information provided for each entry may be compared to the content of a spoken utterance from a user and / or context data corresponding to the context in which the spoken utterance was provided to the user. This comparison may be performed in order to rank and / or prioritize one or more actions over other actions.

[0012] For example, context data generated by a computing device may characterize one or more applications that are currently active at the computing device. Thus, because the alarm system application was active when the user provided the verbal utterance, the "turn on" action characterized by the scheme of the alarm system application may take precedence over the "turn on" action characterized by the scheme of the thermostat application. In some cases, as described in more detail below, the scheme or other application data of an application running in the foreground of a multitasking environment may take precedence over the schemes / other application data of all other applications on the device. In this case, when the application running in the foreground changes from a first application to a second application, the application data of the second application may take precedence over the application data of the first application when determining which application should be used to implement the action specified in the user request.

[0013] In some embodiments, the application data for a particular application may characterize a variety of different actions that can be performed by the particular application. When the automated assistant receives a spoken utterance or other input, a plurality of different actions characterized by a scheme may be ranked and / or prioritized. Thereafter, the highest priority action may be performed in response to the spoken utterance. For example, a user may operate a restaurant reservation application that may be associated with application data characterizing a scheme that identifies a variety of different actions that can be performed by the restaurant application. When a user interacts with the restaurant reservation application, such as when the restaurant reservation application is running in the foreground of a multitasking environment, the user may navigate to a specific interface of the application to select a specific restaurant for making a reservation. While interacting with the specific interface of the application, the user may provide a spoken utterance, such as "Make the reservation for 7:30 PM".

[0014] In response to receiving the spoken utterance, the automated assistant may cause the spoken utterance to be processed in order to identify one or more actions to initiate execution based on the spoken utterance. For example, content data representing the natural language content of the received spoken utterance may be generated at a device receiving the spoken utterance. The content data may be compared to application data representing various different actions that can be performed by the restaurant reservation application. In some implementations, the application data may represent one or more contextual actions that can be performed by the application while the application is displaying a current state, and / or one or more global actions that can be performed by the application regardless of the current state.

[0015] Based on the comparison, one or more actions identified in the application data can be ranked and / or prioritized to determine an appropriate action to initiate in response to the spoken utterance. For example, the application data can characterize a "reservation time" action and a "notification time" action, each of which can be performed by a restaurant reservation application. A correspondence between the content of the received spoken utterance and the actions can be determined to prioritize one action over another. In some embodiments, the application data can include information further characterizing each action, and the information can be compared to the content of the spoken utterance to determine the strength of correlation between the content of the spoken utterance and the information characterizing each action. For example, because the content of the spoken utterance includes the term "reservation", the "reservation time" action can take precedence over the notification time action.

[0016] In some embodiments, the state of the application may be considered when selecting an action to initialize in response to a spoken utterance. For example, based on the previous example, when the user provides the spoken utterance, the restaurant reservation application may not have established a stored reservation when the spoken utterance is received. The state of the restaurant reservation application may be accessed and also compared to the application data when the content of the spoken utterance is compared to the application data. The application data may characterize the application state of one or more actions that can be performed by the reservation application. For example, a reservation time action may be associated with a draft reservation state, while a notification time action may be associated with a stored reservation state. Therefore, when there is no stored reservation, but the user is creating a draft reservation, at least in response to the user providing the spoken utterance, "Booked at 7:30 pm", the reservation time action may take precedence over the notification time action.

[0017] In another example, a user may operate a thermostat application that may be associated with application data characterizing a scheme of various different actions that can be performed by the thermostat application. When the user interacts with the thermostat application, such as when the thermostat application is running in the foreground of a multitasking environment, the user may navigate to a specific interface of the application to select a specific time to set the indoor temperature to a specific value. While interacting with the specific interface of the application, the user may provide a verbal utterance, such as "Set temperature to 68 degrees at 7:00 AM".

[0018] In response to receiving the spoken utterance, the automated assistant may cause the spoken utterance to be processed in order to identify one or more actions to initiate based on the spoken utterance. For example, content data representing the natural language content of the received spoken utterance may be generated at a device receiving the spoken utterance. The content data may be compared to application data representing various different actions that can be performed by the thermostat application. In some embodiments, the application data may represent one or more contextual actions that can be performed by the application while the application is displaying a current state, and / or one or more global actions that can be performed by the application regardless of the current state.

[0019] Based on the comparison, one or more actions identified in the application data can be ranked and / or prioritized to determine an appropriate action to initiate in response to the spoken utterance. For example, the application data may characterize a "set temperature" action and an "eco mode" action, each of which can be performed by a thermostat application. A correspondence between the content of the received spoken utterance and the actions can be determined in order to prioritize one action over another. In some embodiments, the application data may include information further characterizing each action, and the information can be compared to the content of the spoken utterance to determine the strength of correlation between the content of the spoken utterance and the information characterizing each action. For example, because the content of the spoken utterance includes the term "set temperature", the "set temperature" action can take precedence over the "eco mode" action.

[0020] In some embodiments, when an active application is executed at a computing device in response to a spoken utterance provided by a user, a variety of different actions from different applications may be considered. For example, when a user provides a spoken utterance such as "Read my new message" while a non-messaging application (e.g., a stock application) is being presented, the automated assistant may interpret the spoken utterance as being most relevant to an automated assistant action of reading new emails for the user. However, when the user provides the spoken utterance while a social media application is executing in the background of a non-messaging application, the automated assistant may determine whether the state of the social media application is associated with "new messages." If the state and / or context of the background application is associated with "new messages," the automated assistant may initialize the execution of message-related actions through the background application. However, if the state and / or context of the background application is not associated with "new messages," the automated assistant may perform the following automated assistant actions to read any new email messages for the user, and / or if there are no new email messages, the automated assistant may provide a response such as "There are no new messages."

[0021] By providing an automatic assistant that can initialize other application actions in this way, other corresponding applications will not need to pre-load modules for voice control, but can rely on automatic assistants for ASR and / or NLU. This can save client resources that might otherwise be consumed by pre-loading ASR and / or NLU modules for multiple different applications, which can consume a variety of different computing resources. For example, operating multiple different applications, each with its own ASR and / or NLU module, may consume processing bandwidth and / or storage resources. Therefore, the use of the techniques discussed herein can eliminate this waste of computing resources. In addition, these techniques allow a single interface (e.g., a microphone and / or other interface for interacting with an automatic assistant) to control active applications, background applications, and / or automatic assistants. This can eliminate the waste of computing resources that might otherwise be consumed to launch separate applications and / or connect to remote servers to process input.

[0022] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of those embodiments and other embodiments are described in more detail below.

[0023] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more methods described above and / or elsewhere herein. However, other embodiments may include a system consisting of one or more computers and / or one or more robots, the system including one or more processors operable to execute stored instructions to perform methods such as one or more methods described above and / or elsewhere herein.

[0024] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A and Figure 1B Illustrated is a user invoking an automated assistant to control various actions of an application including graphical control elements for controlling other actions.

[0026] Figure 2A and Figure 2B Illustrated is a user accessing a particular application presented in the foreground of a display panel of a computing device while the user is also controlling a separate third-party application through input to an automated assistant.

[0027] Figure 3 A system is illustrated that allows an automated assistant to initiate actions for one or more applications regardless of whether the target applications and / or corresponding graphical control elements are being presented in the foreground of a graphical user interface.

[0028] Figure 4A and Figure 4B Illustrated is a method for controlling a non-assistant application through an automated assistant while accessing a non-assistant application or a separate application distinct from the non-assistant application and the automated assistant.

[0029] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0030] Figure 1A and Figure 1B Views 100 and 140 are illustrated, with a user 102 invoking an automated assistant to control various actions of an application that includes graphical control elements for controlling certain actions, but those graphical control elements may not always be presented when executed. For example, the user 102 may be accessing a computing device 104 that includes a display panel 114 for presenting a graphical user interface 106 of an application. The application may be a media playback application 108 that includes a first graphical control element 110 and a second graphical control element 112. When the media playback application 108 is executed at the computing device 104, the user 102 may control one or more graphical elements presented at the graphical user interface 106. In addition, while the user 102 is viewing the graphical user interface 106, the user 102 may provide spoken utterances to control one or more actions that can be performed by the computing device 104.

[0031] For example, the user may pause the display when they tap a graphical interface element (e.g., a pause button (e.g., Figure 1A)) when providing spoken utterance 116. Spoken utterance 116 may be, for example, "Set to 6 and play my workout playlist". In response to receiving spoken utterance 116, one or more processors of computing device 104 may generate audio data representing spoken utterance 116, and process the audio data to facilitate a response to spoken utterance 116. For example, the one or more processors may process the audio data according to a speech-to-text process for converting audio data into text data. The text data may then be processed by the one or more processors according to a natural language understanding process. In some embodiments, the speech-to-text process and / or the natural language understanding process may be performed exclusively at computing device 104. Alternatively or additionally, the speech-to-text process and / or the natural language understanding process may be performed at a separate server device and / or computing device 104.

[0032] In some implementations, in response to receiving the spoken utterance 116, the automated assistant 130 of the computing device 104 may request and / or access application data 124 corresponding to one or more applications 128 accessible through the computing device 104. The application data 124 may represent one or more actions that can be performed by the one or more applications 128 accessible through the computing device 104. The applications 128 may include the media playback application 108, and the media playback application 108 may provide specific application data 124 that represents one or more actions that can be performed by the media playback application 108. In some implementations, each of the applications 128 may provide application data 124 that represents contextual actions that can be performed by the specific application, depending on whether the specific application is operating according to a specific operating state. For example, the media playback application 108 may provide application data that represents one or more contextual actions that can be performed when the first graphical control element 110 and the second graphical control element 112 are presented at the graphical user interface 106. In this example, the one or more contextual actions may include a volume adjustment action, a pause action, a next action, and / or a previous action.

[0033] In some embodiments, application data 124 may include action schemes 132, which may be accessed by automated assistant 130 and / or action engine 126 for ranking and / or prioritizing actions that can be performed by a particular application. For example, each action entry 134 identified in action scheme 132 may be provided with one or more terms or descriptors associated with a particular action. For example, a particular descriptor may characterize an interface affected by the execution of a particular action, and / or another particular descriptor may characterize a data type that is accessed or otherwise affected by the execution of a particular action. Alternatively or additionally, an application descriptor may characterize one or more applications that may be affected by the execution of a particular action, and / or another particular descriptor may characterize account permissions, account restrictions, network preferences, interface modalities, power preferences, and / or any other specification that may be associated with an application action.

[0034] Furthermore, in some implementations, each of the applications 128 may provide application data 124 that characterizes global actions that can be performed by the particular application, regardless of whether the particular application is operating according to a particular operating state (e.g., whether the application is operating in the foreground of a graphical user interface). For example, the media playback application 108 may provide application data that characterizes one or more global actions that can be performed when the media playback application 108 is executing at the computing device 104.

[0035] In response to receiving the spoken utterance 116, the automated assistant 130 can access the application data 124 to determine whether the spoken utterance 116 is for the automated assistant 130 to initiate the execution of an action of the application 128. For example, the automated assistant 130 can cause the action engine 126 of the computing device 104 to process the application data 124 in response to the spoken utterance 116. The action engine 126 can identify various action entries that represent one or more actions that can be performed by one or more applications 128. In some embodiments, because the computing device 104 is presenting the graphical user interface 106 of the media playback application 108 when the user provides the spoken utterance 116, the action engine 126 can take this context into account when selecting an action to be initiated. For example, the content of the spoken utterance 116 can be compared with the application data 124 to determine whether the content is related to one or more actions that can be performed by the media playback application 108. Each action can be ranked and / or prioritized according to the relevance between the particular action and the content of the spoken utterance 116.

[0036] Alternatively or additionally, each action may be ranked and / or prioritized based on a determined relevance between a particular action and the context in which the user provided the spoken utterance 116. As an example, the action 124 identified by the application data may be a volume adjustment action, which may accept a numeric slot value between 0 and 10 to perform the action. Thus, because the spoken utterance includes the number "6," the content of the spoken utterance 116 has relevance to the volume adjustment action. Alternatively or additionally, because the graphical user interface 106 is currently presenting a "volume" control element (a second element graphical control element 112) that also identifies a number (e.g., "Set 4"), the action engine 126 may also determine that the context of the spoken utterance 116 is relevant to the volume adjustment action.

[0037] In some embodiments, based on the natural language understanding of the spoken utterance 116, the action engine 126 can identify another action corresponding to another portion of the spoken utterance 116. For example, in order to identify an appropriate action to initiate in response to the user 102 saying "play my workout playlist", the action engine 126 can access the application data 124 and compare the action data with this portion of the spoken utterance 116. Specifically, the action engine 126 can access the action scheme 132 and prioritize one or more action entries 134 according to the strength of relevance of each entry to the latter portion of the spoken utterance 116. For example, an action entry 134 that characterizes an action as a "play playlist" action can be prioritized and / or ranked above any other action entry 134. As a result, this highest prioritized and / or highest ranked action entry corresponding to the "play playlist" action can be selected for execution. In addition, the "play playlist" action can include a slot value for identifying a playlist to be played, and thus, the natural language content of the spoken utterance 116 can be used to satisfy the slot value. For example, automated assistant 130 may assign "workout" to the name of the playlist to be played at the slot value to facilitate completion of the "play playlist" action.

[0038] In some implementations, context data characterizing the context in which user 102 provided a request to play a workout playlist may be compared to application data 124 to identify actions that are relevant to the context and the “play my playlist” portion of spoken utterance 116. For example, the context data of application data 124 may characterize graphical user interface 106 as including the text “playing 'relaxing' playlist.” Action engine 126 may compare this text to the text of the action entries to identify actions that are most relevant to the text of graphical user interface 106 and spoken utterance 116. For example, based on the text of graphical user interface 106 including the terms “play” and “playlist,” action engine 126 may rank and / or prioritize the “playlist” action above any other actions.

[0039] like Figure 1B , automated assistant 130 may cause one or more actions to be performed without interfering with the user's access to and / or interaction with media playback application 108. For example, in response to user 102 providing spoken utterance 116, automated assistant 130 may initiate one or more actions for media playback application 108 to perform. The resulting changes to the operation of media playback application 108 may be presented at updated graphical user interface 118, which may display that the "Workout" playlist is playing at computing device 104 in accordance with user 102's request, and the volume is set to "6". Automated assistant 130 may provide output 142 confirming that the request from user 102 has been satisfied and / or an updated graphical user interface 118 may be presented to reflect the changes caused by automated assistant 130 invoking media playback application 108 to perform the action.

[0040] Figure 2A and Figure 2BViews 200 and 240 are respectively illustrated of a user 202 accessing a particular application being presented in the foreground of a display panel 214 of a computing device 204, while the user 202 is also controlling a separate third-party application through input to an automated assistant 230. For example, the user 202 may access an application such as a thermostat application 208, which may be presented at a graphical user interface 206 of the display panel 214. While the user 202 is interacting with the thermostat application 208 (e.g., turning on the heat via a first graphical element 210), the user 202 may provide a spoken utterance 216, such as "Secure the alarm system". From the perspective of the user 202, the user 202 may intend to control a third-party application, such as an alarm system application. However, in order to effectively perform such control by the user 202, the computing device 204 may perform various operations to process this spoken utterance 216 and / or any other input from the user 202.

[0041] For example, in response to receiving the spoken utterance 216 and / or any other input to the automated assistant 230, the automated assistant 230 may cause one or more applications 228 to be queried in order to identify one or more actions that can be performed by the one or more applications 228. In some embodiments, each application 228 may provide an action scheme 232 that may characterize one or more actions that can be performed by the respective application. The action scheme 232 for a particular application 228 may characterize a contextual action that may be performed when the particular application 228 is executing and exhibiting a current state. Alternatively or additionally, the action scheme 232 for a particular application 228 may characterize a global action that may be performed regardless of the state of the particular application 228.

[0042] The spoken utterance 216 may be processed locally at the computing device 204, which may provide a speech-to-text engine and / or a natural language understanding engine. Based on the processing of the spoken utterance 216, the action engine 226 may identify one or more action entries 234 based on the content of the spoken utterance 216. In some embodiments, one or more identified actions may be ranked and / or prioritized based on various different data that the automated assistant 230 can access. For example, the context data 224 may be used to rank one or more identified actions so that the highest ranked actions may be initialized for execution in response to the spoken utterance 216. The context data may characterize one or more features of one or more interactions between the user 202 and the computing device 204, such as the content of the graphical user interface 206, one or more applications executed at the computing device 204, stored preferences of the user 202, and / or any other information that may characterize the context at the user 202. In addition, the assistant data 220 may be used to rank / prioritize one or more identified actions to be performed by a particular application in response to the spoken utterance 216. Assistant data 220 may characterize details of one or more interactions between user 202 and automated assistant 230, the location of user 202, preferences of user 202 regarding automated assistant 230, other devices that provide access to automated assistant 230, and / or any other information that may be associated with an automated assistant.

[0043] In response to the spoken utterance 216, and / or based on accessing one or more action scenarios 232 corresponding to one or more applications 228, the action engine 226 can identify one or more action entries 234 associated with the spoken utterance 216. For example, the content of the spoken utterance 216 can be compared to the action entry corresponding to the thermostat application 208, and determine that the thermostat application 208 does not include an action explicitly labeled with the term "alarm." However, the action engine 226 can compare the content of the spoken utterance 216 to another action entry 234 corresponding to a separate application, such as an alarm system application. The action engine 226 can determine that the alarm system application can perform an action explicitly labeled as "arm," and that the action entry for the "arm" action includes a description that the action is useful for "ensuring" the alarm system. As a result, the action engine 226 can rank and / or prioritize the "arm" action of the alarm system application over any other actions identified by the action engine 226.

[0044] like Figure 2B, automated assistant 230 may initiate execution of the "alert" action as a background process while maintaining thermostat application 208 in the foreground of display panel 214. For example, after user 202 turns on the heat and changes the temperature setting of thermostat application 208, user 202 may enable the alert system to be ensured, as shown in output 242 of automated assistant 230. In this way, background processes may be initiated and streamlined without interfering with any foreground processes in which the user is engaged. This may eliminate the waste of computing resources that may otherwise be consumed switching between foreground applications and / or in reinitializing actions invoked by the user through the foreground application.

[0045] In some embodiments, assistant data 220 can characterize a success metric based on the number of times a particular action and / or a particular application has been invoked by a user but has not been successfully executed. For example, when the automated assistant 230 determines that the alarm system has been ensured, the success metric corresponding to the "alert" action and / or the alarm system application can be modified to reflect the completion of the "alert" action. However, if the "alert" action is not successfully executed, the success metric can be modified to reflect the failure of the "alert" action to be completed. In this way, when the success metric fails to meet a particular success metric threshold, but the user has requested an action corresponding to a failed success metric, the automated assistant 230 can provide the user with a notification about how to proceed with the action. For example, the automated assistant 230 can cause the display panel 214 to present a notification such as "Please open [application_name] to perform the action" in response to receiving a verbal utterance including a request for an action corresponding to a success metric that does not meet the success metric threshold.

[0046] Figure 3The system 300 for enabling the automatic assistant 304 to initialize the action of one or more applications is illustrated, regardless of whether the target application and / or the corresponding graphical control element are being presented in the foreground of the graphical user interface. The automatic assistant 304 can be operated as a part of the assistant application provided at one or more computing devices (e.g., computing device 302 and / or server device). The user can interact with the automatic assistant 304 through the assistant interface 320, which can be a microphone, a camera, a touch screen display, a user interface and / or any other device that can provide an interface between the user and the application. For example, the user can initialize the automatic assistant 304 by providing speech input, text input and / or graphic input to the assistant interface 320, so that the automatic assistant 304 performs functions (e.g., providing data, controlling peripheral devices, accessing agents, generating input and / or output, etc.). The computing device 302 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures, so as to allow the user to control the application 334 of the computing device 302 through the touch interface. In some implementations, computing device 302 may lack a display device, thereby providing audible user interface output rather than providing graphical user interface output. In addition, computing device 302 may provide a user interface for receiving oral natural language input from a user, such as a microphone. In some implementations, computing device 302 may include a touch interface and may lack a camera, but may optionally include one or more other sensors.

[0047] The computing device 302 and / or other third-party client devices can communicate with the server device through a network such as the Internet. In addition, the computing device 302 and any other computing devices can communicate with each other through a local area network (LAN) (e.g., a Wi-Fi network). The computing device 302 can offload computing tasks to the server device to save computing resources at the computing device 302. For example, the server device can host the automatic assistant 304, and / or the computing device 302 can send input received at one or more assistant interfaces 320 to the server device. However, in some embodiments, the automatic assistant 304 can be hosted at the computing device 302, and various processes that can be associated with the automatic assistant operation can be performed at the computing device 302.

[0048] In various implementations, all or less than all aspects of automated assistant 304 may be implemented on computing device 302. In some of those implementations, aspects of automated assistant 304 are implemented by computing device 302 and may interface with a server device, which may implement other aspects of automated assistant 304. The server device may optionally service multiple users and their associated assistant applications via multiple threads. In implementations where all or less than all aspects of automated assistant 304 are implemented by computing device 302, automated assistant 304 may be an application separate from the operating system of computing device 302 (e.g., installed “on top” of the operating system)—or may alternatively be implemented directly by the operating system of computing device 302 (e.g., considered an application of the operating system, but integrated with the operating system).

[0049] In some implementations, automated assistant 304 may include an input processing engine 306 that may employ a plurality of different modules to process input and / or output of computing device 302 and / or server devices. For example, input processing engine 306 may include a speech processing engine 308 that may process audio data received at assistant interface 320 to identify text implemented in the audio data. Audio data may be sent from, for example, computing device 302 to a server device in order to preserve computing resources at computing device 302. Additionally or alternatively, audio data may be processed at computing device 302.

[0050] The process for converting audio data into text may include a speech recognition algorithm that may use a neural network and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data may be parsed by a data parsing engine 310 and made available to an automated assistant 304 as text data that may be used to generate and / or identify command phrases, intentions, actions, slot values, and / or any other content specified by a user. In some embodiments, the output data provided by the data parsing engine 310 may be provided to a parameter engine 312 to determine whether the user has provided the following input, which corresponds to a specific intention, action, and / or routine that can be performed by an automated assistant 304 and / or an application or agent that can be accessed by an automated assistant 304. For example, assistant data 338 may be stored at a server device and / or computing device 302, and may include data defining one or more actions that can be performed by an automated assistant 304, and parameters necessary for performing the action.

[0051] In some implementations, computing device 302 may include one or more applications 334, which may be provided by a third-party entity that is distinct from the entity that provides computing device 302 and / or automated assistant 304. Automated assistant 304 and / or action engine 318 of computing device 302 may access application data 330 to determine one or more actions that can be performed by one or more applications 334. Additionally, application data 330 and / or any other data (e.g., device data 332) may be accessed by automated assistant 304 to generate context data 336, which may characterize the context in which a particular application 334 is executing at computing device 302 and / or a particular user is accessing computing device 302.

[0052] When one or more applications 334 are executed at computing device 302, device data 332 can characterize the current operating state of each application 334 executed at computing device 302. In addition, application data 330 can characterize one or more features of executing applications 334, such as the content of one or more graphical user interfaces presented under the direction of one or more applications 334. Alternatively or additionally, application data 330 can characterize action scenarios that can be updated by the corresponding application and / or by automated assistant 304 based on the current operating state of the corresponding application. Alternatively or additionally, one or more action scenarios of one or more applications 334 can remain static, but can be accessed by action engine 318 in order to determine an appropriate action to initiate through automated assistant 304.

[0053] In some implementations, action engine 318 can initiate execution of one or more actions of application 334, regardless of whether a particular graphical control for the one or more actions is being presented with a graphical user interface of computing device 302. Automated assistant 304 can initiate execution of such actions, and metric engine 314 of automated assistant 304 can determine whether execution of such actions is complete. If it is determined that a particular action is not fully executed, metric engine 314 can modify the metric corresponding to the particular action to reflect that the action being executed was not successful. Alternatively or additionally, if another action is determined to be continuously executed, metric engine 314 can modify the metric corresponding to the other action to reflect the success of causing the intended application to execute the action.

[0054] The action engine 318 may use the metrics determined by the metric engine 314 to prioritize and / or sort the application actions identified by one or more action schemes. The actions may be ranked and / or prioritized to identify appropriate actions to perform initialization, for example, in response to a user providing a specific verbal utterance. The verbal utterance provided by the user when the first application is executed in the foreground as an interface of the computing device 302 may cause the first application to perform an action that may otherwise not be initialized by the user's interaction with the interface. Alternatively or additionally, when the second application is executed in the background and the first application is executed in the foreground, the user may provide a verbal utterance. In response to receiving the verbal utterance, the automated assistant 304 may determine that the verbal utterance corresponds to a specific action that can be performed by the second application, and cause the second application to initialize the execution of the specific action without interrupting the first application in the foreground. In some embodiments, the automated assistant 304 may provide an indication that the specific action is successfully performed by the second application in the background. In some embodiments, although the user interacts with the first application in the foreground, a different user may provide a verbal utterance that causes the second application to perform a specific action in the background.

[0055] Figure 4A and Figure 4B A method 400 for controlling a non-assistant application by an automated assistant while accessing a non-assistant application or a separate application different from the non-assistant application and the automated assistant is illustrated. The method 400 may be performed by one or more computing devices, applications, and / or any other device or module capable of interacting with one or more different applications. The method 400 may include an operation 402 that determines that a user provides a verbal utterance when interacting with an application that can be accessed separately from an automated assistant. For example, the application may be an organizational application for organizing tasks, emails, and schedules. The user may access the application via a portable computing device (e.g., a mobile phone). The user may interact with the application to access incoming emails via a first graphical user interface of the application. When the user interacts with the application at the first graphical user interface, the user may provide a verbal utterance such as "Add this event to my calendar" with reference to an email just received and including an event invitation.

[0056] In response to receiving a spoken utterance, method 400 may proceed from operation 402 to operation 404. Operation 404 may include accessing application data characterizing one or more actions that can be performed by one or more applications. The application data may characterize the one or more actions, as well as the attributes of the one or more actions and / or other information associated with the one or more actions. For example, the application data may be embodied as a scheme file that lists one or more actions that can be performed by one or more applications, and a list of features and / or attributes of each of the one or more actions. The features of a particular action may characterize an output modality and / or an input modality corresponding to the particular action. Alternatively or additionally, the features of a particular action may characterize a data type affected by the execution of the particular action. For example, a "play" action may be listed with corresponding information that lists "speaker" as an output modality affected by the execution of the "play" action, and "audio" as a data type used during the execution of the action (e.g., {"play": [output modality: speaker], [data type: audio], ...}). As another example, a "new event" action may be listed, with "calendar data" listed as the type of data accessed and / or edited during execution of the action, and "add event to calendar" as the description of the "new event" action.

[0057] Method 400 can proceed from operation 404 to optional operation 406, which can include accessing context data of one or more features and / or characteristics that characterize the engagement between a user and an application. The context data can be based on one or more attributes of a computing device during the engagement between a user and an application, one or more other computing devices, and / or any other signal that can be associated with the interaction between a user and an application. For example, when a user interacts with an organization application, the organization application can provide the user with a notification about an incoming email message to the organization application. The context data can be generated by an organization application, an automated assistant, and / or any other application that can be accessed by a computing device. For example, an automated assistant can identify changes at a display panel of a computing device and generate context data based on changes in content provided at the display panel.

[0058] Method 400 may also include operation 408, which determines the correlation between the spoken utterance provided by the user and the data. Determining the correlation may include comparing the content of the spoken utterance (e.g., "add this event to my calendar") with the application data and / or context data. For example, the application data may characterize the aforementioned "new event" action that includes "add an event to a calendar" as a description of the "new event" action. Based at least on the spoken utterance including the terms "add" and "event" and the description of the "new event" action also including the terms "add" and "event", the correlation between the spoken utterance and the "new event" action may be stronger than the differently determined correlation between the spoken utterance and the "play" action. In some embodiments, a machine learning model may be used to identify one or more actions associated with the spoken utterance. For example, a machine learning model that has been trained according to a deep forest and / or deep learning method may be used to identify the correlation between the content of the spoken utterance and the content of the application data.

[0059] In some embodiments, a determined correlation between context data and application data can be determined to identify an appropriate action to perform initialization in response to a spoken utterance from a user. For example, the context data can characterize a notification about a message received while the user is viewing a first graphical user interface. This correlation between application data and context data can be used to further rank one or more actions identified by the application data. For example, in response to an organization application receiving an incoming message, the organization application can modify the application data to indicate that a "new event" action is available for execution. Alternatively or additionally, the organization application can generate updated context data in response to the organization application receiving an incoming message. The updated context data can then be used to rank and / or prioritize one or more actions that can be performed by the organization application and / or another application separate from the automated assistant.

[0060] Method 400 may also include selecting an action from one or more actions based on the determined relevance. The action may be selected based on the ranking and / or priority assigned to the action according to the determined relevance. For example, a "new event" action may take precedence over any other action based on the following application data, which indicates that the "new event" action can be performed by an organization application in a given state and / or in the state of the organization application. Alternatively or additionally, the "new event" action may take precedence over any other action based on contextual data characterizing notifications associated with a verbal utterance provided by a user. In some embodiments, the application data may characterize actions that can be performed by one or more different applications (including an organization application). Therefore, the action selected to be performed may correspond to an application that is different from the automated assistant and is not currently presented at the foreground of the graphical user interface of the operating system of the computing device. In other words, in response to a verbal utterance, the automated assistant may initialize an application different from the organization application and the automated assistant in order to initialize the execution of the selected action.

[0061] Method 400 may proceed from operation 410 to operation 414 of method 412 via a continuation element "A". The continuation element "A" may represent a connection between method 400 and method 412. Method 412 may include operation 414, which determines whether the selected action corresponds to an application with which the user is interacting. When the action corresponds to an application with which the user is interacting, method 412 may proceed from operation 414 to operation 416. Operation 416 may include causing the application to initialize the execution of the selected action through an automated assistant. Alternatively, when the selected action does not correspond to an application that the user is accessing, and / or does not correspond to an application that is presented in the foreground of a display interface of a computing device, method 400 may proceed from operation 414 to operation 418. Operation 418 may include causing another application to initialize the execution of the selected action through an automated assistant.

[0062] From operation 418, method 400 may proceed to optional operation 420, which may include modifying a success metric for a selected action based on whether the application and / or another application completed the selected action. The success metric may correspond to a particular action and / or application, and may reflect the reliability of the application in performing the particular action when invoked by the automated assistant and / or when the application is not executing in the foreground of a graphical user interface. In this way, when a user attempts to initialize certain actions, the actions may be ranked and / or prioritized for selection based at least in part on their respective success metrics. If the success metric for a particular action is low, and / or does not meet a threshold, then at least in response to the automated assistant selecting the particular action for initialization, the user may be prompted to manually initialize the action. Figure 4B and Figure 4AAs shown, method 400 may return from operation 418 and / or optional operation 420 to operation 402 of method 400 via continuation element “B”.

[0063] When the "new event" action corresponds to an organization application presented in the foreground of the display interface, the organization application can perform the "new event" action according to the verbal discourse provided by the user. As a result, the organization application can generate a calendar entry using the verbal discourse and / or notification content provided by the user. When the "new event" action does not correspond to the organization application, but corresponds to another application (such as a social media application or other applications for managing calendars), the automatic assistant can initialize the execution of the "new event" through other applications. In this way, the user does not have to navigate away from the application being presented in the foreground of the display interface. In addition, when the selected action corresponds to the application in the foreground, but can be executed by the graphical user interface of the application, the user can streamline the initialization of the selected action, reducing the delay time that would otherwise be displayed when navigating between the interfaces of the application. This allows a variety of different modes to control a variety of different actions, although the explicit control of those different actions is not currently presented at the display interface. By reducing the amount of graphics data processing that would otherwise be consumed by switching between the graphical interfaces of the application, the waste of GPU bandwidth can be eliminated.

[0064] Figure 5 5 is a block diagram of an example computer system 510. The computer system 510 generally includes at least one processor 514, which communicates with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524, including, for example, a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0065] The user interface input device 522 may include a keyboard, a pointing device such as a mouse, trackball, touch pad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or a communication network.

[0066] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 510 to a user or another machine or computer system.

[0067] Storage subsystem 524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400 and / or implementing one or more of system 300, computing device 104, computing device 204, action engine 126, action engine 226, automated assistant 130, automated assistant 230, automated assistant 304, computing device 302, and / or any other application, device, apparatus, engine, and / or module discussed herein.

[0068] These software modules are usually executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive or a removable media box. The modules that implement the functions of certain embodiments can be stored in the storage subsystem 524 by the file storage subsystem 526, or stored in other machines accessible to the processor 514.

[0069] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0070] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more Figure 5 The computer system depicted in FIG.

[0071] In cases where the systems described herein collect personal information about users (or, as they are often referred to herein, "participants") or may use personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social behavior or activities, occupation, user preferences, or a user's current geographic location), or to control whether and / or how content is received from content servers that may be more relevant to the user. In addition, certain data may be processed in one or more ways prior to storage or use to remove personally identifiable information. For example, a user's identity may be processed so that personally identifiable information about that user cannot be determined, or where geographic location information is obtained, the user's geographic location may be generalized (e.g., to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may have control over how information about users is collected and / or used.

[0072] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining the results and / or one or more advantages described herein may be utilized, and such changes and / or modifications are considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary and actual parameters, dimensions, materials, and / or configurations will depend on the specific application for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein using only routine experiments. Therefore, it should be understood that the aforementioned embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The implementation of the present disclosure relates to each separate feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, is included within the scope of the present disclosure.

[0073] In some embodiments, a method is provided, the method comprising determining, by a computing device, while an application is executed at the computing device, that a user has provided a spoken utterance to an automated assistant but does not explicitly identify any application, wherein the any application is accessible through the computing device. The spoken utterance is received at an automated assistant interface of the computing device, and the automated assistant is accessible separately from the application. The method further comprises, based on determining that the user has provided the spoken utterance, accessing application data representing a plurality of different actions that can be performed by the application executed at the computing device, determining a correlation between the content of the spoken utterance provided by the user and the application data based on the application data, and selecting an action from the plurality of different actions represented by the application data for initialization through the automated assistant. The action is selected based on the correlation between the content of the spoken utterance and the application data. The method further comprises, when the selected action corresponds to one of the plurality of different actions that can be performed by the application executed at the computing device, causing the application to initialize execution of the selected action through the automated assistant.

[0074] These and other implementations of the technology may include one or more of the following features.

[0075] In some embodiments, the application data may further characterize other actions that are executable by one or more other applications that are accessible separately from the automated assistant and the application. In some of those embodiments, the method may further include, when the selected action corresponds to one of the other actions that are executable by another application different from the application executing at the computing device, causing the other application to initiate, by the automated assistant, execution of the selected action.

[0076] In some embodiments, when the user provides the spoken utterance, the application data may identify one or more context actions of the plurality of different actions based on one or more features of a current application state of the application. In some of those embodiments, the one or more context actions are identifiable by the application, and the one or more features characterize a graphical user interface of the application that is presented when the user provides the spoken utterance. In some of those embodiments, the one or more context actions may additionally and / or alternatively be identified by the application based on a state of an ongoing action being performed at the computing device when the user provides the spoken utterance.

[0077] In some embodiments, the method may further include determining a measure of success for one or more of the multiple different actions. The measure of success for a particular action may be based at least on the number of times the particular action has been fully performed in response to the user and / or one or more other users initiating the particular action through the automated assistant. In addition, the action may be further selected based on the measure of success of the action relative to other actions in the multiple different actions. In some of those embodiments, the method may further include, after causing the application and / or another application to initiate execution of the action: determining whether the action was fully performed by the application and / or the other application, and based on whether the action was fully performed by the application and / or the other application, causing the corresponding measure of success of the action to be modified.

[0078] In some embodiments, the method may further include, when the application is executed at the computing device, before determining that the user has provided the verbal utterance, determining that another application provides a notification to the user through an interface of the computing device when the application is executed at the computing device. The application data may include other data characterizing another action that the other application can perform, and in response to the user providing the verbal utterance, the other data may be requested from the other application. In some embodiments, the application data may additionally and / or alternatively identify the multiple different actions that the application can perform, and / or identify descriptive data for each of the multiple different actions. In addition, the specific descriptive data for a specific action in the multiple different actions characterizes two or more attributes of the specific action. In some of those embodiments, the two or more attributes include an action type name and / or an interface type name, wherein the action type name characterizes the action type corresponding to the specific action, and the interface type name corresponds to the interface type for presenting content during the execution of the specific action.

[0079] In some embodiments, determining that the user has provided a spoken utterance directed to the automated assistant but that does not explicitly identify any application accessible via the computing device can include generating, at the computing device, audio data embodying the spoken utterance provided by the user, and processing the audio data at the computing device according to a speech-to-text process and / or a natural language understanding process. In some of those embodiments, the computing device includes one or more processors, and the speech-to-text process and / or the natural language understanding process is performed using one or more of the processors of the computing device.

[0080] In some embodiments, a method is provided, the method comprising determining, by a computing device providing access to an automated assistant, that a user has provided one or more inputs for invoking the automated assistant. The one or more inputs are provided by the user while the application is presenting a current application state. The method further comprises, based on determining that the user has provided the one or more inputs, accessing application data representing a plurality of different actions that can be performed by one or more applications including the application. The application data represents contextual actions that can be performed by the application when the application is presenting the current application state. The method further comprises, based on the application data, determining a correlation between content of a spoken utterance provided by the user while the application is presenting the current application state. The spoken utterance does not explicitly identify any application that can be accessed by the computing device. The method further comprises, based on the correlation between the content of the spoken utterance and the application data, selecting an action from the contextual actions represented by the application data, and, when the selected action corresponds to one of the contextual actions that can be performed by the application when the application is presenting the current application state, causing the application to initiate execution of the selected action by the automated assistant.

[0081] These and other implementations of the technology may include one or more of the following features.

[0082] In some embodiments, the current application state may be displayed by the application when the application is presented in the foreground of a display interface of the computing device and / or another computing device. In some of those embodiments, the method may further include, when the selected action corresponds to another application different from the current application state being displayed, causing the other application to initiate execution of the selected action by the automated assistant.

[0083] In some of those embodiments, when the application displays the current application state, the application data may represent another action, the other action being initiated by selecting one or more graphical interface elements omitted from the display interface. In some of those embodiments, determining that the user has provided a spoken utterance directed to the automated assistant but not explicitly identifying any application accessible through the computing device may include generating audio data at the computing device that embodies the spoken utterance provided by the user, and processing the audio data at the computing device according to a speech-to-text process and / or a natural language understanding process. In some of those embodiments, the computing device may include one or more processors, and the speech-to-text process and / or the natural language understanding process may be performed using the one or more processors in the computing device.

[0084] In some embodiments, a method is provided, the method comprising receiving an indication from an automated assistant that a user has provided a spoken utterance while the user is accessing an application available through a computing device. The spoken utterance does not explicitly identify any application accessible through the computing device, and the automated assistant is accessible separately from the application. The method further comprises, in response to receiving the indication that the user has provided the spoken utterance, providing application data representing one or more context actions. The one or more context actions are identified by the application based on the ability of the application to initialize the execution of the one or more context actions when the application is in a current state, and the one or more context actions are selected from a plurality of different actions based on the current state of the application. The method further comprises, based on providing the application data, causing the automated assistant to determine whether the spoken utterance corresponds to a specific action in the one or more context actions represented by the application data, and when the automated assistant determines that the spoken utterance corresponds to the specific action in the one or more context actions represented by the application data, causing the automated assistant to initialize the execution of the specific action through the application based on the spoken utterance corresponding to the specific action.

[0085] These and other implementations of the technology may include one or more of the following features.

[0086] In some embodiments, the application data may further identify descriptive data for each of the one or more context actions. In addition, the specific descriptive data for a particular context action may characterize two or more attributes of the particular context action. In some of those embodiments, the two or more attributes include an action type name and / or an interface type name, wherein the action type name characterizes an action type corresponding to the particular context action, and the interface type name corresponds to an interface type for presenting content during execution of the particular context action.

[0087] In some embodiments, a method is provided, the method comprising, at a third-party application and while another application is executing at a computing device, receiving a request from an automated assistant to provide application data characterizing one or more actions that can be performed by the third-party application. The request is provided by the automated assistant in response to a user providing one or more inputs to invoke the automated assistant while the other application is executing at the computing device. The method further comprises, in response to receiving the request from the automated assistant, providing the application data to the automated assistant. The application data identifies a specific action that can be performed by the third-party application initialized by the automated assistant. The method further comprises, based on providing the application to the automated assistant, causing the automated assistant to determine whether the spoken utterance provided by the user is directed to initiating performance of the specific action by the third-party application. The spoken utterance does not explicitly identify any application that can be accessed through the computing device. The method further comprises, when the automated assistant determines based on the application data that the spoken utterance is directed to the application, causing the automated assistant to initialize performance of the action by the application based on the action data.

[0088] These and other implementations of the technology may include one or more of the following features.

[0089] In some embodiments, the other applications may be presented in the foreground of a display interface of the computing device, and the third-party application may be omitted from the foreground of the display interface of the computing device. In some of those embodiments, the application data may additionally and / or alternatively identify descriptive data corresponding to the specific action, and the descriptive data characterizes two or more attributes of the specific action. In some of those further embodiments, the two or more attributes include an action type name and / or an interface type name, wherein the action type name characterizes an action type corresponding to the specific action, and the interface type name corresponds to an interface type for presenting content during execution of the specific action.

[0090] Other embodiments may include a non-transitory computer-readable storage medium and / or a computer program storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more methods described above and / or elsewhere herein. However, other embodiments may include a system with one or more processors operable to execute stored instructions to perform methods such as one or more methods described above and / or elsewhere herein.

[0091] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: At the third-party application, while another application is executing on the computing device, receiving a request from the automated assistant to provide application data, the application data representing one or more actions that can be performed by the third-party application, wherein the request is provided by the automated assistant in response to a user providing one or more inputs for invoking the automated assistant while the other application is executing at the computing device; in response to receiving the request from the automated assistant, providing the application data to the automated assistant, wherein the application data identifies a specific action that can be performed by the third-party application initiated via the automated assistant; Based on providing the application data to the automated assistant, the automated assistant determines whether the spoken utterance provided by the user is directed to initiating performance of the particular action by the third-party application, wherein the spoken utterance does not explicitly identify any application accessible via the computing device; and When the automated assistant determines, based on the application data, that the spoken utterance is directed to the third-party application: Based on the application data, the automatic assistant is enabled to initiate execution of the specific action via the third-party application.

2. The method according to claim 1, wherein: The other application is presented in the foreground of a display interface of the computing device, and the third-party application is omitted from the foreground of the display interface of the computing device.

3. The method according to claim 2, wherein: The application data also identifies description data corresponding to the particular action, the description data characterizing two or more attributes of the particular action.

4. The method according to claim 2, wherein: The two or more attributes include: an action type name characterizing a type of action corresponding to the specific action, and / or an interface type name corresponding to a type of interface presenting content during execution of the specific action.

5. A method implemented by one or more processors of a computing device, the method comprising: The auto attendant determines that the user has provided a spoken utterance, wherein the spoken utterance is provided by the user while the application is displaying a current application state, the application being separate from the automated assistant application; receiving application data representing a plurality of different actions that can be performed via the application, wherein the application data is generated by the application and represents a context action that can be performed by the application when the application is presenting the current application state; identifying the spoken utterance provided by the user while the application is presenting the current application state, wherein the spoken utterance does not explicitly identify any application accessible via the computing device; determining whether there is a correlation between the content of the spoken utterance and the application data; In response to determining that there is a correlation between content of the spoken utterance and the application data: Based on the correlation, selecting an action from the context actions represented by the application data; and causing the application to perform the selected operation via the automated assistant; and In response to determining that there is no association between the content of the spoken utterance and the application data: Causing, via the automated assistant, one or more other actions to be performed based on the spoken utterance, the one or more other actions being in addition to the contextual action represented by the application data.

6. The method according to claim 5, wherein: The application is executing in the foreground of the computing device.

7. The method according to claim 5, wherein: The automated assistant initializes the automated assistant to perform the one or more other actions based on the spoken utterance.

8. The method according to claim 5, wherein: The automated assistant initializes another application to perform the one or more actions based on the spoken utterance.

9. The method according to claim 5, wherein: The application data is received prior to recognizing that the spoken utterance is provided by the user.

10. The method according to claim 5, further comprising: receiving other application data representing a plurality of different actions that can be performed by another application of the one or more applications, Wherein, the other application data represents context actions, including actions that can be performed by the other application when the other application is presenting the current application state; and When there is another correlation between the content of the spoken utterance and the other application data: Based on the other correlation, selecting an action from the context actions represented by the other application data; and The other application is caused to perform the selected action via the automated assistant instead of causing the application to perform the selected action.

11. The method according to claim 10, wherein: The other application is executing in the foreground and the application is executing in the background.

12. The method according to claim 10, further comprising: ranking the application and the another application based on the relevance and the second relevance; as well as It is determined that the another application is ranked higher than the application.

13. The method according to claim 5, further comprising: Determining that the application is executing in the foreground; After determining that the application is executing in the foreground and also after receiving the application data, it is determined that the application is no longer executing in the background and that the other application is executing in the foreground.

14. A system comprising one or more processors and a memory operably coupled to the one or more processors, wherein: The memory stores instructions, which, in response to the one or more processors executing the instructions, cause the one or more processors to perform the following operations: The auto attendant determines that the user has provided a spoken utterance, wherein the spoken utterance is provided by the user while the application is displaying a current application state, the application being separate from the automated assistant application; receiving application data representing a plurality of different actions that can be performed via the application, wherein the application data is generated by the application and represents a context action that can be performed by the application when the application is presenting the current application state; identifying the spoken utterance provided by the user while the application is presenting the current application state, wherein the spoken utterance does not explicitly identify any application accessible via the computing device; determining whether there is a correlation between the content of the spoken utterance and the application data; In response to determining that there is a correlation between content of the spoken utterance and the application data: Based on the correlation, selecting an action from the context actions represented by the application data; and causing the application to perform the selected operation via the automated assistant; and In response to determining that there is no association between the content of the spoken utterance and the application data: Causing, via the automated assistant, one or more other actions to be performed based on the spoken utterance, the one or more other actions being in addition to the contextual action represented by the application data.

15. The system of claim 14, wherein: The application is executing in the foreground of the computing device.

16. The system of claim 14, wherein: The automated assistant initializes the automated assistant to perform the one or more other actions based on the spoken utterance.

17. The system of claim 14, wherein: The automated assistant initializes another application to perform the one or more actions based on the spoken utterance.

18. The system of claim 14, wherein: The application data is received prior to recognizing that the spoken utterance is provided by the user.

19. The system of claim 14, wherein: The instructions also include: receiving other application data representing a plurality of different actions that can be performed by another application of the one or more applications, Wherein, the other application data represents context actions, including actions that can be performed by the other application when the other application is presenting the current application state; and When there is another correlation between the content of the spoken utterance and the other application data: Based on the other correlation, selecting an action from the context actions represented by the other application data; and The other application is caused to perform the selected action via the automated assistant instead of causing the application to perform the selected action.

20. The system of claim 19, wherein: The other application is executing in the foreground and the application is executing in the background.

21. A system comprising: a memory storing instructions; One or more processors, the one or more processors being operable to execute instructions for: determining that a user has provided a spoken utterance at a computing device directed to an automated assistant without explicitly identifying any application accessible via the computing device, wherein the spoken utterance is received at an automated assistant interface of the computing device, and wherein the automated assistant is an application separate from the application; based on determining that the user has provided the spoken utterance directed to the automated assistant, accessing application data representative of a plurality of different actions that the application is capable of performing; determining, based on the application data, a correlation between content of the spoken utterance provided by the user and the application data; In response to determining the correlation between content of the spoken utterance provided by the user and the application data: selecting an action from the plurality of different actions represented by the application data based on the content of the spoken utterance; and Causes the application to perform the selected action.

22. The system of claim 21, wherein: The application data includes a solution file, wherein the solution file includes a natural language description of an action, wherein the natural language description of the action represents the multiple different actions that the application can perform.

23. The system of claim 22, wherein: The solution file also includes a characteristic natural language description of the output modality representing the action.

24. The system of claim 22, wherein: The scenario file also includes a natural language description of the characteristics of the input modality that characterizes the action.

25. The system of claim 22, wherein: The solution file also includes a natural language description of the characteristics of the input mode of the action and the output mode of the action.

26. The system of claim 21, wherein: The one or more processors are further operable to execute instructions for: accessing additional application data representing other operations that can be performed via additional applications that are accessible separately from the automated assistant and the application; Wherein, when determining the correlation between the content of the oral utterance provided by the user and the application data, the one or more processors are further to determine the correlation based on the additional application data.

27. The system of claim 21, wherein: The application data identifies one or more contextual actions of the plurality of different actions based on one or more characteristics of a current application state of the application when the user provides the spoken utterance.

28. The system of claim 27, wherein: The one or more context actions are recognized by the application, and the one or more features characterize a graphical user interface of the application that is presented when the user provides the spoken utterance.

29. The system of claim 27, wherein: The one or more context actions are identified by the application based on a state of an ongoing action being performed at the computing device when the user provides the spoken utterance.

30. The system of claim 21, wherein: Upon determining that the user has provided the spoken utterance directed to the automated assistant without explicitly identifying any application accessible via the computing device, the one or more processors are to: Speech-to-text processing is performed on audio data embodying the spoken utterance provided by the user to generate content of the spoken utterance.

31. The system of claim 21, wherein: The one or more processors are possessed by the computing device.

32. A method implemented by one or more processors, the method comprising: determining that a user has provided a spoken utterance, wherein the spoken utterance includes natural language content that does not explicitly identify any application accessible via the computing device, and wherein the spoken utterance is received at an automated assistant interface of an automated assistant of the computing device; based on determining that the user has provided the spoken utterance, accessing application data including a scenario file including natural language descriptions of actions for a plurality of different actions that can be performed by an application, the application being attached to the automated assistant; Based on processing of the natural language content and the natural language description of the action in the solution file, determining: The spoken utterance is directed to the application, and an action among the plurality of different actions that the application is capable of performing; and In response to determining that the spoken utterance is directed to the application and determining the action: The application is caused to perform the action.

33. The method of claim 32, wherein: The solution file also includes a characteristic natural language description of the output modality representing the action.

34. The method of claim 32, wherein: The scenario file also includes a natural language description of the characteristics of the input modality that characterizes the action.

35. The method of claim 32, wherein: The solution file also includes a natural language description of the characteristics of the input mode of the action and the output mode of the action.

36. The method of claim 32, wherein: The application data identifies one or more contextual actions of the plurality of different actions based on one or more characteristics of a current application state of the application when the user provides the spoken utterance.

37. The method of claim 36, wherein: The one or more context actions are recognized by the application, and the one or more features characterize a graphical user interface of the application that is presented when the user provides the spoken utterance.

38. The method of claim 36, wherein: The one or more context actions are identified by the application based on a state of an ongoing action being performed at the computing device when the user provides the spoken utterance.

Images