Automatically assisted action execution and / or dynamic delay of background application requests

Abstract

The invention discloses automatic assistance of dynamic delay of action execution and / or background application requests. Implementations set forth herein allow a user to access a first application at a foreground of a graphical interface and simultaneously employ an automated assistant to respond to notifications originating from a second application. A user may provide an input, such as a spoken utterance, while viewing a first application at a foreground in order to respond to notifications from a second application without performing certain intervention steps that may occur in certain cases. Such intervention steps may include providing a user confirmation that may be displayed in response to the user providing a responsive input for the notification that may be bypassed and / or limited according to a timer time. The period of the timer may be set according to one or more characteristics associated with the notification, the user, and / or any other information that may be associated with the notification received by the user.

Description

Description of the case

[0001] This application is a divisional application of Chinese invention patent application No. 201980093928.X, with a filing date of June 5, 2019. Technical Field

[0002] The present disclosure relates to automatic auxiliary action execution and / or dynamic delay of background application requests. Background Art

[0003] A human may engage in a human-machine conversation with an interactive software application, referred to herein as an "automated assistant" (also referred to as a "digital agent," "chatbot," "interactive personal assistant," "intelligent personal assistant," "assistant application," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., spoken utterances) that may in some cases be converted to text and then processed and / or by providing textual (e.g., typed) natural language input. The automated assistant may respond to the request by providing responsive user interface output, which may include audible and / or visual user interface output.

[0004] When the user is operating other applications, the automated assistant may have limited availability. As a result, the user may try to call the automated assistant to perform certain functions associated with the user and other applications, but eventually terminate the conversation session with the automated assistant when the automated assistant cannot continue. For example, the limited availability or limited functionality of the automated assistant may mean that the user cannot control other applications with the help of voice commands processed by the automated assistant. This may waste computing resources such as network and processing bandwidth, because any processing of spoken words during the conversation session will not result in the execution of any action. In addition, because of this defect, when the user interacts with their corresponding automated assistant, the user can avoid operating other applications that can otherwise provide efficiency for various tasks performed by the user. An example of such a task is to control a separate hardware system such as a heating system, an air conditioning system, or other climate control system via an application installed on the user's computing device. Such avoidance can cause inefficiency in any device that may be assisted by such an application in other ways, such as a smart thermostat and other application-controlled devices within the hardware system, and any person who may benefit from such an application or its control of associated devices.

[0005] Furthermore, many systems may assume that invoking an automated assistant while accessing an application is an indication that the user is no longer interested in further interaction with the application, and cause the application to suspend certain operations or close. This may waste computing resources when the user actually intended to cause the automated assistant to perform actions related to the application and / or another application. In such a situation, any data previously generated during interaction with the application may be lost, thereby forcing the user to reinitialize the application and / or repeat any previous operations performed with the application. Summary of the invention

[0006] Some implementations described herein optionally automatically determine the delay of automatically executing auxiliary actions locally on the client device. As used herein, the "automatic execution" of an action can mean executing an auxiliary action without first requiring explicit confirmatory user interface input. An assistant action can be generated based on a user interface input (e.g., verbal utterance) provided at an automated assistant interface of a client device that implements at least part of an automated assistant (e.g., separately and / or via interaction with one or more remote automated assistant components). During the delay, further user interface input can be provided to prevent and / or modify the execution of the auxiliary action. However, in the absence of such a user interface input, the automatic execution of the auxiliary action can occur. The duration of the delay is dynamically determined based on one or more dynamic considerations depending on the user interface input. For example, when the user input is a verbal utterance, the duration of the delay can be based on one or more confidence metrics, and in some implementations, the confidence metric can be a speech-to-text conversion based on a verbal utterance. Therefore, the duration of the delay can be a confidence metric of an item utilized when determining an action and / or a time slot value of an action. Additionally, for example, the duration of the delay may additionally or alternatively be based on: the length of the slot value for the action and determined from the spoken utterance; the historical likelihood of slot values ​​(across user groups and / or for a user); the commonality of the natural language content of the spoken utterance underlying the action and / or slot value; the location of the user and / or device when the spoken utterance is provided; and events and / or other information characterizing the context in which the spoken utterance is provided.

[0007] One or more factors described above (and elsewhere in this article) are used to seek to minimize the waiting time for performing the auxiliary action when determining the duration of the delay, while selectively delaying the execution of the auxiliary action when the factor indicates that the action may have been improperly interpreted by the automated assistant and / or incorrectly transmitted by the user. In addition, the duration of the delay in the execution of the auxiliary action can be related to the degree to which the factor indicates such improper interpretation and / or transmission. In these and other ways, the waiting time for performing the auxiliary action is at least selectively reduced, which can further reduce the duration of the entire dialogue session when one or more additional user inputs and / or auxiliary responses follow the execution. In addition, in these and other ways, the waiting time for performing the auxiliary action is at least selectively increased to provide the user with an opportunity to prevent the execution of the action, thereby no longer wasting computer and / or network resources to perform undesirable assistant actions and performing subsequent actions to remedy the undesirable assistant actions. Therefore, more generally, the implementation disclosed in this article seeks to balance the reduction of the waiting time for performing the auxiliary action with the reduction of the utilization of computer and / or network resources wasted to perform undesirable auxiliary actions.

[0008] Some implementations described herein additionally or alternatively relate to an automated assistant that provides a user with the ability to view foreground applications and simultaneously use the automated assistant to control background applications. For example, an automated assistant can be used to respond to application notifications from background applications without wasting computing resources for either application. In some implementations, a notification based on the operation of a second application can be presented to the user while the user is operating a first application. The user can choose to respond to the notification while actively viewing and / or interacting with the first application in the foreground of the display panel. In order to respond to the notification, the user can provide input to the automated assistant, which can determine that the input is in response to a notification from the second application. The response input can be processed to identify an action to be performed by a second application such as sending a message, and the action can be performed in the background with or without further confirmation from the user.

[0009] The automated assistant can determine whether to query the user for confirmation based on a variety of different information available to the automated assistant. For example, the user may receive a notification related to a second application (e.g., "John says: 'Are you here?') while the user is operating a first application (e.g., a podcast application) in the foreground of a graphical user interface of a computing device. When the user provides a spoken utterance in response to seeing the notification (e.g., "I'm on the way."), the automated assistant can use the content of the spoken utterance to generate a response message (e.g., "Draft message: 'I'm on the way."). Based on information associated with and / or characterizing the notification, the automated assistant can choose to render the action-impending notification on the graphical user interface of the computing device, or alternatively, choose to bypass providing the action-impending notification.

[0010] In some implementations, an action impending notification may be rendered by an automated assistant to provide the user with an opportunity to confirm and / or modify the impending action before executing the impending action. In addition, when it is determined that the impending action is an action that the user will not generally expect to confirm and / or modify, the action impending notification may be bypassed. For example, when a user operates a first application such as a map application at a computing device and receives a notification of a second application such as a messaging application at the computing device, the message may be notified to the user. For example, the received message may be from a friend name "Matt" to whom the user often sends messages, and the notification may be "Matt: 'What is your ETA?'". After viewing the notification that temporarily appears in the foreground of the graphical interface of the computing device, the user may respond with a spoken word such as "25 minutes". In response to receiving the spoken word, the automated assistant may determine a confidence metric indicating the confidence in predicting whether the user will confirm and / or modify the message. If the confidence metric meets a threshold, thereby indicating that the predicted user does not expect to confirm and / or modify the message, the message may be transmitted without providing an action impending notification. However, if the confidence metric does not satisfy the threshold, then the action imminence notification may be rendered for a duration and / or until the user elects to confirm and / or modify the corresponding action request.

[0011] In some implementations, the automated assistant can bias the decision to select an application to receive an action request based on which application most recently provided output and / or received input. For example, because the second application most recently provided a notification about a message from "Matt," the automated assistant can select the second application for an action. Thus, when the automated assistant selects the second application for a subsequent action, the automated assistant can generate an action request to facilitate the second application to perform a particular action.

[0012] In some implementations, the automated assistant can interact with the second application, and / or the second application can interact with the automated assistant via an application programming interface (API). The automated assistant and the second application can interact with each other to facilitate the automated assistant to generate an action request corresponding to a specific action that can be performed by the second application. For example, based on the second application providing the notification, the second application can use the API to push action data to the automated assistant to assist the automated assistant in selecting the action to which the action request should be directed. For example, following the example mentioned above, the second application can push action data such as "{Response_Message(recipient: ["Matt",...]; body: [message content,...]), which the automated assistant can use to generate an action request. Specifically, the action request may include data that relates natural language content to the "body" of the "Response_message" action.

[0013] When determining how to respond to a spoken utterance from a user, the automated assistant may also generate a confidence metric. The confidence metric may reflect the calculated likelihood that the user will confirm and / or modify the action request before providing the action request to the second application. The confidence metric may also reflect the perceived cost of incorrectly performing the action (e.g., computational cost in a network or other resource). For example, if it is determined that the confidence metric does not meet a specific threshold, the automated assistant may render an action impending notification on the display panel. The action impending notification may indicate that the user may confirm and / or modify the action request before the action request is provided to the second application via the automated assistant. However, if it is determined that the confidence metric meets a specific threshold, the automated assistant may bypass rendering the action impending notification on the display panel. In addition, in addition to bypassing the presentation of the action impending notification on the display panel, the automated assistant may provide the action request to the second application so that the second application performs the action (e.g., sending the message "25 minutes" to a stored contact named "Matt").

[0014] In some implementations, when an impending action notification is rendered on a display panel, the duration during which the impending action notification is rendered on the display panel may be based on a confidence metric. For example, the confidence metric may be determined using data characterizing one or more different properties and / or aspects of the input from the user and / or the determined action to be performed. For example, when the input from the user corresponds to a response message that is close to a specific message that the user and / or one or more other users have a history of sending, the confidence metric may be at least higher relative to another confidence metric corresponding to another response message that the user and / or one or more other users rarely send. Additionally or alternatively, when the input from the user corresponds to a response message for a recipient to whom the user frequently sends messages, the confidence metric may be at least higher relative to another confidence metric corresponding to another response message for another recipient to whom messages are not frequently sent. Additionally or alternatively, the confidence metric can be based on: the importance of a particular action for which the input is intended when the input corresponds to an automatically corrected message, the location of the user when the input is provided, an event associated with or ongoing when the input is provided, the amount and / or type of data incorporated into the input, and / or any other information that may be associated with the input.

[0015] The above description is provided as an overview of some implementations of the present disclosure. Below, further descriptions of these and other implementations are described in more detail.

[0016] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. However, other implementations may include a system of one or more computers and / or one or more robots including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0017] It should be understood that all combinations of the above concepts and additional concepts described in more detail herein are contemplated to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 1E and Figure 1F Illustrate the dynamic delay employed in various situations before an action is initiated.

[0019] Figure 2A , Figure 2B and Figure 2C Illustrate a dynamic delay that is generated and employed before initiating a user-requested action.

[0020] Figure 3 A system for automatically determining whether to delay—and by how much to delay—before automatically performing an application action via an automated assistant is illustrated.

[0021] Figure 4 Methods of dynamically delaying action execution based on a variety of different factors and bypassing action execution delays based on such factors are illustrated.

[0022] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0023] Figures 1A to 1F Illustrated in various different situations and before the action is initiated how the duration of the dynamic delay of the action can be adopted. Specifically, Figure 1A A view 100 of a user 102 controlling a first application rendered in the foreground of a graphical user interface 106 is illustrated, while a second application provides notifications and is executing in the foreground of the graphical user interface 106. The first application may be a thermostat application 110, which may provide a first user interface 112 for controlling different features of the thermostat application 110 and a second user interface 114 for controlling the temperature setting of the thermostat application 110. The second application may be an alarm system application 108 executing in the background, or at least not entirely in the foreground, of the graphical user interface 106. While operating in the background, the alarm system application 108 may cause a notification 132 to be pushed to an operating system and / or automated assistant 130 of a computing device 104. The notification provided by the alarm system application 108 may then be rendered at a display panel 138 of the computing device 104.

[0024] Notification 132 may indicate that the alarm system controlled by alarm system application 108 is not secure (i.e., not actively monitoring the residence of user 102). Computing device 104 may optionally cause output 118 to be rendered in response to the notification being pushed from alarm system application 108. Notification 132 may be rendered while user 102 is interacting with thermostat application 110 and turning on the heat in their home. To simplify securing the alarm system without having to abruptly stop what they are doing with thermostat application 110, user 102 may provide input to automated assistant 130, which may be waiting for input from user 102, as indicated by state 116 of automated assistant 130.

[0025] Figure 1BA view 140 of a user 102 providing a spoken utterance 136 for controlling a background application while interacting with a foreground application is illustrated. Specifically, the user 102 may provide a spoken utterance 136 such as "secure the alarm system" that may be received as input 134 to the automated assistant 130. The automated assistant 130 may process the input 134 to identify the intent that the user 102 is intending to initiate. In some implementations, the action engine 126 of the computing device 104 may generate an action request corresponding to an action of one or more applications 128 accessible via the computing device 104 and / or the intent of the application.

[0026] In some implementations, the computing device 104 may also generate one or more confidence metrics for determining whether to provide a certain amount of delay before providing an action request to the alarm system application 108. In other words, given the current context, there may be certain actions that the user will want to confirm or modify before the action is initiated and other actions that the user may not expect to confirm or modify before the action is initiated. The determination of whether to implement a delay can be based on the assistant data 120 and / or the context data 124, which can be processed to determine a confidence metric that can provide an indication of whether to implement a delay before providing an action request to the alarm system application 108.

[0027] For example, when the context data 124 and / or the assistant data 120 indicate that the user has previously provided the spoken utterance 136 multiple times without subsequently requesting to modify the resulting action, a confidence metric can be generated to eliminate delays before the action is initialized. Alternatively or additionally, the context data 124 can indicate that the alarm system application 108 has provided a recent notification 132 relative to other notifications pushed by other applications 128 accessible via the computing device 104. Therefore, because the spoken utterance 136 and / or the action request corresponds to an action to be performed by the alarm system application 108, and the alarm system application 108 provides the latest notification 132, a confidence metric can be generated so that delays are bypassed or otherwise not implemented before initiating an action to cause the alarm system to protect. In some implementations, a trained machine learning model provided at the computing device 104 and / or another device in communication with the computing device 104 can be used to process various data to generate one or more confidence metrics. For example, assistant data 120 and / or context data 124 can be processed using a trained machine learning model to generate a confidence metric for determining whether to provide a delay before initiating an action.

[0028] Figure 1CA view 150 of an automated assistant 130 is illustrated in which a delay is bypassed before initiating an action based on a determined confidence metric. Specifically, after determining the confidence metric, the automated assistant 130 can determine whether to employ a delay before initiating the action requested by the user 102. In some implementations, when the confidence metric meets a certain threshold, the action request generated by the automated assistant 130 can be transmitted to the alarm system application 108 without incorporating a delay. Instead, the automated assistant 130 can transmit the action request to the background application via the operating system so that the background application performs the action. In response to receiving the action request, the alarm system application 108 can perform an action to cause the alarm system to protect, and optionally, provide another notification 152 indicating that the action was successfully performed. The automated assistant 130 can also provide an output 154 indicating that the alarm system is protected, and can thereafter provide an updated state 156 indicating that the automated assistant 130 is waiting for further input.

[0029] As a result of bypassing the delay before initializing the action, the wasted waiting time exhibited before the initialization action can be eliminated so that the execution of the action can be accelerated. This may be particularly important in the case of an alarm system that should operate with minimal waiting time in order to protect the user 102. In addition, as the user continues to interact with the automated assistant 130 to control the application 128, this enables the waiting time between all applications 128 to be reduced. For example, as a result of the automated assistant 130 being used to frequently perform certain actions, the corresponding confidence metrics of each action and / or verbal utterance can be increased, thereby reducing the delay between the user's input to the automated assistant 130 and the initialization of the corresponding action. In addition, by adopting dynamic delays in some cases, an adaptive process can be provided to the user 102, through which the user confirms or modifies the action request that the user only starts to request via the automated assistant 130. This can assist in training one or more trained machine learning models, thereby providing a more accurate confidence metric, according to which the implementation of the delay is bypassed and / or the duration of the delay is selected.

[0030] Figure 1D A view 160 of a user 102 providing a spoken utterance 164 is illustrated, which may correspond to an action request that may be delayed according to a dynamic duration selected via the automated assistant 130. The spoken utterance 164 may be provided by the user 102 while a foreground application such as the thermostat application 110 and a background application such as the alarm system application 108 are executing at the computing device 104. The spoken utterance 164 may be processed as input 162 to the automated assistant 130 while the user 102 is viewing the thermostat application 110. In response to receiving the spoken utterance 164, the automated assistant 130 may identify an action and / or application corresponding to the spoken utterance 164.

[0031] Figure 1E A view 170 of a computing device 104 is illustrated in which a confidence metric is determined in response to a spoken utterance 164 and an action impending notification 174 is provided based on the confidence metric. The confidence metric may be based on a variety of different data that may indicate that the spoken utterance 164 provided by the user 102 is less frequent than the spoken utterance 136. As a result, the confidence metric generated based on the spoken utterance 164 may be less than the confidence metric based on the spoken utterance 136. In some implementations, the confidence metric and / or another confidence metric may be generated at the computing device 104 to determine the duration of a timer 172, which may provide a real-time expiration of a delay before providing an action request to a corresponding application. For example, when the confidence metric does not meet a threshold for omitting the action impending notification 174, the confidence metric may still be used to determine the duration of the timer 172.

[0032] In some implementations, the confidence metric may be based on the amount of content provided in the spoken utterance 164, and thus the duration of the timer 172 may be proportional to or increase with the length of the spoken utterance from the user 102. Alternatively or additionally, the confidence metric may be generated based on whether a majority of the content of the spoken utterance contains terms that are commonly used by the user 102 and / or one or more other users. Thus, when the spoken utterance 164 includes terms that are least commonly used by the user 102 and / or one or more other users, the confidence metric may be lower relative to when the user includes more commonly used terms. Additionally, when less commonly used terms are included in the spoken utterance, the duration of the timer 172 may be set to a length that is at least longer relative to the set duration of the timer 172 when the user 102 includes more commonly used terms.

[0033] In some implementations, impending action notification 174 can provide the user with the ability to modify the impending action of setting the heat to 70° by tapping on a portion of impending action notification 174. Alternatively or additionally, the user can provide a subsequent verbal utterance such as "confirmed," or tap on a different portion of impending action notification 174, to confirm the action request before automated assistant 130 submits the action request to thermostat application 110. Alternatively or additionally, user 102 can wait for timer 172 to expire, such that the action request will be submitted to thermostat application 110 when timer 172 expires.

[0034] Figure 1FA view 180 of the automated assistant 130 providing an action request to the thermostat application 110 in response to the expiration of the timer 172 is illustrated. Because the user 102 allowed the timer 172 to expire, the automated assistant 130 can confirm the expiration of the timer and, in response, provide the generated action request to the thermostat application 110. The action request can cause the thermostat application 110 to initiate an action to modify the thermostat setting to 70 degrees, as indicated by output 184. In addition, the automated assistant 130 can provide an updated state 182 indicating that the automated assistant 130 is waiting for further input from the user 102. The user 102 can continue to interact with the thermostat application 110 and / or the alarm system application 108 via the graphical user interface 106, verbal input, and / or any other input that the computing device 104 can receive. However, each verbal input can be processed to identify the desired action, determine whether to delay for a duration before initiating the action, and optionally determine the duration of the delay based on the action, the verbal input, one or more confidence metrics, and / or any other information that can be associated with the verbal input.

[0035] FIG. 2A to FIG. 2C The example uses a delay before initiating an action requested by the user 206. Specifically, Figure 2A A view 200 of a user 206 invoking an automated assistant to perform an action and when the computing device 208 is not actively executing a third-party application in the foreground of the graphical user interface 202 of the computing device 208 is illustrated. For example, the automated assistant accessible via the computing device 208 may be in an idle mode, or otherwise awaiting input from the user 206, as indicated by the state 204. While accessing the computing device 208, the user 206 may provide a spoken utterance 210, such as "Call, Kay," which may correspond to a request to the automated assistant to initiate a phone call to a contact (e.g., "Kay") stored in association with the account of the user 206. The spoken utterance 210 may be received at a microphone of the computing device 208 and converted into audio data, which may be processed on the device using speech-to-text processing and / or natural language understanding processing. Based on the processing of the audio data, an action request may be generated for transmission to a corresponding application to initiate an action, such as a phone call application that may perform a call action. However, based on the content of spoken utterance 210 , the action request, and / or any other information that may be associated with the interaction between user 206 and the automated assistant, a confidence metric may be generated to determine whether to delay initiation of the call action.

[0036] In some implementations, information characterizing the interaction between user 206 and the automated assistant may be processed using a trained machine learning model to assist in generating a confidence metric. For example, with the user's prior permission, information associated with user 206 and a contact may be processed to determine how often user 206 communicates with the contact. In addition, the information may also be used to determine how often the user invokes the automated assistant to perform a phone call. If user 206 frequently contacts the contact that is the subject of spoken utterance 210 (e.g., at least more frequently than one or more other contacts) and user 206 frequently invokes the automated assistant to perform a phone call (e.g., at least more frequently than the user invokes the automated assistant to perform other actions), the confidence metric may meet a threshold for bypassing the initialization delay of the requested action. In other words, the automated assistant may bypass the call action delay made in response to spoken utterance 210, and may initiate a call to "Kay" via the automated assistant. However, when it is determined that the confidence metric does not meet a specific confidence metric threshold that may vary depending on the context in which user 206 is interacting with the automated assistant, the automated assistant may choose to perform a delay before the action is initialized. In some implementations, the confidence metric threshold can be static or dynamic, and can also be based on a variety of different data such as the user, the context of the interaction, one or more actions, one or more applications, other data generated using a trained machine learning model, and / or any other information that can be used to generate a metric.

[0037] Figure 2B A view 220 of an automated assistant having an action imminent notification 224 rendered at a graphical user interface 202 is illustrated. In some implementations, an optional timer 222 may be rendered with the action imminent notification 224. The timer 222 may provide a real-time expiration indication over the duration of the delay, and may begin to expire when the action imminent notification 224 is provided at the graphical user interface 202. In addition, the action imminent notification 224 may include a cancel element 226 that may cancel an impending action (e.g., call an action) when the cancel element 226 is selected during the duration of the delay. In addition, the user may click on a portion of the action imminent notification 224 that does not include the cancel element 226 or the timer 222 in order to confirm the impending action. By confirming the impending action before the delay timer 222 expires, the user 206 may cause the impending action to be initialized before the delay expires.

[0038] For example, Figure 2CAs illustrated in view 230 of , user 206 can initiate execution of a call action by confirming an action impending notification 224 and / or waiting for expiration of timer 222. The automated assistant can initialize execution of the call action, which can result in rendering an application interface 232 in the foreground of graphical user interface 202. In some implementations, in response to user 206 confirming the action via action impending notification 224 or waiting for expiration of delay timer 222, the trained machine learning model can be modified on the device to enable generation of a subsequent confidence metric. In this way, when the user subsequently confirms the call action more easily in a given context, the confidence metric can increase due to processing one or more inputs via the trained machine learning model. Once a subsequent iteration of the confidence metric satisfies a confidence metric threshold, the automated assistant can bypass delaying the call action in certain contexts.

[0039] In some implementations, the data may characterize the interaction between the user 206 and the automated assistant when an action is requested, when an action impending notification is provided, when an action request is transmitted to another application, and / or during the execution of an action. The data may be used to train a machine learning model for processing input received according to a federated learning process. For example, a federated learning process may be implemented using one or more server devices that communicate with one or more client devices. When a user interacts with their corresponding automated assistant via their corresponding client device, each client device may provide data characterizing the interaction between the user and the automated assistant with the user's prior permission. The data may be used to train a machine learning model, and the trained machine learning model and / or the gradient derived therefrom may be used by each client device to generate a confidence metric. Thus, when multiple different users confirm certain action impending requests, cancel certain action requests, and / or otherwise use features for bypassing action delays, the trained machine learning model may be updated for use in generating subsequent confidence metrics.

[0040] For example, when multiple different users confirm an alarm system "turn on" action when interacting with their respective automated assistants in their respective homes, a federated learning process can be used to train such an augmented machine learning model that will reflect similar interactions. As a result, the confidence metric subsequently generated in response to the user requesting the alarm system "turn on" action in their home can meet a particular threshold, which can result in bypassing delays in the execution of the alarm system "turn on" action. In this way, wasted waiting time that might otherwise be performed by the automated assistant can be eliminated, thereby simplifying certain actions that the user would not expect to confirm or modify. In addition, some processes can facilitate users to modify and / or confirm certain actions that seem important or critical to the user, such as sending messages or calling certain contacts with certain titles, and / or in certain situations such as driving a vehicle, operating a particular smart device, passing through a dangerous area, and / or interacting with an automated assistant when the user may want an additional opportunity to confirm or modify the requested action.

[0041] Figure 3 A system 300 for automatically determining whether to delay before automatically executing an application action via an automated assistant 304 - and the amount of delay - is illustrated. The automated assistant 304 may operate as part of an assistant application provided at one or more computing devices such as a computing device 302 and / or a server device. A user may interact with the automated assistant 304 via an assistant interface 320, which may be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user may initialize the automated assistant 304 by providing speech, text, and / or graphical input to the assistant interface 320 to cause the automated assistant 304 to perform a function (e.g., provide data, control peripherals, access an agent, generate input and / or output, etc.). The computing device 302 may include a display device, which may be a display panel including a touch interface, the touch interface being used to receive touch input and / or gestures that allow a user to control an application 334 of the computing device 302 via the touch interface. In some implementations, the computing device 302 may not have a display device, thereby providing an audible user interface output instead of a graphical user interface output. In addition, computing device 302 may provide a user interface such as a microphone for receiving spoken natural language input from a user. In some implementations, computing device 302 may include a touch interface and may not have a camera, but may optionally include one or more other sensors.

[0042] Computing device 302 and / or other third-party client devices can communicate with the server device via a network such as the Internet. In addition, computing device 302 and any other computing devices can communicate with each other via a local area network (LAN) such as a Wi-Fi network. Computing device 302 can offload computing tasks to the server device in order to save computing resources at computing device 302. For example, the server device can host automated assistant 304, and / or computing device 302 can transmit input received at one or more assistant interfaces 320 to the server device. However, in some implementations, automated assistant 304 can be hosted at computing device 302, and various processing associated with automated auxiliary operations can be performed at computing device 302.

[0043] In various implementations, all or less than all aspects of automated assistant 304 may be implemented on computing device 302. In some of these implementations, aspects of automated assistant 304 are implemented via computing device 302 and may interact with a server device that may implement other aspects of automated assistant 304. The server device may optionally serve multiple users and their associated auxiliary applications via multiple threads. In implementations in which all or less than all aspects of automated assistant 304 are implemented via computing device 302, automated assistant 304 may be an application separate from the operating system of computing device 302 (e.g., installed “on top” of the operating system)—or may alternatively be implemented directly by the operating system of computing device 302 (e.g., considered an application of the operating system, but integrated with the operating system).

[0044] In some implementations, automated assistant 304 may include an input processing engine 308 that may employ a plurality of different modules and / or an engine that processes input and / or output to computing device 302 and / or a server device. For example, input processing engine 308 may include a speech processing engine 316 that may process audio data received at assistant interface 320 to identify text embodied in the audio data. The audio data may be transmitted from, for example, computing device 302 to a server device in order to conserve computing resources at computing device 302. Additionally or alternatively, the audio data may be processed at computing device 302.

[0045] The processing of converting audio data into text may include a speech recognition algorithm, which may use a neural network and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data may be parsed by a data parsing engine 310 and available as text data for an automated assistant 304, which may be used to generate and / or identify command phrases, intentions, actions, time slot values, and / or any other content specified by a user. In some implementations, the output data provided by the data parsing engine 310 may be provided to a parameter engine 312 to determine whether the user has provided input corresponding to a specific intention, action, or routine that can be performed by an automated assistant 304 and / or an application or agent that can be accessed via an automated assistant 304. For example, assistant data 322 may be stored at a server device and / or a computing device 302, and may include data defining one or more actions that can be performed by an automated assistant 304 and parameters necessary for performing these actions. Assistant data 322 generated via input processing engine 308 can be processed by output generation engine 324 to cause automated assistant 304 to provide output to a user via assistant interface 320 and / or initiate one or more actions associated with one or more applications 334 .

[0046] In some implementations, computing device 302 may include one or more applications 334 that may be provided by a third-party entity that is distinct from the entity providing computing device 302 and / or automated assistant 304. Automated assistant 304 and / or action engine 318 of computing device 302 may access application data 330 to determine one or more actions that can be performed by one or more applications 334. Additionally, application data 330 and / or any other data (e.g., device data 306) may be accessed by automated assistant 304 to generate context data 336, which may characterize the context in which a particular application 334 is executing at computing device 302 and / or the context in which a particular user is accessing automated assistant 304 and / or computing device 302.

[0047] When one or more applications 334 are executing at computing device 302, device data 306 can characterize the current operating state of the respective applications 334 executing at computing device 302. In addition, application data 330 can characterize one or more features of the executing applications 334, such as the content of one or more graphical user interfaces rendered at the direction of one or more applications 334. In some implementations, action engine 318 can initiate the execution of one or more actions of application 334 under the direction of automated assistant 304. In addition, action engine 318 and / or automated assistant 304 can use the metrics determined by confidence metric engine 314 to prioritize and / or sort the application actions identified by automated assistant 304 in response to the spoken utterance. Additionally or alternatively, delay duration engine 326 can use the metrics to determine whether to delay an action to be initiated in response to a user and the duration of the delayed action.

[0048] Initialization of the action may be delayed so as to provide the user with an opportunity to confirm or modify the action request before the automated assistant provides the action request to a particular application. As an example, the computing device 302 may be executing a first application in the foreground of a graphical user interface and a second application in the background of the graphical user interface. During this time, the second application may provide a notification in the foreground of the graphical user interface, and the notification may include the content to which the user is being requested to respond. For example, the second application may be a messaging application, and the notification may correspond to a new message from a particular contact. Data representing the notification may be stored with application data 330, which may be accessible by the automated assistant 304. In response to answering the notification from the second application, the user may provide input to the assistant interface 320 of the computing device 302 in response to the content of the notification. Audio data representing the spoken utterance may be processed at the input processing engine 308, and thereafter, input data based on the spoken utterance may be processed at the action engine 318. The action engine 318 may identify that the user is intending to invoke and / or initialize an application and / or action beyond the spoken utterance. Additionally, confidence measurement engine 314 may also process input data or other data to determine whether initiation of an action should be delayed.

[0049] Confidence measurement engine 314 can determine a confidence metric that can be used to determine whether to delay the initialization of the requested action. Confidence measurement engine 314 can use application data 330, device data 306, context data 336, assistant data 322 and / or any other information that can be associated with the requested action. For example, application data 330 can indicate the frequency with which the user requests to perform the requested action. Alternatively or additionally, context data 336 can characterize the context in which the user provides a verbal utterance for requesting the execution of the action. Alternatively or additionally, assistant data 322 can characterize the historical interaction between the user and automated assistant 304. Based on some or all of this data, confidence measurement engine 314 can generate one or more confidence metrics that can be used to determine whether to delay the initialization of the requested action.

[0050] When the determined confidence metric indicates that the initialization of the requested action should be delayed, the delay duration engine 326 can use the confidence metric to determine the length of the duration of the delay. For example, in some implementations, the length of the duration of the delay can be proportional to the amount of natural language content provided in the spoken utterance. Alternatively or additionally, the amount of delay can be proportional to the frequency with which the user causes the identified action to perform the requested action. Alternatively or additionally, the length of the delay duration can be determined based on the location and / or context of the user characterized by the context data 336. For example, when it is indicated that the user provided a spoken utterance during an event stored in his calendar, the action can be delayed when the automated assistant determines that the action is not related to the calendar event. However, when it is determined that the action is related to the calendar event, such as when the user is requesting the automated assistant to send a message to the invitee of the calendar event, the automated assistant can bypass the delay in the initialization of the action. This can reduce the waiting time often shown when requesting to perform certain actions, and also improve the accuracy of certain actions that the user may expect to reflect or otherwise give extra attention. By improving such accuracy, computing resources associated with repetitive and / or corrective actions may be conserved because the user will repeat and / or corrective actions less frequently.

[0051] Figure 4A method 400 for dynamically delaying the execution of an action based on various different factors and bypassing the execution delay of other actions based on such factors is illustrated. The method 400 can be performed by one or more computing devices, applications, and / or any other means, modules, and / or engines capable of initiating the execution of an action. The method 400 may include an operation 402 of determining whether a spoken utterance is detected at a computing device. The spoken utterance may be a verbal input provided by one or more users to one or more interfaces that are part of or communicate with a computing device. The spoken utterance may be, for example, "TellMatthew I'm on my way". When the computing device is not executing an application in the foreground, the computing device is executing an application in the foreground, or the computing device is executing one or more applications in the foreground and executing one or more applications in the background, the spoken utterance may be provided at the computing device. The foreground and the background may refer to a graphical user interface rendered at a display panel of the computing device. The foreground of the graphical user interface may be an area of ​​the "top layer" of the background of the graphical user interface, so that it is at least the most visible relative to the background area for a user who is viewing the graphical user interface.

[0052] When a spoken utterance is detected at operation 402, method 400 may proceed from operation 402 to operation 404. Alternatively, when a spoken utterance is not detected at operation 402, the computing device may continue to monitor spoken utterances or other inputs from one or more users. Operation 404 may include identifying an application to which the spoken utterance is directed. In some implementations, determining that the application to which the spoken utterance is directed may be based on the natural language content of the spoken utterance. For example, the content of the spoken utterance may be compared to application data of one or more different applications, and a correlation between the content and the application data may be determined. A specific application that is most relevant to the content of the spoken utterance may be identified as a desired target of the spoken utterance.

[0053] Method 400 may also include an operation 406 of generating an action request based on the spoken utterance. In some implementations, one or more actions may be identified based on a specific application that is determined to be the target of the spoken utterance. For example, a computing device may determine that a messaging application is the target of "Tell Matt I'm on my way", and therefore, one or more actions that can be performed by the messaging application may be identified. The identified actions may include, for example, a send message action, which may include one or more time slot values ​​such as, but not limited to, a contact name and message content (e.g., Send_Message{Contact_Name(Name); Message_Content(Text); ...}). Therefore, because the send message action includes a time slot value for the name, and the spoken utterance is synonymous with sending a message, the send message action may be determined to be the target action of the spoken utterance. In some implementations, to generate the action request, the natural language content of the spoken utterance may be processed to identify one or more values ​​of each time slot value for the selected action. For example, the name "Matthew" may be assigned to the "contact_name" slot value, and the text "I'm on my Way" may be assigned to the "message_content" slot value.

[0054] Method 400 may also include an operation 408 of generating a confidence metric based on the action request and / or the spoken utterance. The confidence metric may indicate the degree of correlation between the spoken utterance and the content of the action request. Alternatively or additionally, the confidence metric may be used to predict whether the user will modify or confirm the action before the action request is transmitted to the identified application. For example, the degree of correlation between the spoken utterance and the content of the action request may depend on the accuracy of the speech-text processing and / or natural language understanding processing. Therefore, when the degree of correlation is low, the confidence metric may also be low, or otherwise indicate that the accuracy of the speech-text processing and / or natural language understanding processing is insufficient. However, when the degree of correlation is high, the confidence metric may also be high, or otherwise indicate that the speech-text processing and / or natural language understanding processing is at least sufficiently accurate. In some implementations, the confidence metric may be based on the total number of words provided in the spoken utterance, the frequency of use of specific words in the spoken utterance, the actions involved in the spoken utterance, the applications involved in the spoken utterance, and / or any other information that may be used as the basis for application measurement.

[0055] Method 400 may also include an operation 410 of determining whether the confidence metric predicts or otherwise indicates further confirmation and / or modification of the action. In other words, when the confidence metric is particularly low, the confidence metric may indicate that the user is likely to confirm or modify the action before the automated assistant initializes the action. Alternatively, when the confidence metric is relatively high, the confidence metric may indicate that the user is likely not to confirm or modify the action before the automated assistant initializes the action. When the confidence metric indicates that the user is unlikely to confirm or modify the action before the automated assistant initializes the action, method 400 may proceed to operation 420. Operation 420 may include initializing the execution of the action based on or in response to a spoken utterance. However, when the confidence metric indicates that the user is likely to confirm or modify the action before the automated assistant initializes the action, method 400 may proceed to operation 412.

[0056] Operation 412 may include determining the duration of the delay before initializing the action and / or transmitting the action request to the identified application. In some implementations, the duration of the delay may be based on a confidence metric. Alternatively or additionally, the duration of the delay may be based on a confidence metric and / or one or more other metrics. For example, in some implementations, the duration of the delay may be dynamically adjusted based on information characterizing the context in which the user provides the spoken utterance. For example, the information may characterize the user's location, an application in the foreground of a graphical user interface, an application in the background of a graphical user interface, a notification recently provided to the user via a computing device or another computing device, the frequency of execution of the action, the frequency of use of the identified application, the frequency of use of the spoken utterance content, and / or any other information that may be related to the request to perform the action on the application.

[0057] In some implementations, method 400 may include an operation 414 of rendering an action impending notification on a display interface. The action impending notification may indicate to the user that the action will be performed after the delay period expires. In some implementations, the action impending notification may be rendered when the confidence metric indicates that the action request will be confirmed or modified before the action is initialized. The action impending notification may include a dynamic graphic element that characterizes the expiration of the delayed duration in real time. During the duration of the delay, the user may choose to confirm and / or modify the action request before the action request is transmitted from the automated assistant to the identified application. In addition, in some implementations, the action impending notification may characterize the content provided by the user via a spoken word. In this way, the user is able to view the content that will be used for the action before the action is initialized. However, it is assumed that when the confidence metric meets the threshold for bypassing the delayed action, such viewing of the content is unnecessary. Otherwise, when the confidence metric does not meet the threshold for bypassing the delayed action, the delay timer may be called and / or the action impending notification may be rendered at the display interface to delay the action for a duration. In addition, this duration may be based on the confidence metric and / or any other information that may be associated with the automated assistant.

[0058] Method 400 may also include an operation 416 of initializing a real-time timer for indicating the expiration of the duration of the delay. The real-time timer representing the expiration of the duration of the delay may be initialized regardless of whether the action imminent notification is rendered at the display interface. In this way, although the action imminent notification is not rendered in some implementations, the user may still have the opportunity to confirm or modify the action request before the action is initialized. However, when the action imminent notification is rendered at the display interface, the real-time timer may appear as a graphical element that changes over time to indicate the expiration of the duration of the delay.

[0059] The method 400 may also include an operation 418 of determining whether there is an action modification during the duration of the delay. When it is determined that the user has not requested a modification during the duration of the delay, the method 400 may proceed to operation 420 to initialize the action. However, when there is an indication that the user requested to modify the action or action request during the duration of the delay, the method 400 may return to operation 408. In operation 408, another confidence metric may be generated based on the modified action and / or the modified action request. Alternatively, in some implementations, when there is an indication that the user requested to modify the action or action request during the duration of the delay, the method 400 may proceed to operation 420 to send the modified action request to the identified application and / or initialize the execution of the modified action via the automated assistant.

[0060] Figure 55 is a block diagram of an example computer system 510. The computer system 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 - including, for example, a memory 525 and a file storage subsystem 526, a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input devices and output devices enable a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is connected to corresponding interface devices in other computer systems.

[0061] The user interface input device 522 may include a keyboard, a pointing device such as a mouse, a trackball, a touch pad or a drawing tablet, a scanner, a touch screen mounted in a display, an audio input device such as a speech recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 510 or on a communication network.

[0062] User interface output devices 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways to output information from the computer system 510 to a user or another machine or computer system.

[0063] Storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules and / or engines described herein. For example, storage subsystem 524 may include logic to perform selected aspects of method 400, and / or implement one or more of computing device 104, computing device 208, computing device 302, automated assistant 304, server device, action engine, confidence metric engine, delay duration engine, and / or any other application, device, apparatus, and / or engine discussed herein.

[0064] These software modules and / or engines are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the memory subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules and / or engines that implement the functionality of certain implementations may be stored by the file storage subsystem 526 in the storage subsystem 524 or in other machines accessible to the processor 514.

[0065] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as desired. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0066] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some implementations. Figure 5 Many other configurations of computer system 510 may include more or fewer components than the computer system depicted.

[0067] In situations where the systems described herein collect personal information about users (or, as they are often referred to herein, "participants") or may use personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about a user's social network, social actions or activities, occupation, a user's preferences, or a user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, a user's identity may be processed so that personally identifiable information about the user cannot be determined, or a user's geographic location may be summarized when geographic location information (such as city, zip code, or state level) is obtained so that the user's specific geographic location cannot be determined. Thus, users may control how information about users is collected and / or used.

[0068] In some implementations, a method implemented by one or more processors includes an operation such as determining at a computing device that a user provides a spoken utterance to an interface of the computing device, wherein the computing device provides access to an automated assistant via the interface. The method may also include determining, based on the user providing the spoken utterance, that the spoken utterance is for an application that is separately accessible from the automated assistant. The method may also include generating an action request for the application based on the natural language content of the spoken utterance and a confidence metric for predicting whether the action request will be confirmed or modified before the action request is provided to the application. The method may also include determining, based on the confidence metric, whether an action impending notification should be rendered in the foreground of a graphical user interface, wherein the action impending notification characterizes the content of the action request. The method may also include, when the confidence metric indicates that the action impending notification should be rendered, the method further includes: causing the action impending notification to be rendered in the foreground of the graphical user interface. The method may also include, when the confidence metric indicates that the action impending notification should not be rendered, the method further includes: bypassing causing the action impending notification to be rendered in the foreground of the graphical user interface, and causing the action request to be provided to the application via the automated assistant.

[0069] In some implementations, the action impending notification includes an animated graphic element that represents the real-time expiration of a delay that the action request will be provided to the application via the automated assistant. In some implementations, the duration of the delay that the action request will be provided to the application via the automated assistant is based on a confidence metric. In some implementations, rendering the action impending notification in the foreground of the graphical user interface includes: rendering the action impending notification in the foreground during the duration of the delay or until the user takes an action to confirm and / or modify the action request. In some implementations, generating the confidence metric includes: processing input data representing one or more features of the historical interaction between the user and the automated assistant. In some implementations, the method may also include when the confidence metric indicates that the action impending notification should be rendered, the method also includes: determining that the user confirmed the action impending notification rendered in the foreground of the graphical user interface within a duration; and in response to determining that the user confirmed the action impending notification, causing the action request to be provided to the application via the automated assistant.

[0070] In some implementations, the method may further include, when the confidence metric indicates that the impending action notification should be rendered: determining that the user has not acknowledged the impending action notification rendered in the foreground of the graphical user interface for a duration; and in response to determining that the user has not acknowledged the impending action notification, modifying the trained machine learning model based on the user not acknowledging the impending action notification. In some implementations, determining whether the spoken utterance is directed to the application includes: identifying that the application provides a latest notification, wherein the latest notification corresponds to an incoming message from the sender.

[0071] In some implementations, generating the confidence metric includes identifying a sender of the incoming message, and wherein the value of the confidence metric is generated based on the sender. In some implementations, the action imminent notification includes other natural language content based on the spoken utterance and characterizes a draft message in response to the incoming message. In some implementations, generating the confidence metric includes: determining the number of characters and / or words included in the draft message in response to the incoming message, wherein the degree of confidence that the action request will be confirmed and / or modified is proportional to the number of characters and / or words included in the draft message. In some implementations, generating the confidence metric includes: determining the degree of correlation between the text content of the draft message and the natural language content of the spoken utterance, wherein a higher degree of correspondence between the text content of the draft message and the natural language content of the spoken utterance is associated with a higher confidence metric. In some implementations, when the confidence metric indicates that the action imminent notification should be rendered, the method further includes: if a confirmation or modification of the user request is not received before the expiration of the delay, causing the action request to be provided to the application via the automated assistant after the delay.

[0072] In other implementations, a method implemented by one or more processors is described as including operations such as determining at a computing device that a user provides a spoken utterance to an interface of a computing device in which a first application and a second application are being executed. In some implementations, the first application is being rendered in the foreground of a graphical user interface accessible via a display panel of the computing device, and the computing device provides access to an automated assistant that is separately accessible from the first application and the second application. In some implementations, the method may also include determining whether the spoken utterance is for the first application and / or the second application based on the user providing the spoken utterance and when the first application is rendered in the foreground of the graphical user interface. In some implementations, the method may also include, when it is determined that the spoken utterance is for the second application, generating an action request for the second application based on the natural language content of the spoken utterance and a confidence metric for predicting whether the action request will be confirmed and / or modified before the action request is provided to the application. In some implementations, the method may also include determining whether to render an action imminent notification in the foreground of the graphical user interface based on the confidence metric, wherein the action imminent notification includes a graphical element characterizing the duration of the delay before providing the action request to the second application. In some implementations, when the confidence metric indicates that the action imminent notification should be rendered, the method further includes: causing the action imminent notification to be rendered in the foreground of the graphical user interface. In some implementations, the method may also include: when the confidence metric indicates that the action imminent notification should be rendered, the method further includes: bypassing causing the action imminent notification to be rendered in the foreground of the graphical user interface; and causing the action request to be provided to the second application via the automated assistant.

[0073] In some implementations, the duration of the delay before the action request is provided to the second application is based on a confidence metric, and causing the action imminent notification to be rendered in the foreground of the graphical user interface includes causing the action imminent notification to be rendered in the foreground during the duration of the delay or until the user takes an action to confirm and / or modify the action request. In some implementations, generating the confidence metric includes processing input data characterizing one or more features of historical interactions between the user and the automated assistant, the user and the first application, and / or the user and the second application. In some implementations, the method may also include when the confidence metric indicates that the action imminent notification should be rendered: determining that the user confirmed the action imminent notification rendered in the foreground of the graphical user interface within the duration; and in response to determining that the user confirmed the action imminent notification, causing the action request to be provided to the second application via the automated assistant. In some implementations, the method may also include when the confidence metric indicates that the action imminent notification should be rendered: determining that the user did not confirm the action imminent notification rendered in the foreground of the graphical user interface within the duration. In some implementations, the method may also include: in response to determining that a user has not confirmed an impending action notification, causing a trained machine learning model to be modified based on the user has not confirmed an impending action notification, wherein the trained machine learning model is used to generate a confidence metric.

[0074] In some implementations, determining whether the spoken utterance is for the first application and / or the second application includes: identifying a specific application that provided a latest notification, wherein the latest notification corresponds to an incoming message from a sender, and wherein the second application is a specific application. In some implementations, generating a confidence metric includes identifying a sender of the incoming message, and wherein the duration of a delay before an action request is provided to the second application is based on the sender. In some implementations, the action-imminent notification includes natural language content based on the spoken utterance and characterizes a draft message in response to the incoming message. In some implementations, generating a confidence metric includes: determining a number of characters and / or words included in a draft message in response to the incoming message, wherein the duration of a delay before an action request is provided to the second application is based on the number of characters and / or words included in the draft message.

[0075] In some implementations, generating the confidence metric includes: determining a degree of correlation between the text content of the draft message and the natural language content of the spoken utterance, wherein a higher degree of correspondence between the text content of the draft message and the natural language content of the spoken utterance is associated with a shorter duration of the delay before the action request is provided to the second application. In some implementations, the graphical element represents in real time the expiration of a duration for which the action impending notification will be rendered on the display panel. In some implementations, the method may also include when the confidence metric indicates that the action impending notification should be rendered, the method also includes: if no confirmation or modification of the user request is received before the expiration of the delay, causing the action request to be provided to the application via the automated assistant after the delay.

[0076] In other implementations, a method implemented by one or more processors is described as including operations such as determining at a computing device that a user provides a spoken utterance to an automated assistant, wherein the computing device accesses the automated assistant via one or more automated assistant interfaces. The method may also include, based on determining that the user provides a spoken utterance, identifying an action to be performed in response to the spoken utterance provided by the user. The method may also include determining, based on the spoken utterance provided by the user and / or the action to be performed in response to the spoken utterance, a duration of a delay before initiating an action via an action request provided by the automated assistant. The method may also include, during the real-time expiration of the duration of the delay, monitoring another input from the user for confirming or modifying the action request before initiating the action via the automated assistant. The method may also include: when, based on the monitoring, it is determined that the user provides another input for confirming or modifying the action request during the duration of the delay: in response to the user providing the another input during the duration of the delay, causing the automated assistant to at least temporarily bypass the initialization action. The method may also include: when, based on the monitoring, it is determined that the user has not provided another input for confirming or modifying the action request during the duration of the delay: in response to determining that the user has not provided another input during the duration of the delay, causing the automated assistant to initialize the execution of the action.

[0077] In some implementations, the method may further include, based on the user providing the spoken utterance, accessing context data characterizing one or more properties of a context in which the user provided the spoken utterance, wherein determining the duration of the delay is based on the context data. In some implementations, the method may further include, based on the user providing the spoken utterance, determining whether to render an action-imminent notification via a display panel of the computing device for the duration of the delay, wherein the action-imminent notification characterizes a real-time expiration of the duration of the delay. In some implementations, determining the duration of the delay before initiating the action via the automated assistant includes processing audio data characterizing the spoken utterance using a trained machine learning model.

[0078] Although many implementations have been described and illustrated herein, various other devices and / or structures for performing functions and / or obtaining one or more of the results and / or advantages described herein, and each of these variants and / or modified forms is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications for which the teachings are used. Those skilled in the art will recognize or be able to determine many equivalents of the specific implementations described herein using only routine experiments. Therefore, it is to be understood that the aforementioned implementations are presented only by way of example and within the scope of the attached claims and their equivalents, the implementations may be practiced in a manner different from that specifically described and claimed. The implementations disclosed herein are directed to each independent feature, system, product, material, kit, and / or method described herein. In addition, if these features, systems, products, materials, kits, and / or methods are not mutually contradictory, any combination of two or more features, systems, products, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface; determining, based on processing the spoken utterance, that the spoken utterance is an action to be performed by an application, wherein the application is separate from the automated assistant; determining, based on the spoken utterance, a duration of a delay before initiating the action via an action request provided by the automated assistant; causing an imminent action notification to be rendered in the foreground of a graphical user interface within the duration of the delay, wherein the imminent action notification comprises: a representation of the content of the action requested, and a dynamic graphical representation of a remaining duration of said duration of said delay; during the duration of the delay, monitoring for a confirmation input from the user to confirm the action or a modification input from the user to modify the action; When it is determined based on the monitoring that the user has provided the modification input for modifying the action during the duration of the delay: causing the automated assistant to at least temporarily bypass initiating the action; and When it is determined based on the monitoring that the user has provided the confirmation input for confirming the action request during the duration of the delay or that the user has allowed the duration of the delay to expire without providing the confirmation input or the modification input: The automated assistant is caused to initiate performance of the action by providing the action request.

2. The method according to claim 1, further comprising: Context data characterizing one or more properties of a context in which the user provided the spoken utterance is accessed, wherein determining the duration of the delay is further based on the context data.

3. The method according to claim 1, wherein: Determining the duration of the delay based on the spoken utterance includes: generating a confidence measure based on the natural language content of the spoken utterance; and The duration of the delay is determined based on the confidence metric.

4. The method according to claim 3, wherein: Generating the confidence metric is further based on input data characterizing one or more features of historical interactions between the user and the automated assistant.

5. The method according to claim 1, wherein: Determining the action to be performed by the spoken utterance for the application further comprises: The application is determined based on that the application has provided a most recent notification, wherein the most recent notification corresponds to an incoming message from a sender.

6. The method according to claim 5, wherein: Determining the duration of the delay is further based on a sender of the incoming message.

7. The method according to claim 1, wherein: Determining the duration of the delay based on the spoken utterance includes: The duration of the delay is determined based on a number of words in the spoken utterance and / or a number of characters in the words.

8. The method according to claim 1, wherein: The actions include sending a message, wherein the natural language content of the spoken utterance includes words to be included in the message, and wherein determining the duration of the delay based on the spoken utterance includes: The duration of the delay is determined based on the number of characters in the words and / or the number of words.

9. The method according to claim 1, wherein: Determining the duration of the delay based on the spoken utterance includes: determining one or more confidence measures based on the speech-to-text conversion of the spoken utterance; and Determining the duration of the delay is based on the one or more confidence metrics.

10. The method according to claim 1, wherein: Determining the duration of the delay based on the spoken utterance includes: determining a level of similarity between the action and previous user actions based on historical user data; and The duration of the delay is determined based on the similarity level.

11. A method implemented by one or more processors, the method comprising: determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface; determining, based on processing the spoken utterance, that the spoken utterance is an action to be performed by an application, wherein the application is separate from the automated assistant; Based on the spoken utterance, determining to render an action impending notification in a foreground of a graphical user interface for a duration of a delay before initiating the action via an action request provided by the automated assistant, wherein the action impending notification comprises: a representation of the content of the action requested, and a dynamic graphical representation of a remaining duration of said duration of said delay; during the duration of the delay, monitoring for a confirmation input from the user to confirm the action or a modification input from the user to modify the action; When it is determined based on the monitoring that the user has provided the modification input for modifying the action during the duration of the delay: causing the automated assistant to at least temporarily bypass initiating the action; and When it is determined based on the monitoring that the user has provided the confirmation input for confirming the action request during the duration of the delay or that the user has allowed the duration of the delay to expire without providing the confirmation input or the modification input: The automated assistant is caused to initiate performance of the action by providing the action request.

12. The method according to claim 11, wherein: The actions include sending a message to an additional user of an additional computing device.

13. The method according to claim 12, further comprising: The duration of the delay is determined based on the spoken utterance.

14. The method according to claim 13, wherein: The message is determined based on the spoken utterance, and wherein determining the duration of the delay based on the spoken utterance comprises: The duration of the delay is determined based on a length of the message.

15. A method implemented by one or more processors, the method comprising: determining that a user provided a spoken utterance to an interface of a computing device, wherein the computing device provides access to an automated assistant via the interface; determining, based on processing the spoken utterance, that the spoken utterance is an action to be performed by an application, wherein the application is separate from the automated assistant; determining, based on the action, a duration of a delay before initiating the action via an action request provided by the automated assistant; causing an imminent action notification to be rendered in the foreground of a graphical user interface within the duration of the delay, wherein the imminent action notification comprises: a representation of the content of the action requested, and a dynamic graphical representation of a remaining duration of said duration of said delay; during the duration of the delay, monitoring for a confirmation input from the user to confirm the action or a modification input from the user to modify the action; When it is determined based on the monitoring that the user has provided the modification input for modifying the action during the duration of the delay: causing the automated assistant to at least temporarily bypass initiating the action; and When it is determined based on the monitoring that the user has provided the confirmation input for confirming the action request during the duration of the delay or that the user has allowed the duration of the delay to expire without providing the confirmation input or the modification input: The automated assistant is caused to initiate performance of the action by providing the action request.

16. The method according to claim 15, wherein: Determining the duration of the delay based on the action includes: determining a level of similarity between the action and previous user actions based on historical user data; and The duration of the delay is determined based on the similarity level.

17. The method according to claim 15, wherein: The action comprises sending a message, and wherein determining the duration of the delay based on the spoken utterance comprises: The duration of the delay is determined based on a number of characters in the message and / or a number of words in the message.

18. A method implemented by one or more processors, the method comprising: determining that a user provided a spoken utterance to an interface of a computing device, wherein the spoken utterance is provided while both a first application and a second application are executed at the computing device; Based on the spoken utterance, determining that the first application is a desired target of the spoken utterance; generating an action to be performed by the first application based on processing the spoken utterance; determining, based on the spoken utterance, a duration of a delay before initiating the action via an action request provided to the first application; causing an imminent action notification to be rendered in the foreground of a graphical user interface within the duration of the delay, wherein the imminent action notification comprises: a representation of the content of the action requested, and a dynamic graphical representation of a remaining duration of said duration of said delay; During the duration of the delay, monitoring for a confirmation input from the user to confirm the action or a modification input from the user to modify the action; and When it is determined based on the monitoring that the user has provided the confirmation input for confirming the action request during the duration of the delay or that the user has allowed the duration of the delay to expire without providing the confirmation input or the modification input: causing the execution of the action to be initiated by providing the action request to the first application, Wherein, the action request is provided to the first application in response to determining that the first application is the desired target of the spoken utterance.

19. The method according to claim 18, further comprising: Context data characterizing one or more properties of a context in which the user provided the spoken utterance is accessed, wherein determining the duration of the delay is further based on the context data.

20. The method according to claim 18, wherein: Determining the duration of the delay based on the spoken utterance includes: generating a confidence measure based on the natural language content of the spoken utterance; and The duration of the delay is determined based on the confidence metric.

21. The method according to claim 20, wherein: Generating the confidence metric is further based on input data characterizing one or more features of one or more historical interactions of the user.

22. The method according to claim 18, wherein: Determining that the first application is the desired target of the spoken utterance is based on the first application having provided a latest notification, wherein the latest notification corresponds to an incoming message from a sender.

23. The method according to claim 22, wherein: Determining the duration of the delay is further based on a sender of the incoming message.

24. The method according to claim 18, wherein: Determining that the first application is the desired target of the spoken utterance includes comparing content of the spoken utterance to first application data of the first application.

25. The method of claim 18, wherein: Determining the duration of the delay based on the spoken utterance includes: The duration of the delay is determined based on a number of words in the spoken utterance and / or a number of characters in the words.

26. The method of claim 18, wherein: The actions include sending a message, wherein the natural language content of the spoken utterance includes words to be included in the message, and wherein determining the duration of the delay based on the spoken utterance includes: The duration of the delay is determined based on the number of characters in the words and / or the number of words.

27. The method according to claim 18, wherein: Determining the duration of the delay based on the spoken utterance includes: determining one or more confidence measures based on the speech-to-text conversion of the spoken utterance; and The duration of the delay is determined based on the one or more confidence metrics.

28. The method of claim 18, wherein: Determining the duration of the delay based on the spoken utterance includes: determining a level of similarity between the action and previous user actions based on historical user data; and The duration of the delay is determined based on the similarity level.

29. A method implemented by one or more processors, the method comprising: determining that a user provides a spoken utterance to an interface of a computing device, wherein the spoken utterance is provided while both a first application and a second application are executed at the computing device; Based on the spoken utterance, determining that the first application is a desired target of the spoken utterance; generating an action to be performed by the first application based on processing the spoken utterance; Based on the spoken utterance, determining whether to render an action-imminent notification in a foreground of a graphical user interface within a duration of a delay before initiating the action via an action request, wherein the action-imminent notification comprises: a representation of the content of the action requested, and a dynamic graphical representation of a remaining duration of said duration of said delay; When determining to render the action imminent notification: rendering the action-imminent notification prior to any initiation of execution of the action; When determining to bypass rendering of the action imminent notification: automatically initiating performance of the action by providing the action request to the first application, Wherein, in response to determining that the first application is the desired target of the spoken utterance, the action request is provided to the first application.

30. The method of claim 29, further comprising: Context data characterizing one or more properties of a context in which the user provided the spoken utterance is accessed, wherein determining whether to render the action imminent notification is further based on the context data.

31. The method of claim 29, wherein: Determining whether to render the impending action notification based on the spoken utterance includes: generating a confidence measure based on the natural language content of the spoken utterance; and A determination is made based on the confidence metric whether to render the action imminent notification.

32. The method according to claim 31, wherein: Generating the confidence metric is further based on input data characterizing one or more features of one or more historical interactions of the user.

33. The method of claim 29, wherein: Determining whether to render an impending action notification based on the spoken utterance includes: Whether to render the action imminent notification is determined based on a number of words in the spoken utterance and / or a number of characters in the words.

34. The method of claim 29, wherein: Determining whether to render the impending action notification based on the spoken utterance includes: determining one or more confidence measures based on the speech-to-text conversion of the spoken utterance; and A determination is made whether to render the action imminent notification based on the one or more confidence metrics.

35. The method of claim 29, wherein: Determining whether to render the impending action notification based on the spoken utterance includes: determining a level of similarity between the action and previous user actions based on historical user data; and A determination is made whether to render the imminent action notification based on the similarity level.

36. A method implemented by one or more processors, the method comprising: determining that a user provides a spoken utterance to an interface of a computing device, wherein the spoken utterance is provided while both a first application and a second application are executed at the computing device; Determining that the spoken utterance is for the first application, wherein determining that the spoken utterance is for the first application is based on one or both of the following: the first application provided output more recently than the second application, and the first application receiving user input more recently than the second application; generating an action to be performed by the first application based on processing the spoken utterance; causing an imminent action notification to be rendered in the foreground of a graphical user interface, wherein the imminent action notification includes a representation of the content of the action request; During the duration that the action-imminent notification is rendered, monitoring for a confirmation input from the user to confirm the action or a modification input from the user to modify the action; and When it is determined based on the monitoring that the user has provided the confirmation input for confirming the action request or that the user has refrained from providing the confirmation input or the modification input: causing the execution of the action to be initiated by providing the action request to the first application, Wherein, in response to determining that the spoken utterance is directed to the first application, the action request is provided to the first application.

37. The method of claim 36, wherein: The first application is executed in the background of the computing device.

38. The method of claim 37, wherein: Determining that the spoken utterance is for the first application is based on the first application providing the output more recently than the second application, and wherein the output corresponds to a notification of an incoming message from a sender.

39. The method of claim 38, wherein: The duration that the action imminent notification is rendered is determined based on the sender.

40. The method of claim 36, wherein: Determining that the spoken utterance is for the first application further comprises: comparing content of the spoken utterance with first application data of the first application; and A determination is made that the spoken utterance is for the first application based on comparing the content of the spoken utterance to the first application data.

41. The method of claim 36, wherein: Determining that the spoken utterance is for the first application further includes determining that the spoken utterance is for the first application based on the first application executing in the foreground of the interface of the computing device.

42. The method of claim 36, further comprising: Context data characterizing one or more properties of a context in which the user provided the spoken utterance is accessed, wherein the duration for which the action imminent notification is rendered is determined based on the context data.

43. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 42.

44. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 42.

45. A system comprising one or more processors for executing the method of any one of claims 1 to 42.

Images